Developing an automatic crawling system for populating a digital repository of professional development resources: A pilot study

Abstract

This study is a part of the larger project that develops a sustainable digital repository of professional development resources on emerging data standards and technologies for data organization and management in libraries. Toward that end, the project team developed an automated workflow to crawl for, monitor, and classify relevant web objects into searchable professional development categories. In this study we present the rapid prototyping approach and the iteration and ranking processes used to develop the automatic crawling system. In its final form, the repository will constitute self-sustaining continuing education resources for the cataloging and metadata community.