Sitemap XML Format - Department Of Computer Sciencesitemap XML格式-计算机科学系.pptVIP

Efficient, Automatic Web Resource Harvesting Michael L. Nelson, Joan A. Smith and Ignacio Garcia del Campo Old Dominion University Computer Science Dept Norfolk VA 23529 USA {mln, jsmit, dgarcia}@ Presentation Overview Introduction OAI-PMH Complex data formats as Metadata MOD_OAI Demo Quantitative Evaluation Representation Problem Counting Problem Sitemaps Conclusion References Introduction What is a web crawler ? A program or automated script which browses the World Wide Web in a methodical, automated manner. What makes Web crawling difficult ?: Large volume Fast rate of change and Dynamic page generation Crawling Difficulties The large volume implies that the crawler can only download a fraction of the web pages within a given time, so it needs to prioritize its downloads. The high rate of change implies that by the time the crawler is downloading the last pages from a site, it is very likely that new pages have been added to the site, or that pages have already been updated or even deleted. Problems Two problems associated with conventional web crawling techniques: Counting Problem :A crawler cannot know if all resources at a web site have been discovered and crawled. Representation Problem :The human-readable format of the resources are not always suitable for machine processing. Mod_oai A Solution For Counting And Representation Solution: Via an Apache module: mod_oai implements OAI-PMH + MPEG-21 DIDL OAI-PMH: count everything (linked or not) using “List” verbs MPEG-21 DIDL: capture everything using a complex-object format and automated metadata extraction OAI-PMH Web servers do not have the capability to answer questions of the form “what resources do you have?” and “what resources have changed since 2004-12-27?” Using OAI-PMH a web crawler can quickly get an update on the latest changes to a site. Requests only resources that are new or have changed since its la