Efficient Crawling Through URL Ordering：（高效的爬行通过URL命令）.pdfVIP

Efficient Crawling Through URL Ordering Junghoo Cho Hector Garcia-Molina Lawrence Page Department of Computer Science Stanford, CA 94305 {cho, hector, page}@ Abstract In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more “important” pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this problem. We also experimentally evaluate the ordering schemes on the Stanford University Web. Our results show that a crawler with a good ordering scheme can obtain important pages significantly faster than one without. 1 Introduction A crawler is a program that retrieves Web pages, commonly for use by a search engine [Pin94] or a Web cache. Roughly, a crawler starts off with the URL for an initial page p . It retrieves p , 0 0 extracts any URLs in it, and adds them to a queue of URLs to be scanned. Then the crawler gets URLs from the queue (in some order), and repeats the process. Every page that is scanned is given to a client that saves the pages, creates an index for the pages, or summarizes or analyzes the content of the pages. Crawlers are widely used today. Crawlers for the major search engines (e.g., AltaVista, InfoSeek, Excite, and Lycos) attempt to visit most text Web pages, in order to build content indexes. Other crawlers may also visit many pages, but m