基于网络爬虫的有效URL缓存外文文献翻译.docVIP

外文资料原文 Efficient URL Caching for World Wide Web Crawling Andrei Z. Broder IBM TJ Watson Research Center 19 Skyline Dr Hawthorne, NY 10532 HYPERLINK mailto:abroder@abroder@ Marc Najork Microsoft Research 1065 La Avenida Mountain View, CA 94043 HYPERLINK mailto:najork@najork@ Janet L. Wiener Hewlett Packard Labs 1501 Page Mill Road Palo Alto, CA 94304 HYPERLINK mailto:janet.wiener@janet.wiener@ ABSTRACT Crawling the web is deceptively simple: the basic algorithm is (a)Fetch a page (b) Parse it to extract all linked URLs (c) For all the URLs not seen before, repeat (a)—(c). However, the size of t