Web信息体系结构 high_perf_crawlers.pptVIP

High Performance Crawling MW Ch.2 Crawling the web Mercator- A Scalable, Extensible Web Crawler(1999) High-Performance Web Crawling (2001) Course slides on Spider by Prof. Li (2003) Web是一个有向图 系统框图 Web Graph-Search Algorithms I PROCEDURE SPIDER1(G) Let ROOT := any URL from G Initialize STACK stack data structure Let STACK := push(ROOT, STACK) Initialize COLLECTION big file of URLpairs While STACK is not empty, URLcurr := pop(STACK) PAGE := look-up(URLcurr) STORE(URLcurr, PAGE, COLLECTION) For every URLi in PAGE, push(URLi, STACK) Return COLLECTION Depth-first Search What is wrong with algorithm I? 如果web graph有回路会导致 Algorithm will not halt 遇到汇聚的结构会出现 Pages will replicated in collection Inefficiently large index Duplicates to annoy user Crawl 完整吗？ SPIDER1 only reaches pages in the the connected web subgraph where ROOT page lives Web是强连通的？ Completeness Observations (1) Completeness is not guaranteed Impossible to guarantee each connected subgraph is sampled How to make it better: more seeds, more diverse seeds, port scanner maybe help Completeness Observations (2) Search Engine Practice Wish to maximize subset of web indexed. Maintain (secret) set of diverse seeds (grow this set opportunistically, e.g. when X complains his/her page not indexed). Register new web sites on demand New registrations are seed candidates. A Correct Spidering Algorithm PROCEDURE SPIDER2(G) Let ROOT := any URL from G Initialize STACK stack data structure Let STACK := push(ROOT, STACK) Initialize COLLECTION big file of URLpairs While STACK is not empty, | Do URLcurr := pop(STACK) | Until URLcurr is not in COLLECTION PAGE := look-up(URLcurr) STORE(URLcurr, PAGE, COLLECTION) For every URLi in PAGE, push(URLi, STACK) Return COLLECTION A More Efficient Correct Algorithm PROCEDURE SPIDER3(G) Let ROOT := any URL from G Initialize STACK stack data structure Let STACK := push(ROOT, STACK) Initialize COLLECTION big file of URLpairs | Initialize VISITED b