可扩展Web信息搜集系统的设计、实现与应用初探新版9.pptVIP

两阶段映射模型 已知：H : hosts on the web ; N : main-controllers; M : elements in the Array A. 则有：{ (h , n) | m =f1(h), n=f2(m), h∈H, m∈M, n∈N } 要求：(Ui ∩ Uj) = ? ；(Hi ∩ Hj) = ? ；(Ni ∩ Nj) = ? 负载平衡 通信量低 各主控之间不重复工作 有利于后续工作 * * jgyj 两阶段映射举例 1 … 1 2 … 2 … … 9 … 9 1 … 1 2 … 2 … … 9 … 9 1 … 1 11 … 11 2 … 2 11 … 11 … … 10 … 10 11 … 11 N11 1 … 1 2 … 2 … … 10 … 10 N2(5001..10000) N2 shift (9546..10000) N10 shift (49546..50000) N1 shift (4546..5000) N1 (45001..45555) Array A URLs N2 (45556..46110) N9 (49440..50000) N1(1..5000) N10(45001..50000) N1(1..4545) N2(5001..9545) N10(45001..49545) N1(1..5000) N2(5001..10000) N9(40001..45000) a) 稳定状态 b)增加节点 c) 减少节点 设 N = 10 and M = 50000 * * jgyj 提纲 问题提出 可扩展网页搜集系统 网页搜集系统的动态配置 海量网页应用研究初步 中国Web大小、形状和结构 工作总结 * * jgyj 术语介绍 Web直径 网页出度，网页入度 存在于导航功能中的称为导航影响入度 存在于认可功能中的称为认可影响入度 有效入度 权威型网页，目录型网页 * * jgyj 天网搜集记录 第三次搜集数据具有代表性 覆盖了中国89.6%的网站，45.2%的网页。 类似于宽度优先搜索的策略 覆盖了93.2%有影响力的网站 * * jgyj 基本统计数据 平均每个网站有网页548.72个。 网页文字平均为12.92 KB，网站在各省之间的分布方差为24.18。 其中教育网有8144个网站，网站分布方差为16.14； 科技网有732个网站，网站分布方差为27.68。 * * jgyj * bhj oo * bhj oo Researchers Map the Web - Press Release ALTAVISTA, COMPAQ AND IBM RESEARCHERS CREATE WORLD LARGEST, MOST ACCURATE PICTURE OF THE WEB Bow Tie Theory Shows the Web is Not as Connected as Previously Thought SAN JOSE, PALO ALTO SAN MATEO, Calif. -- May 11, 2000 -- Scientists from IBM Research, Compaq Corporate Research Laboratories and AltaVista Company have completed the first comprehensive map of the World Wide Web, and uncovered divisive boundaries between regions of the Internet that can make navigation difficult or, in some cases, impossible. Previous studies, based on small samplings of the Web, suggested that there was a high degree of connectivity between sites as evidenced by recent reports on the small world Web and 19 degrees of separation. Contrary to those preliminary findings, the new study -- based on analysis of more than 500 million pages -- found that the World Wide Web is fundamentally