网页去重方法研究和算法设计.pdfVIP

25 2 ( ) V o l. 25 N o. 2 2010 4 JOU RNAL O F ZH ENGZHOU U N IV ER S ITY O F L IGHT INDU STRY ( N atu ral Sc ence) A pr. 2010 : 20 10) 网页去重方法研究及算法设计 张素智, 樊得强 (郑州轻工业学院计算机与通信工程学院, 河南郑州 450002) : 针对在网页检索结果中经常出现内容相同的冗余页面的问题, 为了按用户偏好对搜索结果 进行筛选, 提出了一种通过中文字主题要素学习网页内容的网页去重算法. 该算法通过抽取网页 要素中用户偏好的短语来抽取网页的内容, 根据学习的网页内容计算其相似度来判断网页的重复 度. 经与相似方法比较, 结果表明, 该方法能够完成针对中文内容网页的去重, 具有较高的查全率 和查准率. : 中文字主题要素; 模糊匹配; 去重算法; 查全率; 差准率 : TM 9122 : A Study on duplicated rem ovalm ethodW eb pages and algorithm design ZHANG Suzh , FAN Deq ang (College of Comp. and Com. Eng., Zhengzhou Univ. of Light Ind., Zhengzhou 450002, China) Abstract: In v ew of theW eb page retr eval result, the u ser frequently obta ns the sam e content redundant page. In order to so lve th s problem, one k nd of dupl catedW eb pages rem ova l a lgor thm accord ng to the u sers nterest s proposed to study W eb pages content on e lem ents of Ch nese top c. T h s m ethod fo llow s the three step s: F rst, extract phrase of the user s nterest w h ch s Ch nese e lem ents content ofW eb pages, then study W eb pages content extracted and f na lly, ca lculates the r sm lar ty acco rd ng to the study new s content to judge the heavy mu lt pl c ty of the new s hom epage. Compared w th sm l ar m ethods, the exper m ental resu lt nd cated tha,t n v ew o f the Ch nese con tent w eb pages dupl cated, th s m ethod obta ns the h gh recall and the accuracy rat o. K ey words: e lem ents o f Ch nese subjec;t fuzzy m a tch ng; dupl cate rem oval algo r thm; recall rat o; accura cy rat o