中文网页分类研究与系统实现_段军峰.pdfVIP

2007Vol34 6 * 清华大学计算机科学与技术系 北京100084 近年来, 网页分类研究逐渐成为网络挖掘和文本挖掘的研究热点, 针对中文网页分类的研究也日益增多本 文详 介绍了一个中文网页分类系统的设计与实现, 并且提出了一些网页预处理和特征处理的方法实验结果表明, 本文的方法可以使中文网页分类性能由81. 5% 提高至94. 9% 中文网页分类, 网页预处理, 特征抽取 Study and System Implementation of Chinese Web- DUA N Jun-Fen H UANG We-i T on LU Yu-Chan Department of Computer Scien ce and Technolo y, T sin hua University, Beijin 100084 Abstract W eb- pa e classification has become a hotspot in the fields of Web minin and text minin in recent years. St udies tow ards Chinese Web-pa e are more and more. A detailed desi n and implementation of a Chinese Web-pa e classification sy st em is introduced, and some met hods on Web-pa e preprocessin and feature processin are proposed. Experimental results show that the proposed methods can improve the effectiveness of Chinese W eb-pa e classification from 81. 5% to 94. 9% . Keywords Chinese W eb-pa e classificat ion, W eb-pa e preprocessin , Feature extraction 3 : HT ML 1 XML , W eb H T ML , XML , 2005 , Goo le 82 , 10 , , , Web W eb , , , , : 1 , H T M L , , ; 2 W eb , ; 3 , ; 4 , , , 2 1 2. 1 2. 2 1 4 , : 60% , 6 * 座机电话号码 , : ; , : ; , : 210 , HT ML 2. 2. 4 词条选择 , 2. 2. 1 H T ML 解析 , HT ML html , , H T ML , XML 3. 3 HT ML : , ICT CLA S[ 6] , , st yle, script, applet HT- , ML , , metaHT M L , 2. 3 , a href URL , headbody headmeta , a title, metaa, body , , , : , , , : , , text, , anchor, H T ML , 3. 2 , H T ML HT ML 2. 4 textanchor , table, , : Document fre- quency , DF Information Ga1n, IG Mu- [ 1,2] H T ML , tual information, MI x-test, CHI , , T erm Stren th, T S , 3. 2 [ 3, 4] 4 , , : IG CH I , , ; DF IG CH I , DF ; MI 2. 2. 2 英文词法分析与中文分词 , , , t : DF , , , , t , , , , , , , , ChineseSe menter[ 5] , , , , , f r eq w , C w t t i ICT CLAS[ 6] C C num C i i , , ICT CLA S f r eq w , C w C , num t i t i C C i i , , [ 8, 9] , , n- ram 2. 5 2. 2. 3 停用词删除与英文词干提取 , , , , t o, : , o, , 580 637 , P c | d : j i |di | P