文本分类中基于方差的改进特征提取算法.pdfVIP

第28卷 第24期 计算机工程与设计 2007年l2月 Vo1．28 No．24 Computer Engineering and Design Dec．2007 文本分类中基于方差的改进特征提取算法 吕 佳 (重庆师范大学数学与计算机科学学院运筹学与系统工程重庆市市级重点实验室，重庆400047) 摘 要：特征提取算法TFIDF是文本分类中常用的衡量特征权重的算法，但该算法没有考虑特征词在类间和类内的分布情 况，导致算法无法反映特征词在分布比例中量上的差异。为此，引入方差来描述特征词在类间和类内的分布情况，并利用方 差来修正TFIDF权重。仿真实验结果表明，同传统TFIDF算法相比，改进TFIDF算法能得到更好的分类结果。 关键词：文本分类；特征提取；改进TFIDF；权重；方差 中图法分类号：TP391 文献标识码：A 文章编号：1000．7024(2007)24．6039—03 Improved feature selection algorithm based on variance in text categorization LU Jia (Chongqing Key Lab of Operations Research and System Engineering，College of Mathematics and Computer Science， Chongqing Normal University,Chongqing 400047，China) Abstract：The traditional TFIDF algorithm is a common method that is used to measure feature weight in text categorization．However, the algorithm doesn t take the distribution of feature terms in inter-class and intra—class into consideration，which makes the algorithm can t effectively weigh the distribution proportion of feature term s．In order to solve the problem，variance in inter-class and intra—class which describes the distribution offeature term s is used to revise TFIDF weight．Comp~ed with traditional TFIDF algorithm ，the results of simulation experiment have demonstrated that the improved TFIDF algorithm Can get better classification results． Key words：text categorization；feature selection；improved TFIDF；weight；variance 面影响；②特征词选取的好坏与否取决于其能否代表类和文 0 引 言 档的属性，TF值高的特征词，如果在所有文档中TF值都高， 随着网络和信息技术的飞速发展，人们可获得的知识越 则难以说明其到底代表哪个文档。反文档频率(inverse docu． 来越多，但是面对具体的知识，要在这浩瀚的信息世界中快速 ment frequency，IDF)是特征项在文档集分布情况