一种新的基于统计的自动文本分类方法.pdfVIP

1一种新的基于统计的自动文本分类方法 刘斌1 黄铁军2 程军3 高文1 1 中国科学院计算技术研究所 100080 北京 2 中国科学院研究生院 100080 北京 3 中国科学院文献情报中心 100080 北京 摘要: 自动文本分类就是在给定的分类体系下，让计算机根据文本的内容确定与它相关联的类 别。为了提高分类性能，本文提出了中文文本多层次特征提取方法和基于核的距离加权KNN 算法。 多层次特征提取方法在汉字、常用词表和专业词表三个层次上提取文档的统计特征，能够更好地反 映文档的统计分布。基于核的距离加权KNN 算法解决了样本的多峰分布、边界重叠问题和分类器的 精确分类决策问题。实际应用中，互联网和文本库提供了大量经过粗分类的训练文本，但普遍存在 样本质量较差的问题，本文通过样本重要性分析技术解决此问题。实验系统证明了新方法的有效性。 关键词: 自动文本分类；多层次特征提取；基于核的距离加权KNN 算法；样本重要性分析 中图分类号: TP391 A New Statistical-based Method in Automatic Text Classification 1 2 3 1 LIU Bin HUANG Tiejun CHENG Jun GAO Wen 1 Institute of Computing Technology, Chinese Academy of Sciences 100080 Beijing 2 Graduate School of Chinese Academy of Sciences 100080 Beijing 3 The Library of Chinese Academy of Sciences 100080 Beijing Abstract: Automatic text classification is defined as the task to assign pre-defined category labels to documents. To improve the classification performance, this article puts forward the multi-level feature selection method and the kernel-based distance-weighted KNN algorithm. We extract the statistical text features on three different levels as Chinese letters, the common wordlist and the professional wordlist, which can represent more statistical character of the document set. The kernel-based weighted KNN algorithm solves the multi-peak distribution problem and the overlap boundary problem of the sample set, as well as the classifier’s precise decision problem. In practical use, the Internet and text databases provide many pre-classified training sam