面向文本聚类的语义加权研究-通信与信息系统专业毕业论文.docxVIP

第 第 II 页 词语与词语之间的相似度指的是两个词语在不同的上下文中可以 互相替换使用而不改变文本的句法语义结构的程度。本文利用了基于 《知网》的语义相似度计算方法，对特征向量进行基于语义相似度的加 权。实验研究发现，基于语义相似度的特征向量加权虽然能够提高最终 文本聚类效果，但是其效果并没有基于词语相关度的加权方法这么明 显。本文也尝试着寻找了一些原因和解决的方法。 最后本文对于把几种基于语义的加权方法叠加使用的可行性方法 进行了探讨。实验研究发现，本文提出的这种叠加方案是完全可行的， 比只使用任何一种语义加权方法的效果都要好,大约有 5%的提高。 关键词：向量空间模型，文本聚类，词语相关度，词语相似度 第 第 III 页 RESEARCH OF FEATURE VECTOR VALUE WEIGHTED BASED ON SEMANTIC ANALYSIS IN CHINESE TEXT CLUSTERING ABSTRACT With the rapid development of Internet, people find it difficult to find their useful information through Internet. The application of text mining is very broad, while text clustering is an important part of text mining. Text clustering is an important research branch of clustering method and it is the application of clustering method used in text processing field. But text clustering has some common problems such as high-dimensional text object, a document collection may need hundreds of thousands of words to express; the sparse of text object, a lot of words are rarely used. However the literal meanings of text object and potential semantic relations could hardly be excavated. First, this paper reviews some concept and methods of Chinese text pre-processing, text representation, text similarity calculation, and document feature vector reduction, clustering algorithms and clustering efficiency evaluation. Then this paper discusses the main problem of text clustering in excavation of literal meaning of text object and semantic relations and presents several methods of feature vector value weighted based on semantic analysis: based on word part of speech, based on the term location, based on term length, based on the term correlation and based on term similarity. The term correlation means the probability of two terms appearing in a certain language environment. In general we use document co-occurrence frequency, paragraph co-occurrence frequency and sentence co-occurrence frequency to measure term correlation. Study shows that the method of 第 第 PAGE