数据挖掘论文：基于web文献的数据挖掘研究应用01.doc

数据挖掘论文：基于web文献的数据挖掘研究应用 【中文摘要】随着高等教育的大众化,高校人数由过去的几十万上升到几百万,国家也会提供大量的资金资助大量科研项目,每年都会有数以万计的文献产生。由于Web文献的大量累积,人们很难从海量的文献数据中寻找到有用的信息,也就起不到提高工作效率的作用。本文的主要就是利用数据挖掘技术从大量的文献数据中找到有用的信息,以便进一步的指导工作。为了选择适合大量文献数据的数据挖掘算法,本文首先对数据挖掘的理论知识做了简要的介绍,给出了文本相似度计算的一般流程和公式,对几种聚类算法进了分析比较,发现一些不足的地方。根据聚类效果的评估原则和增量聚类算法的思想,设计了一个基于内聚度的增量聚类算法,弥补了上面几种算法的不足,然后通过相关实验对该聚类算法的参数进行了优选。查阅相关文献和分析PaperPass软件的检测结果,得出了一个计算文献相似度的计算方法,以便对文献抄袭现象进行检查。根据采用空间向量计算文本相似度的方式,改进了计算相似度的算法。为了获取大量的Web文献数据,本文研究了爬虫的相关知识,设计并实现了一个文献聚集爬虫。本文为了应用上面的算法和为用户提供可操作的平台,设计了一个基于Web文献的数据挖掘系统。本文对该系统的目标和特点进行了分析,选择了相关的技术路线,完成了系统架构、功能及主要模块的划分与设计,设计了系统数据库。最后,给出了系统的运行部署方法和相关功能的演示。 【英文摘要】With the development of higher education, the number of university students has been increased from hundred thousand to several million during the past few years, the government will provide substantial fundings, and thus a large number of research projects are generated each year. Due to the accumulation of a large number of Web documents, it is difficult to find useful information from the mass of literature data, let alone improve the efficiency. The main purpose of this thesis is to find useful information from a large number of literature data for further guidance by using data mining technology.To find data mining algorithms suited for a large number of literature datas, firstly, this thesis gives a brief introduction to theoretical knowledge of data mining, and gives a general similarity calculation process and formula of the text, where we present an analysis of several clustering algorithms and find some deficiencies. According to the principles of clustering effect sassessment and the thinking of incremental clusterings, we design a cohesion-based incremental clustering algorithm, which makes up the deficiency of several above-mentioned algorithms. Then the parameters of the clustering algorithm are optimized by some relevant experiments. By referring to relevant literatures and analysizing the test results of PaperPass software, a method for caculating