数据质量论文应用于数据质量核查的几项数据挖掘技术研究.doc

数据质量论文：应用于数据质量核查的几项数据挖掘技术研究 【中文摘要】质量低劣的数据已经成为影响企业正确决策的关键因素,成为制约信息服务的瓶颈。因此,如何高效的管理数据,提高数据质量,使其成为决策部门的有效依据,是具有较高研究价值和实际意义的问题。本文由此背景出发,根据不同类型的数据错误,采用相应检测方法,并通过实现具体程序,验证方法的有效性。本文首先介绍了数据质量的定义、分类、评价指标以及数据质量提高技术。然后总结了数据清洗技术的原理方法。最后针对不同的数据错误类型,给出了相应的解决方法。本文重点给出了异常数据和相似重复记录的检测方法。本文充分考虑数据内部之间的联系,采用基于关联规则的思想对异常数据进行检测。首先,将数据集中的数据进行转换,使其满足关联规则挖掘的条件；然后,在训练集中找出所有的频繁项集,由频繁项集生成关联规则,并将其放入规则库；最后,将测试集中的记录与规则库中的规则进行比对,以此判断记录是否异常。通过实验证明此方法检测异常数据效果良好。本文采用基于权值分组的方法来检测相似重复记录。根据识别事物的能力给不同的属性分配相应的权值,以此提高检测精度；根据关键字段将大数据集分割成不相交的小数据集,再在小数据集中检测相似重复记录,以减少匹配次数；采用位置编码方法计算字段相似度,解决了英文缩写问题和中文字符匹配问题；使用多趟查找技术克服了字符敏感问题。实验证明此方法能够快速准确的检测相似重复记录。 【英文摘要】Data in poor quality has become a key factor for enterprise to do the right decision, and a bottleneck of information service. Therefore, how to manage data efficiently and improve the quality to make data an effective basis for decision-making department is a problem with high research value and practical significance. In this context, this dissertation according to the different types of data errors through implementing specific program uses the appropriate solutions to verify the validity of the method.First, this dissertation introduces the definition of data quality, classification, evaluation index and the technology of improving the data quality. Second, summarize the principle and the method of data cleansing techniques. Finally, give the corresponding solutions for different error types especially on the duplicate records and similar abnormal data detection method.Fully considering the link within data, this dissertation detects abnormal data using the idea based on association rules. Firstly, convert the data in the dataset to meet the conditions for mining association rules. Secondly, find all the frequent item sets in the training set and generate the association rules from the frequent item sets and put them into the rule base. Finally, compare the records in the test set and rules in t