基于网络的数据清洗技术分析-计算机应用技术专业论文.docxVIP

哈尔滨工业大学工程硕士学位论文 哈尔滨工业大学工程硕士学位论文 - - II - Abstract As time goes by, more and more data is produced and stored in the information systems, some even reached TB or PB level. While some factor for example, out-dated, wrong value, duplication and confliction affect the quality of data. All the data clean methods are based on known FD/CFD, while in fact, not all the FD/CFDs are known in advance. To solve the data quality problems and overcome the drawback of current approaches, a Web-based data clean method is proposed in this paper. There are vast amounts of information on the internet can be used to support data cleaning work. When a tuple is checked whether it can meet the quality requirements, some text patterns can be obtained on the internet based on the relation data to support the judgment. And the information on the interment can also be used to clean the data that have quality problems. To propose the Web-based data cleaning framework, firstly, the definitions of data quality problems, data clean contents and Web-based data cleansing techniques are given. Then we classify the data. The correct tuples are used to search the text patterns, and the data with quality problems are to be cleaned by using the text patterns and the information on the Web. The Web-based data cleansing framework are made by three parts. The first part is data quality problem detection, the second part is text pattern access, and the last part is data cleansing based on the text pattern. The data quality problem detection part interacts with the others. The text pattern access part contains keywords generation and pattern generation. The data cleansing contains feasible tuples selection, tuple cleaning and other strategies. At last, abundant experiments have been done to verify the efficiency and effectiveness of the algorithms proposed in this paper. Keywords: data cleaning, entity resolution, Web information, text pattern dependency. - - PAGE IV - 目 录 摘 要 I HYPERLINK \l _bookmark0 ABS