Chapter 3. 数据预处理.ppt

* * * MK 08.11.02: This needs to be updated to reflect changes. * * * * * * * * * * * * Removed 09/09/10: V-optimal: with the least histogram variance (weighted sum of the original values that each bucket represents) * * * * * * * PCA and Wavelet are forms of data compression also. * * * * * 08.09.12 MK: Slide moved here from Ch. 3. Added examples. * * * * * * * Should we briefly introduce ontology? Ontology has been used extensively in bio-medical data warehouses. -- JP We should introduce this part using 1-2 slides -JH * * * 第3章: 数据预处理 数据预处理: 综述 数据质量 数据预处理的主要任务 数据清理 数据集成 数据归约 数据变换与数据离散化 小结 * 小结 数据质量: accuracy, completeness, consistency, timeliness, believability, interpretability 数据清理: e.g. missing/noisy values, outliers 数据集成from multiple sources: Entity identification problem Remove redundancies Detect inconsistencies 数据归约 Dimensionality reduction Numerosity reduction Data compression 数据变换和数据离散化 Normalization Concept hierarchy generation * References D. P. Ballou and G. K. Tayi. Enhancing data quality in data warehouse environments. Comm. of ACM, 42:73-78, 1999 T. Dasu and T. Johnson. Exploratory Data Mining and Data Cleaning. John Wiley, 2003 T. Dasu, T. Johnson, S. Muthukrishnan, V. Shkapenyuk.?Mining Database Structure; Or, How to Build a Data Quality Browser. SIGMOD’02 H. V. Jagadish et al., Special Issue on Data Reduction Techniques. Bulletin of the Technical Committee on Data Engineering, 20(4), Dec. 1997 D. Pyle. Data Preparation for Data Mining. Morgan Kaufmann, 1999 E. Rahm and H. H. Do. Data Cleaning: Problems and Current Approaches. IEEE Bulletin of the Technical Committee on Data Engineering. Vol.23, No.4 V. Raman and J. Hellerstein. Potters Wheel: An Interactive Framework for Data Cleaning and Transformation, VLDB’2001 T. Redman. Data Quality: Management and Technology. Bantam Books, 1992 R. Wang, V. Storey, and C. Firth. A framework for analysis of data quality research. IEEE Trans. Knowledge and Data Engineering, 7:623-640, 1995 * * * *