数据挖掘CHAPTER数据预处理创新.pptVIP

* 自动产生概念分层 Some concept hierarchies can be automatically generated based on the analysis of the number of distinct values per attribute in the given data set 含不同值最多的属性放在层次的最低层 Note: Exception—weekday, month, quarter, year country province_or_ state city street 15 distinct values 65 distinct values 3567 distinct values 674,339 distinct values * Summary Data preparation is a big issue for both warehousing and mining Data preparation includes Data cleaning and data integration Data reduction and feature selection Discretization A lot a methods have been developed but still an active area of research Data Reduction, Transformation, Integration Data Quality Major Tasks in Data Preprocessing Data Cleaning and Data Integration Data Cleaning i. Missing Data and Misguided Missing Data ii. Noisy Data iii. Data Cleaning as a Process Data Integration Methods Data Reduction Data Reduction Strategies Dimensionality Reduction i. Principal Component analysis ii. Feature Subset Selection iii. Feature Creation Numerosity Reduction i. Parametric Data Reduction: Regression and Log-Linear Models ii. Mapping Data to a New Space: Wavelet Transformation iii. Data Cube aggregation iv. Data Compression v. Histogram analysis vi. Clustering vii. Sampling: Sampling without Replacement, Stratified Sampling Data Transformation and Data Discretization Data Transformation: Normalization Data Discretization Methods i. Binning ii. Cluster Analysis iii. Discretization Using Class Labels: Entropy-Based Discretization iv. Discretization Without Using Class Labels: Interval Merge by ?2 Analysis Concept Hierarchy and Its Formation i. Concept Hierarchy Generation for Numerical Data ii. Concept Hierarchy Generation for Categorical Data iii. Automatic Concept Hierarchy Generation * References E. Rahm and H. H. Do. Data Cleaning: Problems and Current Approaches. IEEE Bulletin of the Technical Committee on Data Engineering. Vol.23, No.4 D. P. Ballou and G. K. Tayi. Enhancing data quality in data warehouse en