第三章 数据预处理2015-1.ppt

* 由自然划分离散化 3-4-5 规则 如果最高有效位包含 3, 6, 7 or 9 个不同的值, partition the range into 3 个等宽区间（7：2-3-2分成3个区间） 2, 4, or 8 不同的值, 区域分成 4 个等宽区间 1, 5, or 10 不同的值, 区域分成5 个等宽区间 类似地，逐层使用此规则 * 分类数据的概念分层 Categorical Data 用户/专家在模式级显式地指定属性的偏序 streetcitystatecountry 通过显式数据分组说明分层 {厄巴纳，香槟，芝加哥}Illinois 只说明属性集 系统自动产生属性偏序，根据 每个属性下不同值的数据 启发式规则：相比低层，高层概念的属性通常有较少取值 E.g., street city state country 只说明部分属性值 * 自动产生概念分层 Some concept hierarchies can be automatically generated based on the analysis of the number of distinct values per attribute in the given data set 含不同值最多的属性放在层次的最低层 Note: Exception—weekday, month, quarter, year country province_or_ state city street 15 distinct values 65 distinct values 3567 distinct values 674,339 distinct values * Summary Data preparation is a big issue for both warehousing and mining Data preparation includes Data cleaning and data integration Data reduction and feature selection Discretization A lot a methods have been developed but still an active area of research Data Reduction, Transformation, Integration Data Quality Major Tasks in Data Preprocessing Data Cleaning and Data Integration Data Cleaning i. Missing Data and Misguided Missing Data ii. Noisy Data iii. Data Cleaning as a Process Data Integration Methods Data Reduction Data Reduction Strategies Dimensionality Reduction i. Principal Component analysis ii. Feature Subset Selection iii. Feature Creation Numerosity Reduction i. Parametric Data Reduction: Regression and Log-Linear Models ii. Mapping Data to a New Space: Wavelet Transformation iii. Data Cube aggregation iv. Data Compression v. Histogram analysis vi. Clustering vii. Sampling: Sampling without Replacement, Stratified Sampling Data Transformation and Data Discretization Data Transformation: Normalization Data Discretization Methods i. Binning ii. Cluster Analysis iii. Discretization Using Class Labels: Entropy-Based Discretization iv. Discretization Without Using Class Labels: Interval Merge by ?2 Analysis Concept Hierarchy and Its Formation