数据挖掘之数据预处理.pptVIP

Data preprocessing;outline;Data in the real world is dirty Incomplete: occupation=“ ” Noisy: Salary=“-10” Inconsistent: M,F; 0,1;Why is Data Preprocessing Important?;Major Tasks in Data Preprocessing;;Data cleaning ;Missing values;How to Handle Missing Data?;Noisy data;How to Handle Noisy Data?;Binning method;How can we smooth out the data;Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25,26, 28, 29, 34 * Partition into (equidepth) bins: -Bin 1: 4, 8, 9, 15 -Bin 2: 21, 21, 24, 25 -Bin 3: 26, 28, 29, 34 * Smoothing by bin means: -Bin 1: 9, 9, 9, 9 -Bin 2: 23, 23, 23, 23 -Bin 3: 29, 29, 29, 29 * Smoothing by bin boundaries: -Bin 1: 4, 4, 4, 15 -Bin 2: 21, 21, 25, 25 -Bin 3: 26, 26, 26, 34;Clustering;Regression;Combined computer and human inspection;Data integration and transformation;Data integration;data value conflicts;Redundancy in Data Integration;Data transformation;Normalization;;Suppose that the minimum and maximum values for the attribute income are $12000 and $98000, respectively. We would like map income to the range [0.0,1.0].By min-max normalization, a value of $ 73600 for income is transformed to ?(0.716);Suppose that the mean and stand deviation of the values for the attribute income are $54000 and $16000,respectively. With z-score normalization, a value of $73600 for income is transformed to ?(1.225) ;Suppose that the recorded values of A range from -986 to 917. the maximum absolute value of A is 986.To normalize by decimal scaling, we therefore divide each value by 1000(i.e., j=3) so that -986 normalizes to -0.986;attributes construction;Data reduction;Data cube aggregation;;Dimensionality reduction;Basic heuristic methods of attribute subset selection;;Data compression;Lossy data compression;Numerosity reduction;Histograms;Cluster Analysis;Sampling;;Discretization and concept hierarchy generation;Discretization;Concept hierarchies;A concept hierarchy for the attribute price;concept hierarchy generation for numeric data;concep