3国立联合大学资讯管理学系资料探勘课程(陈士杰).ppt

Data Mining: Concepts and Techniques Course 2資料預處理Data Preprocessing ? Outline Why preprocess the data? (為何要做資料的預處理) Descriptive data summarization (資料的摘要性描述) Data cleaning (資料清理) Data integration and transformation (資料整合與轉換) Data reduction (資料縮減) Discretization and concept hierarchy generation (離散化與概念分層的產生) Summary ? Why Data Preprocessing? Data in the real world is dirty – 只要人一多，什麼樣的怪腳都可能會出現!! Incomplete (不完整的): lacking attribute values, lacking certain attributes of interest e.g., occupation=“ ” Noisy (含噪音的): containing errors or outliers e.g., Salary=“-10” Inconsistent (不一致的): containing discrepancies in codes or names e.g., Age=“42” Birthday=“03/07/1997” e.g., Was rating “1,2,3”, now rating “A, B, C” e.g., discrepancy between duplicate records Why Is Data Dirty? Incomplete data may come from “Not applicable (不合用)” data value when collected Different considerations between the time when the data was collected and when it is analyzed. Human/hardware/software problems Noisy data (incorrect values) may come from Faulty data collection instruments (如: 問卷設計不良) Human or computer error at data entry Errors in data transmission Inconsistent data may come from Different data sources Functional dependency (功能相依性) violation (e.g., modify some linked data) Why Is Data Preprocessing Important? No quality data, no quality mining results! Quality decisions must be based on quality data e.g., duplicate or missing data may cause incorrect or even misleading statistics. Data warehouse needs consistent integration of quality data Data extraction, cleaning, and transformation comprises the majority of the work of building a data warehouse Multi-Dimensional Measure of Data Quality A well-accepted multidimensional view: Accuracy (精確性) Completeness (完整性) Consistency (一致性) Timeliness (及時性) Believability (可信度) Value added (附加價值) Interpretability (可解釋性) Accessibility (易接受) Major Tasks in Data Preprocessing Data cleaning (資料清理) Fill in missing values, smooth noisy data, identify or