数据挖掘标准规范工具和发展趋势概述.pptx

数据挖掘;本章内容;8.1 数据挖掘标准与规范; 数据挖掘相关标准 CRISP-DM（交叉行业数据挖掘过程标准，Cross Industry Standard Process for Data Mining）。SPSS、NCR以及DaimlerChrysler三个在数据挖掘领域经验丰富的公司发起建立一个社团，目的建立数据挖掘方法和过程的标准 ;Crisp - DM;Business Understanding Phase Understand the business objectives What is the status quo? Understand business processes Associated costs/pain Define the success criteria Develop a glossary of terms: speak the language Cost/Benefit Analysis Current Systems Assessment Identify the key actors Minimum: The Sponsor and the Key User What forms should the output take? Integration of output with existing technology landscape Understand market norms and standards;Business Understanding Phase Task Decomposition Break down the objective into sub-tasks Map sub-tasks to data mining problem definitions Identify Constraints Resources Law e.g. Data Protection Build a project plan List assumptions and risk (technical/ financial/ business/ organisational) factors;Data Understanding Phase Collect Data What are the data sources? Internal and External Sources (e.g. Axiom, Experian) Document reasons for inclusion/exclusions Depend on a domain expert Accessibility issues Are there issues regarding data distribution across different databases/legacy systems Where are the disconnects?;Data Understanding Phase Data Description Document data quality issues Compute basic statistics Data Exploration Simple univariate data plots/distributions Investigate attribute interactions Data Quality Issues Missing Values: Understand its source Strange Distributions ;Data Preparation Phase Integrate Data Joining multiple data tables Summarisation/aggregation of data Select Data Attribute subset selection Rationale for Inclusion/Exclusion Data sampling Training/Validation and Test sets;Data Preparation Phase Data Transformation Using functions such as log Factor/Principal Components analysis Normalization/Discretisation/Binarisation Clean Data Handling missing values/Outliers Data Construction Derived Attributes;The Mo