Data Mining – Best Practices数据挖掘–最佳实践.ppt

EFFECTIVE PREDICTIVE MODELING- DATA,ANALYTICS AND PRACTICE MANAGEMENT Richard A. Derrig Ph.D. OPAL Consulting LLC Karthik Balakrishnan Ph.D. ISO Innovative Analytics CANE Meeting Southbridge, MA September 26, 2008 Topics in Predictive Modeling Introduction Explanatory vs. Predictive Modeling Data Quality Data augmentation Data adjustments Predictive Problems/software issues Analytic Methods of Interest Introduction Research by IBM indicates only 1% of data collected by organizations is used for analysis Predictive Modeling and Data Mining widely embraced by leading businesses 2002 Strategic Decision Making survey by Hackett Best Practices determined that world class companies adopted predictive modeling technologies at twice the rate of other companies Important commercial application is Customer retention: 5% increase in retention ? 95% increase in profit It costs 5 to 10 times more to acquire new business Another study of 24 leading companies found that they needed to go beyond simple data collection Models Explanatory vs Predictive Explanatory Model The association of some target or decision variable with explanatory variables through mathematical formulations in which it is assumed that all variables are known with certainty. Example: Underwriting Model, decision to accept or reject is target Models Explanatory vs Predictive Predictive Model The association of some target or decision variable with predictor variables through mathematical formulations in which it is assumed that all predictor variables are known with possibly varying probabilities . Example: Claim Investigation Model, decision to investigate is target Data Mining Data Mining, also known as Knowledge-Discovery in Databases (KDD), is the process of automatically searching large volumes of data for patterns. In orde