DM10 Evaluation and Credibility .pptVIP

Evaluation and Credibility How much should we believe in what was learned? Outline Introduction Classification with Train, Test, and Validation sets Handling Unbalanced Data; Parameter Tuning Cross-validation Comparing Data Mining Schemes Introduction How predictive is the model we learned? Error on the training data is not a good indicator of performance on future data Q: Why? A: Because new data will probably not be exactly the same as the training data! Overfitting – fitting the training data too precisely - usually leads to poor results on new data Evaluation issues Possible evaluation measures: Classification Accuracy Total cost/benefit – when different errors involve different costs Lift and ROC curves Error in numeric predictions How reliable are the predicted results ? Classifier error rate Natural performance measure for classification problems: error rate Success: instance’s class is predicted correctly Error: instance’s class is predicted incorrectly Error rate: proportion of errors made over the whole set of instances Training set error rate: is way too optimistic! you can find patterns even in random data Evaluation on “LARGE” data, 1 If many thousands of examples are available, including several hundred examples from each class, then how can we evaluate our classifier method? Evaluation on “LARGE” data, 2 A simple evaluation is sufficient Randomly split data into training and test sets usually 2/3 for train, 1/3 for test Build a classifier using the train set and evaluate it using the test set. Classification Step 1: Split data into train and test sets Classification Step 2: Build a model on a training set Classification Step 3: Evaluate on test set Re-train? Unbalanced data Sometimes, classes have very unequal frequency Attrition prediction: 97% stay, 3% attrite in a month medical diagnosis: 90% healthy, 10% disease eCommerce: 99% don’t buy, 1% buy Security: 99.99% of Americans are not terrorists Similar situation with multiple cla