机器树学习相关内容_Decision Tree Learning.pptVIP

Compared with FIND-S and CANDIDATE-ELIMINAITON ID3 vs. CANDIDATE-ELIMINATION Source of the picture: /wiki/William_of_Ockham Issues Avoid overfitting the data Incorporating continuous-valued attributes Attributes with many values Unknown attribute values Attributes with various costs * Overfitting in Decision Trees Consider adding noisy training example #15: Sunny, Hot, Normal, Strong, PlayTennis=No, what is the new tree? * The Tree becomes… * Overfitting Consider error of hypothesis h over training data: errortrain(h) entire distribution D of data: errorD(h) Hypothesis h∈H overfits training data if there is alternative hypothesis h’∈H such that errortrain(h)errortrain(h’), but errorD(h) errorD(h’) . * Overfitting in Decision Trees Learning See /~tom/mlbook.html for software and data for more experiments * Avoid Overfitting How can we avoid overfitting? Stop growing when data split not statistically significant (previous example) Grow full tree, then post-prune How to select “best” tree? Measure performance over both training data and separate validation data Use a measure such as Minimum Description Length (MDL) to minimize both size(tree) and size(misclassifications(tree)) * Avoid Overfitting How can we avoid overfitting? Stop growing when data split not statistically significant (previous example) Grow full tree, then post-prune How to select “best” tree? Measure performance over both training data and separate validation data Minimum Description Length (MDL): minimize size(tree)+size(misclassifications(tree)) * Reduced-Error Pruning Split data into training and validation set Do until further pruning is harmful: Evaluate impact on validation set of pruning each possible node (i.e. removing the subtree rooted at the node, and making the node a leaf) Greedily remove the one that most improves validation set accuracy * Effect of Reduced-Error Pruning * What if data is limited? Try rule post-pruning! Rule Post-Pruning Convert tree to equivalent set of rules (o