Ensemble Methods Bagging and Boosting.ppt

Ensemble Methods: Bagging and Boosting Lucila Ohno-Machado HST951 Topics Combining classifiers: Ensembles (vs. mixture of experts) Bagging Boosting Ada-Boosting Arcing Stacked Generalization Combining classifiers Examples: classification trees and neural networks, several neural networks, several classification trees, etc. Average results from different models Why? Better classification performance than individual classifiers More resilience to noise Why not? Time consuming Overfitting Bagging Breiman, 1996 Derived from bootstrap (Efron, 1993) Create classifiers using training sets that are bootstrapped (drawn with replacement) Average results for each case Bagging Example (Opitz, 1999) Boosting A family of methods Sequential production of classifiers Each classifier is dependent on the previous one, and focuses on the previous one’s errors Examples that are incorrectly predicted in previous classifiers are chosen more often or weighted more heavily Ada-Boosting Freund and Schapire, 1996 Two approaches Select examples according to error in previous classifier (more representatives of misclassified cases are selected) – more common Weigh errors of the misclassified cases higher (all cases are incorporated, but weights are different) – does not work for some algorithms Boosting Example (Opitz, 1999) Ada-Boosting Define ek as the sum of the probabilities for the misclassified instances for current classifier Ck Multiply probability of selecting misclassified cases by bk = (1 – ek)/ ek “Renormalize” probabilities (i.e., rescale so that it sums to 1) Combine classifiers C1…Ck using weighted voting where Ck has weight log(bk) Arcing Arcing-x4 (Breiman, 1996) For the ith example in the training set, mi refers to the number of times that it was misclassified by the previous K classifiers Probability pi of selecting example i in the next classifier K-1 is Empirical determination Bias plus variance decomposition Geman, 1992 Bias: how close the average cla