An Introduction to Boosting.ppt

An Introduction to Boosting Yoav Freund Banter Inc. Plan of talk Generative vs. non-generative modeling Boosting Alternating decision trees Boosting and over-fitting Applications Toy Example Computer receives telephone call Measures Pitch of voice Decides gender of caller Generative modeling Discriminative approach Ill-behaved data Traditional Statistics vs. Machine Learning Comparison of methodologies A weak learner The boosting process Adaboost Binary labels y = -1,+1 margin(x,y) = y [St at ht(x)] P(x,y) = (1/Z) exp (-margin(x,y)) Given ht, we choose at to minimize S(x,y) exp (-margin(x,y)) Main property of adaboost If advantages of weak rules over random guessing are: g1,g2,..,gT then in-sample error of final rule is at most (w.r.t. the initial weights) A demo www.cs.huji.ac.il/~yoavf/adabooost Adaboost as gradient descent Discriminator class: a linear discriminator in the space of “weak hypotheses” Original goal: find hyper plane with smallest number of mistakes Known to be an NP-hard problem (no algorithm that runs in time polynomial in d, where d is the dimension of the space) Computational method: Use exponential loss as a surrogate, perform gradient descent. Margins view Adaboost et al. One coordinate at a time Adaboost performs gradient descent on exponential loss Adds one coordinate (“weak learner”) at each iteration. Weak learning in binary classification = slightly better than random guessing. Weak learning in regression – unclear. Uses example-weights to communicate the gradient direction to the weak learner Solves a computational problem What is a good weak learner? The set of weak rules (features) should be flexible enough to be (weakly) correlated with most conceivable relations between feature vector and label. Small enough to allow exhaustive search for the minimal weighted training error. Small enough to avoid over-fitting. Should be able to calculate predicted label very efficiently. Rules can be “specialists”