ensemble_methods.pptx

An Introduction to Ensemble MethodsBoosting, Bagging, Random Forests and MoreYisong YueSupervised LearningGoal: learn predictor h(x) High accuracy (low error)Using training data {(x1,y1),…,(xn,yn)}PersonAgeMale?Height 55”Alice1401Bob1011Carol1301Dave810Erin1100Frank911Gena800h(x) = sign(wTx-b)PersonAgeMale?Height 55”Alice1401Bob1011Carol1301Dave810Erin1100Frank911Gena800Male?YesNoAge9?Age10?YesYesNoNo1010Splitting Criterion: entropy reduction (info gain)Complexity: number of leaves, size of leavesOutlineBias/Variance TradeoffEnsemble methods that minimize varianceBaggingRandom ForestsEnsemble methods that minimize biasFunctional Gradient DescentBoostingEnsemble SelectionGeneralization Error“True” distribution: P(x,y) Unknown to usTrain: h(x) = y Using training data S = {(x1,y1),…,(xn,yn)}Sampled from P(x,y)Generalization Error: L(h) = E(x,y)~P(x,y)[ f(h(x),y) ] E.g., f(a,b) = (a-b)2PersonAgeMale?Height 55”James1111Jessica1401Alice1401Amy1201Bob1011Xavier910Cathy901Carol1301Eugene1310Rafael1211Dave810Peter910Henry1310Erin1100Rose700Iain811Paulo1210Margaret1001Frank911Jill1300Leon1010Sarah1200Gena800Patrick511PersonAgeMale?Height 55”Alice1401Bob1011Carol1301Dave810Erin1100Frank911Gena800yh(x)Generalization Error:L(h) = E(x,y)~P(x,y)[ f(h(x),y) ] …Bias/Variance TradeoffTreat h(x|S) has a random function Depends on training data SL = ES[ E(x,y)~P(x,y)[ f(h(x|S),y) ] ]Expected generalization errorOver the randomness of SBias/Variance TradeoffSquared loss: f(a,b) = (a-b)2Consider one data point (x,y)Notation: Z = h(x|S) – y ? = ES[Z]Z-? = h(x|S) – ES[h(x|S)]Expected ErrorES[(Z-?)2] = ES[Z2 – 2Z? + ?2] = ES[Z2] – 2ES[Z]? + ?2 = ES[Z2] – ?2ES[f(h(x|S),y)] = ES[Z2] = ES[(Z-?)2] + ?2Bias/Variance for all (x,y) is expectation over P(x,y).Can also incorporate measurement noise.(Similar flavor of analysis for other loss functions.)VarianceBiasExampleyxh(x|S)h(x|S)h(x|S)BiasVarianceBiasVarianceVarianceBiasES[(h(x|S) - y)2] = ES[(Z-?)2] + ?2Z = h(x|S) – y ? = ES[Z]Variance