《BagBoo Bagging the Gradient》.pdf

BagBoo: Bagging the Gradient Boosting 3rd in RR (track I and II) 1st in nDCG (track II) and 2nd in nDCG (track I) Dmitry Pavlov and Cliff Brunk aka JOKER aka team_404 Yandex Labs () {dmitry-pavlov,cliff}@yandex-team.ru Yandex • Yandex.ru Y • Leading search engine in Russia w/65%+ search market share • Labs office in Palo Alto, CA • We are doing a lot of technology innovations • … and We are hiring! • Come chat with us or send a message Bagging (Breiman) • Ensemble of models – Sampling data – Voting models • Random Forest – Models are functions of iid random vectors – Assume Model = Tree WLOG from now on • Nice properties – Variance reduction – Resistance to overfitting – Efficient parallelizable computation Gradient Boosting (Friedman) • Ensemble of models – Each next model learned to optimize residual error – Weight in the linear combination are optimized – Randomness/Stochastisity similar to Bagging • Nice properties – Bias reduction • Hard to parallelize BagBoo: combined Bagging and Boosting • Combine the best of both worlds: – get highly parallelizable algorithm with bias and variance reduction properties 1. Input: Training data D, Nbag and Nboo iter. 2. Output: Random Forest of Nbag x Nboo trees 3. For i=1 to Nbag do D[i] := SampleData(D) ; # samp. feats and records BT[i] := BoostedTree(D[i], Nboo) ; EndFor 4. Output: additive model \sum_i { BT[i] } BagBoo: highly parallelizable algorithm • Accurate – Excellent results in contests and on TREC benchmarks • Fast – Can train many trees fast • Gotchas – need to control learning rate – winning the contest with many trees