boosting算法理论与应用研究.pdfVIP

第 三 届 CCF 大 数 据 学 术 会 议 投 稿 Boosting 算法理论与应用研究 于廷照 张文生 ( 中科院自动化所, 北京 100190) 摘 要 作为机器学习领域最经典算法之一，Boosting 是一种集成学习算法，并广泛应用于机器学习与模式识别各领域。 Boosting 的理论研究分为可学习理论和统计学两个角度。Boosting 最初从弱可学习理论角度阐明了由弱到强的提升算法，从 理论上证明了一组优于随机猜测的弱学习器通过集成，可提升为在训练集上任意精度的强学习器。从统计学的角度看， Boosting 是一种叠加模型，理论上二者的等价性已经证明。本文首先从可学习的角度出发，回顾了Boosting 算法弱可学习理 论，并提出面临的问题及挑战，包括对高维数据的有效性及Margin 理论。随即说明Boosting 算法理论研究分支，并详细回 顾了当前最为流行的几种经典Boosting 算法，及在Boosting 理论框架下的新应用。最后，在探讨Boosting 算法未来研究趋 势的基础上总结全文。 关键词 Boosting ；弱可学习理论；Margin 理论；集成学习；AdaBoost Research on Boosting theory and application Yu Ting-Zhao Zhang Wen-Sheng (Institute of Automation, Chinese Academy of Science, Beijing, 100190) Abstract Boosting is one of the most popular ensemble algorithm in machine learning, and it has been widely used in machine learning and pattern recognition. There are mainly two frameworks of Boosting, PAC theory and statistical theory. Boosting is first proposed from theory of weak learnability which illustrates the theory of boosting a group of weak learners into a strong learner. It is demonstrated that after a finite number of iterations, the combination of weak learners could be boosted into any accuracy on training set, and the only requirement for a weak learner is that the accuracy be slightly better than random guess. From statistical point of view, Boosting is an additive model, the equivalence between these two models has already been proved. In this paper, we start by reviewing the theory of weak learnability from PAC perspective, and propose the challenge Boosting may face, which includes the effectiveness for high dimension data and the Margin theory. Then we describe various Boo