机器学习中的随机优化算法2.53.doc

机器学习中的随机优化算法 Stochastic Optimization Algorithm in Machine Learning PAGE I 中文摘要 对于机器学习中的数值优化问题，考虑到其规模和维数都比较大，传统的方法难以高效的解决这一问题。近些年来，针对大规模的机器学习问题做了很多研究，比较重要的一类方法是 随机算法。优化方法主要分为一阶梯度方法和二阶牛顿方法两大类，针对一阶方法的改进和研究比较成熟和完善，一阶方法又可以分为两类，原始方法和对偶方法，原始方法比较有代表性的有SAG、SAGA、SVRG，对偶方法有SDCA和SPDC两种。此外，加速方法如Catalyst、Katyusha，其收敛速度为一阶方法所能达到的最好结果。二阶方法目前是机器学习领域研究的重要方向，其收敛速度要优于一阶方法，但是其实践中会有一些难度，比较实用的是L-BFGS方法及其随机算法的改进。 本文将详细全面的叙述机器学习中各种随机算法，介绍随机算法的发展历程，研究方向及研究热点，最后通过数值试验比较了几种常见随机算法，以给读者直观的数值效果。 关键词 ：大规模机器学习，随机算法，优化方法 PAGE II Abstract For the optimization problem in machine learning field, traditional method have difficulties in solving the high dimension and big data problem . In recent years, there are many researches in large scale machine learning problems, especially stochastic algorithms. Generally, stochastic method can divided into two parts. One is first-order gradient method and the other is second- order Newton method. There is more improvement and research in first order method, and the first order method is more mature and perfect. There are two classes for first order method. For the primal class, SVRG, SAG, SAGA is the representation, and SDCA, SPDC for dual class. Otherwise, the acceleration method such as catalyst and katyusha, which has the optimal con-vergence speed for first order method, is put forward in last two years. Second order method is one important research area, and it has better convergence but not better performance because it has to compute the hessian matrix, one useful method is L-BFGS and its variants. In this paper, the author will introduce stochastic algorithms in machine learning area in detail. In the end, numerical experiments compare some common algorithm and give a direct view to readers. Key Words : Large-scale machine learning problem, Stochastic algorithm, Optimization method PAGE 27 目 录 TOC \o 1-3 \h \z \u 中文摘要 I Abstract II 目 录 1 引 言 1 1 基础知识 4 2 一阶方法 4 2.1 梯度下降法（Gradient Descent） 4 2.2 随机梯度方法（Stochastic Gradient