机器学习中的随机优化算法2.docVIP

Stochastic Optimization Algor ithm i n Machine Learning 中文摘要 对于机器学习中的数值优化问题，考虑到其规模和维数都比较大，传统的方法难以高效的解决这一同题。近些年来，针对大规模的机器学习问题做了很多研究，比较币要的一类方法是随机算法。优化方法主要分为一阶梯度方法和二阶牛顿方法两大类，针对一阶方法的改进和研宄比较成熟和完善，一阶方法又可以分为两类，原始方法和对偶方法，原始方法比较有代表性的有G、G，、、SVRG,对偶方法有5乁和SYD〔：两种。此外，加速方法如Catal ”t、Katyusha,其收敛速度为一阶方法所能达到的最好结果。二阶方法目前是机器学习领域研宄的重要方向，其收敛速度要优于一阶方法，但是其实践中会有一些难度，比较实用的是L一BFGS方法及其随机算法的改进。 本文将详细全面的叙述机器学习中各种随机算法，介绍随机算法的发展历程，研究方向及研究热点，最后通过数值试验比较了几种常见随机算法，以给读者直观的数值效果， 关键词 ．大规模机器学习，随机算法，优化方法 Abstract For the optimization prOb1em in machine learning field, traditional method have difficulties in solving the high dimension and big data problem In recent years, there are many researches in large scale machine learning problems, especially stochastic algorithms. Generally, stochastic method can divided into two parts. One is first—order gradient method and the other is second— order Newton method. There is more improvement and research in first order rnethod, and the first order method is more mature and perfect, There are two classes for first order method. For the primal class, SVRG, SAG, SAGA is the representat ion, and SDCA, SPDC for dual class, Otherwise, the acceleration method such as catalyst and katyusha, which has the optimal con—vergence speed for first order method, is put forward in last two years. Second order method is one important research area, and it has better convergence but not better performance because it has to compute the hessian matrix, one useful method is L=BFGS and its variants. In this paper, the author will introduce stochastic algori thins in machine learning area in detail. In the end, numerical experiments compare some common algorithm and give a direct view to readers. Key Words Large—scale machine learning probl em, Stochastic a gori thm, Optimi zat ion method 11 1 Abstract 11 3, 2 Limited—memorvBFGS 近此年来，随着科学技术不断地进步发展，尤其是在计算机、通信等领域的进步以及互联网不断的普及。人们产生数据速率大大提高的同时又能够容易廉价地获取和存储数据。数据中蕴含是大量信息，但是如何在大量数据中发掘出有效的信息是一个很田难的问题。为了解决该问题，人们尝试采用各种方式。其中，比较好的处理方式是使用机器学习，数理统计的