SupplementaryMaterialOptimizationwithFirst-Order.pdfVIP

Supplementary Material Optimization with First-Order Surrogate Functions Julien Mairal - INRIA LEAR Project-Team, Grenoble, France Abstract In this paper, we study optimization methods consisting of iteratively minimizing surrogates of an objective function. By proposing several algorithmic variants and simple convergence analyses, we make two main contributions. First, we provide a unified viewpoint for several first- order optimization techniques such as accelerated proximal gradient, block coordinate descent, or Frank-Wolfe algorithms. Second, we introduce a new incremental scheme that experimentally matches or outperforms state-of-the-art solvers for large-scale optimization problems typically arising in machine learning. Outline: This document starts with the ICML paper formatted in one column, without any modification. It is followed by several appendices. In Appendix A, we present simple math- ematical definitions. Appendix B contains useful mathematical results, which are used in the paper. In Appendix C, we present various mechanisms to build first-order surrogate functions; it is in fact a more rigorous version of Section 2.2, where all claims are proved. In Appendix D, we present the block Frank-Wolfe optimization scheme. Finally, all proofs of propositions are given in Appendix E, and Appendix F contains additional experimental results. 1 Introduction The principle of iteratively minimizing a majorizing surrogate of an objective function is often called majorization-minimization (Lange et al., 2000). Each iteration drives the objective function downhill, thus giving the hope of finding a local optimum. A large number o