Practical Large-Scale Optimization for Max-Norm Regularization.pdfVIP

Practical Large-Scale Optimization for Max-Norm Regularization Jason Lee Benjamin Recht Institute of Computational and Mathematical Engineering Department of Computer Sciences Stanford University University of Wisconsin-Madison email: jl115@ email: brecht@ Ruslan Salakhutdinov Nathan Srebro Brain and Cognitive Sciences and CSAIL Toyota Technological Institute at Chicago Massachusetts Institute of Technology email: nati@ email: rsalakhu@ Joel A. Tropp Computing and Mathematical Sciences California Institute of Technology email: jtropp@ Abstract The max-norm was proposed as a convex matrix regularizer in [1] and was shown to be empirically superior to the trace-norm for collaborative ?ltering problems. Although the max-norm can be computed in polynomial time, there are currently no practical algorithms for solving large-scale optimization problems that incor- porate the max-norm. The present work uses a factorization technique of Burer and Monteiro [2] to devise scalable ?rst-order algorithms for convex programs involving the max-norm. These algorithms are applied to solve huge collabora- tive ?ltering, graph cut, and clustering problems. Empirically, the new methods outperform mature techniques from all three areas. 1 Introduction A foundational concept in modern machine learning is to construct models for data by balancing the complexity of the model against ?delity to the measurements. In a wide variety of applications, such as collaborative ?ltering, multi-task learning, multi-class learning and clustering of multivariate observations, matr