Recent Advances in Structured Sparse Models.pdf

Recent Advances in Structured Sparse Models Julien Mairal Willow group - INRIA - ENS - Paris 21 September 2010 LEAR seminar At Grenoble, September 21st, 2010 Julien Mairal Recent Advances in Structured Sparse Models 1/46 Acknowledgements Francis Rodolphe Guillaume Bach Jenatton Obozinski Julien Mairal Recent Advances in Structured Sparse Models 2/46 What this talk is about? Sparse Linear Models Not only sparse, but also structured! Solving challenging optimization problems Developping new applications of sparse models in computer vision and machine learning Related publications: [1] J. Mairal, R. Jenatton, G. Obozinski and F. Bach. Network Flow Algorithms for Structured Sparsity. NIPS, 2010 [2] R. Jenatton, J. Mairal, G. Obozinski and F. Bach. Proximal Methods for Hierarchical Sparse Coding. arXiv:1009.2139v1 [3] R. Jenatton, J. Mairal, G. Obozinski and F. Bach. Proximal Methods for Sparse Hierarchical Dictionary Learning. ICML, 2010 Julien Mairal Recent Advances in Structured Sparse Models 3/46 Sparse Linear Model: Machine Learning Point of View Let (y i , xi )ni=1 be a training set, where the vectors x i are in Rp and are called features. The scalars y i are in {?1,+1} for binary classification problems. {1, . . . ,N} for multiclass classification problems. R for regression problems. In a linear model, on assumes a relation y ≈ w?x, and solves min w∈Rp 1 n n∑ i=1 ?(y i ,w?xi ) ︸ ︷︷ ︸ data-fitting + λ?(w)︸ ︷︷ ︸ regularization . Julien Mairal Recent Advances in Structured Sparse Models 4/46 Sparse Linear Models: Machine Learning Point of View A few examples: Ridge regression: min w∈Rp 1 2n n∑ i=1 (y i ?w?xi )2 + λ‖w‖22. Linear SVM: min w∈Rp 1 n n∑ i=1 max(0, 1? y iw?xi ) + λ‖w‖22. Logistic regression: min w∈Rp 1 n n∑ i=1 log ( 1 + e?y iw?xi ) + λ‖w‖22. The squared ?2-norm induces smoothness in w. When one knows in advance that w should be sparse, one should use a sparsity-inducing regularization such as the ?1-norm. [Chen et al., 1999, Tibshirani, 1996] The purpose