Efficient Online and Batch Learning using Forward Backward Splitting：（有效的在线学习和批处理使用向前向后分裂）.pdfVIP

Journal of Machine Learning Research 10 (2009) ??-?? Submitted 2/2009; Revised 7/2009; Published 10/2009 Efficient Online and Batch Learning using Forward Backward Splitting John Duchi jduchi@ Computer Science Division University of California, Berkeley Berkeley, CA 94720 USA Yoram Singer singer@ Google 1600 Amphitheatre Parkway Mountain View, CA 94043 USA Editor: Yoav Freund Abstract We describe, analyze, and experiment with a framework for empirical loss minimization with regularization. Our algorithmic framework alternates between two phases. On each iteration we first perform an unconstrained gradient descent step. We then cast and solve an instantaneous optimization problem that trades off minimization of a regularization term while keeping close proximity to the result of the first phase. This view yields a simple yet effective algorithm that can be used for batch penalized risk minimization and online learning. Furthermore, the two phase approach enables sparse solutions when used in conjunction with regularization functions that promote sparsity, such as ℓ . We derive concrete and very simple algorithms for minimization 1 2 of loss functions with ℓ , ℓ , ℓ , and ℓ regularization. We also show how to construct efficient 1 2 2 ∞ algorithms for mixed-norm ℓ /ℓ regularization. We further extend the algorithms and give efficient