Coordinate Descent slide.pdf

Coordinate descent Geoff Gordon Ryan Tibshirani Optimization 10-725 / 36-725 1 Adding to the toolbox, with stats and ML in mind We’ve seen several general and useful minimization tools ? First-order methods ? Newton’s method ? Dual methods ? Interior-point methods These are some of the core methods in optimization, and they are the main objects of study in this field In statistics and machine learning, there are a few other techniques that have received a lot of attention; these are not studied as much by those purely in optimization They don’t apply as broadly as above methods, but are interesting and useful when they do apply ... our focus for the next 2 lectures 2 Coordinate-wise minimization We’ve seen (and will continue to see) some pretty sophisticated methods. Today, we’ll see an extremely simple technique that is surprisingly efficient and scalable Focus is on coordinate-wise minimization Q: Given convex, differentiable f : Rn → R, if we are at a point x such that f(x) is minimized along each coordinate axis, have we found a global minimizer? I.e., does f(x+ d · ei) ≥ f(x) for all d, i ? f(x) = minz f(z)? (Here ei = (0, . . . , 1, . . . 0) ∈ Rn, the ith standard basis vector) 3 x1 x2 f A: Yes! Proof: ?f(x) = ( ?f ?x1 (x), . . . ?f ?xn (x) ) = 0 Q: Same question, but for f convex (not differentiable) ... ? 4 x1 x 2 f x1 x 2 ?4 ?2 0 2 4 ? 4 ? 2 0 2 4 ● A: No! Look at the above counterexample Q: Same question again, but now f(x) = g(x) + ∑n i=1 hi(xi), with g convex, differentiable and each hi convex ... ? (Non-smooth part here called separable) 5 x1 x 2 f x1 x 2 ?4 ?2 0 2 4 ? 4 ? 2 0 2 4 ● A: Yes! Proof: for any y, f(y)? f(x) ≥ ?g(x)T (y ? x) + n∑ i=1 [hi(yi)? hi(xi)] = n∑ i=1 [?ig(x)(yi ? xi) + hi(yi)? hi(xi)]︸ ︷︷ ︸ ≥0 ≥ 0 6 Coordinate descent This suggests that for f(x) = g(x) + ∑n i=1 hi(xi) (with g convex, differentiable and each hi convex) we can use coordinate descent to find a minimizer: start with some initial guess x(0), and repeat for k = 1, 2, 3, .