Discussion of Least angle regression by Efron et al.pdf

a r X i v : m a t h / 0 4 0 6 4 7 0 v 1 [ m a t h .S T ] 2 3 J u n 2 0 0 4 The Annals of Statistics 2004, Vol. 32, No. 2, 469–475 DOI: 10.1214/009053604000000067 c? Institute of Mathematical Statistics, 2004 DISCUSSION OF “LEAST ANGLE REGRESSION” BY EFRON ET AL. By Saharon Rosset and Ji Zhu IBM T. J. Watson Research Center and Stanford University 1. Introduction. We congratulate the authors on their excellent work. The paper combines elegant theory and useful practical results in an in- triguing manner. The LAR–Lasso–boosting relationship opens the door for new insights on existing methods’ underlying statistical mechanisms and for the development of new and promising methodology. Two issues in particu- lar have captured our attention, as their implications go beyond the squared error loss case presented in this paper, into wider statistical domains: ro- bust fitting, classification, machine learning and more. We concentrate our discussion on these two results and their extensions. 2. Piecewise linear regularized solution paths. The first issue is the piecewise linear solution paths to regularized optimization problems. As the discussion paper shows, the path of optimal solutions to the “Lasso” regularized optimization problem β?(λ) = argmin β ‖y ?Xβ‖22 + λ‖β‖1(2.1) is piecewise linear as a function of λ; that is, there exist ∞>λ0 >λ1 > · · ·> λm = 0 such that ?λ≥ 0, with λk ≥ λ≥ λk+1, we have β?(λ) = β?(λk)? (λ? λk)γk. In the discussion paper’s terms, γk is the “LAR” direction for the kth step of the LAR–Lasso algorithm. This property allows the LAR–Lasso algorithm to generate the whole path of Lasso solutions, β?(λ), for “practically” the cost of one least squares calculation on the data (this is exactly the case for LAR but not for LAR– Lasso, which may be significantly more computationally intensive on some data sets). The important practical consequence is that it is not necessary This is an electronic reprint of the original article published by the In