Sampling algorithms for l2 regression and applications.ppt

Sampling algorithms for l2 regression and applications Michael W. Mahoney Yahoo Research /homes/mmahoney (Joint work with P. Drineas and S. (Muthu) Muthukrishnan) SODA 2006 Regression problems “Induced” regression problems Regression problems, definition Exact solution Singular Value Decomposition (SVD) Questions … Creating an induced subproblem The induced subproblem Our results Our results, cont’d Back to induced subproblems … Conditions on the probabilities, SVD Conditions on the probabilities, interpretation Conditions for the probabilities Computing “good” probabilities Critical observation Critical observation, cont’d Critical observation, cont’d Application: CUR-type decompositions * * We seek sampling-based algorithms for solving l2 regression. We are interested in overconstrained problems, n d. Typically, there is no x such that Ax = b. sampled “rows” of b sampled rows of A scaling to account for undersampling There is work by K. Clarkson in SODA 2005 on sampling-based algorithms for l1 regression (? = 1 ) for overconstrained problems. Pseudoinverse of A Projection of b on the subspace spanned by the columns of A U (V): orthogonal matrix containing the left (right) singular vectors of A. S: diagonal matrix containing the singular values of A. ?: rank of A. Computing the SVD takes O(d2n) time. The pseudoinverse of A is Can sampling methods provide accurate estimates for l2 regression? Is it possible to approximate the optimal vector and the optimal value value Z2 by only looking at a small sample from the input? (Even if it takes some sophisticated oracles to actually perform the sampling …) Equivalently, is there an induced subproblem of the full regression problem, whose optimal solution and its value Z2,s aproximates the optimal solution and its value Z2? Algorithm Fix a set of probabilities pi, i=1…n, summing up to 1. Pick r indices from {1…n} in r i.i.d. trials, with respect to the pi’s. For each sampled index j, keep the j-th row of A and