Analysisandextensionofspectralmethodsfornonlineardimensionalityreduction.pdfVIP

Analysis and Extension of Spectral Methods for Nonlinear Dimensionality Reduction Fei Sha FEISHA @CIS .UPENN .EDU Lawrence K. Saul LSAUL @CIS .UPENN .EDU Department of Computer Information Science, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA 19104 Abstract quite tractable—involving (for example) nearest neighbor searches, least squares fits, dynamic programming, eigen- Many unsupervised algorithms for nonlinear di- value problems, and semidefinite programming. mensionality reduction, such as locally linear embedding (LLE) and Laplacian eigenmaps, are One large family of algorithms for manifold learning con- derived from the spectral decompositions of sists of approaches based on the spectral decomposition of sparse matrices. While these algorithms aim to sparse matrices (Chung, 1997). Algorithms in this family preserve certain proximity relations on average, include locally linear embedding (LLE) (Roweis Saul, their embeddings are not explicitly designed to 2000) and Laplacian eigenmaps (Belkin Niyogi, 2003). preserve local features such as distances or an- The matrices in these algorithms are derived from sparse gles. In this paper, we show how to construct a weighted graphs whose nodes represent high dimensional low dimensional embedding that maximally pre- inputs and whose edges indicate neighborhood relations.