Kernel Methods for Deep Learning.pdfVIP

Kernel Methods for Deep Learning Youngmin Cho and Lawrence K. Saul Department of Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, Mail Code 0404 La Jolla, CA 92093-0404 {yoc002,saul}@ Abstract We introduce a new family of positive-definite kernel functions that mimic the computation in large, multilayer neural nets. These kernel functions can be used in shallow architectures, such as support vector machines (SVMs), or in deep kernel-based architectures that we call multilayer kernel machines (MKMs). We evaluate SVMs and MKMs with these kernel functions on problems designed to illustrate the advantages of deep architectures. On several problems, we obtain better results than previous, leading benchmarks from both SVMs with Gaussian kernels as well as deep belief nets. 1 Introduction Recent work in machine learning has highlighted the circumstances that appear to favor deep archi- tectures, such as multilayer neural nets, over shallow architectures, such as support vector machines (SVMs) [1]. Deep architectures learn complex mappings by transforming their inputs through mul- tiple layers of nonlinear processing [2]. Researchers have advanced several motivations for deep architectures: the wide range of functions that can be parameterized by composing weakly non- linear transformations, the appeal of hierarchical distributed representations, and the potential for combining unsupervised and supervised methods. Experiments have also shown the benefits of deep learning in several interesting applications [3, 4, 5]. M