ModelSelectionforSupportVector.PDFVIP

Model Selection for Support Vector Machines Olivier Chapelle, Vladimir Vapnik ATT Research Labs 100 Schulz drive Red Bank, NJ 07701 chapelle,vlad @ { } Abstract New functionals for parameter (model) selection of Support Vector Ma- chines are introduced based on the concepts of the span of support vec- tors and rescaling of the feature space. It is shown that using these func- tionals, one can both predict the best choice of parameters of the model and the relative quality of performance for any value of parameter. 1 Introduction Support Vector Machines (SVMs) implement the following idea : they map input vectors into a high dimensional feature space, where a maximal margin hyperplane is constructed [6]. It was shown that when training data are separable, the error rate for SVMs can be characterized by h = R2/M 2, (1) where R is the radius of the smallest sphere containing the training data and M is the mar- gin (the distance between the hyperplane and the closest training vector in feature space). This functional estimates the VC dimension of hyperplanes separating data with a given margin M. To perform the mapping and to calculate R and M in the SVM technique, one uses a positive de?nite kernel K(x, x′) which speci?es an inner product in feature space. An example of such a kernel is the Radial Basis Function (RBF), x x′ 2 2 K(x, x′) = e?|| ? || /2σ . This kernel has a free parameter σ and more generally, most kernels require some param- eters to be set. When treating noisy data with SVMs, another parameter, penalizing the training errors, also needs to be set. The problem of choosing the values