Comparison of SVM Regression with Least Square Method.docVIP

NC 2569 Under Review in Neural Computation, Revised June 10, 2002 Selecting the Loss Function for Robust Linear Regression Vladimir Cherkassky Yunqian Ma Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455, U.S.A. {cherkass, myq}@ece.umn.edu Abstract This paper addresses selection of the loss function for regression problems with finite data. It is well-known (under standard regression formulation) that for a known noise density there exist an optimal loss function under an asymptotic setting (large number of samples), i.e. squared loss is optimal for Gaussian noise density. However, in real-life applications the noise density is unknown and the number of training samples is finite. Robust statistics provides prescriptions for choosing the loss function using only general information about noise density; however robust statistics is based on asymptotic arguments and may not work well for finite sample problems. For such practical situations, we suggest using Vapnik’s -insensitive loss function. We propose a practical method for setting the value of as a function of known number of samples and (known or estimated) noise variance. First we consider commonly used unimodal noise densities (such as Gaussian and Laplacian). Empirical comparisons for several representative linear regression problems indicate that the proposed loss function yields more robust performance and improved prediction accuracy, in comparison with commonly used squared loss and least-modulus loss, especially for noisy high-dimensional data sets. We also performed comparisons for symmetric bimodal noise densities (where large errors occur more likely than small errors). For such (bimodal) noise, the proposed -insensitive loss consistently provides improved prediction accuracy (in comparison with other loss functions), for both low-dimensional and high-dimensional problems. Introduction _________________________________________________________ Es