人工智能课件AI2014S_Chap06KernelMethodsandSVM幻灯片.pptxVIP

Artificial Intelligence Kernel Methods and SVMDonghui WangAI Institute@ZJU2014.3ContentsKernel methodsMaximum margin classifiersReferences:Bishop. “Pattern Recognition and Machine Learning”, Chapter 6,7. 2006.Kernel MethodsTwo approaches to use training dataSlow to learn but fast at making prediction.Fast to learn but slow at making prediction.Use training data only in learning phase:Obtain a point estimate of the parameter vector wDetermine a posterior distribution over wPredictions for new inputs are based purely on the learned parameter vector wUse training data in both of learning phase and prediction phase:Kept and use training data also during the prediction phase (memory-based)Typically require to measure the similarity of any two vectors in input spaceE.g. nearest neighbors classifierKernel function and kernel trickSimilarity metric in feature space can be defined by kernel function:The kernel is symmetric function:The simplest example of a kernel function:Kernel trick (kernel substitution)The general idea is that, if we have an algorithm formulated in such a way that the input vector x enters only in the form of scalar products (e.g. inner products x, y), then we can replace that scalar product with some other choice of kernel k(x,y) and allow new learning algorithm to work efficiently in the high dimensional feature space.E.g. perceptron, SVM, PCA, CCA, Fishers linear discriminant analysis and cluster analysisDual representationsM parametersIn linear regression model, we have regularized SSE function:Let:Substitute into J(w) to obtain dual representation:The prediction for a new input x:N parametersN×M design matrixN×N Gram matrixIt’s memory-based method!Constructing kernelsMethods #1:Choose a feature space mapping φ(x) and use it to find the corresponding kernel.Constructing kernelsMethods #2:Construct kernel function directly, but we must ensure the function we choose is a valid kernel (it can be represented by a scalar product in some feature space).Ex