svm-德克萨斯奥斯汀-机器学习组.pptVIP

Support Vector Machines Perceptron Revisited: Linear Separators Binary classification can be viewed as the task of separating classes in feature space: Linear Separators Which of the linear separators is optimal? Classification Margin Distance from example xi to the separator is Examples closest to the hyperplane are support vectors. Margin ρ of the separator is the distance between support vectors. Maximum Margin Classification Maximizing the margin is good according to intuition and PAC theory. Implies that only support vectors matter; other training examples are ignorable. Linear SVM Mathematically Let training set {(xi, yi)}i=1..n, xi?Rd, yi ? {-1, 1} be separated by a hyperplane with margin ρ. Then for each training example (xi, yi): For every support vector xs the above inequality is an equality. After rescaling w and b by ρ/2 in the equality, we obtain that distance between each xs and the hyperplane is Then the margin can be expressed through (rescaled) w and b as: Linear SVMs Mathematically (cont.) Then we can formulate the quadratic optimization problem: Which can be reformulated as: Solving the Optimization Problem Need to optimize a quadratic function subject to linear constraints. Quadratic optimization problems are a well-known class of mathematical programming problems for which several (non-trivial) algorithms exist. The solution involves constructing a dual problem where a Lagrange multiplier αi is associated with every inequality constraint in the primal (original) problem: The Optimization Problem Solution Given a solution α1…αn to the dual problem, solution to the primal is: Each non-zero αi indicates that corresponding xi is a support vector. Then the classifying function is (note that we don’t need w explicitly): Notice that it relies on an inner product between the test point x and the support vectors xi – we will return to this later. Also keep in mind that solving the optimization problem involved computing