AAI-Lecture7-Machine-learning(机器学习).pptVIP

Regression: Minimizing Loss y = w1 x + w0 Linear algebra givesan exact solution to the minimization problem Linear Algebra Solution Linear Regression X: 3, 6, 4, 5 Y: 0, -3, -1, -2 f(x)=w1x+w0 w1=-1, w0 =3 Minimizing quadratic loss Recaculate w0,w1 Another quiz: X(2,4,6,8), Y(2,5,5,8) Don’t Always Trust Linear Models Regression by Gradient Descent w = any point loop until convergence do: for each wi in w do: wi = wi – α ? Loss(w) ? wi Multivariate Regression You learned this in math class too hw(x) = w ? x = w xT = Σi wi xi The most probable set of weights, w*(minimizing squared error): w* = (XT X)-1 XT y Overfitting To avoid overfitting, don’t just minimize loss Maximize probability, including prior over w Can be stated as minimization: Cost(h) = EmpiricalLoss(h) + λ Complexity(h) For linear models, consider Complexity(hw) = Lq(w) = ∑i | wi |q L1 regularization minimizes sum of abs. values L2 regularization minimizes sum of squares Regularization and Sparsity L1 regularization L2 regularization Cost(h) = EmpiricalLoss(h) + λ Complexity(h) Outline Machine Learning Classification (Na?ve Bayes) Regression (Linear, Smoothing) Linear Separation (Perceptron, SVMs) Non-parametric classification (KNN) Linear Separator Perceptron Perceptron Algorithm Start with random w0, w1 Pick training example x,y Update (α is learning rate) w1 ? w1+α(y-f(x))x w0 ? w0+α(y-f(x)) Converges to linear separator (if exists) Picks “a” linear separator (a good one?) What Linear Separator to Pick? What Linear Separator to Pick? Maximizes the “margin” Support Vector Machines Non-Separable Data? Not linearly separable for x1, x2 What if we add a feature? x3= x12+x22 See: “Kernel Trick” X1 X2 X3 Outline Machine Learning Classification (Na?ve Bayes) Regression (Linear, Smoothing) Linear Separation (Perceptron, SVMs) Non-parametric classification (KNN) Nonparametric Models If the process of learning good values for parameters is prone to over