Recap_forStu.docxVIP

Linear regression (Lec 1, Ng’s Lecture note 1) Approximate y as a linear function of x, theta’s are parameters, letting x0=1, view both theta and x as vectors Define the hypotheses: Then how do we pick or learn theta? Make h(x) close to y Cost Function (least square cost function that gives rise to the ordinary least squares regression model): Gradient descent algorithm (start with some initial theta, and repeatedly performs the update, alpha is the learning rate): Batch gradient descent (has to scan through the entire training set before taking a single step-a costly operation if m is large) Stochastic gradient descent Note: here J is a convex quadratic function. The gradient descent converges to the global minimum. The normal equations (using matrix derivative formulations) ? But XTX is not invertible sometimes. If not, use regularization. If use L2 norm regularization? Ringe regression Locally weighted linear regression (LWR) Parametric vs. non-parametric LWR is a non-parametric algorithm The (unweighted) linear regression algorithm that we saw earlier is known as a parametric learning algorithm because it has a fixed, finite number of parameters (the θ), which are fit to the data The term non-parametric (roughly) refers to the fact that the amount of stuff we need to keep in order to represent the hypothesis grows linearly with the size of the training set. Classification and logistic regression (Lec 2, Ng’s Lecture note 2) Regression vs classification Logistic regression is a linear classification model, although it is called regression Hypothesis of logistic regression Compact form: MLE (Conditional) Likelihood: Log-likelihood: Problem: Optimization method: a) Gradient ascent; (property of sigmoid function, g’(z)=g(z)(1-g(z))) b) The Newton’s method Bayesian learning (Lec 3, 4, Ng’s Lecture note 2) Bayes rule: MLE vs. MAP Too few trials ? cause problems ? laplace smoothing (Conjugate prior of binomial is Beta d