ModelAssessmentandSelection.pptVIP

Model Assessment and Selection Chapter 7 Neil Weisenfeld, et al. Introduction Generalization performance of a learning method: Measure of prediction capability on independent test data Guide model selection Give quantitative assessment of chosen model Chapter shows key methods and how to apply them to model selection Bias, Variance, and Model Complexity Bias-Variance trade-off again Generalization: test sample vs. training sample performance Training data usually monotonically increasing performance with model complexity Measuring Performance target variable Vector of inputs Prediction model Typical Choices of Loss function Generalization Error Test error aka. Generalization error Note: This expectation averages anything that is random, including the randomness in the training sample that it produced Training error average loss over training sample not a good estimate of test error (next slide) Training Error Training error - Overfitting not a good estimate of test error consistently decreases with model complexity drops to zero with high enough complexity Categorical Data same for categorical responses Typical Choices of Loss functions: Loss Function for General Densities For densities parameterized by theta: Log-likelihood function can be used as a loss-function Two separate goals Model selection: Estimating the performance of different models in order to choose the (approximate) best one Model assessment: Having chosen a final model, estimating its prediction error (generalization error) on new data Ideal situation: split data into the 3 parts for training, validation (est. prediction error+select model), and testing (assess model) Typical split: 50% / 25% / 25% Remainder of the chapter: Data-poor situation = Approximation of validation step either analytically (AIC, BIC, MDL, SRM) or by efficient sample reuse (cross-validation, bootstrap) Bias-Variance Decomposition Then for an input point , using unit-square loss and regression fit: