Model Assessment, Selection and Averaging课件.ppt

Model Assessment, Selection and Averaging Presented by: Bibhas Chakraborty Goals Model Selection: estimating the performance of different models in order to choose the best one. Model Assessment: having chosen a final model, estimating its generalization error on new data. Model Averaging: averaging the predictions from different models to achieve improved performance. Splitting the data Split the dataset into three parts: Training set: used to fit the models. Validation set: used to estimate prediction error for model selection. Test set: used to assess the generalization error for the final chosen model. Bayesian Information Criterion (BIC) Model selection tool applicable in settings where the fitting is carried out by maximization of a log-likelihood. Motivation from Bayesian point of view. BIC tends to penalize complex models more heavily, giving preference to simpler models in selection. Its generic form is: Bayesian Model Selection Suppose we have candidate models with corresponding model parameters Prior distribution: Posterior probability: Compare two models via posterior odds: The second factor on the RHS is called the Bayes factor and describes the contribution of the data towards posterior odds. Bayesian Approach Continued Unless strong evidence to the contrary, we typically assume that prior over models is uniform (non-informative prior). Using Laplace approximation, one can establish a simple (but approximate) relationship between posterior model probability and the BIC. Lower BIC implies higher posterior probability of the model. Use of BIC as model selection criterion is thus justified. AIC or BIC? BIC is asymptotically consistent as a selection criterion. That means, given a family of models including the true model, the probability that BIC will select the correct one approaches one as the sample size becomes large. AIC does not have the above proper