模型估计与选择-BCMIHomepage.pptVIP

* Model Assessment Selection * 自助法 自助法过程图解： 重复实验 样本 训练样本 Model Assessment Selection Model Assessment Selection Dept. Computer Science Engineering, Shanghai Jiaotong University * Model Assessment Selection * Outline Bias, Variance and Model Complexity The Bias-Variance Decomposition Optimism of the Training Error Rate Estimates of In-Sample Prediction Error The Effective Number of Parameters The Bayesian Approach and BIC Minimum Description Length Vapnik-Chernovenkis Dimension Cross-Validation Bootstrap Methods * Model Assessment Selection * Bias, Variance Model Complexity * Model Assessment Selection * Bias, Variance Model Complexity The standard of model assessment : the generalization performance of a learning method Model: Prediction Model: Loss function: * Model Assessment Selection * Bias, Variance Model Complexity Error: training error, generalization error Typical loss function: * Model Assessment Selection * Bias, Variance Model Complexity 模型选择： 估计不同模型的性能，以便选择最好的模型 模型评估： 从选定的模型，估计新样本的预测值 解决方法： 交叉校验：数据集=训练集+校证集+测试集 * Model Assessment Selection * Bias-Variance Decomposition Basic Model: The expected prediction error of a regression fit . The more complex the model, the lower the (squared) bias but the higher the variance. * Model Assessment Selection * Bias-Variance Decomposition For the k-NN regression fit the prediction error: For the linear model fit * Model Assessment Selection * Bias-Variance Decomposition The in-sample error of the Linear Model The model complexity is directly related to the number of parameters p. For ridge regression the square bias * Model Assessment Selection * Bias-Variance Decomposition Regularized fit Closest fit Estimation Variance Model bias Estimation bias Truth Realization Closest fit in population RESTRICED MODEL SPACE MODEL SPACE Schematic of the behavior of bias and variance * Model Assessment Selection * Optimism of the Training Error Rate Training Error True Error is e