FAFU机器学习05-1 Model Evaluation课件.pptxVIP

Foundations of Machine LearningModel Evaluation2023/11/4Model EvaluationLesson 4- 1 2023/11/4Model EvaluationLesson 4- 2Q: How do we estimate the performance of these models using different machine learning algorithms? 2023/11/4Model EvaluationLesson 4- 3Q: How do we estimate the performance of these models with different parameters?Blue: Observed dataGreen: true distributionPolynomial Curve Fitting Red: Predicted curve 2023/11/4Model EvaluationLesson 4- 4Q: How do we estimate the performance of machine learning model?Answer:① We want to estimate the generalization performance, the predictive performance of our model on future (unseen) data.②We want to increase the predictive performance by tweaking the learning algorithm and selecting the best performing model from a given hypothesis space.③we want to compare different algorithms, selecting the best-performing one as well as the best performing model from the algorithm’s hypothesis space. Basic Concepts2023/11/4Model EvaluationLesson 4- 5 i.i.d.: Independent and identically distribution means that all samples have been drawn from the same probability distribution and are statistically independent from each other. Accuracy : the number of correct predictions a divided by the number of samples m Error Rate : the number of wrong predictions b divided by the number of samples m Basic Concepts2023/11/4Model EvaluationLesson 4- 6Error（误差）: generally speaking, the difference between expected output value from the model and real sample value Training error（训练误差）: empirical error（经验误差）, is the error we get applying the model to the same data from which we trained.Test error（测试误差）: is the error that we incur on new data..Generalization error（泛化误差）: out-of-sample error, is a measure of how accurately an algorithm is able to predict outcome values for unseen dataPractically, the test error is used to estimate generalization errorTheoretically, generalization error bound is employed Basic Concepts2023/11/4Model Ev