数据挖掘与监督学习中的模型选择 Data Mining and Model Choice in Supervised Learning.ppt

AIC and BIC: different theories BIC : bayesian choice between m models Mi . For each model P(?i / Mi). The posterior probability of Mi knowing the data x is proportional to P(Mi) P(x/Mi). With equal priors P(Mi): The most probable model Mi a posteriori is the one with minimal BIC. Beijing, 2008 * AIC and BIC: different uses BIC favourises more parsimonious models than AIC due to its penalization AIC (not BIC) is biased : if the true model belongs to the family Mi , the probability that AIC chooses the true model does not tend to one when the number of observations goes to infinity. It is inconsistent to use AIC and BIC simultaneously Other penalisations such as theory? Beijing, 2008 * 3.3 Limitations Refers to a “true” which generally does not exist, especially if n tends to infinity. “Essentially, all models are wrong, but some are useful ” G.Box (1987) Penalized likelihood cannot be computed for many models: Decision trees, neural networks, ridge and PLS regression etc. No likelihood, which number of parameters? Beijing, 2008 * 4. Predictive modelling In Data Mining applications (CRM, credit scoring etc.) models are used to make predictions. Model efficiency: capacity to make good predictions and not only to fit to the data (forecasting instead of backforecasting: in other words it is the future and not the past which has to be predicted). Beijing, 2008 * Classical framework Underlying theory Narrow set of models Focus on parameter estimation and goodness of fit Error: white noise Data mining context Models come from data Algorithmic models Focus on control of generalization error Error: minimal Beijing, 2008 * The black-box problem and supervised learning (N.Wiener, V.Vapnik) Given an input x, a non-deterministic system gives a variable y = f(x)+e. From n pairs (xi,yi) one looks for a function which approximates the unknown function f. Two conceptions: A good approximation is a function close to f A good approximation is a function whi