15ModelSelection.PDFVIP

15 Model Selection 15.1 KLIC Y Suppose a random sample is y = y1; ; :::; yn has unknown density f(y) = f(yi): Y A model density g(y) = g(yi): How can we assess the “…t”of g as an approximation to f? One useful measure is the Kullback-Leibler information criterion (KLIC) Z   f(y) KLIC(f; g) = f(y) log dy g(y) You can decompose the KLIC as Z Z KLIC(f; g) = f(y) log f(y)dy f(y) log g(y)dy = Cf E log g(y) R The constant Cf = f(y) log f(y)dy is independent of the model g: Notice that KLIC(f; g) 0; and KLIC(f; g) = 0 i¤ g = f: Thus a “good” approximating  model g is one with a low KLIC. 15.2 Estimation Let the model density g(y; ) depend on a parameter vector : The negative log-likelihood function is Xn () = log g(yi; ) = log g(y; ) L i=1 and the MLE is ^ = argmin (): Sometimes this is called a “quasi-MLE”when g(y; ) is acknowl-  L edged to be an approximation, rather than the truth. Let the minimizer of E log g(y; ) be written 0 and called the pseudo-true value. This value also minimizes KLIC(f; g()): As the likelihood divided by n is an estimator of E log g(y; ); the MLE ^ converges in probability to 0: That is,