信道匹配算法用于混合模型——挑战EM算法Channels.pptVIP

信道匹配算法用于混合模型——挑战EM算法Channels’ Matching Algorithm for Mixture Models——A Challenge to the EM Algorithm 鲁晨光 Chenguang Lu lcguang@Hpage: /lcg/; /lcg/english/ This ppt may be downloaded from /lcg/CM/CM4mix.ppt  1. Mixture Models: Guessing Parameters There are about 70 thousand papers with EM in titles. See / True model: P*(Y) and P*(X|Y) produces P(X)=P*(y1)P*(X|y1)+P*(y2)P*(X|y2)+… Predictive model：P(Y) and {θj} produces Q(X)=P(y1)P(X|θ1)+P(y2)P(X|θ2)+… Gaussian distribution: P(X| θj)=Kexp[-(X-cj)2/(2dj2)] Iterative algorithm to guess P(Y) and {(cj ，dj)} till Kullback-Leibler divergence or relative entropy 开始 P(X) Q(X) Q(X)≈ P(X) Iterating 2. The EM Algorithm for Mixture Models The popular EM algorithm and its convergence proof Likelihood is negative general entropy negative general joint entropy E-step: put P(yj|xi, θ) into Q M-step：Maximize Q. Convergence Proof: 1) Q’s increasing makes H(Q||P) -- 0; 2) Q is increasing in every M-step and no-decreasing in every E-step. 3. Problems with the Convergence Proof of the EM Algorithm 1). There is a counterexample against the convergence proof [1,2]: Real and guessed model parameters and iterative results For the true model, Q*=P(XN,Y|θ*)= - 6.031N； After first M-step, Q=P(XN,Y|θ)= - 6.011N, which is larger. 2). E-step might decrease Q, such as in the above example (talked later). [1].Dempster, A. P., Laird, N. M., Rubin, D.B.: Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, Series B 39, 1–38 (1977). [2] Wu, C. F. J.: On the Convergence Properties of the EM Algorithm. Annals of Statistics 11, 95–10 (1983). 0 -6.011N Target