AIML-EMalgorithm-1.pptVIP

Machine Learning Expectation Maximization Tom M. Mitchell Expectation Maximization (EM) When to use: Data is only partially observable Unsupervised clustering (target value unobservable) Supervised learning (some instance attributes unobservable) Some uses: Train Bayesian Belief Networks Unsupervised clustering (AUTOCLASS) Learning Hidden Markov Models Generating Data from Mixture of k Gaussians Each instance x generated by 1. Choosing one of the k Gaussians with uniform probability 2. Generating an instance at random according to that Gaussian EM for Estimating k Means (1/2) Given: Instances from X generated by mixture of k Gaussian distributions Unknown means ?1,…,?k of the k Gaussians Don’t know which instance xi was generated by which Gaussian Determine: Maximum likelihood estimates of ?1,…,?k Think of full description of each instance as yi = xi, zi1, zi2 where zij is 1 if xi generated by jth Gaussian xi observable zij unobservable EM for Estimating k Means (2/2) EM Algorithm: Pick random initial h = ?1, ?2 then iterate E step: Calculate the expected value E[zij] of each hidden variable zij, assuming the current hypothesis h = ?1, ?2 holds. M step: Calculate a new maximum likelihood hypothesis h = ?1, ?2, assuming the value taken on by each hidden variable zij is its expected value E[zij] calculated above. Replace h = ?1, ?2 by h = ?1, ?2. EM Algorithm Converges to local maximum likelihood h and provides estimates of hidden variables zij In fact, local maximum in E[ln P(Y|h)] Y is complete (observable plus unobservable variables) data Expected value is taken over possible values of unobserved variables in Y General EM Problem Given: Observed data X = {x1,…, xm} Unobserved data Z = {z1,…, zm} Parameterized probability distribution P(Y|h), where Y = {y1,…, ym} is the full data yi = xi ? zi h are the parameters Determine: h that (locally) maximizes E[ln P(Y|h)] Many uses: Train Bayesian belief networks Unsupervised clustering (e.g.,