Parallelized Variational EM for Latent Dirichlet Allocation An Experimental Evaluation of S.pdfVIP

Parallelized Variational EM for Latent Dirichlet Allocation: An Experimental Evaluation of Speed and Scalability Ramesh Nallapati, William Cohen and John Lafferty Machine Learning Department Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213, USA {nmramesh,wcohen,lafferty }@ Abstract α θ z w β Statistical topic models such as the Latent Dirichlet Al- N K location (LDA) have emerged as an attractive framework M to model, visualize and summarize large document collec- tions in a completely unsupervised fashion. Considering Figure 1. Graphical representation of LDA: each plates the enormous sizes of the modern electronic document col- represents repeated sampling as many times as the number lections, it is very important that these models are fast and in the right-hand bottom corner of the plate. scalable. In this work, we build parallel implementations of the variational EM algorithm for LDA in a multiproces- • For each document d = 1, , M sor architecture as well as a distributed setting. Our ex- periments on various sized document collections indicate – sample mixing proportions θd ∼ Dir(|α) that while both the implementations achieve speed-ups, the – For each position i = 1, , N , sample: