A permutation-augmented sampler for DP mixture models.pdf

A permutation-augmented sampler for DP mixture models Percy Liang pliang@ University of California, Berkeley Michael Jordan jordan@ University of California, Berkeley Ben Taskar taskar@ University of Pennsylvania Abstract We introduce a new inference algorithm for Dirichlet process mixture models. While Gibbs sampling and variational methods fo- cus on local moves, the new algorithm makes more global moves. This is done by intro- ducing a permutation of the data points as an auxiliary variable. The algorithm is a blocked sampler which alternates between sampling the clustering and sampling the permutation. The key to the efficiency of this approach is that it is possible to use dynamic program- ming to consider all exponentially many clus- terings consistent with a given permutation. We also show that random projections can be used to effectively sample the permutation. The result is a stochastic hill-climbing algo- rithm that yields burn-in times significantly smaller than those of collapsed Gibbs sam- pling. 1. Introduction Dirichlet process (DP) mixture models (Antoniak, 1974) have been usefully employed as a clustering methodology in a variety of applied areas such as bioin- formatics (Xing et al., 2004), vision (Sudderth et al., 2006), and topic modeling (Teh et al., 2006). By treat- ing the number of mixture components as random, DP mixtures provide an appealing nonparametric ap- proach to mixture modeling in which the complexity of the model adapts to the complexity inherent in the data. Posterior inference for DP mixtures is challenging, and a variety of inference algorithms have been specialized Appearing in Proceedings of the 24 th International Confer- ence on Machine Learning, Corvallis, OR, 2007. Copyright 2007 by the author(s)/owner(s). to the DP mixture setting, including samplers (Ish- waran James, 2001; Escobar West, 1995), varia- tional approximations (Blei Jordan, 2005; Kurihara et al., 2007), and other search algorithms (Daume, 2007). A diffic