ANeuralAutoregressiveTopicModel.PDFVIP

A Neural Autoregressive Topic Model Hugo Larochelle Stanislas Lauly Departement′ d’informatique Departement′ d’informatique Universite′ de Sherbrooke Universite′ de Sherbrooke hugo.larochelle@usherbrooke.ca stanislas.lauly@usherbrooke.ca Abstract We describe a new model for learning meaningful representations of text docu- ments from an unlabeled collection of documents. This model is inspired by the recently proposed Replicated Softmax, an undirected graphical model of word counts that was shown to learn a better generative model and more meaningful document representations. Speci?cally, we take inspiration from the conditional mean-?eld recursive equations of the Replicated Softmax in order to de?ne a neu- ral network architecture that estimates the probability of observing a new word in a given document given the previously observed words. This paradigm also allows us to replace the expensive softmax distribution over words with a hierar- chical distribution over paths in a binary tree of words. The end result is a model whose training complexity scales logarithmically with the vocabulary size instead of linearly as in the Replicated Softmax. Our experiments show that our model is competitive both as a generative model of documents and as a document repre- sentation learning algorithm. 1 Introduction In order to leverage the large amount of available unlabeled text, a lot of research has been devoted to developing good probabilistic models of documents. Such models are usually embedded with latent variables or topics, whose role is to capture salient statistical patterns in the co-occurrence of words within documents. The most popular model is latent Dirichlet allocation (LDA) [1], a directed graphical model i