a note on topical strongnstrong-grams.pdfVIP

A Note on Topical N-grams Xuerui Wang and Andrew McCallum Department of Computer Science University of Massachusetts Amherst Amherst, MA 01003 USA {xuerui,mccallum}@ Technical Report UM-CS-2005-071 December 24, 2005 Abstract Most of the popular topic models (such as Latent Dirichlet Allocation) have an underlying assump- tion: bag of words. However, text is indeed a sequence of discrete word tokens, and without considering the order of words (in another word, the nearby context where a word is located), the accurate meaning of language cannot be exactly captured by word co-occurrences only. In this sense, collocations of words (phrases) have to be considered. However, like individual words, phrases sometimes show polysemy as well depending on the context. More noticeably, a composition of two (or more) words is a phrase in some context, but not in other contexts. In this paper, we propose a new probabilistic generative model that automatically determines unigram words and phrases based on context and simultaneously associates them with mixture of topics, and show very interesting results on large text corpora. 1 Introduction n-gram phrases (or collocations) are fundamentally important in many areas of natural language processing (e.g., parsing, machine translation and information retrieval). Phrase as the whole carries more information than the sum of its individual components, thus it is much more crucial in determining the topics of document collections than individual words. However, most of the top