a hierarchical bayesian language model based on pitman.pdfVIP

A Hierarchical Bayesian Language Model based on Pitman-Yor Processes Yee Whye Teh School of Computing, National University of Singapore, 3 Science Drive 2, Singapore 117543. tehyw@.sg Abstract data, and smoothing methods are indispensible for proper training of n-gram models. We propose a new hierarchical Bayesian A large number of smoothing methods have n-gram model of natural languages. Our been proposed in the literature (see (Chen and model makes use of a generalization of Goodman, 1998; Goodman, 2001; Rosenfeld, the commonly used Dirichlet distributions 2000) for good overviews). Most methods take a called Pitman-Yor processes which pro- rather ad hoc approach, where n-gram probabili- duce power-law distributions more closely ties for various values of n are combined together, resembling those in natural languages. We using either interpolation or back-off schemes. show that an approximation to the hier- Though some of these methods are intuitively ap- archical Pitman-Yor language model re- pealing, the main justification has always been covers the exact formulation of interpo- empirical—better perplexities or error rates on test lated Kneser-Ney, one of the best smooth- data. Though arguably this should be the only ing methods for n-gram language models. real justification, it only answers the question of Experiments verify that our model gives w