faster and smaller strongnstrong-gram language models.pdfVIP

Faster and Smaller -Gram Language Models Adam Pauls Dan Klein Computer Science Division University of California, Berkeley adpauls,klein@ Abstract and Raj, 2001; Hsu and Glass, 2008), our meth- ods are conceptually based on tabular trie encodings -gram language models are a major resource wherein each -gram key is stored as the concatena- bottleneck in machine translation. In this pa- tion of one word (here, the last) and an offset encod- per, we present several language model imple- ing the remaining words (here, the context). After mentations that are both highly compact and presenting a bit-conscious basic system that typifies fast to query. Our fastest implementation is such approaches, we improve on it in several ways. as fast as the widely used SRILM while re- quiring only 25% of the storage. Our most First, we show how the last word of each entry can compact representation can store all 4 billion be implicitly encoded, almost entirely eliminating -grams and associated counts for the Google its storage requirements. Second, we show that the -gram corpus in 23 bits per -gram, the most deltas between adjacent entries can be efficiently en- compact lossless representation to date, and coded with simple variable-length encodings. Third, even more compact than recent lossy compres- we investigate block-based schemes that minimize