Portability of Syntactic Structure for Language Modeling.pdfVIP

a r X i v : c s / 0 1 0 8 0 2 2 v 1 [ c s .C L ] 2 9 A u g 2 0 0 1 PORTABILITY OF SYNTACTIC STRUCTURE FOR LANGUAGE MODELING Ciprian Chelba Microsoft Speech.Net / Microsoft Research One Microsoft Way Redmond, WA 98052 chelba@ ABSTRACT The paper presents a study on the portability of statistical syntactic knowledge in the framework of the structured lan- guage model (SLM). We investigate the impact of porting SLM statistics from the Wall Street Journal (WSJ) to the Air Travel Information System (ATIS) domain. We compare this approach to applying the Microsoft rule-based parser (NLPwin) for the ATIS data and to using a small amount of data manually parsed at UPenn for gathering the intial SLM statistics. Surprisingly, despite the fact that it per- forms modestly in perplexity (PPL), the model initialized on WSJ parses outperforms the other initialization meth- ods based on in-domain annotated data, achieving a signifi- cant 0.4% absolute and 7% relative reduction in word error rate (WER) over a baseline system whose word error rate is 5.8%; the improvement measured relative to the minimum WER achievable on the N-best lists we worked with is 12%. 1. INTRODUCTION The structured language model uses hidden parse trees to assign conditional word-level language model probabilities. The model is trained in two stages: first the model param- eters are intialized from a treebank and then an N-best EM variant is employed for reestimating the model parameters. Assuming that we wish to port the SLM to a new do- main we have four alternatives for initializing the SLM: ? manual annotation of sentences with parse structure. This is expensive, time consuming and requires linguistic exper- tise. Consequently, only a small amount of data could be annotated this way. ? parse the training sentences in the new domain using an automatic parser ([1], [2], [3]) trained on a domain where a treebank is available already ? use a rule-based domain-independent parser ([4]) ? port the SLM s