NLP自然语言处理—N-gramlanguagemodel.ppt

* * * * * * * * * * * * * * * * * * * * * * * CS 388: Natural Language Processing:N-Gram Language Models Raymond J. Mooney University of Texas at Austin Language Models Formal grammars (e.g. regular, context free) give a hard “binary” model of the legal sentences in a language. For NLP, a probabilistic model of a language that gives a probability that a string is a member of a language is more useful. To specify a correct probability distribution, the probability of all sentences in a language must sum to 1. Uses of Language Models Speech recognition “I ate a cherry” is a more likely sentence than “Eye eight uh Jerry” OCR Handwriting recognition More probable sentences are more likely correct readings. Machine translation More likely sentences are probably better translations. Generation More likely sentences are probably better NL generations. Context sensitive spelling correction “Their are problems wit this sentence.” Completion Prediction A language model also supports predicting the completion of a sentence. Please turn off your cell _____ Your program does not ______ Predictive text input systems can guess what you are typing and give choices on how to complete it. N-Gram Models Estimate probability of each word given prior context. P(phone | Please turn off your cell) Number of parameters required grows exponentially with the number of words of prior context. An N-gram model uses only N?1 words of prior context. Unigram: P(phone) Bigram: P(phone | cell) Trigram: P(phone | your cell) The Markov assumption is the presumption that the future behavior of a dynamical system only depends on its recent history. In particular, in a kth-order Markov model, the next state only depends on the k most recent states, therefore an N-gram model is a (N?1)-order Markov model. N-Gram Model Formulas Word sequences Chain rule of probability Bigram approximation N-gram approximation Estimating Probabilities N-gram conditional probabilities can be estimated from ra