Lecture 4 The MaximumEntropy Stewpot讲座34最大熵炖锅.ppt

600.465 - Intro to NLP - J. Eisner The Maximum-Entropy Stewpot Probability is Useful We love probability distributions! We’ve learned how to define use p(…) functions. Pick best output text T from a set of candidates speech recognition (HW2); machine translation; OCR; spell correction... maximize p1(T) for some appropriate distribution p1 Pick best annotation T for a fixed input I text categorization; parsing; part-of-speech tagging … maximize p(T | I); equivalently maximize joint probability p(I,T) often define p(I,T) by noisy channel: p(I,T) = p(T) * p(I | T) speech recognition other tasks above are cases of this too: we’re maximizing an appropriate p1(T) defined by p(T | I) Pick best probability distribution (a meta-problem!) really, pick best parameters ?: train HMM, PCFG, n-grams, clusters … maximum likelihood; smoothing; EM if unsupervised (incomplete data) Bayesian smoothing: max p(?|data) = max p(?, data) =p(?)p(data|?) Probability is Flexible We love probability distributions! We’ve learned how to define use p(…) functions. We want p(…) to define probability of linguistic objects Trees of (non)terminals (PCFGs; CKY, Earley, pruning, inside-outside) Sequences of words, tags, morphemes, phonemes (n-grams, FSAs, FSTs; regex compilation, best-paths, forward-backward, collocations) Vectors (decis.lists, Gaussians, na?ve Bayes; Yarowsky, clustering/k-NN) We’ve also seen some not-so-probabilistic stuff Syntactic features, semantics, morph., Gold. Could be stochasticized? Methods can be quantitative data-driven but not fully probabilistic: transf.-based learning, bottom-up clustering, LSA, competitive linking But probabilities have wormed their way into most things p(…) has to capture our intuitions about the ling. data An Alternative Tradition Old AI hacking technique: Possible parses (or whatever) have scores. Pick the one with the best score. How do you define the score? Completely ad hoc! Throw anything you want into the stew Add a bonus for this, a