Maximum Entropy Model课件.ppt

Maximum Entropy Model LING 572 Fei Xia 02/08/07 Topics in LING 572 Easy: kNN, Rocchio, DT, DL Feature selection, binarization, system combination Bagging Self-training Topics in LING 572 Slightly more complicated Boosting Co-training Hard (to some people): MaxEnt EM History The concept of Maximum Entropy can be traced back along multiple threads to Biblical times. Introduced to NLP area by Berger et. al. (1996). Used in many NLP tasks: Tagging, Parsing, PP attachment, LM, … Outline Main idea Modeling Training: estimating parameters Feature selection during training Case study Main idea Maximum Entropy Why maximum entropy? Maximize entropy = Minimize commitment Model all that is known and assume nothing about what is unknown. Model all that is known: satisfy a set of constraints that must hold Assume nothing about what is unknown: choose the most “uniform” distribution ? choose the one with maximum entropy Ex1: Coin-flip example(Klein Manning 2003) Toss a coin: p(H)=p1, p(T)=p2. Constraint: p1 + p2 = 1 Question: what’s your estimation of p=(p1, p2)? Answer: choose the p that maximizes H(p) Coin-flip example (cont) Ex2: An MT example(Berger et. al., 1996) An MT example (cont) An MT example (cont) Ex3: POS tagging(Klein and Manning, 2003) Ex3 (cont) Ex4: overlapping features(Klein and Manning, 2003) Modeling the problem Objective function: H(p) Goal: Among all the distributions that satisfy the constraints, choose the one, p*, that maximizes H(p). Question: How to represent constraints? Modeling Reference papers (Ratnaparkhi, 1997) (Ratnaparkhi, 1996) (Berger et. al., 1996) (Klein and Manning, 2003) ? Different notations. The basic idea Goal: estimate p Choose p with maximum entropy (or “uncertainty”) subject to the constraints (or “evidence”). Setting From training data, collect (a, b) pairs: a: thing to be predicted (e.g., a class in a classification problem) b: the context Ex: POS tagging: a=NN b=the words in a window and previous two