formal multiple-bernoulli models for language modeling.pptVIP

Formal Multinomial and Multiple-Bernoulli Language Models Don Metzler Overview Two formal estimation techniques MAP estimates [Zargoza, Hiemstra, Tipping, SIGIR’03] Posterior expectations Language models considered Multinomial Multiple-Bernoulli (2 models) Bayesian Framework(MAP Estimation) Assume textual data X (document, query, etc) is generated by sampling from some distribution P(X | θ) parameterized by θ Assume some prior over θ. For each X, we want to find the maximum a posteriori (MAP) estimate: θX is our (language) model for data X Multinomial Modeling assumptions: Why Dirichlet? Conjugate prior to multinomial Easy to work with Multinomial How do we set α? α = 1 = uniform prior = ML estimate α = 2 = Laplacian smoothing Dirichlet-like smoothing: Multiple-Bernoulli Assume vocabulary V = A B C D How do we model text X = D B B D? In multinomial, we represent X as the sequence D B B D In multiple-Bernoulli we represent X as the vector [0 1 0 1] denoting terms B and D occur in X Each X represented by single binary vector Multiple-Bernoulli(Model A) Modeling assumptions: Each X is a single sample from a multiple-Bernoulli distribution parameterized by θ Use conjugate prior (multiple-Beta) Multiple-Bernoulli(Model A) Problems with Model A Ignores document length This may be desirable in some applications Ignores term frequencies How to solve this? Model X as a collection of samples (one per word occurrence) from an underlying multiple-Bernoulli distribution Example:V = A B C D, X = B D D BRepresentation: { [0 1 0 0], [0 0 0 1], [0 0 0 1], [0 1 0 0] } Multiple-Bernoulli(Model B) Modeling assumptions: Each X is a collection (multiset) of indicator vectors sampled from a multiple-Bernoulli distribution parameterized by θ Use conjugate prior (multiple-Beta) Multiple-Bernoulli(Model B) How do we set α, β? α = β = 1 = uniform prior = ML estimate But we want smoothed probabilities… One possibility: Another approach… Another way to formally estimate