Modeling Long Distance Dependence n Language Topic Mixtures 在长距离依赖建模语言主题的混合物.ppt

Modeling Long Distance Dependence in Language: Topic Mixtures Versus Dynamic Cache Models Rukmini.M Iyer, Mari Ostendorf Introduction Statistical language models part of state-of-the-art speech recognizers Word sequence: s w1,w2,…,wT /s Consider the word sequence to be a Markov process N-gram Probability: Introduction Use bigram representations Bigram probability Trigram in experiments Although n-gram models are powerful, but are constrained to n. Dependencies longer than n ? The problem of representing long distance dependencies has been explored. Introduction Sentence-level: words can co-occur in a sentence due to topic, grammatical dependencies Article or conversation-level: The subject of an article of a conversation Dynamic cache language models address the second issue Increase the likelihood of a word given that it has been observed previously in the article Other methods: Trigger language models Context-free grammars Mixture Model Framework N-gram level: Sentence level: Pk: bigram model for k-th class λk: mixture weight Mixture Model Framework Two main issues: Automatic clustering to handle data not explicitly marked for topic dependence Robust parameter estimation Clustering Specified by hand or determined automatically Agglomerative clustering Partition into desired number of topics Performed at the article level Reduce computation The initial clusters can be singleton units of data Clustering let the desired number of clusters be C* and initial number of clusters be C Find the best clusters, say Ai and Aj, to maximize some similarity criterion Sij Merge Ai and Aj and decrement C If current number of C=C*, then stop;otherwise go to Step 2 Clustering Similarity measure: Set-intersection measure Combination of inverse document frequencies |Ai|: the number of unique words in cluster i |Aw|: the number of clusters containing the word w Nij: the normalization factor Advantage of idf measure: High-frequency words, such as func