NaturalLanguageProcessing(3b)ZhaoHai赵海Department分析.pptVIP

10/10/2006 Natural Language Processing (3b) Zhao Hai 赵海Department of Computer Science and EngineeringShanghai Jiao Tong University2010-2011?zhaohai@cs.sjtu.edu.cn Outline Lexicons and Lexical Analysis Collocation Lexicons and Lexical Analysis (254) Collocation (35) Hypothesis Testing (1) One difficulty that we have glossed over so far is that high frequency and low variance can be accidental. For example, if the two constituent words of a frequent bigram like new companies are frequently occurring words (as new and companies are), then we expect the two words to co-occur a lot just by chance, even if they do not form a collocation. Lexicons and Lexical Analysis (255) Collocation (36) Hypothesis Testing (2) What we really want to know is whether two words occur together more often than chance. Assessing whether or not something is a chance event is one of the classical problems of statistics. It is usually couched in terms of hypothesis testing. Lexicons and Lexical Analysis (256) Collocation (37) Hypothesis Testing (3) We formulate a null hypothesis H0 that there is no association between the words beyond chance occurrences, compute the probability p that the event would occur. If H0 were true, and then reject H if p is too low (typically if beneath a significance level of p 0.05, 0.01, 0.0005, or 0.001) and retain H0 as possible otherwise. Lexicons and Lexical Analysis (257) Collocation (38) Hypothesis Testing (4) How can we apply the methodology of hypothesis testing to the problem of finding collocations? We first need to formulate a null hypothesis which states what should be true if two words do not form a collocation. For such a free combination of two words we will assume that each of the words w1 and w2 is generated completely independently of the other, and so their chance of coming together is simply given by: Lexicons and Lexical Analysis (258) Collocation (39) Hypothesis Testing (5) The model imp