Lectur_2-Mathmatic_Foundations-v2.ppt

* * * * * * * * * * * * * * * * * * * * * Entropy Measures the amount of information in a random variable The unit is bits. The least (average) number of bits needed to encode a message The average information of X The expectation of self information – E(I(X)=E(-log2p(x))= Σ p(x)(-log2p(x)) = H(X) * 傈呸戴罪筑箭蝶喜葬历斡钳映肄储去迎粕凑淆达夯底恃地婴震矿科西笨嚣Lectur_2-Mathmatic_Foundations-v2Lectur_2-Mathmatic_Foundations-v2 Entropy: Example Toss Coin (evenly coin)，Ω={H,T} p(H)=0.5, p(T)=0.5 H(p)=-0.5log20.5+(-0.5log20.5)=1 Toss Coin (unevenly coin) ，Ω={H,T} p(H)=0.2, p(T)=0.8, H(p)=0.722 p(H)=0.01, p(T)=0.99, H(p)=0.081 32side –die – H(p)=-32((1/32)log2(1/32))=5 - 21=2, 25=32(perplexity) ? 掷不均匀硬币 * 扛场烂猜由痒杭敖怕判醋搐袜变象针崩贼幂磋挠迪疤泉刻库泻徐郸哄搅脾Lectur_2-Mathmatic_Foundations-v2Lectur_2-Mathmatic_Foundations-v2 Entropy Entropy ≥0 H(p)=0 only when the value of X is determinate, hence providing no new information –?x∈Ω, p(x)=1; ?y∈Ω, p(y)=0 if y≠x Entropy increases with the message length – |Ω|=n，H(p)≤-log2n – Maximal Entropy for uniform distribution 2 outputs ，H(p)=1bit 32 outputs, H(p)=5bits 4.3 billion outputs，H(p)=32bits * 忙吟凯絮归自鳞被铲娇酶族莫稳磕帘脂苞氢谈赏沥环狙秉逝胺姆勃兰科尿Lectur_2-Mathmatic_Foundations-v2Lectur_2-Mathmatic_Foundations-v2 Perplexity in NLP Define How many possible outputs for each experiment Perplexity in NLP a common way of evaluating language models. A language model is a probability distribution over entire sentences or texts Uniform distribution of words in lexicon Largest Entropy Highest perplexity Hardest to predicate * 巾冉权嗣龚刷喧没头咳胎舵辱游蝗曳肿掏搔愈唤跌菩莲拳飘癌痘鼠甜反愁Lectur_2-Mathmatic_Foundations-v2Lectur_2-Mathmatic_Foundations-v2 Perplexity in NLP: A Example The average sentence xi in the test sample could be coded in 190 bits The lowest perplexity that has been published on the Brown Corpus (1 million words of American English of varying topics and genres) It is about 247 bits per word, or 1.75 bits per letter using a trigram model. It is often possible to achieve lower perplexity on more s