07信息论_03.pptVIP

信息论 郑伟zhengw@ Last Class Conditional Entropy(条件熵) The expected entropy of Y after we have observed a value x?X, is called the conditional entropy H(Y|X): Noisy Channel Example噪声信道 Another Example of H(X|Y) Take p(X) over {0,…,500} with p = (?,1/1000,…,1/1000) with entropy H(X) = ? + ??log 1000 = 4.983… bits. If we ‘learn’ that x is not 0, then we increase the entropy: p(x|“x≠0”)=(0,1/500,…,1/500) with H(X|“x≠0”)=8.966… bits. We learned information, yet the entropy/uncertainty increased? Think: Not finding your wallet in the likely place. The expected uncertainty (=conditional entropy) goes down:H(X|“x=0?”) = ? H(X| “x=0”) + ? H(X|“x≠0”) = 4.483… bits. Think: Average value of H(X|X’)? 平均意义下，条件作用使熵减小。 Asymmetry of H(X|Y) If the relation between X and Y is asymmetric, then we will have in general: H(X|Y) ≠ H(Y|X). Example of an asymmetric channel. Assume again H(X) = 1 andnote that H(Y) = H((3/4,1/4)) = 0.811… We have H(X|Y) = 0.689… bits. On the other hand H(Y|X) = 0.5 bits. Check with H(X,Y) = 3/2 and the chain rule:H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y) Chain Rule for Entropy For random variables X1,…,Xn we have the Chain rule: Mutual Information For two variables X,Y the mutual information I(X;Y) is the amount of certainty regarding X that we learned after observing Y. Hence I(X;Y) = H(X)–H(X|Y). Note that now X and Y can be interchanged using the chain rule:  Think of I(X;Y) as the ‘overlap’ between X and Y. 互信息是对称的！ All Together Now Asymmetric Channel About Mutual Information Mutual information is the central notion in information theory. It quantifies how much we learn about X by observing Y. When X and Y are the same we get: I(X;X) = H(X), hence entropy is called ‘self information’. It can be generalized to more than two variables: Expectation of What? Mutual information can be viewed as an expectation: Relative Entropy The relative entropy or Kullback Leibler distance between two distributions p and q is defined by Example KL Distan