07信息论_04.pptVIP

信息论 郑伟 浙江大学信息与通信工程研究所 Chapter 8: Channel Capacity 通信系统模型：信道部分 Using Noisy Channels Shannon’s noisy channel-coding theorem shows that unreliable channels can be used for reliable communication if we code our messages cleverly. More specifically, the theorem states that each (discrete, memoryless) channel has a capacity C≥0, such that each “bits per transmission” rate R C is achievable if we use long enough codes. (Achievable here means that the error probability ?(n) tends to zero as the length n of the codes grows.) Defining Channels Channel Capacities Some Example Capacities A noiseless binary channel has H(X|Y)=0, hence for the mutual information I(X;Y)=H(X), which is maximized by p(0)=p(1)=?. Hence C = maxp I(X;Y) = 1 bit. 每次使用信道最多传输1bit信息 Channel Capacity Properties [Cover Thomas, Section 8.3]: Because the capacity of a channel is the maximum of mutual informations I(X;Y) it holds that C≥0 and that C ≤ log|X| and C ≤ log|Y| as I(X,Y)≤log|X|, log|Y|. I(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X) Furthermore, C=0 if and only if H(X,Y)=H(X)+H(Y), which means that X and Y are independent. Although for simple channels one can often calculate the capacity explicitly, for general transition probabilities p(Y|X) and large sets X,Y determining the capacity C is a optimization problem of the concave function I(X;Y). Binary Erasure Channel Noisy Dialing Entropies of Noisy Dialing A Bigger Picture Why Shannon’s Noisy Channel Theorem Holds Typical versus Atypical AEP for Binary Values Example of a Binary Random Variable X:Let p(X=0) = 1/3, p(X=1) = 2/3 such that H(X) = 0.918296. The Asymptotic Equipartition Property claims that p(Xn) will converge to a probability distribution over {0,1}n for which 2(0.918296±?) n typical elements have probability 2 –(0.918296±?) nand the atypical elements are negligible. For ?=0.01 and n=10 we should have 542 ~ 623 elements with probability 0.00161 ~ 0.00184 In truth we have no elements with probabilities in this range. Let’s try aga