Informationchapter1-new3分解.ppt

Information Theory, Coding and Cryptography;Claude Elwood Shannon Father of Information Theory Born: 30 April 1916 in Gaylord, Michigan, USADied: 24 Feb 2001 in Medford, Massachusetts, USA “A Mathematical Theory of Communication” (1948). ;计分方式 ;REFERENCES l C.E.Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal , vol.27, pp.379-423, July, 1948 l《信息论》 ---傅祖芸，电子工业出版社 l“Information Theory, Coding and Cryptography” 《信息论、编码与密码学》 ---Ranjan Bose,机械工业出版社 l《应用信息论基础》 ---朱雪龙，清华大学出版社;信息论的方法应用及取得的成果;信息论的方法应用及取得的成果;信息论的方法应用及取得的成果;信息论的方法应用及取得的成果;信息论的方法应用及取得的成果;信息论的方法应用及取得的成果;信源编码与数据压缩-关键理论进展的十个里程碑[Kieffer 1993];;;;多用户信息论与网络通信;;Chapter 1Source Coding;Probability and probability distribution;; Self-Information ;Joint and Conditional probability Joint probability conditional probability; Conditional Self-Information of the event X=xi given Y=yj ;Average Self-Information of a random variable X is called the entropy, it’s given by ;entropy of the source: average amount of information conveyed per source symbol extremum property: If there are L possible events, the information per symbol lies between 0 and log2L bits, the maximum value being achieved when all probabilities are equal ( ). ;Average Mutual information between two random variables X and Y is given by ;Average Conditional Self-Information the information (or uncertainty) in X having observed Y. ;27;relationship ;互信息;互信息; 1. 互信息 (1) yj对xi的互信息 I(xi;yj) 即： I(xi;yj)= I(xi)- I(xi/yj) p(xi) ——先验概率：信源发xi的概率 p(xi/yj)——后验概率：信宿收到yj后， 推测信源发xi的概率 [含义] 互信息I(xi;yj) =自信息I(xi) - 条件自信息I(xi/yj) ＊I(xi) __信宿收到yj之前，对信源发xi的不确定度 ＊ I(xi/yj) __信宿收到yj之后，对信源发xi的不确定度 ＊ I(xi;yj) __收到yj而得到(关于xi )的互信息 =不确定度的减少量; (2) xi对yj的互信息 I(yj;xi) [含义] 信源发xi前、后， 信宿收到yj的不确定度的减少;2. 互信息的性质 (1) 对称性——I(xi ;yj) = I(yj ;xi)