情报理论.ppt

today’s lecture -the performance of the Huffman code and its limit -extensions of the Huffman code - block coding contribute to reduce the average codeword length -Shannon’s source coding theorem - theoretical bound of the source coding (or, data compression) -block Huffman code with unequal block length -run-length Huffman code * a source code should - be immediately decodable (this implies that the code is uniquely decodable) - have small average codeword length A Huffman code seems a good code, but “how good” is it? to answer to this question, we would like to investigate a theoretical bound on the average codeword length. * we consider a theorem which gives the lower-bound limit of the average codeword length Assume that we have an information source S which has M symbols, and also assume that the i-th messages occurs with probability p_i. We would like to construct an immediately decodable code C by defining a codeword for each message symbol. Theorem: part 1: the average codeword length L of C is H_1(S) or more. part 2: there is a code whose code length L is less than H_1(S) + 1 (a proof will be given later) This theorem implies we cannot make the average codeword length smaller than the entropy. Interestingly, this lower bound is achievable. We can construct a code whose code length L is smaller than the entropy plus one. * To give a proof of the previous theorem, we need one lemma which is named after Shannon. In the lemman we use probabilities p_1, ..., p_M which are probabilities of messages of S. Lemma: for any positive numbers q_1,.., q_M with q_1 + ... +q_M = 1, the inequality given in the slide holds. The inequality holds by equation if and only if p_i = q_i. * proof of the Shannon’s lemma We would like to show the inequality which is given at the top of this slide. Consider (lhs – rhs) of the inequality and translate the base of the logarithm. The log-value is lower-bounded as illustrated in the graph, and we finally see that (lhs