Paper 3, Section I, G

Coding and Cryptography
Part II, 2015

Let AA be a random variable that takes each value aa in the finite alphabet A\mathcal{A} with probability p(a)p(a). Show that, if each l(a)l(a) is an integer greater than 0 and 2l(a)1\sum 2^{-l(a)} \leqslant 1, then there is a decodable binary code c:A{0,1}c: \mathcal{A} \rightarrow\{0,1\}^{*} with each codeword c(a)c(a) having length l(a)l(a).

Prove that, for any decodable code c:A{0,1}c: \mathcal{A} \rightarrow\{0,1\}^{*}, we have

H(A)El(A)H(A) \leqslant \mathbb{E} l(A)

where H(A)H(A) is the entropy of the random variable AA. When is there equality in this inequality?