The empirical distribution of good codes
Let the kth-order empirical distribution of a code be defined as the proportion of k-strings anywhere in the codebook equal to every given k-string. We show that for any fixed k, the X th-order empirical distribution of any good code (i.e., a code approaching capacity with vanishing probability of error) converges in the sense of divergence to the set of input distributions that maximize the input/output mutual information of X channel uses, This statement is proved for discrete memoryless channels as well as a large class of channels with memory, If k grows logarithmically (or faster) with blocklength, the result no longer holds for certain good codes, whereas for other good codes, the result can be shown for k growing as fast as a certain fraction of blocklength.
divergence discrete memoryless channels approximation of output statistics channel capacity error- correcting codes gaussian channels shannon theory