The empirical distribution of rate-constrained source codes
Let X = (X-1,X-....) be a stationary ergodic finite-alphabet source, X-n denote its first n symbols, and Y-n be the codeword assigned to X-n by a lossy source code. The empirical kth-order joint distribution Q(k)[X-n, Y-n](x(k), y(k)) is defined as the frequency of appearances of pairs of k-strings (x(k), y(k)) along the pair (X-n, Y-n). Our main interest is in the sample behavior of this (random) distribution. Letting I(Q(k)) denote the mutual information I(X-k; Y-k) when (X-k, Y-k) similar to Q(k) we show that for any (sequence of) lossy source code(s) of rate  infinity) 1/k I(Q(k)[X-n, Y-n]) <= R + 1/k H (X-1(k)) - (H) over bar (X) a.s. where (H) over bar (X) denotes the entropy rate of X. This is shown to imply, for a large class of sources including all independent and identically distributed (i.i.d). sources and all sources satisfying the Shannon lower bound with equality, that for any sequence of codes which is good in the sense of asymptotically attaining a point on the rate distortion curve Q(k)[X-n, Y-n]double right arrow(d) P-X(k),(k)(Y) a.s. whenever P-X(k),(k)(Y) is the unique distribution attaining the minimum in the definition of the kth-order rate distortion function. Consequences of these results include a new proof of Kieffer's sample converse to lossy source coding, as well as performance bounds for compression-based denoisers.
empirical distribution lossy source codes shannon lower bound compression-based denoising good codes pointwise bounds strong converse