Rate-distortion approach to databases: Storage and content-based retrieval
This paper investigates the relationship between rate-distortion theory and efficient content-based data retrieval from high-dimensional databases. We consider database design as the encoding of a data object sequence, and retrieval from the database as the decoding of the sequence using side information (i.e., the query) available only at the decoder. We show that, in this setting, the optimal asymptotic tradeoff between the search time R-s (bits per data object read from the storage device) and the expected search accuracy D-s (relevance of the retrieved data set) is given by the Wyner-Ziv solution with a side-information-dependent distortion measure. Moreover, the data indexing and retrieval problem is, in general, inseparable from the data compression problem. Data items selected by the search procedure, which can be stored in the disk with a limited total rate of R-r greater than or equal to R-s, need to be presented at a prescribed expected reconstruction quality D-r. This is, hence, a problem of scalable source coding or successive refinement, albeit with differing layer distortion measures to quantify search and reconstruction quality, respectively. We derive a single-letter characterization of all achievable quadruples {R-s, R-r, D-s, D-r}, and prove conditions for "successive refinability" without rate loss. Finally, we show that the special case D-s = D-r = 0 is nontrivial and of practical interest in this context, as it can impose "acceptable" search and reconstruction qualities for each individual data item and for the entire query space with high probability, in contradistinction with standard average distortion requirements. The region of achievable {R-s,R- R-r} is obtained by adapting Rimoldi's characterization to a new regular scalable coding problem.
databases content-based retrieval scalable coding approximate similarity searching successive refinability without rate loss wyner-ziv problem zero-one distortion measures