SDR : Efficient Neural Re-ranking using Succinct Document Representation (2022)

Abstract: BERT based ranking models have achieved superior performance on various information retrieval tasks. However, the large number of parameters and complex self-attention operations come at a significant latency overhead. To remedy this, recent works propose late-interaction architectures, which allow pre-computation of intermediate document representations, thus reducing latency. Nonetheless, having solved the immediate latency issue, these methods now introduce storage costs and network fetching latency, which limit their adoption in real-life production systems. In this work, we propose the Succinct Document Representation (SDR) scheme that computes highly compressed intermediate document representations, mitigating the storage/network issue. Our approach first reduces the dimension of token representations by encoding them using a novel autoencoder architecture that uses the document’s textual content in both the encoding and decoding phases. After this token encoding step, we further reduce the size of the document representations using modern quantization techniques. Evaluation on MSMARCO’s passage re-reranking task show that compared to existing approaches using compressed document representations, our method is highly efficient, achieving 4x–11.6x higher compression rates for the same ranking quality. Similarly, on the TREC CAR dataset, we achieve 7.7x higher compression rate for the same ranking quality.

## introduction information retrieval (ir) systems traditionally comprise of two stages: retrieval and ranking. given a user query, the role of the retrieval stage is to quickly retrieve a set of candidate documents * both authors contributed equally to the paper. † work carried out while working at amazon. baseline figure 1 : mrr@10 performance vs. document corpus size tradeoff, measured on the msmarco-dev dataset. bert split is a distilled late-interaction model with reduced vector width and no compression ( § 4.2). for mrr@10 above 0.35, sdr is 4x-11.6x more efficient compared to the baseline. from a (very large) search index. retrieval is typically fast but not accurate enough; in order to improve the quality of the end result for the user, the candidate documents are re-ranked using a more accurate but computationally expensive algorithm. neural approaches have achieved the state of the art ranking performance in ir applications @xcite . transformer networks such as bert @xcite consistently show better ranking effectiveness at the cost of a higher computational cost and latency @xcite . to rank k documents, the ranker is called k times with an input of the form (query, document), where the query is the same, but the document is different. several works @xcite @xcite @xcite @xcite have proposed to modify bert-based rankers in a way that allows part of the model to compute query and document representations separately, and then produce the final score using a low-complexity interaction block; we denote these models as late-interaction rankers. such approaches pre-compute document representations to improve latency significantly. next, at runtime the model computes the query representation (once), retrieves the precomputed document representations, and is only required to run a low-complexity interaction block k times to produce the final ranking score. precomputing document representations has shown to significantly reduce latency and at the same time retain comparable scores to bert models @xcite . however, this does not account for additional storage and/or network fetching latency costs. the representations typically consist of the contextual token embeddings in a transformer model, which consume orders of magnitude more storage than storing the entire corpus search index (cf. § 5.1). in this work, we propose succinct document representation (sdr), a general scheme for compressing document representations. it enables lateinteraction rankers to be efficient in both latency and storage, while maintaining high ranking quality. sdr is suitable for any ranking scheme that uses contextual embeddings, and achieves extreme compression ratios (2-3 orders of magnitude) with little to no impact on retrieval accuracy. sdr consists of two major components: (1) embedding dimension reduction using an autoencoder with side information and (2) distribution-optimized quantization of the reduced-dimension vectors. in sdr, the autoencoder consists of two subnetworks: an encoder that reduces the vector's dimensions and a decoder that reconstructs the compressed vector. the encoder's output dimension represents the tradeoff between reconstruction fidelity and storage requirements. to improve the compression-reliability tradeoff, we leverage static token embeddings, which are available since the ranker has access to the document text (as it needs to render it to the user), and are computationally cheap to obtain. we feed these embeddings to both the encoder and decoder as side information, allowing the autoencoder to focus more on storing "just the context" of a token, and less on its original meaning that is available in the static embeddings. ablation tests verify that adding the static vectors significantly improves the compression rates for the same ranking accuracy. since data storage is measured in bits rather than floating-point numbers, sdr uses quantization techniques to reduce storage size further. given that it is hard to evaluate the amount of information in each of the encoder's output dimensions, we perform a randomized hadamard transform on the vectors, resulting in (1) evenly spread information across all coordinates and (2) transformed vectors that follow a gaussian-like distribution. we utilize known quantization techniques to represent these vectors using a small number of bits, controlling for the amount of quantization distortion. existing late-interaction schemes either ignore the storage overhead, or consider basic compression techniques, such as a simple (1 layer) autoencoder and float16 quantization. however, this is insufficient to reach reasonable storage size @xcite ; furthermore, this results in an increased fetching latency. for the msmarco dataset, we used a distilled model with a reduced vector width @xcite as the initial pre-trained weights for the late-interaction model. on top of this, we used a non-linear autoencoder consisting of 2 dense layers followed by float16 quantization, a natural extension of @xcite . this baseline achieves compression rates of 30x with no noticeable reduction in retrieval accuracy (measured with the official mrr@10 metric). in comparison with this strong baseline, our sdr scheme achieves an additional compression rate of between 4x to 11.6x with the same ranking quality, reducing document representation size to the same order of magnitude as the retrieved text itself. in figure 1 we include a high-level presentation of the baseline, a variant of our method with float16 quantization, and our full method. for the trec car dataset, for which we do not have a reduced-width baseline, we used a bert model as the pre-trained weights for the late-interaction model. the baseline with 2 dense layers and float16 quantization achieves a 30x compression rates with a slight reduction in accuracy. the sdr scheme reaches the same quality while improving compression rate by another 7.7x. to summarize, here are the contribution of this work foot_0 : late-interaction models. the idea of running several transformer layers for the document and the query independently, and then combining them in the last transformer layers, was developed concurrently by multiple teams: prettr @xcite , earl @xcite , dc-bert @xcite , dipair @xcite , and the deformer @xcite . these works show that only a few layers where the query and document interact are sufficient to achieve results close to the performance of a full bert ranker at a fraction of the runtime cost. for each document, the contextual token vectors are stored in a cache and retrieved during the document ranking phase. this impacts both storage cost as well as latency cost of fetching these vectors during the ranking phase. mores @xcite , extends lateinteraction models, where in the last interaction layers only the query attends to the document (and not vice-versa). as document are typically much longer, this results in additional performance improvements with similar storage requirements. col-bert @xcite ) is another variant that runs all transformer layers independently for the query and the document, and the interaction between the final vectors is done through a sumof-max operator. a similar work, the transformer-kernel (tk) @xcite , has an interaction block based on a low-complexity kernel operation. both colbert and tk result in models with lower runtime latency at the expense of a drop in ranking quality. however, the storage requirements for both approaches are still significant. some of the works above acknowledge the issue of storing the precomputed document representations and proposed partial solutions. in col-bert @xcite , the authors proposed to reduce the dimension of the final token embedding using a linear layer. however, even moderate compression ratios caused a large drop in ranking quality. in the prettr model @xcite , it was proposed to address the storage cost by using a standard auto-encoder architecture and the float16 format instead of float32. again, the ranking quality drops even with moderate compression ratios (they measured up to 12x). several other works @xcite @xcite @xcite proposed representing the queries and documents as vectors (as opposed to a vector per token), and using dot product as the interaction block. while this ranker architecture approach is simple (and can also be used for the retrieval step via an approximate nearest neighbor search such as faiss @xcite , scann @xcite or the pinecone managed service foot_1 ), the overall ranking quality is generally lower compared to methods that employ a query-document crossattention interaction. for that reason these methods are used mainly for first-stage retrieval, followed by a reranking step. compressed embeddings. our work reduces storage requirements by reducing the number of bits per floating-point value. quantization gained attention and success in reducing the size of neural network parameters @xcite @xcite and distributed learning communication costs @xcite @xcite @xcite . specifically, compressing word embeddings has been studied as an independent goal. @xcite studied the effect of quantized word embeddings on downstream applications and proposed a metric for quantifying this effect with simple linear models that operate on the word embeddings directly. as our work is concerned with compressing contextual embeddings, these methods do not apply since the set of possible embeddings values is not bounded by the vocabulary size. nevertheless, as in @xcite , we also observe that simple quantization schemes are quite effective. our work uses recent advances in this area to further reduce storage requirements for document representation, which, to the best of our knowledge, were not previously attempted in this context. ## succinct document representation (sdr) our work is based on the late-interaction architecture @xcite @xcite @xcite , which separates bert into l independent layers for the documents and the queries, and t -l interleaving layers, where t is the total number of layers in the original model, e.g., 12 for bert-base. naively storing all documents embeddings consumes a huge amount of storage with a total of m • h • 4 bytes per document, where m is the average number of tokens per document and h is the model hidden size (384 for the distilled version and 768 for the bert version). for msmarco, with 8.8m documents and @xmath0.9, it leads to a high storage cost of over a terabyte, which is not affordable except in large production systems. our compression scheme for the document representations consists of two sequential steps, (i) dimensionality reduction and (ii) block-wise quantization, described in § 3.1 and § 3.2 respectively. ## experimental settings in this section we describe the datasets used to evaluate the competing approaches for ranking documents given a query. next, we describe the baseline and the different configurations of sdr with emphasis on how we measure the compression ratio. ## evaluation results in this section, we present the end to end latency results ( § 5.1), show compression ratios and quality tradeoff of the sdr scheme ( § 5.2). we then examine how the proposed autoencoder ( § 5.3) compares with other baselines and present additional measurements ( § 5.4). ## conclusion in this paper, we proposed a system called sdr to solve the storage cost and latency overhead of existing late-interaction transformer based models for passage re-ranking. the sdr scheme uses a novel autoencoder architecture that uses static token embeddings as side information to improve encoding quality. in addition, we explored different quantization techniques and showed that the recently proposed eden performs well in our use case and presented extensive experimentation. overall, the sdr scheme reduces pre-computed document representation size by 4x-11.6x compared to a baseline that uses existing approaches. in future work, we plan to continue investigating means to reduce pre-computed document representation size.we believe that additional analysis of bert's vector and their interaction with the context would be fundamental in such an advancement.