C ol BERT v2: Effective and Efficient Retrieval via Lightweight Late Interaction (2022)

Abstract: Neural information retrieval (IR) has greatly advanced search and other knowledge-intensive language tasks. While many neural IR methods encode queries and documents into single-vector representations, late interaction models produce multi-vector representations at the granularity of each token and decompose relevance modeling into scalable token-level computations. This decomposition has been shown to make late interaction more effective, but it inflates the space footprint of these models by an order of magnitude. In this work, we introduce ColBERTv2, a retriever that couples an aggressive residual compression mechanism with a denoised supervision strategy to simultaneously improve the quality and space footprint of late interaction. We evaluate ColBERTv2 across a wide range of benchmarks, establishing state-of-the-art quality within and outside the training domain while reducing the space footprint of late interaction models by 6–10x.

## introduction neural information retrieval (ir) has quickly dominated the search landscape over the past 2-3 years, dramatically advancing not only passage and document search @xcite but also many knowledge-intensive nlp tasks like opendomain question answering @xcite , multi-hop claim verification @xcite , and open-ended generation @xcite . many neural ir methods follow a single-vector similarity paradigm: a pretrained language model is used to encode each query and each document into a single high-dimensional vector, and relevance is modeled as a simple dot product between both vectors. an alternative is late interaction, introduced in colbert @xcite , where queries and documents are encoded at a finergranularity into multi-vector representations, and by decomposing relevance modeling into tokenlevel computations, late interaction aims to reduce the burden on the encoder: whereas single-vector models must capture complex query-document relationships within one dot product, late interaction encodes meaning at the level of tokens and delegates query-document matching to the interaction mechanism. this added expressivity comes at a cost: existing late interaction systems impose an order-of-magnitude larger space footprint than single-vector models, as they must store billions of small vectors for web-scale collections. considering this challenge, it might seem more fruitful to focus instead on addressing the fragility of single-vector models @xcite by introducing new supervision paradigms for negative mining @xcite , pretraining @xcite , and distillation @xcite . indeed, recent single-vector models with highlytuned supervision strategies @xcite sometimes perform on-par or even better than "vanilla" late interaction models, and it is not necessarily clear whether late interaction architectures-with their fixed token-level inductive biases-admit similarly large gains from improved supervision. in this work, we show that late interaction retrievers naturally produce lightweight token representations that are amenable to efficient storage off-the-shelf and that they can benefit drastically from denoised supervision. we couple those in colbertv2, 1 a new late-interaction retriever that employs a simple combination of distillation from a cross-encoder and hard-negative mining ( §3.2) to boost quality beyond any existing method, and then uses a residual compression mechanism ( §3.3) to reduce the space footprint of late interaction by 6-10× while preserving quality. as a result, col-bertv2 establishes state-of-the-art retrieval quality both within and outside its training domain with a competitive space footprint with typical singlevector models. when trained on ms marco passage ranking, colbertv2 achieves the highest mrr@10 of any standalone retriever. in addition to in-domain quality, we seek a retriever that generalizes "zeroshot" to domain-specific corpora and long-tail topics, ones that are often under-represented in large public training sets. to this end, we evaluate col-bertv2 on a wide array of out-of-domain benchmarks. these include three wikipedia open-qa retrieval tests and 13 diverse retrieval and semanticsimilarity tasks from beir @xcite . in addition, we introduce a new benchmark, dubbed lotte, for long-tail topic-stratified evaluation for ir that features 12 domain-specific search tests, spanning stackexchange communities and using queries from gooaq @xcite . lotte focuses on relatively long-tail topics in its passages, unlike the open-qa tests and many of the beir tasks, and evaluates models on their capacity to answer natural search queries with a practical intent, unlike many of beir's semanticsimilarity tasks. on 22 of 28 out-of-domain tests, colbertv2 achieves the highest quality, outperforming the next best retriever by up to 8% relative gain, while using its compressed representations. this work makes the following contributions: ## colbertv2 we now introduce colbertv2, which improves the quality of multi-vector retrieval models ( §3.2) while reducing their space footprint ( §3.3). ## evaluation we now evaluate colbertv2 on passage retrieval tasks, testing its quality within the training domain ( §5.1) as well as outside the training domain in zero-shot settings ( §5.2). unless otherwise stated, we compress colbertv2 embeddings to @xmath0. ## conclusion we introduced colbertv2, a retriever that advances the quality and space efficiency of multivector representations. we hypothesized that cluster centroids capture context-aware semantics of the token-level representations and proposed a residual representation that leverages these patterns to dramatically reduce the footprint of multi-vector systems off-the-shelf. we then explored improved supervision for multi-vector retrieval and found that their quality improves considerably upon distillation from a cross-encoder system. the proposed colbertv2 considerably outperforms existing retrievers in within-domain and out-of-domain evaluations, which we conducted extensively across 28 datasets, establishing state-of-the-art quality while exhibiting competitive space footprint.