J ina- C ol BERT -v2: A General-Purpose Multilingual Late Interaction Retriever (2024)

Abstract: Multi-vector dense models, such as ColBERT, have proven highly effective in information retrieval. ColBERT’s late interaction scoring approximates the joint query-document attention seen in cross-encoders while maintaining inference efficiency closer to traditional dense retrieval models, thanks to its bi-encoder architecture and recent optimizations in indexing and search. In this paper, we introduce a novel architecture and a training framework to support long context window and multilingual retrieval. Leveraging Matryoshka Representation Loss, we further demonstrate that the reducing the embedding dimensionality from 128 to 64 has insignificant impact on the model’s retrieval performance and cut storage requirements by up to 50%. Our new model, Jina-ColBERT-v2, demonstrates strong performance across a range of English and multilingual retrieval tasks,

## introduction neural retrieval has gained popularity in recent years following the arrival of capable pre-trained language models (plms) @xcite @xcite . two types of approaches have been employed to apply plms to retrieval. sparse neural retrieval systems, such as splade @xcite , represent texts as weighted bags of words that are interpreted as sparse high-dimensional vectors for maximum inner product search (mips). dense retrievers similarly encode queries and documents as dense vectors, capturing relevance signals through spatial relationships extending beyond exact term matching. most dense retrievers encode a query or document as a single vector, commonly the result of mean-pooling or the [cls]-embedding over the transformer's final layer token embeddings. in contrast, recent multi-vector retrievers like colbert @xcite generalize this embedding process to maintain an embedding for each token, computing relevance scores as a function of the similarities of query and document tokens instead. to make the colbert usable in practice, the output dimensionality is restricted to be much smaller than the single-vector models. this * work done while at jina ai. colbert-xm @xcite does this by using parameter extensions for each additional language, and @xcite trains solely on machine-translated english msmarco data to get effective heterogeneous multilingual performance. these approaches, however, come with trade-offs in terms of model usability and training data diversity. other multilingual multi-vector models like bge-m3 @xcite produce extremely large token representations that limit their practical utility for first-stage retrieval. in this work, we propose jina-colbert-v2, which introduces an improved training recipe for colbert models with the following features: training with diverse weakly-supervised data: we additionally pretrain our modified plm with rotary position embedding and train on large-scale unlabeled text pairs from various corpus with a weakly-supervised single-vector contrastive objective. a second-stage of colbert finetuning with labeled triplet data and supervised distillation is used to further boost its performance. ## related work in this section, we discuss related work in singleand multi-vector retrieval, as well as the non-english late-interaction retrievers from which our training recipe draws inspiration. ## pair training to leverage an abundance of text pairs with varying richness of semantic structure, we draw inspiration from common practices in single-vector embedding model training and begin with training on these text pairs, focusing on optimizing the embedding model's performance on general semantic similarity and relatedness tasks. this weakly-supervised stage is in contrast to previous colbert works, which typically start directly from a plm like bert with triplet training on 32-way or 64-way retrieval triplets consisting of a query, a positive passage, and multiple mined negatives. ## results we evaluate jina-colbert-v2 on four widely used benchmarks, beir, lotte, and miracl and mmarco. for general english performance, we use the same subset of 14 retrieval and text-similarity tasks from the beir benchmark as in @xcite . additionally, we assess performance on the lotte benchmark, which focuses on long-tail queries, and the miracl and mmarco benchmarks @xcite , which assess non-english retrieval performance. we report ndcg@10 for the beir and miracl collections, mrr@10 for mmarco, and success@5 for lotte. scores are reported on the test split for beir, development split for miracl and mmarco, and search test split for lotte. we use the same maximum query/document lengths as reported in @xcite , and use the default (32/300) for miracl and mmarco. finally, comparing against colbert-xm's zero-shot evaluation on mmarco in ## ablation studies in this section we present short ablation studies on modifications to three various aspects of colbert modeling and training. ## conclusion this work presents jina-colbert-v2, a capable multilingual colbert model that is the result of improvements to its architecture and training process. we implement modifications to the model architecture that yield efficiency gains with effectively no downside, and subsequently train it on a heterogeneous mix of data of varying tasks, languages, and supervision structures in order to bolster its performance as a general purpose retriever. our ablation experiments demonstrate the sensitivity of colbert to modifications to its representations.