Large Dual Encoders Are Generalizable Retrievers (2022)

Abstract: It has been shown that dual encoders trained on one domain often fail to generalize to other domains for retrieval tasks. One widespread belief is that the bottleneck layer of a dual encoder, where the final score is simply a dot-product between a query vector and a passage vector, is too limited compared to models with fine-grained interactions between the query and the passage. In this paper, we challenge this belief by scaling up the size of the dual encoder model while keeping the bottleneck layer as a single dot-product with a fixed size. With multi-stage training, scaling up the model size brings significant improvement on a variety of retrieval tasks, especially for out-of-domain generalization. We further analyze the impact of the bottleneck layer and demonstrate diminishing improvement when scaling up the embedding size. Experimental results show that our dual encoders, G eneralizable T 5-based dense R etrievers (GTR), outperform previous sparse and dense retrievers on the BEIR dataset significantly. Most surprisingly, our ablation study finds that GTR is very data efficient, as it only needs 10% of MS Marco supervised data to match the out-of-domain performance of using all supervised data.

## introduction typical neural retrieval models follow a dual encoder paradigm @xcite @xcite . in this setup, queries and documents are encoded separately into a shared fixed-dimensional embedding space, where relevant queries and documents are represented in each other's proximity. then, approximated nearest neighbor search @xcite is applied to efficiently retrieve relevant documents given an encoded input query. while dual encoders are popular neural retrievers, the expressiveness of the model is limited by a bottleneck layer consisting of only a simple dotproduct between query and passage embeddings. @xcite ; @xcite argued that the dot-product (or cosine similarity) between the embeddings might not be powerful enough to capture the semantic relevance. similarly, @xcite suggested that dual encoder models have "issues for out-of-distribution data" and models with more interactions between queries and documents have better generalization ability. in this paper, we challenge this belief by scaling up the dual encoder model size while keeping the bottleneck as a single dot-product with a fixed size. note that scaling up a dual encoder is different from scaling up pretrained language models, such as bert @xcite and t5 @xcite , because of the presence of the bottleneck layer. while increasing the model size can greatly increase model capacity, for dual encoders with a fixed bottleneck embedding size, the interactions between queries and documents are still limited by a simple dot-product. to test this hypothesis, we take advantage of the t5 architecture and checkpoints to build encoders of up to 5 billion parameters while keeping the the research question we ask is: can scaling up dual encoder model size improve the retrieval performance while keeping the bottleneck layers as a single dot-product with a fixed size? only the encoder is taken from the pre-trained t5 models, and the two towers of the dual encoder share parameters. ## evaluation results we present three groups of experiments to study a) the in-domain performance on ms marco, b) the out-of-domain generalization performance on beir, and c) the data efficiency. ## ablation study and analysis in this section we present ablations and analysis to further illustrate the effects of scaling up, the impact of fine-tuning and pre-training, and the gtr model's behavior. ## inference latency a caveat of scaling up is the increase in latency overhead. therefore, we investigate the inference speed, in microseconds (ms), for all gtr models with a batch size of 1 and an input length of 128. we found the latency increases from 17 ms, 34 ms, 96 ms to 349 ms. to put it in context, the gtr-base model has a close latency compared to tas-b while the largest gtr-xxl model has a similar latency with re-ranking models @xcite . with the recent work towards making large models efficient with sparsification, distillation and prompttuning, we hope the inference time for large dual encoders can be significantly reduced in the future. ## conclusion this paper presents the generalizable t5 retriever (gtr), a scaled-up dual encoder model with a fixed-size dot-product bottleneck layer. we show that scaling up the model size brings significant improvement on retrieval performance across the board on the beir zero-shot retrieval benchmark, especially for out-of-domain generalization. the gtr-xxl model performs at the level of state-ofthe-art performance on beir, outperforming many models that use earlier interactions between queries and documents. this sheds light on the research direction to continue enhancing the single vector representation model through better backbone encoders. moreover, our in-depth analysis reveals the impact of scaling up under the scenarios of different training stages, pre-training strategies, fine-tuning datasets, and bottleneck sizes, as well as how scaling up influences the retrieved document lengths. our findings can inform future work and is an integral part of the joint effort to improve dual encoder models.