S am T o N e: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives (2023)

Abstract: Dual encoders have been used for retrieval tasks and representation learning with good results. A standard way to train dual encoders is using a contrastive loss with in-batch negatives. In this work, we propose an improved contrastive learning objective by adding queries or documents from the same encoder towers to the negatives, for which we name it as “contrastive loss with SAMe TOwer NEgatives” (SamToNe). By evaluating on question answering retrieval benchmarks from MS MARCO and MultiReQA, and heterogenous zero-shot information retrieval benchmarks (BEIR), we demonstrate that SamToNe can effectively improve the retrieval quality for both symmetric and asymmetric dual encoders. By directly probing the embedding spaces of the two encoding towers via the t-SNE algorithm (van der Maaten and Hinton, 2008), we observe that SamToNe ensures the alignment between the embedding spaces from the two encoder towers. Based on the analysis of the embedding distance distributions of the top-1 retrieved results, we further explain the efficacy of the method from the perspective of regularisation.

## introduction the dual encoder architecture applied to information retrieval has shown excellent performance in a wide range of tasks @xcite @xcite . recently, the information retrieval community has transitioned towards deep learning models that leverage large unsupervised corpus pretraining @xcite , which offers more powerful semantic and contextual representation for queries and documents. these models can be successfully applied to scoring tasks, e.g. @xcite , or retrieval tasks, e.g. @xcite . in contrast, classic searchqa show that sharing a projection layer in asymmetric dual encoders (ade-spl) @xcite may not guarantee that the embeddings from the two encoder towers are in coinciding parameter spaces. however samtone can effectively achieve that. a dual encoder @xcite @xcite consists of two encoding towers that map queries and documents, respectively, into a shared low-dimensional dense representation, namely, the embedding space. the model is usually optimized by a contrastive loss @xcite , which moves the embeddings of the queries and documents from the same positive examples closer to each other, and the embeddings from negative examples farther away. training the dual encoder in batches allows to use, for each question, the passages that answer all the other questions within the batch as negatives @xcite , namely "in-batch negatives". at indexing time, all the documents in a corpus are encoded via bulk inference and indexed. to run retrieval, a query is encoded and its most relevant documents can be retrieved through nearest neighbours search ( @xcite over the embedding space using a measure of similarity, e.g. the dot-product or cosine distance of the embedding vectors. motivation. in this work, we consider two major types of dual encoder architectures: "symmetric dual encoder" (sde) 1 , with parameters shared between two encoder towers, and "asymmetric dual encoder" (ade), with two distinctly parameterized encoder towers. @xcite demonstrated that sharing projection layers can significantly improve the performance of ades. they empirically explained the efficacy of sde and ade-spl by claiming that the shared projection layers help mapping the embeddings of the two encoder towers into a coinciding parameter space. by repeating this embedding space analysis on a variety tasks, we find that ade-spl may not be enough to ensure that the embedding spaces from two encoder towers are coinciding, as shown in figure 1 . this motivates us to further improve the dual encoder retrieval quality beyond the architectural change explored in @xcite . although the projection layers are shared, our analyses suggest that an extra mechanism, other than using the standard contrastive loss with in-batch negatives, is required to ensure the adjacency of the embeddings of a ground truth pair. contributions. in this paper, we propose an improved training objective for dual encoder models: contrastive loss with same tower negatives (samtone). in section 3, we demonstrate its usefulness on a variety of information retrieval tasks, including both tasks with in-task fine-tuning and a zero-shot benchmark suite. across all the tasks explored, samtone performs competitively comparing to the traditional training setup, with a significant improvement on the metrics averaged across tasks. finally, through an analysis of the produced embeddings, in section 4, we further make evident the superiority of samtone from the perspective of regularisation. 1 this kind of dual encoders have also been called "siamese" or "twin" dual encoders. 2 method dual encoder architecture. we follow the standard setup of information retrieval: given a query, q, and a corpus of retrieval candidates, p, the goal is to retrieve k relevant candidates, p @xmath0. the candidate can be a phrase, a sentence, a passage, or a document. recent research @xcite demonstrated that sharing projection layers can significantly improve the performance of ades and we use this shared projection layer for ades (ade-spl) throughout our experiments. figure 2 illustrates the sde and ade-spl architectures we use in this work. our dual encoders are initialized from pre-trained t5.1.1 encoders @xcite . contrastive loss. a standard way to train a dual encoder model is optimizing an in-batch sampled softmax loss for contrastive learning @xcite : ## conclusions evaluating on qa retrieval tasks and zero-shot generalization benchmarks, we demonstrate that training with samtone can significantly improve the dual encoder retrieval quality. with t-sne maps of query and document embeddings, we show that the embedding spaces from the two encoding towers of models trained with samtone are better aligned. through the distributions of similarity distances between the embeddings of queries and their nearest neighbours, we empirically explain the efficacy of samtone from a regularisation prospective. in general, we recommend using samtone to train dual encoders for information retrieval tasks.