BERM : Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval (2023)

Abstract: Dense retrieval has shown promise in the first-stage retrieval process when trained on in-domain labeled datasets. However, previous studies have found that dense retrieval is hard to generalize to unseen domains due to its weak modeling of domain-invariant and interpretable feature (i.e., matching signal between two texts, which is the essence of information retrieval). In this paper, we propose a novel method to improve the generalization of dense retrieval via capturing matching signal called BERM. Fully fine-grained expression and query-oriented saliency are two properties of the matching signal. Thus, in BERM, a single passage is segmented into multiple units and two unit-level requirements are proposed for representation as the constraint in training to obtain the effective matching signal. One is semantic unit balance and the other is essential matching unit extractability. Unit-level view and balanced semantics make representation express the text in a fine-grained manner. Essential matching unit extractability makes passage representation sensitive to the given query to extract the pure matching information from the passage containing complex context. Experiments on BEIR show that our method can be effectively combined with different dense retrieval training methods (vanilla, hard negatives mining and knowledge distillation) to improve its generalization ability without any additional inference overhead and target domain data.

## introduction dense retrieval encodes the texts to dense embeddings and efficiently gets the target texts via approximate nearest neighbor search @xcite . compared with the traditional word-to-word exact matching methods such as bm25 @xcite , dense retrieval can capture the relevance at the semantic level of two ## related work dense retrieval estimates the relevance via representations of two texts. dpr @xcite combines dense retrieval with pre-trained models for open-domain question answering @xcite . besides, some methods focus on obtaining more valuable negatives @xcite @xcite . some methods use a more powerful reranker for knowledge distillation @xcite . recently, the generalization of dense retrieval has received attention. @xcite performs the examination of the generalization of dense retrieval. beir @xcite is proposed as the benchmark to evaluate the zero-shot ability of information retrieval models. modir @xcite uses the data from source and target domains for ad-versarial training to perform unsupervised domain adaptation. genq @xcite and gpl @xcite generate queries and pseudo labels for domain adaptation. contriever @xcite uses contrastive pre-training on large corpus (wikipedia and cc-net @xcite ). coco-dr @xcite performs unsupervised pre-training on target domain and introduces distributional robust optimization. gtr @xcite scales up the model size to improve the generalization. @xcite introduce sparse retrieval to achieve better generalization. improvement of generalization of dense retrieval in previous studies comes from the adaptation of the target domain, knowledge from large pretraining corpus, and assistance of sparse retrieval but not dense retrieval itself. they need to obtain the target domain data in the training or increase the complexity of the system. in this paper, we introduce a novel method to improve the generalization of dense retrieval without target domain data and additional modules via learning the generalizable representation for matching. one thing must be emphasized that the methods of multi-view dense retrieval @xcite also divide a passage into multiple units, but our method is essentially a completely different method. multi-view dense retrieval uses multiple representations to fully express a passage from multiple views, which focuses on in-domain retrieval. our method uses multiple units to make the model learn to extract essential matching unit from the passage containing complex context, which is domain-invariant for generalization. in our method, multiple units are only used as the constraint for optimization in training and only a single representation is used in inference. learning-based sparse retrieval such as coil @xcite and splade @xcite also aim to express fine-grained token-level semantics but they need multiple vectors to represent tokens in passage (coil) or sparse-vector of vocabulary size (splade) and calculates the score by tokento-token matching, which is not suitable for dense retrieval that uses single dense vector to perform representation and dot product. ## motivation dense retrieval is hard to generalize to unseen domains due to its weak modeling of domain-evenly aggregate the semantics of the units into text representation to implicitly and comprehensively express each unit of the passage. ## our method this section introduces the implementation of our method (figure 3 ). our method optimizes the relationship between the representations and the units in the passage. therefore, before training, we perform unit segmentation and annotate the essential matching unit for the datasets. then, we design loss functions according to the requirements of r1 and r2 and combine these functions with task loss of dense retrieval (contrastive loss) for joint training. ## experiments this section introduces the experimental setups and analyzes the results. ## conclusion in this paper, we propose an effective method called berm to improve the generalization ability of dense retrieval without target domain data and additional modules. the basic idea of berm is learning the domain-invariant feature, that is, matching signal. to achieve it, we introduce a novel concept of dense retrieval to represent the matching information between two texts, the matching representation. further, we propose two requirements for matching and text representations as the constraint in the training of dense retrieval to enhance the ability to extract essential matching information from the passage according to different queries under the premise of balanced expression of the text. the two requirements unlock the ability of dense retrieval to capture matching signal without additional interaction. experimental results show that berm is a flexible method that can be combined with different dense retrieval training methods without inference overhead to improve the out-of-domain generalization ability. in domain adaptation setting, our method is also effective and performs better than baselines.