DADA : Distribution-Aware Domain Adaptation of PLM s for Information Retrieval (2024)

Abstract: Pre-trained language models (PLMs) exhibit promise in retrieval tasks but struggle with out-of-domain data due to distribution shifts.Addressing this, generative domain adaptation (DA), known as GPL, tackles distribution shifts by generating pseudo queries and labels to train models for predicting query-document relationships in new domains.However, it overlooks the domain distribution, causing the model to struggle with aligning the distribution in the target domain.We, therefore, propose a Distribution-Aware Domain Adaptation (DADA) to guide the model to consider the domain distribution knowledge at the level of both a single document and the corpus, which is referred to as observation-level feedback and domain-level feedback, respectively.Our method effectively adapts the model to the target domain and expands document representation to unseen gold query terms using domain and observation feedback, as demonstrated by empirical results on the BEIR benchmark.

## introduction recent advances in pretrained language models (plms) @xcite have significantly enhanced our ability to retrieve information @xcite . these advancements are particularly pronounced when the data closely aligns with what the plm was originally trained on, denoted as "indomain" data. however, challenges emerge when dealing with "out-of-domain" (ood) data, characterized by substantial disparities from the training data. studies such as @xcite have highlighted these challenges. in response to these issues, various domain adaptation (da) methods have been proposed. a widely ## related work this section delves into the landscape of plms within information retrieval (ir) and examines the evolution of da techniques that have paved the way for our proposed method. ## proposed method in this section, we provide a detailed explanation of the methods employed in our study, focusing on the integration of the dada into the gpl framework. we begin with a concise overview of gpl, followed by an in-depth description of how dada is integrated into gpl, covering both training and inference perspectives. ## conclusion we have introduced a novel method called dada with the specific aim of enhancing the capabilities of plms for addressing domain shifts in information retrieval. unlike traditional methods that resolve domain shifts through dataset updates based on observation-level feedback, dada takes a direct approach by updating the loss function with a multi-level feedback vector, which integrates domain-level feedback and observation-level feedback. this innovative approach enables dada to dynamically adjust to varying data landscapes, making it particularly effective in scenarios where rapid adaptation is essential. compared to an existing method like gpl on the beir benchmark, dada demonstrates superior performance across various subtasks, highlighting its potential in handling unseen domains. while dada shows promise, further research could investigate integration with other dataset update adaptation techniques, contributing to the development of more robust and versatile information retrieval systems.