Transfer-Aware Data Selection for Domain Adaptation in Text Retrieval (2025)

Abstract: Domain adaptation is widely adopted in text retrieval scenarios where large labeled data is unavailable. To improve model adaptability, existing methods try to expand more source datasets. However, we found from experiments that indiscriminately using a large amount of source data from various text tasks does not guarantee improved adaptability, but may negatively impact model performance. To tackle this issue, we propose Trait, a framework that can effectively improve model adaptability by selecting beneficial data without evaluating all source data. Specifically, we first divide multiple source datasets into data chunks of the same size as the minimum selection unit to form the whole selection space. Then we devise an iterative process that includes Bayesian optimization-based selection and transfer-aware chunk evaluation to incrementally select beneficial chunks. To reduce unnecessary evaluation costs, we also design backtracking and pruning actions to adjust the selection subspace. Extensive experimental results show that Trait not only achieves average state-of-the-art for few-shot on nine target datasets by evaluating only 4% of BERRI source data, but also is very competitive for zero-shot compared with LLM-based rankers.

## introduction in many realistic text retrieval tasks, acquiring large labeled training data is challenging due to the high cost of manual annotations @xcite @xcite . to tackle this issue, a technique called domain adaptation is widely adopted @xcite , which trains the model on large labeled source datasets from related tasks (e.g., ms marco @xcite and nq @xcite ), and then adapts it to the target dataset. ## problem statement let @xmath0, ..., d m } denote documents and @xmath1. text retrieval requires a retrieval model to learn a ranking function r : d × @xmath2, such that for any d i , d @xmath3, there is: in many realistic retrieval scenarios, it is hardly possible to obtain large labeled training data. to solve it, traditional domain adaptation in text retrieval uses all available source datasets from related tasks for training (see train on evaluate target retrieval dataset model train evaluate target retrieval dataset model a subset of examples : query : document : relevance label headline retrieval tasks ( , , ) source datasets {s 1 , s 2 , . . . , s k } and a target retrieval dataset t , each s k (1 ≤ @xmath4, d) with relevance label l, the goal of traditional domain adaptation is to improve the adaptation of a model r(•, •; θ(s)) to t , which is trained on the aggregated source pool @xmath5. however, as shown in figure 1 , various source data can lead to different effects on model adaptability. therefore, identifying a subset of source data beneficial to adaptability is the research problem of this paper (i.e., fig. 2 (b) ), named data selection for domain adaptation in text retrieval, which is formulated as follows: problem 1 given a set of source data @xmath6, a target retrieval dataset t , the goal of data selection for domain adaptation in text retrieval is to identify a subset s ⊂ s with s ≪ |s|, such that the model r(•, •; θ( s)) trained on s achieves enhanced performance on t . ## methodology we introduce trait, a framework that efficiently selects beneficial source data to enhance model adaptation without computationally expensive evaluation of the entire source data. as depicted in figure 3 , it begins by initializing the selection subspace (section 4.1). then, the chunk selection module leverages bayesian optimization to select a high-value data chunk from the subspace (section 4.2.1). thereafter, the transfer-aware module evaluates the selected chunk, which generates a new evaluation record and adjusts the subspace dynamically (section 4.2.2). these updates are returned to facilitate the next selection. this selection-evaluation process iterates until the stopping conditions are met, outputting a trait-ranker trained on a subset of selected chunks. (a) divided source datasets … data chunk 𝑺 𝟏 𝑺 𝟐 (b) selection subspace 𝓢 ′ +1 +1 +1 … initial selection: none select a random chunk from each source 𝑺 𝑲 chunk selection … (c) vectors represented candidates in 𝓢 ′ none + none (initial) none + none + … vector 𝒔 figure 4: selection space structuring includes: (a) divide source datasets; (b) specify the selection subspace s ′ ; (c) design vector to represent candidates in s ′ . ## conclusion we show that indiscriminately using various source data may reduce model adaptability to the target retrieval task. we propose trait, a data selection framework that accumulates beneficial source data by selectively evaluating data chunks in limited iterations, thus effectively improving model adaptability. extensive experiments demonstrate the effectiveness of trait for domain adaptation in both zero-shot and few-shot settings.