CAPSTONE : Curriculum Sampling for Dense Retrieval with Document Expansion (2023)

Abstract: The dual-encoder has become the de facto architecture for dense retrieval. Typically, it computes the latent representations of the query and document independently, thus failing to fully capture the interactions between the query and document. To alleviate this, recent research has focused on obtaining query-informed document representations. During training, it expands the document with a real query, but during inference, it replaces the real query with a generated one. This inconsistency between training and inference causes the dense retrieval model to prioritize query information while disregarding the document when computing the document representation. Consequently, it performs even worse than the vanilla dense retrieval model because its performance heavily relies on the relevance between the generated queries and the real query. In this paper, we propose a curriculum sampling strategy that utilizes pseudo queries during training and progressively enhances the relevance between the generated query and the real query. By doing so, the retrieval model learns to extend its attention from the document alone to both the document and query, resulting in high-quality query-informed document representations. Experimental results on both in-domain and out-of-domain datasets demonstrate that our approach outperforms previous dense retrieval models.

## introduction text retrieval aims to find the relevant documents for a given query from a large collection of documents, playing an indispensable role in opendomain question answering @xcite , fact verification @xcite and retrievalaugmented generation @xcite . at the early stage, sparse retrieval methods such as tf-idf or bm25 dominated passage retrieval by relying mainly on lexical term matching * work done during internship at microsoft research asia. †corresponding authors. to compute relevance between the query and document. recently, there has been a surge of research interest in neural network-based dense retrieval @xcite . different from sparse retrieval, dense retrieval resorts to neural encoders to compute the dense representations of the query and document. this enables dense retrieval to infer the relevance between them at the semantic level rather than the surface level, thus circumventing the term mismatch problem suffered by the sparse retrieval models. in recent years, the dual-encoder architecture has been a standard workhorse for dense retrieval. one major disadvantage of this architecture is that it can only partially extract the interactions between the query and document, since it encodes them separately. by comparison, the cross-encoder architecture can effectively capture the deep correlation between them by taking the concatenation of the query and document as input. by directly concatenating the query and document, the cross-encoder gains an advantage in capturing interactions, but also loses the advantage of pre-computing document representations during inference. therefore, cross-encoder cannot wholly replace dual-encoder. to enhance the retrieval models' ability to capture interactions between queries and documents while maintaining retrieval efficiency, previous work mainly focuses on generating query-informed document representations. one approach, known as late interaction @xcite , involves encoding the query and document independently in the early layers, while the later layers model their interactions. late interaction combines dual-encoder and cross-encoder, making a trade-off between retrieval efficiency and performance. on the other hand, @xcite proposed a promising retrieval architecture, dual-cross-encoder. as shown in figure 1 (c), this architecture computes the query-related document representation by expanding the document with a real or pseudo query. compared with late interaction, dual-crossencoder (i.e., dense retrieval with document expansion) gets the query-related document representation without sacrificing the retrieval efficiency at inference. however, there exists a discrepancy between training and inference in the current dualcross-encoder retriever. specifically, during training, the document is expanded using a real query, whereas during inference, the document is enriched with a generated query. this discrepancy causes the learned retriever overly focus on the query, yet neglect the document, when computing the document representation. during inference, if the generated query q′ significantly differs from the user-input query q, the query-related document representation will be misled by q′, thus degrading the performance. that is why the dual-cross-encoder even underperforms the vanilla dual-encoder. to address this issue, @xcite proposed a solution by computing multiview document representations using different generated queries for each document. while multiview document representations improve retrieval performance, they come at the cost of significantly increased retrieval latency, which scales linearly with the number of views. in this paper, we propose capstone, a curriculum sampling for dense retrieval with document expansion, to bridge the gap between training and inference for dual-cross-encoder. our motivation is to expect the dual-cross-encoder retrieval model can utilize both the document d and pseudo query q′ to compute the query-informed document representation. to achieve this, we train the dual-cross-encoder retriever by gradually increasing the relevance of the pseudo query q′ to the gold query q. specifically, at the early training stage, a pseudo query q′ irrelevant to q is selected, causing the retriever to solely rely on the document. as we progress to the late training stage, a highly related pseudo query q′ is chosen, allowing the retriever to learn to augment the document representation with the pseudo query. by doing so, we alleviate the discrepancy between training and inference. during inference, if the user-input query q is similar to the pseudo query q′, then q′ will contribute more to making the target document d be retrieved. otherwise, the retrieval model will mainly rely on the relevance between q and d. to summarize, the main contributions of this paper are as follows: (1) we propose a curriculum learning approach to bridge the gap between train-ing and inference for dense retrieval with document expansion, further improving the query-informed document representation foot_0 . (2) we propose to compute the typical document representation rather than using multiview document representations at inference, which balances the retrieval efficiency and performance. (3) to verify the effectiveness, we apply our proposed approach to two different dense retrieval models, dpr @xcite and cocondenser, and conduct extensive experiments on three in-domain retrieval datasets, and the zero-shot beir benchmark. experimental results show our proposed approach brings substantial gains over competitive baselines. (4) to the best of our knowledge, this is the first time that document expansion is successfully applied to dense retrieval without incurring extra retrieval latency. ## preliminary in this section, we introduce the definition of text retrieval and three architectures for dense retrieval. task description. text retrieval is meant to find the most relevant m documents d + = {d + 1 , d + 2 , . . . , d + m } for the given query q from a large corpus @xmath0, d 2 , . . . , d n } with n documents (m ≪ n ). ## methodology in section 3.1, we first identify the discrepancy between training and inference of dce. then, we introduce how to bridge the gap with curriculum learning in section 3.2. finally, we will show our proposed inference method in section 3.3. ## conclusion this work proposes the curriculum sampling for dense retrieval with document expansion, which enables dense retrieval models to learn much better query-related document representations. in addition, we propose to compute the typical representation of different document views, which balances inference efficiency and effectiveness. our experimental results on the in-and out-of-domain datasets verify the effectiveness of the curriculum sampling and typical representation.