Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation (2022)

Abstract: Dense retrieval models, which aim at retrieving the most relevant document for an input query on a dense representation space, have gained considerable attention for their remarkable success. Yet, dense models require a vast amount of labeled training data for notable performance, whereas it is often challenging to acquire query-document pairs annotated by humans. To tackle this problem, we propose a simple but effective Document Augmentation for dense Retrieval (DAR) framework, which augments the representations of documents with their interpolation and perturbation. We validate the performance of DAR on retrieval tasks with two benchmark datasets, showing that the proposed DAR significantly outperforms relevant baselines on the dense retrieval of both the labeled and unlabeled documents.

## introduction retrieval systems aim at retrieving the documents most relevant to the input queries, and have received substantial spotlight since they work as core elements in diverse applications, especially for open-domain question answering (qa) @xcite . open-domain qa is a task of answering the question from a massive amount of documents, often requiring two components, a retriever and a reader @xcite . specifically, a retriever ranks the most questionrelated documents, and a reader answers the question using the retrieved documents. traditional sparse retrieval approaches such as bm25 @xcite and tf-idf rely on term-based matching, hence suffering from the vocabulary mismatch problem: the failure of retrieving relevant documents due to the lexical difference from queries. to tackle such a problem, recent research focuses on dense retrieval models to generate learnable dense representations for queries and documents @xcite . despite their recent successes, some challenges still remain in the dense retrieval scheme for a couple of reasons. first, dense retrieval models need a large amount of labeled training data for a decent performance. however, as figure 1 shows, the proportion of labeled query-document pairs is extremely small since it is almost impossible to rely on humans for the annotations of a large document corpus. second, in order to adapt a retrieval model to the real world, where new documents constantly emerge, handling unlabeled documents that are not seen during training should obviously be considered, but remains challenging. to automatically expand the query-document pairs, recent work generates queries from generative models @xcite or incorporates queries from other datasets @xcite , and then generates extra pairs of augmented queries and documents. however, these query augmentation schemes have serious and obvious drawbacks. first, it is infeasible to augment queries for every document in the dataset (see the number of unlabeled documents in figure 1 ), since generating and pairing queries are quite costly. second, even after obtaining new pairs, we need extra training steps to reflect the generated pairs on the retrieval model. third, this query augmentation method does not add variations to the documents but only to the queries, thus it may be suboptimal to handle enormous unlabeled documents. since augmenting additional queries is costly, the question is then if it is feasible to only manipulate the given query-document pairing to handle numerous unlabeled documents. to answer this, we first visualize the embeddings of labeled and unlabeled documents. figure 1 shows that there is no distinct distributional shift between labeled and unlabeled documents. thus it could be effective to manipulate only the labeled documents to handle the nearby unlabeled documents as well as the labeled documents. using this observation, we propose a novel document augmentation method for a dense retriever, which not only interpolates two different document representations associated with the labeled query (figure 2 (b) ), but also stochastically perturbs the representations of labeled documents with a dropout mask (figure 2 (c) ). one notable advantage of our scheme is that, since it manipulates only the representations of documents, our model does not require explicit annotation steps of query-document pairs, which makes it highly efficient. we refer to our overall method as document augmentation for dense retrieval (dar). we experimentally validate our method on standard open-domain qa datasets, namely natural question (nq) @xcite and triv-iaqa @xcite (tqa), against various evaluation metrics for retrieval models. the experimental results show that our method significantly improves the retrieval performances on both the unlabeled and labeled documents. furthermore, a detailed analysis of the proposed model shows that interpolation and stochastic perturbation positively contribute to the overall performance. our contributions in this work are threefold: • we propose to augment documents for dense retrieval models to tackle the problem of insufficient labels of query-document pairs. • we present two novel document augmentation schemes for dense retrievers: interpolation and perturbation of document representations. dense retriever dense retrieval models @xcite have gained much attention, which generate dense representations for queries and documents. however, dense retrieval faces a critical challenge from limited training data. recent work has addressed such a problem by generating extra query-document pairs to augment those pairs to the original dense retrieval model @xcite @xcite , or by regularizing the model (rosset et al., 2019). however, unlike ours that automatically augments data during a training phase, these methods require extensive computational resources for an additional generation step of explicitly querydocument pairing before training the retriever. data augmentation since data augmentation is crucial to the performance of deep neural networks, it is widely applied to diverse domains @xcite , where interpolation and perturbation are dominant methods. mixup interpolates two items, such as pixels of images, to augment the training data @xcite , which is also adopted for nlp @xcite . however, none of the previous work has shown the effectiveness of mixup when applied to retrieval tasks. besides interpolation, @xcite and @xcite proposed perturbation over words, and @xcite proposed perturbation over word embeddings. @xcite and @xcite perturbed text embeddings to generate diverse sentences and to augment positive sentence pairs in unsupervised learning. in contrast, we address dense retrieval, perturbing document representations with dropout @xcite in a supervised setting with labeled documents. ## method we begin with the definition of dense retrieval. dense retrieval given a pair of query q and document d, the goal of dense retrieval is to correctly calculate a similarity score between them from the dense representations q and d, as follows: a dense retrieval scheme generally uses the negative sampling strategy to distinguish the relevant query-document pairs from irrelevant pairs, which generates an effective representation space for queries and documents. we specify a relevant query-document pair as (q, d + ) ∈ τ + , and an irrelevant pair as (q, d -) ∈ τ -, where the objective function is as follows: interpolation with mixup as shown in interpolation of figure 2 , we aim at augmenting the document representation located between two labeled documents to obtain more query-document pairs, which could be useful to handle unlabeled documents in the middle of two labeled documents. to achieve this goal, we propose to interpolate the positive and negative documents (d + , d -) for the given query q, adopting mixup @xcite . note that, since the input documents to the encoder e d are discrete, we use the output embeddings of documents to interpolate them, as follows: ## conclusion we presented a novel method of augmenting document representations focusing on dense retrievers, which require an extensive amount of labeled query-document pairs for training. specifically, we augment documents by interpolating and perturbing their embeddings with mixup and dropout masks. the experimental results and analyses on multiple benchmark datasets demonstrate that dar greatly improves retrieval performances.