Noisy Self-Training with Synthetic Queries for Dense Retrieval (2023)

Abstract: Although existing neural retrieval models reveal promising results when training data is abundant and the performance keeps improving as training data increases, collecting high-quality annotated data is prohibitively costly. To this end, we introduce a novel noisy self-training framework combined with synthetic queries, showing that neural retrievers can be improved in a self-evolution manner with no reliance on any external models. Experimental results show that our method improves consistently over existing methods on both general-domain (e.g., MS-MARCO) and out-of-domain (i.e., BEIR) retrieval benchmarks. Extra analysis on low-resource settings reveals that our method is data efficient and outperforms competitive baselines, with as little as 30% of labelled training data. Further extending the framework for reranker training demonstrates that the proposed method is general and yields additional gains on tasks of diverse domains.

## introduction as an important task, information retrieval (ir) refers to the task of finding relevant texts from a large collection of passages or documents to satisfy the specific information needs of users. the information need is usually expressed as a short textual query, and the task is formulated as retrieving texts that are most relevant to the given query. recently, impressive achievements have been made in neural retrieval models through adopting large-scale pre-trained models @xcite . dual-encoders typically serve as the backbone architecture, which enables retrieving relevant knowledge from collections with millions or billions of passages in a fraction of time @xcite . unlike traditional term-matching-based lexical retrievers, such as tf-idf @xcite or bm25 @xcite , which can be applied without any training, neural retrieval methods normally require training on a sufficient number of human-labelled query-passage pairs to work well. nevertheless, due to the high cost of human annotations, the number of available query-passage pairs is way smaller compared to the size of passage collections (500k at most @xcite v.s. 21m passages @xcite ). moreover, the situation becomes increasingly worse in out-of-domain applications, where only a few or no training examples are available. directly applying neural retrieval models trained on high-resource datasets typically achieves low out-of-domain performance and lags behind their lexical counterparts @xcite . in this work, we aim to improve the performance of state-of-the-art neural retrieval models on both general domain and out-of-domain datasets, by using automatically synthesised queries. for this purpose, we use a query generator to automatically synthesise queries for each passage in the target dataset, which we then use for pre-training, based on a self-training objective @xcite . more specifically, instead of directly training the model on synthetic query-passage pairs @xcite , we use a neural retrieval model that was trained on the labelled dataset as the teacher to generate soft labels for each synthetic query, providing a supervision signal which is more robust to noise in the generated data. this ameliorates issues with the synthetic queries which are often generic or ambiguous, meaning that the query is only weakly related to its originating passage, and accordingly, other passages may match the synthetic query equally well or better. ## conclusion in this paper, we present a novel noisy self-training framework for neural retrieval models. it shows that when combined with automatically generated queries, neural retrievers can be improved in a selfevolution manner without relying on any external models. empirical results on both general-domain and out-of-domain benchmarks confirm the superiority of our proposed method, significantly outperforming a wide range of competitive existing models. we further adapt our method to show it can be applied to improve an expensive reranker.