Exploring efficient zero-shot synthetic dataset generation for Information Retrieval (2024)

Abstract: The broad integration of neural retrieval models into Information Retrieval (IR) systems is significantly impeded by the high cost and laborious process associated with the manual labelling of training data. Similarly, synthetic training data generation, a potential workaround, often requires expensive computational resources due to the reliance on large language models. This work explored the potential of small language models for efficiently creating high-quality synthetic datasets to train neural retrieval models. We aim to identify an optimal method to generate synthetic datasets, enabling training neural reranking models in document collections where annotated data is unavailable. We introduce a novel methodology, grounded in the principles of information theory, to select the most appropriate documents to be used as context for question generation. Then, we employ a small language model for zero-shot conditional question generation, supplemented by a filtering mechanism to ensure the quality of generated questions. Extensive evaluation on five datasets unveils the potential of our approach, outperforming unsupervised retrieval methods such as BM25 and pretrained monoT5. Our findings indicate that an efficiently generated “silver-standard” dataset allows effective training of neural rerankers in unlabeled scenarios. To ensure reproducibility and facilitate wider application, we will release a code repository featuring an accessible API for zero-shot synthetic question generation.

## introduction deep learning is at the heart of many current breakthroughs in ai in a wide range of fields. typically, such progress is attributed to better computational capabilities, superior algorithms, and a larger corpus of high-quality training data. particularly in the information retrieval (ir) field, significant gains against traditional baselines are obtained when a large amount of labelled data is available @xcite @xcite . however, manual data labelling is expensive and labor-intensive, highlighting the urgency to devise methods that can automatically produce higher quality training data to unlock the potential of neural retrieval models for unlabelled data collections. recent strides in large language models offer a new avenue of generating synthetic training data to train neural retrieval models @xcite . present strategies largely fall into two categories, finetune-based and prompt-based. the former necessitates annotated data to train a language model to craft questions given a document text and, optionally, a correct answer. in contrast, the prompt-based method capitalizes on expensive language models to generate a question in a zero-shot fashion, using a document as context. although both techniques are effective, they still have some drawbacks. the finetune-based approach is a supervised method, thus requiring the acquisition of labelled data. moreover, even though publicly available models can be adopted, these inevitably bear inherent biases from their training dataset, which can be a limiting factor in adapting to the target domain. on the other hand, the prompt-based approach, often linked to large models, comes with steeper costs, be it for model execution or through paid apis. this particularly restricts its applicability in low-resource environments. another overlooked problem that is rooted in both approaches is that in ir the target document collection for which synthetic questions are being generated usually contains millions of documents. it is therefore common to randomly select some documents as seeds to generate the synthetic dataset. however, some documents can be bad examples, leading the generator to produce unuseful or invalid questions, wasting computation resources. in this work, we explore the limits of promptbased small language models in generating highquality synthetic training data. specifically, we hypothesize that these models can efficiently and quickly create a synthetic dataset, which can then empower neural retrieval models to outperform traditional unsupervised techniques such as bm25. our approach starts with an innovative filtering technique rooted in information theory measures to identify and exclude non-representative documents. we then investigate various small language models and generation strategies across diverse document collections, gauging their capacity for producing relevant questions. to further improve the quality of the generated dataset, we also explore filtering techniques to remove less suitable questions. lastly, we assess the performance of simple neural retrieval models trained with the best synthetic datasets. our contributions can be summarized as follows: (1) an innovative method grounded in information theory principles for discovering outliers within a document collection; (2) the development and validation of techniques to estimate the quality of synthetic generated questions; (3) an extensive benchmark of the quality of synthetic datasets for document retrieval, derived from several small language models and generation strategies, totalling 150 unique configurations; (4) publicly available off-the-shelf software tool for creating synthetic datasets for a given document collection available at https://github.com/ieeta-pt/synqgen . ## related work the field of synthetic data generation has seen significant advances with the advent of deep learning, mostly thanks to the transformer-based large language models capability of generating coherent text @xcite . following the same trend, generating synthetic training data for information retrieval became a viable option to replace the labour-intensive data annotation process @xcite . on the one hand, we have the finetune-based approaches initially popularized by @xcite as the doc2query technique, where the main idea was to train a sequence-to-sequence model to generate a question given a document as input. however, its purpose was not to build a synthetic dataset, but rather to add the generated questions to the document to aid lexical models. then, @xcite improved the initial approach by adopting t5 as the generator model. more recently, @xcite showed that sequence-to-sequence models are prone to "hallucination", suggesting the incorporation of pretrained relevance models to weed out inaccurate questions. meanwhile, ma et al. (2021); thakur et al. (2021); wang et al. (2022 ) adopted a similar methodology, but with the primary objective to construct a synthetic dataset for training neural retrieval models in unlabelled document collections. opposed to the previous trend, zero-shot question generation, also known as prompt-based, has recently emerged as a promising alternative that involves generating questions without training a generation model specifically for that task. large language models (llms) are typically used in zeroshot question generation, given their capability of generating coherent text and being easily conditioned to produce the desired output without needing extra training. for instance, @xcite and @xcite obtained promising results in the creation of zero-shot synthetic datasets for information retrieval by using llms, namely @xcite . nevertheless, the deployment of llms on a larger scale remains challenging due to their extensive computational resource requirements. our work resonates most with the approach presented by @xcite , given the shared focus on zero-shot question generation utilizing language models for ir. notwithstanding, in this work, we focused on only exploring small language models (from 70m to 1.3b parameters) while entirely concentrating on the problem of effectively and efficiently producing a synthetic dataset for information retrieval. as such, contrary to previous works, herein we explore the limits of zero-shot question generation with small language models by evaluating the impact of different language models and generation strategies, as well as a mechanism for document outlier detection. ## methods this section details all the individual components that we explored in order to generate a synthetic dataset for document retrieval, followed by the evaluation methodology. ## experiments and results this section outlines the performed experiments and their outcomes. we first introduce the document collections used for the benchmarks. following this, we present experiments that validate our assumptions: the use of information theory for outlier document elimination and the employment of retrieval models for question quality assessment. lastly, we disclose the results of the benchmarks themselves. ## ablation studies in this section, we present an ablation study designed to understand the impact of each proposed method on the overall pipeline. ## conclusion and future work this work demonstrated that smaller language models can efficiently generate high-quality synthetic datasets for neural retrieval model training. our approach shows that utilizing information theory principles for document selection and a small language model for zero-shot question generation can outper-form methods like bm25 and pretrained monot5 in certain scenarios. future work could focus on refining the downstream benchmark by also levering dense retrieval models and adopting stronger reranker models. our findings bring us closer to broader neural retrieval model integration, mitigating data labelling and computational resource challenges.