Expand, Highlight, Generate: RL -driven Document Generation for Passage Reranking (2023)

Abstract: Generating synthetic training data based on large language models (LLMs) for ranking models has gained attention recently. Prior studies use LLMs to build pseudo query-document pairs by generating synthetic queries from documents in a corpus. In this paper, we propose a new perspective of data augmentation: generating synthetic documents from queries. To achieve this, we propose DocGen, that consists of a three-step pipeline that utilizes the few-shot capabilities of LLMs. DocGen pipeline performs synthetic document generation by (i) expanding, (ii) highlighting the original query, and then (iii) generating a synthetic document that is likely to be relevant to the query. To further improve the relevance between generated synthetic documents and their corresponding queries, we propose DocGen-RL, which regards the estimated relevance of the document as a reward and leverages reinforcement learning (RL) to optimize DocGen pipeline. Extensive experiments demonstrate that DocGen pipeline and DocGen-RL significantly outperform existing state-of-theart data augmentation methods, such as InPars, indicating that our new perspective of generating documents leverages the capacity of LLMs in generating synthetic data more effectively. We release the code, generated data, and model checkpoints to foster research in this area.

## synthetic document generation in this section, we present the details of our proposed method, docgen, and its reinforcement learning extension, docgen-rl. task definition. given a query @xmath0, where q is a set of queries, the goal of synthetic document generation is to generate a document d, such that d is relevant to q, and use @xmath1, q} collection to train a retriever or reranking model. ## experimental setup retrieval methods. we use a two-stage retrieval architecture (nogueira and cho, 2019) consisting of initial retrieval with bm25, followed by a neural reranker. the collection is indexed using pyserini. 5 we retrieve 1000 candidate documents for each query using bm25. subsequently, we rerank the candidate documents using monot5, which is an adaptation of the t5 model @xcite ) for text ranking proposed by @xcite . we fine-tune monot5-base (220m parameters) with a constant learning rate of 10 -3 and an equal number of positive and negative examples in each batch of size 64. we did not conduct experiments with the 3b and 11b versions due to their computational cost. baselines for data augmentation. we select the state-of-the-art data augmentation method using llms for the passage reranking task, inpars, as our main baseline. in addition to the inpars, we replicate genread @xcite that recently has set a new state-of-the-art for the open-domain question answering task, in which the focus is not on data augmentation for training rerankers, but the goal is to generate a relevant document for the given query and then generate the final answer from the generated document. we consider the few-shot variation of genread as another baseline for generating documents given the query. we use the same prompts released by the author. 6 moreover, we replicate query2doc (q2d) @xcite that generate documents given a query in order to expand the query and improve retrieval effectiveness using the expanded query. we adapt q2d for data augmentation by using their prompt and generating documents. we use the generated documents for building the training data. to ensure a fair comparison, we do not compare our results with inpars-v2 @xcite and promptagator @xcite results. inpars- retriever data augmentor nq ms marco dl'20 hotpotqa fever ndcg mrr map ndcg ndcg ndcg first-stage retriever bm25 (replicated) .329 .187 .286 .480 .633 .651 rerankers monot5 inpars (bonifacio et al., 2022) .335 .259 .360 .576 --monot5 inpars (replicated) .337 .223 .357 .569 .627 .653 monot5 genread (replicated) .368 .230 .354 .570 .629 .668 monot5 q2d (replicated) .309 .158 .252 .437 .610 .634 rerankers w/ docgen monot5 docgen (ours) .467 † .275 † .398 † .580 † .647 † .693 † monot5 docgen-rl (ours) .517 * .332 * .421 * .618 * .663 * .720 * human annotations monot5 -.567 .381 .491 .714 .695 .802 v2 uses the reranker that is trained on ms marco to filter the generated synthetic data; after the synthetic data filtering and generation, inpars-v2 first warms up a passage reranker using the whole training set of ms marco and then continue training the reranker on the generated synthetic data. however, docgen has not used the supervision from ms marco for either synthetic data filtering or passage reranker training. promptagator performs first-stage retrieval (full ranking) with customized prompts per dataset, which is different from our approach where we focus on reranking with a single prompt for all the datasets. to have a fair comparison between data augmentation approaches in our replication, we use the same llm (bloom-560m) and same filtering approach for all of the data augmentation methodologies. we then generate an equal amount of synthetic data which is 100, 000 synthetically generated queries or documents followed by @xcite . dataset and metrics. we conduct our experiments on the ms marco-passage collection @xcite and the trec deep learning track (trec-dl'20) @xcite , as well as the beir versions of natural questions (nq) @xcite , hot-potqa @xcite , and fever @xcite datasets. we report the official metrics for each dataset, which are ndcg@10 for beir's datasets (nq, hotpotqa, and fever), mrr@10 for ms marco, and ndcg@10 and map@1000 for trec-dl'20. the ms marcopassage dataset contains about 1 million natural language queries (average length: 7.5 words) and has been extensively used to train neural retrievers for ranking because of the large number of queries. following prior work on ms marco @xcite @xcite @xcite , we use the dev set (∼ 7k queries) for our empirical evaluation. trec-dl'20 consists of 54 queries, and the retrieval for these queries is based on the passage corpus of ms marco. we follow the common practice on trec-dl @xcite and use ndcg@10 and m ap @1000 for evaluation. llm selection. in our preliminary experiments, we examine four llms, namely, llama-7b @xcite , alpaca-lora @xcite , flan-t5-xxl (chung et al., 2022), bloom-560m @xcite . we observe that the smallest and most efficient model, bloom-560m, consistently generates higher-quality documents compared to the other models for our document generation task. therefore, we select bloom-560m as the main llm for our experiments. we found that alpaca-lora and flan-t5 tend to generate less informative and shorter text. llama tends to produce noisy text, e.g., latex code. training policy llm with rl. we train the policy network for 30k episodes, 5 epochs per batch with a batch size of 2 and a learning rate of 2 × 10 -6 . we set the other hyperparameters to the optimal parameters that are suggested in @xcite . if the agent (i.e., llm) chooses an action (i.e., token), which is not a token in the given query that is going to be highlighted, we assign a penalty reward of -0.25, drawing inspiration from @xcite that gives the same penalty for finding hint keywords from a text to do summarization. ## results in this section, we address the following research questions, assessing the effectiveness of our proposed methods, docgen and docgen-rl, from different perspectives: monot5 improve the proposed method? main results (rq1 and rq2). ## discussion gap between synthetic and realistic data. we investigate to what extent different data augmentation methods are close to the human data in terms of the average length of queries and documents, as well as the average number of word overlaps within the query and document. ## conclusions this paper presents a new perspective for constructing training datasets in information retrieval without human labels. it proposes two novel query-todocument generators, called docgen and docgen-rl. the motivation behind this approach is to overcome the limitations of existing methods that generate queries from documents. additionally, we automatically enhance low-quality queries be-7 urlhttps ://huggingface.co/castorini/monot5-basemsmarco fore generating documents. extensive experiments demonstrate that both docgen and docgen-rl significantly outperform existing approaches, including inpars. this work showcases the potential of leveraging llms for document generation and highlights the significance of document augmentation in information retrieval. we emphasize the importance of improving the quality of training datasets. the findings of this study contribute to advancing the state-of-the-art in ir and provide valuable insights for future research in this domain.