It’s All Relative! – A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction (2024)

Abstract: Large language models (LLMs) have shown promising ability to generate synthetic query-document pairs by prompting with as few as 8 demonstrations. This has enabled building better IR models, especially for tasks with no training data. Typically, such synthetic query generation (QGen) approaches condition on an input context (e.g. a text document) and generate a query relevant to that context, or condition the QGen additionally on the relevance label (e.g. relevant vs irrelevant) to generate queries across relevance buckets. However, we find that such QGen approaches are sub-optimal as they require the model to reason about the desired label and the input from a handful of examples. In this work, we propose to reduce this burden of LLMs by generating queries simultaneously for different labels. We hypothesize that instead of asking the model to generate, say, an irrelevant query given an input context, asking the model to generate an irrelevant query relative to a relevant query is a much simpler task. Extensive experimentation across nine IR datasets shows that synthetic queries generated in such a fashion translates to better downstream performance.

## introduction recently, large language models (llms) (gpt-3:brown et al. ( 2020) 2023 )) have been applied to automatically generate task-specific data which can be used to train downstream models (section 6). for example, promptagator @xcite and inpars @xcite use as few as 8 task-specific demonstrations and prompt the llm to generate synthetic queries for new documents. these demonstrations are pairs of inputoutput examples where the input is a document text and the output is a query relevant to that context. however, this process only generates "relevant" synthetic queries. to create the corresponding negative or "irrelevant" pairs, a retriever is used to retrieve documents for each synthetic query, from which the hard negatives are constructed. finally, a task-specific downstream model is trained using this synthetic data. @xcite forgo the retriever step, and instead generate queries across different relevance buckets (e.g. relevant and irrelevant) by providing the relevance label along with the document as the context. although, they find conditioning query generation (qgen) on the different relevance labels does outperform approaches such as promptagator, the downstream model trained on the task-specific synthetic data is outperformed by the traditional transfer learning model, where a model is trained on a related dataset and directly applied to the target task. in this work, we propose a novel few-shot qgen approach where we prompt the model to generate queries across different relevance labels relative to each other. for example, given an input document and two relevance labels (relevant and irrelevant), the model is required to generate both a relevant query and an irrelevant query. by forcing the model to generate an irrelevant query after generating a relevant query, forces the model to condition on the previously generated query along with the document context (figure 1 ). we hypothesize this is a much more simpler task for the model in comparison to generating, say, an irrelevant query from only the document context or the label alone. this is inspired from the observation that both humans and llms find the task of comparing two things with respect to each other a simpler task, as opposed to judging things in isolation @xcite and such a binary comparison protocol is specially actively applied for llm alignment @xcite . to evaluate our hypothesis, we conduct zero-shot experiments with ir datasets from the beir @xcite benchmark. we generate synthetic prompt llm generate synthetic queries generate-relevant-only label-conditioned generate-pairwise filter synthetic queries retriever relevant examples irrelevant examples synthetic training data previous work proposed work passage: {passage} passage: {passage} label: {irrelevant /relevant} passage: {passage} query: {relevant query} query: {irrelevant query / relevant query} query-1:{relevant query} query-2:{irrelevant query} task: is the {query} relevant to {passage}? output: {yes / no} where we prompt the model to generate both the relevant and the irrelevant query at the same time, forcing the model to condition the irrelevant query generation on the previous generated query. we compare the proposed approach with previous qgen approaches, generate-relevant-only, where only relevant queries are generated followed by a retriever to retrieve irrelevant examples and, label-conditioned, where the query generation is conditioned on one relevance label, at a time, to generate a query. next, the generated queries are filtered automatically to ensure self-consistency. and the resulting synthetic data is used for downstream training. ## synthetic query generation we follow promptagator's few-shot query generation (qgen) setup, which comprises of three steps, namely, prompt-based query generation, query filtration, and, downstream model training. ## experimental setup our goal is to improve relevance prediction for tasks with no training data, for which we generate sufficient synthetic data and train a downstream model. we experiment with six english ir tasks from the beir benchmark @xcite , under zero-shot settings, where we generate querydocument examples for each task. we construct the prompt using a fixed set of examples from the ms-marco dataset @xcite , which covers a variety of domains, and generate new queries using the respective target task's document corpora. 1 we summarize all the datasets in subsection a.1. ## results and discussion we report our main results in qgen prompt source qgen model ndcg@10 ms-marco label-conditioned 40.05 generate-pairwise 47.69 fiqa label-conditioned 45.44 generate-pairwise 56.45 however, for the fever task, generate-pairwise is not the best performing model. this is probably because there is a mis-match between the irrelevant queries of the test split and the generated irrelevant queries. since fever is a factverification task, the irrelevant queries of the test split (derived from sentence claims) are entitycentric while the generated queries are more general or concept-focused (check do the trends hold if task-specific exemplars are used in the qgen prompt? yes! as mentioned above, we choose to use the same set of exemplars from ms-marco to generate queries for all tasks, while promptagator use 8 task-specific exemplars. therefore, for one of the tasks, fiqa-2018, we re-run two qgen models, namely label-conditioned and generate-pairwise, but this time constructing all prompts using 10 exemplars from the training corpus of fiqa-2018. from do the trends hold for smaller downstream models? yes! for many real-world applications inference matters more than fine-tuning, since infer-ence needs to run several times and often on large set of documents. 3 therefore, we aim to examine the qgen performance across smaller downstream models. in ## case study: qgen for fine-grained relevance prediction we follow chaudhary et al. ( 2023 ) and conduct a limited study where we use esci @xcite , a shopping relevance dataset, to generate training data for another shopping dataset wands @xcite . unlike the beir benchmark, where there were only two relevance labels, both esci and wands are multi-class and have nuanced relevance labels (details in subsection a.1). in this setting, we construct prompts for all the qgen models using examples from esci. for example, for the generate-relevant-only, we use the same setting as described in subsection 3.2, with the difference that instead of selecting exemplars from ms-marco, we select 10 queryproduct pairs from esci with the relevance label as exact, and as before, we use a retriever to retrieve the hard negatives. for all the other models, we generate queries across all relevance labels. for example, for the label-conditioned model, where we condition the query generation on the label, we select 10 exemplars, ensuring at least two examples for each of the four relevance labels are included. next, we generate queries for each wands product and each of the four relevance labels. for our proposed generate-pairwise model, earlier, we prompted the model to generate a relevant query followed by an irrelevant query. given that in this case there are four relevance labels, we adapt the task of pairwise generation to also take into account for which two labels the queries need to be generated. specifically, we prompt the model with the product and an instruction outlining for which two of the four labels the model should generate the queries ( ## related work llms have been extensively explored for synthetic text generation across different tasks, such as, structured prediction @xcite ), text classification @xcite @xcite @xcite , information retrieval @xcite , to name a few. llms for query generation as mentioned above, promptagator @xcite task-specific prompting for synthetic text generation the most common prompting strategy for text generation has been to use seed exemplars where the prompt is guided by a labeldescription (for example, self-instruct @xcite and attrprompt @xcite ) for text classification tasks). specifically, attrprompt find that simply conditioning on class often leads to lack of diversity in generated examples and propose to condition on other attributes such as length and style. more recently, @xcite propose targen to generate synthetic data for text classification tasks, without using seed exemplars. instead, targen aim to generate diverse examples by incorporating task-specific elements (e.g. linguistic information about the task) in the text generation process. they first generate these taskspecific elements (these are not actual exemplars but intermediate hints useful for the task) and similar to chaudhary et al. (2023) they also condition their example generation on the target label. the generated examples are run through a llm with a task-specific prompt to remove examples, similar to subsection 2.3. the above works suggest that adding task-specific attributes is a useful prompting strategy in addition to the final task labels, which aligns well with our finding as well, where using a ranking-based conditioning (e.g. generate-pairwise and iterative-pairwise) is helpful for ranking tasks. llms for relevance judgements complementary to our work, llms have also been directly used for relevance judgements or ranking documents @xcite @xcite via prompting. @xcite propose the pairwise ranking prompting (prp) that uses a query and a pair of documents as prompt to the llms for ranking. specifically, they find that instead of asking the model to rank all documents in a list, providing documents in pair simplifies the ranking problem. in addition to using binary relevance labels (relevant and irrelevant), some existing works @xcite also prompt llms with intermediate relevance labels (e.g. partially relevant) to improvie fine-grained relevance prediction. ## next steps in this work, we have shown that by aligning the query generation task with the downstream ranking task, we are able to generate synthetic data useful to the downstream task. in all the tasks, the query generation is conditioned on the document or a product context. however, there are tasks within beir such as the duplicate-question retrieval task where the task is retrieving a duplicate or similar question. given there is no conditioning context, it will be interesting to check how to adapt our best performing pairwise generation for such a task. as we saw in ## limitations in this paper, we evaluate the generated query quality by running the downstream model for each qgen approach, which amounts to running a large set of experiments across different tasks resulting in computation and time cost. we believe that having evaluation techniques which could simulate the downstream performance without requiring to run the downstream models every time, will save a lot of compute and time for researchers exploring the synthetic text generation space. ## statement of ethics the use of pretrained llms for language generation always carries the risk of bias propagation, and since in this case, we are training a downstream model on the generated synthetic data, there is a chance this risk propagates further down the pipeline.