Disentangling Questions from Query Generation for Task-Adaptive Retrieval (2024)

Abstract: This paper studies the problem of information retrieval, to adapt to unseen tasks. Existing work generates synthetic queries from domain-specific documents to jointly train the retriever. However, the conventional query generator assumes the query as a question, thus failing to accommodate general search intents. A more lenient approach incorporates task-adaptive elements, such as few-shot learning with an 137B LLM. In this paper, we challenge a trend equating query and question, and instead conceptualize query generation task as a “compilation” of high-level intent into task-adaptive query. Specifically, we propose EGG, a query generator that better adapts to wide search intents expressed in the BeIR benchmark. Our method outperforms baselines and existing models on four tasks with underexplored intents, while utilizing a query generator 47 times smaller than the previous state-of-the-art. Our findings reveal that instructing the LM with explicit search intent is a key aspect of modeling an effective query generator.

## introduction information retrieval has significantly facilitated the process of locating relevant documents in response to user requests. with the advent of dense retrieval @xcite , prior works have concentrated on the supervised alignment of latent spaces within query and passage encoders @xcite @xcite . however, this requires collecting labeled data across numerous domains, which are often unavailable. in such scenarios, where only given the target corpus, existing work focuses on zeroshot query generation to form a synthetic dataset @xcite . representing approaches include genq @xcite , train-figure 1 : overview of our method. given a document, conventional zero-shot query generator generates questions, while few-shot query generator performs incontext learning with few-shot examples. in contrast, our method reflects diverse search intents utilizing metaprompts to enhance the generalizability of the query generator. fact checking can be viewed as transaction, where the retriever determines whether the given claim is supported or not, and argument retrieval is similarly so. citation prediction, presenting a title as the query, represents a navigational intent for a specific document, and entity retrieval exhibits a mixture of the intents. we challenge the prevailing trend of equating queries with questions in training. though question form may align with information intent, web search intents are categorized to include more diverse intents @xcite such as informational, navigational, and transactional. while concentrating on the first category has paid off, given the prevalence of datasets aligned with informational intent, generalization is critical, by four non-qa datasets in beir @xcite tasks in figure 1 . we thus interpret query generation task as a "compilation" of high-level intent into task-adaptive query, to reflect diverse search intents @xcite @xcite . this task has been mostly delegated to in-context learning: promptagator-few leverages flan 137b @xcite as a few-shot query generator to better capture the latent intent in query-document pairs. however, challenges may arise in in-context learning @xcite due to small model size @xcite , limited context length @xcite , or poor quality of examples @xcite . to overcome the reliance on expensive fewshot examples and enable the use of small lms as query generators, we propose egg (efficient generalized generator), which leverages metaprompt foot_1 to incorporate unique search intents. our system comes in two model sizes: egg-flan is for scenarios where the model is too small to support in-context learning, for which, we adapt instructions for diverse search intents. however, we find the lm generated queries lack sufficient diversity, thus we ensure the instruction to diversify those. in cases where the model size is just enough to support in-context learning (egg-llama), we first generate prototype queries with meta-prompt as candidates for in-context query-document pairs. another key distinction is, we utilize retriever feedback, specifically, relevance ranking. egg not only surpasses both zero-shot and few-shot baselines but also outperforms previous state-of-the-art methods. our method effectively covers undersupported intents which existing methods often struggle with. to summarize, our contributions are as follows: (1) we introduce egg, which employs meta-prompt and retriever feedback, to integrate unique search intents into query generation. (2) our method outperforms the baselines, demonstrating its effectiveness. (3) despite the much smaller size of the query generator, our method achieves the highest overall performance. task intent eq fact checking transactional claim argument retrieval transactional argument citation prediction navigational title entity retrieval mixed entity in conventional zero-shot methods, e q is often considered as either the term 'query' @xcite @xcite , or 'question' @xcite , interchangeably. we challenge this assumption by defining e q as query adaptive to each task, by interpreting e q as task-specific attribute incorporated it into the meta-prompt. ## related work in scenarios where only the target corpus is available @xcite @xcite , existing works create synthetic labels by generating queries from documents. a common approach is to train a query generator on large qa datasets @xcite . recent work prompts llms to generate synthetic queries from documents. there has been active research on training neural rerankers with synthetic queries @xcite @xcite . in-pars @xcite performs few-shot learning, while inpars-light @xcite extends inpars by employing cost efficient recipes. udapdr @xcite generates a small number of queries using gpt-3 and iteratively generates a large number of queries with cheaper model to train rerankers. @xcite leverages small lms to generate questions starting with common initiators, such as what, how, and when. meanwhile, training task-adaptive retriever is underexplored, only being studied by promptagator @xcite , which leverages few-shot examples to capture the latent intents, albeit at much higher computational costs. another line of study provides specific instructions along with the queries and builds a generalpurpose retriever @xcite @xcite . this involves large sizes of llms during inference, while our method establishes small, task-specific retrievers. ## conclusion in this work, we present egg, a novel approach designed to overcome the shortcomings of previous query generation methods. we propose two designs for egg, distinguished by their model sizes, to improve the diversity and quality of synthetic queries while effectively capturing the search intent of the task using meta-prompts. our approach demonstrates superior performance across four tasks, suggesting a promising direction for query generation involving underexplored search intents.