Relevance-aware Diverse Query Generation for Out-of-domain Text Ranking (2024)

Abstract: Domain adaptation presents significant challenges for out-of-domain text ranking, especially when supervised data is limited. In this paper, we present ReadQG (Relevance-Aware Diverse Query Generation), a method to generate informative synthetic queries to facilitate the adaptation process of text ranking models. Unlike previous approaches focusing solely on relevant query generation, our ReadQG generates diverse queries with continuous relevance scores. Specifically, we propose leveraging soft-prompt tuning and diverse generation objectives to control query generation according to the given relevance. Our experiments show that integrating negative queries into the learning process enhances the effectiveness of text ranking models in out-of-domain information retrieval (IR) benchmarks. Furthermore, we measure the quality of query generation, highlighting the underlying beneficial characteristics of negative queries. Our empirical results and analysis also shed light on potential directions for more advanced data augmentation in IR. The data and code have been released.

## introduction many domain-specific tasks lack supervised data, posing challenges for many neuarl approaches. recently, @xcite introduce an out-ofdomain (ood) information retrieval (ir) benchmark across diverse scenarios and domains. their findings indicate that many neural text ranking models demonstrate limited effectiveness in such contexts. these tasks primarily struggle with adaptation @xcite , highlighting the issues of insufficient task-and domain-specific labeled data. to address this, one line of research propose utilizing synthetic training data for adapting text ranking models @xcite @xcite . these approachees employ generative models to first learn document-toquery mapping from rich-resource datasets such as msmarco @xcite . subsequently, a document and its generated query can be treated as a relevant pair for fine-tuning text ranking models. recently, these methods have been further refined with instruction-tuned large langugage models (llms) @xcite . such llm-driven query generation can produce more informative query through incontext @xcite or few-shot learning @xcite . compared to these works, in this study, we introduce relevance-aware diverse query generation (readqg), aiming to generate more informaive synthetic queries with lightweight generative models. specifically, we generate both positive and (hard) negative queries from the same document, as illustrated in figure 1 . our hypothesis is that a set of diverse relevance-aware queries can enhance relevancy representation of texts in unseen domains. to achieve this, we develop the instruction prompt and relevance prompt embeddings. the instruction prompt directs llms to generate query, while rel-evance prompt captures and conrols the relevance dynamic between docuemnt and multiple queries. moreover, we develop two strategies to diversify our generated queries: self-contrastive loss and sequence calibration loss @xcite . these strategies prevent readqg from degeneration (i.e., falling back to naive relevant query generation). finally, to exploit positive and negative queries genereted by readqg, we integrate the querybased objectives into the training process of text ranking models. our experiments demonstrates that models fine-tuned on our synthetic data outperform the original model in terms of passage re-ranking efftiveness on the beir benchmark. in addition, we define and propose two metrics to measure the quality of generated queries. we observe that the query exhibiting both diversity and relevancy provide useful signals for passage re-ranking models to learn, emphasizing the importance of hard negative query. to sum up, we propose a domain adaptation pipeline with readqg, tailored for out-of-domain text ranking. our empirical results show that hard negative queries could provide useful signals. further, the domain adaptation pipeline is built with lightweight generators and text ranking models, achieving improved effectiveness but more efficient in terms of inference time and computational costs. more details can be found in our results (section 6.1) and our analysis (section 6.2). ## methodologies in this work, we utilize synthesized out-of-domain training queries to tackle the domain-mismatch issues. we will first provide an overview of our domain adaptation pipeline in section 3.1; it is also illustrated in figure 2 . following this are the details of the two main stages in ths pipeline, including outof-domain data augmentation and domain-adaptive fine-tuning. ## semi-supervised msmarco document-centric pairs to fine-tune readqg, we collect training pairs for query generation using the msmarco passage ranking dataset @xcite . we utilize this dataset, along with the pseudo-relevance of bm25 hard negatives foot_4 , which are predicted by the off-the-shelf ranking model, minilm. 8 this cross-encoder is fine-tuned on ms marco triplets. the procedure is illustrated in figure 4 . ## experimental setups we will first report the setup of the two stages in the proposed pipeline: data augmnetation using readqg (section 3.2), and domain adaptive passage re-ranking (section 3.3) fine-tuned on the synthetic training data. ## empirical results in this section, we first validate the text ranking effectiveness using the synthetic data constructed by readqg. then, we explore the query generation effectiveness via the aforementioned three performance metrics. ## conclusion in this study, we present relevance-aware diverse queries generation and validate several setups for constructing more informative queries. the generation of negative query can benefit from appropriate soft-prompt tuning and diverse generation constraints, resulting in a more effective learning process of text ranking models. thus, we consider the negative query generation as a potential research direction. there are several other avenues for future work, including (1) scaling up readqg or prompting larger llms for negative queries; (2) mining hard negative documents with hard negative query;