SQUARE : Unsupervised Retrieval Adaptation via Synthetic Data (2025)

Abstract: Pre-trained retrieval models often face challenges in zero-shot retrieval for knowledge-based question answering, as different tasks rely on different corpora. We introduce SQUARE (Synthetic QUery-based Adaptive REtrieval), a novel method for corpus-specific unsupervised retrieval customization. SQUARE leverages LLMs to generate grounded synthetic question-answer pairs from the corpus, which are then used to fine-tune the retriever. A filtering mechanism based on the synthetic answers is employed to ensure high quality of tuning data. Extensive experiments on various datasets demonstrate superior performance of SQUARE compared to zero-shot retrieval and other customization methods, highlighting the value of corpus adaptation for effective retrieval.

## introduction large language models (llms) frequently use retrieval-augmented generation to improve their question-answering capabilities @xcite @xcite . it is based on retrieving relevant information from a large text corpus, allowing llms to produce more accurate and comprehensive responses. retrieval is essential for grounding llm responses, ensuring consistency and relevance to the information within the corpus @xcite . the accuracy of the retrieval process often directly determines the quality of the llms' answers. pre-trained dense embedding models are widely used for retrieval and often demonstrate strong zeroshot performance across diverse corpora @xcite @xcite . however, their effectiveness can decline when the target corpus significantly deviates from the training data @xcite . to overcome this bottleneck, customizing the retrieval model has the potential to yield significant improvements. existing customization methods often rely on labeled querydocument pairs from the corpus @xcite , employ-ing techniques like adaptor approaches, parameterefficient fine-tuning @xcite , and full model fine-tuning. however, obtaining such labeled data can be costly and time-consuming. moreover, some fine-tuning approaches often require substantial amounts of labeled data, limiting their practicality. unsupervised customization provides a more practical approach to customize retrieval, eliminating the need for costly labeled query-document pairs. this approach is designed to leverage the target corpus directly, which is readily available. existing unsupervised techniques include promptagator @xcite , which uses llms to generate relevant queries from passages (with filtering), and document expansion @xcite , where llms generate and append synthetic queries to each document. however, they have major bottlenecks. promptagator's effectiveness is limited by its inability to handle queries related to multiple documents (due to its strict oneto-one query-document pairing) and its weak error filtering (relying solely on round-trip consistency with generated synthetic samples). document expansion suffers from scalability issues, requiring synthetic query generation for the entire corpus to avoid retrieval bias. to overcome these limitations, we introduce square (synthetic query-based adaptive retrieval), a novel unsupervised retrieval customization method. square generates both synthetic queries and corresponding answers for each document of the corpus. these synthetic answers are then used to robustly filter noisy or irrelevant queries and identify additional relevant documents for each query. the resulting synthetic querydocument pairs are used to fine-tune the retrieval model via supervised customization techniques. we specifically selected newer retrieval benchmarks-bright @xcite , longembed @xcite , and mteb-law @xcite -from the mteb leaderboard @xcite to evaluate square. these benchmarks, not widely used in pre-training, provide an opportunity to measure how well our unsupervised customization generalizes to unseen data. our results demonstrate significant performance gains compared to the zero-shot baselines and other unsupervised tuning methods, showcasing the effectiveness of square in improving retrieval accuracy for various llm applications. ## method in this section, we present square, a novel unsupervised framework designed to enhance retrieval performance. the complete workflow of square is described in fig. 1 . given a corpus @xmath0, d 2 , ..., d n } containing n documents, our goal is to extract information to generate synthetic data for customizing a retrieval model, ultimately improving retrieval accuracy within d. we leverage an llm which inputs and produces text. for each document d k , we prompt llm with an instruction prompt i p to generate a synthetic query qk and its corresponding answer âk : qk , âk = llm(i p , d k ). details of the prompting strategy are provided in appendix a. the generated synthetic query-answer-document triplets are represented as @xmath1, âi , d i )} n @xmath2. unlike document expansion @xcite , square doesn't necessitate generating synthetic data for every document. for large corpora, generating data for a representative subset is sufficient, significantly enhancing scalability. our experiments confirm that adapting with a subset of documents effectively boosts retrieval performance. the generated answers âk are crucial for refining the relationship between the synthetic query qk and the documents in d. we employ a pre-trained embedding model e that transforms text into numerical vectors, enabling similarity comparisons. we compute the similarity between the embedding of the generated answer, e(â k ), and the embedding of each document d i in the corpus, e(d i ): if the similarity score between e(â k ) and e(d k ) is higher than the threshold η k (â k , d), we consider (q k , d k ) a relevant query-document pair and add it to our filtered set: these filtering and expansion methods offer two key advantages. first, they exploit the direct relationship between generated answers and source documents, leading to more robust relevance learning, especially beneficial in complex questionanswering scenarios requiring reasoning and inference (e.g., bright benchmarks @xcite ). second, ## related works square is distinct from hyde @xcite , which generates hypothetical documents from the query at inference time and averages their embeddings. this not only incurs a significant computational cost during inference-requiring multiple llm invocations and embedding calculations per query-but also differs fundamentally from square. square generates synthetic queries and answers directly from the documents during the tuning phase, ensuring all knowledge is grounded in the provided corpus and adding no overhead at inference beyond a single query embedding. @xcite also differs in its problem setting, as it begins with real queries to generate synthetic documents for fine-tuning rerankers. in contrast, square is unsupervised, generates synthetic queries, and adapts embedding models for retrieval. we see these approaches as complementary and square could potentially generate the initial queries for @xcite . other related works focus on synthetic data generation but for different goals. the primary aim of 2 https://www.anthropic.com/news/ claude-3-5-sonnet @xcite is to create large, diverse, multilingual datasets for training general, zero-shot embedding models. conversely, square focuses on constructing task-specific models by generating synthetic data tailored to a given document set. @xcite is much closer to promptagator @xcite , focusing on efficient synthetic query generation using small language models. however, it inherits promptagator's key limitations, namely an inability to effectively handle queries that span multiple documents and a filtering method vulnerable for its reliance on roundtrip consistency. furthermore, its simple generation prompts may limit its ability to create the complex queries necessary for specialized domains. ## limitations and future works this paper presents square, an effective unsupervised framework for customizing retrieval models to specific corpora. by generating and refining synthetic query-answer pairs, square achieves significant performance improvements compared to baselines. although our study utilized state-of-the-art llms for the generation of synthetic data, it would be beneficial for future research to investigate the performance of square when employing smaller, open-source llms. this exploration would provide valuable insights into the accessibility and broader applicability of our approach. furthermore, we intentionally excluded beir datasets from our evaluation due to the extensive pre-training of the initial embedding models on these datasets, which could potentially obscure the true impact of square. expanding square's capabilities to include multilingual data expansion is another important avenue for future development.