ARL 2: Aligning Retrievers with Black-box Large Language Models via Self-guided Adaptive Relevance Labeling (2024)

Abstract: Retrieval-augmented generation enhances large language models (LLMs) by incorporating relevant information from external knowledge sources. This enables LLMs to adapt to specific domains and mitigate hallucinations in knowledge-intensive tasks. However, existing retrievers are often misaligned with LLMs due to separate training processes and the inherent black-box nature of LLMs. To address this challenge, we propose ARL2, a retriever learning technique that harnesses LLMs as labelers. ARL2 leverages LLMs to annotate and score adaptive relevance evidence, enabling the retriever to learn from robust LLM supervision. Furthermore, ARL2 incorporates a self-training strategy to minimize the cost of API calls. Extensive experiments demonstrate the effectiveness of ARL2, achieving accuracy improvements of 5.4% on NQ and 4.6% on MMLU compared to the state-of-the-art methods. Additionally, ARL2 exhibits robust transfer learning capabilities and strong zero-shot generalization abilities.

## introduction retrieval-augmented generation (rag) is a widely used technique for tailoring large language models (llms) @xcite @xcite to specific domains and tasks. by incorporating information from external knowledge sources, rag enhances llms by prompting them with relevant evidence @xcite @xcite , without the need for expensive fine-tuning @xcite . these knowledge sources serve as a non-parametric reference, allowing llms to access up-to-date and customized corpora for answering questions. rag has shown promising results in improving llm response accuracy for target tasks, while also helping to mitigate llm hallucination @xcite . the practice of rag for state-of-the-art llms often involves directly using standard retrievers (e.g. google search @xcite , bm25 @xcite ) or off-the-shelf dense retrievers (e.g., dpr @xcite , contriever @xcite ) trained with supervised relevance signals. however, the performance of these methods is limited by the mismatch between the retrieval and downstream tasks, as the retrieved similar documents may not always be useful for the queries despite their relevance. in fact, retrieved documents with similar topics but irrelevant content may even mislead the llm's predictions @xcite . to address the challenge of adapting retrievers for llms, several works propose joint training of retrievers and language models @xcite @xcite . however, these methods require training the llms from scratch, which is impractical for cutting-edge llms due to their prohibitive training costs and black-box nature. the recent replug method @xcite offers a solution by refining the retriever for black-box llms. replug utilizes language modeling scores of the answers as a proxy signal to train the dense retriever. however, such supervision for retriever training is indirect and may not be discriminative enough, especially when the questions could be directly answered through the parametric knowledge of the llm. therefore, effectively adapting retrievers for black-box llms remains an unsolved challenge. to enhance the retrieval model's performance, we present our approach, arl2 foot_0 , which leverages guidance from llms through self-guided adaptive relevance labeling. unlike existing methods that rely on indirect supervision via attention or answerbased language modeling scores, arl2 leverages the llms' capabilities to directly assess document relevance, results in the curation of high-quality relevance labels to train better retrievers. the advantages of using llm-annotated relevance labels for retriever training are threefold. firstly, arl2 can effectively distinguish truly useful documents from similar but irrelevant ones, providing valuable positives and hard negatives for training. secondly, it enables the creation of diverse training data beyond a single target dataset, surpassing the limitations of methods like replug. thirdly, arl2 can reversely generate diverse questions from unlabeled documents, enhancing data diversity and facilitating effective generalization under challenging few-shot or zero-shot scenarios. to reduce the cost of data curation due to frequent llm calls for relevance annotation, we propose an adaptive self-training strategy. this strategy empowers the retriever to identify and label confident and well-trained data points, reducing the reliance on costly llm interactions. furthermore, we introduce a cluster-driven prompt demonstration metric to ensure the diversity and high quality of the constructed data. this enables learning a strong retriever from a smaller amount of highquality and diverse annotated data. our experiments encompass both open-domain qa datasets (nq and tqa) and a specific-domain qa dataset (mmlu). the results demonstrate that our framework significantly enhances the perfor-mance of both the retriever and rag-based question answering, achieving an accuracy improvement of 5.4% on nq and 4.6% on mmlu. moreover, with the diverse curated data and self-training strategy, our method exhibits strong transferability to specific domains with limited training data and delivers promising results even in the challenging zero-shot generalization setting. in summary, our contributions are as follows: (1) we propose arl2, a retrieval augmentation framework that effectively aligns retrieval models with black-box llms. arl2 leverages the llm as a labeler to assign relevance scores with robust llm self-guided supervision. (2) we incorporate a cluster-driven prompt demonstration metric to ensure the generation of high-quality data. additionally, we explore a self-training strategy for the retriever to reduce the computational cost of llm calls. (3) extensive experiments demonstrate that our retrieval augmentation framework not only improves the performance of llms across various question-answering tasks but also exhibits strong transfer and zero-shot generalization capabilities. retrieval-augmented llms. rag have been widely used for language modeling @xcite ), question answering @xcite @xcite , and domain adaptation @xcite . to align retrievers with llms, most rag methods integrate a pre-trained retriever with a generator and subsequently undergo an end-to-end fine-tuning process to effectively capture knowledge @xcite . among them, atlas @xcite leverages retrieved documents as latent variables and fine-tunes retrieval models with four designed loss. aar @xcite identifies the llm's preferred documents through fid cross-attention scores @xcite , and fine-tuning the retriever with hard negative sampling. however, these methods are inapplicable to black-box llm as they require accessing llm parameters. the only exception is replug @xcite , which conducts supervised training by evaluating the kl divergence between the probability distributions of the retrieved documents and llm's likelihood. ## methodology in arl2, we employ llms to explicitly label relevance scores between questions and evidence, thereby generating relevance supervision for training an llm-aligned retriever. arl2 addresses two key challenges: (1) how can we effectively utilize the llm to construct a diverse and high-quality training set of relevance labels? (section 3.1) and ( 2 ) how can we train the retriever using the provided relevance supervision and further leverage the retriever to inform adaptive llm annotation? an overview of our proposed arl2 method is presented in figure 1 . ## conclusion we introduce arl2, a retrieval-augmentation framework that harnesses the capabilities of llms as annotators for retriever learning. unlike conventional approaches that face misalignment due to separate training processes and the inherent complexity of llms, our framework dynamically leverages llms to annotate and assess relevant evidence. this enables the retriever to benefit from robust llm supervision. additionally, we incorporate a self-training strategy to mitigate the cost associated with api calls. through extensive experimentation, we demonstrate the effectiveness of arl2, which enhances accuracy in open-domain qa tasks, exhibits robust transfer learning capabilities, and showcases strong zero-shot generalization abilities.