PRCA : Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter (2023)

Abstract: The Retrieval Question Answering (ReQA) task employs the retrieval-augmented framework, composed of a retriever and generator. The generators formulate the answer based on the documents retrieved by the retriever. Incorporating Large Language Models (LLMs) as generators is beneficial due to their advanced QA capabilities, but they are typically too large to be fine-tuned with budget constraints while some of them are only accessible via APIs. To tackle this issue and further improve ReQA performance, we propose a trainable Pluggable Reward-Driven Contextual Adapter (PRCA), keeping the generator as a black box. Positioned between the retriever and generator in a Pluggable manner, PRCA refines the retrieved information by operating in a token-autoregressive strategy via maximizing rewards of the reinforcement learning phase. Our experiments validate PRCA’s effectiveness in enhancing ReQA performance on three datasets by up to 20% improvement to fit black-box LLMs into existing frameworks, demonstrating its considerable potential in the LLMs era.

## introduction retrieval question answering (reqa) tasks involve generating appropriate answers to given questions, utilizing relevant contextual documents. to achieve this, retrieval augmentation is employed @xcite @xcite , and comprised of two key components: a retriever and a generator. the retriever's role is to retrieve relevant documents from a large corpus in response to the question, while the generator uses this contextual information to formulate accurate answers. such systems alleviate the problem of hallucinations @xcite , thereby enhancing the overall accuracy of the output. showcases the traditional method where a query is processed by a retriever that scans a corpus to fetch the top-k documents and then fed to a white-box generator. the lower section introduces our proposed prca method, which processes extracted top-k documents from the retriever before feeding them to black-box generator to achieve better performance for in-domain tasks. recent advances in large language models (llms) such as the generative pre-trained transformer (gpt) series @xcite @xcite have demonstrated remarkable potential, notably in their zero-shot and few-shot abilities within the realm of qa tasks. owing to these capabilities, llms are excellent choices as generators within the retrievalaugmented framework. however, due to the vast parameters of llms, fine-tuning them becomes exceedingly difficult within a limited computation budget. furthermore, certain llms such as @xcite are closed-source, making it impossible to fine-tune them. to achieve optimal results on specific datasets, fine-tuning retrievalaugmented models becomes necessary @xcite @xcite . previous attempts to integrate llms into the retrievalaugmented framework have met with partial suc-cess but also come with limitations. @xcite utilized the logits from the final layer of the llms when calculating the loss function, which may not be available to certain powerful llms that served via apis. @xcite involved frequently invoking pricy llms and overlooked the impact of the input token length on the accuracy and effectiveness of the system. to overcome these hurdles, we propose a trainable pluggable reward-driven context adapter (prca) that enables one to fine-tune the adapter instead of llms under the retrieval-augmented framework on specific datasets and achieve higher performance. furthermore, prca distills the retrieved documents information guided by rewards from the generator through reinforcement learning. the distillation of retrieval information through prca reduces the length of text input to the generator and constructs a context of superior quality, which mitigates the hallucination issues during the answer generation. as shown in figure 1 , prca is placed between the retriever and the generator, forming a prca-based paradigm where both the generator and the retriever remain frozen. in general, the introduction of the prca-based paradigm brings the following advantages: ## conclusion in conclusion, this research successfully introduces a prca-based paradigm for reqa tasks, tackling the inherent challenges of fine-tuning llms in the retrieval-enhancement framework, especially given their vast parameter size and closed-source natures. prca innovatively distills retrieved documents via generator rewards, leading to a marked improvement in the reqa task's performance. experimental outcomes consistently demonstrate the robustness and effectiveness of prca when paired with various retrievers and generators, indicating its potential to be widely deployed as an adapter on the reqa task.