Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLM s (2024)

Abstract: The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the help of a search engine remains an unresolved issue. Most existing methods solve this problem through the results of preliminary answers or reasoning done by the LLM itself, but this incurs excessively high computational costs. This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in LLMs with a slim proxy model, to enhance the LLM’s knowledge acquisition process. We employ a proxy model which has far fewer parameters, and take its answers as heuristic answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM. We only conduct retrieval for the missing knowledge in questions that the LLM does not know. Extensive experimental results on five datasets with two LLMs demonstrate a notable improvement in the end-to-end performance of LLMs in question-answering tasks, achieving or surpassing current state-of-the-art models with lower LLM inference costs.

## introduction large language models (llms) have demonstrated significant prowess in various natural language processing (nlp) tasks @xcite , attributed to their advanced language comprehension and generation capabilities. despite being trained on extensive text corpora, these models occasionally produce hallucinated content @xcite . to tackle this problem, the integration of retrieval systems with llms has been proposed, enabling access to external knowledge bases for more accurate and reliable text generation. retrieval-augmented generation (rag) involves using a retrieval system to supplement llms with relevant external information, thereby improving text generation quality @xcite . yet, recent studies have suggested that retrieval may not always be beneficial. in cases where llms can adequately respond without external knowledge, retrieval may introduce irrelevant information, potentially degrading performance @xcite @xcite . therefore, it is critical to determine when retrieval is necessary for user questions @xcite . the challenge lies in identifying questions that exceed the llms' intrinsic knowledge and require external retrieval, due to the prevalence of content hallucination. efforts to address this challenge can be categorized into two groups: (1) the first group of methods involves fine-tuning llms for rag scenarios, allowing them to autonomously signal the need for external knowledge @xcite @xcite . this method, while effective, demands substantial computational resources and risks diminishing the llms' general capabilities due to potential catastrophic forgetting @xcite . (2) the second category avoids direct tuning of llms, assessing the necessity for retrieval based on the quality of the generated content or specific indicators within it @xcite . however, this approach still has its drawbacks, as it requires multiple inferences, thereby increasing both the inference costs and the latency of responses to user questions. in light of this, we put forward a question: is it feasible to employ a proxy model with a relatively smaller parameter size to facilitate effective retrieval results for an llm? theoretically, existing decoder-only language models share similar transformer structures, and they are pre-trained on some common text corpora, such as common crawl web pages, books, and wikipedia pages @xcite @xcite @xcite . therefore, it is possible for them to reach a consensus on relative mastery over different knowledge and the necessity of retrieval. our preliminary quantitative analysis, shown in section 4.6, also supports this hypothesis. the experimental results show that on questions well understood by the llm, the relatively smaller language model also has considerable knowledge. the gap between larger and smaller llms mainly manifests in questions they do not understand. this further validates the possibility of employing a proxy model to help determine the necessity of retrieval. based on our analysis, in this paper, we introduce a novel approach, called slimplm (slim proxy language model), which leverages a relatively smaller language model as a "proxy model" to help determine when and how to perform retrieval for llms. specifically, for a user question, slimplm first uses the proxy model to generate a preliminary "heuristic answer". this heuristic answer serves two purposes. first, it is evaluated by a lightweight model designed to assess the necessity for retrieval. if this evaluation shows that the heuristic answer is of high quality, it implies that the question may be addressed directly by llms without additional information retrieval. in contrast, a lower-quality answer triggers the retrieval process to identify and supplement missing knowledge. to facilitate this, slimplm utilizes the heuristic answer again to generate multiple queries, each reflecting a specific aspect of the initial response. these queries are then individually assessed for their need for retrieval, filtering out queries that do not require retrieval. by this means, the remaining queries can retrieve more relevant knowledge that is lacking in llms. the integration of slimplm into existing rag frameworks offers a flexible and effective enhancement without notably increasing computational costs or response latency. experimental results across five commonly used questionanswering datasets validate slimplm's effectiveness in determining the necessity for retrieval and improving retrieval results. our contributions are threefold: (1) we pro-pose a novel approach that leverages a small proxy model to generate heuristic answers, helping determine when and how to perform retrieval for llms. ## methodology in this paper, we aim to leverage a relatively smaller model as the proxy model to determine whether the user-issued question requires supplementary retrieval results and further provide clues for retrieving relevant knowledge. our method, slimplm, can be flexibly used as a plug-in to various retrieval-augmented generation scenarios, without additional training requirements. the illustration of our method is shown in figure 1 . ## experiments we conduct experiments on five widely used question-answering (qa) datasets and compare the performance of our method with several baselines. ## conclusion in conclusion, our research proposes a new paradigm for rag, utilizing a smaller llm as proxy model. based on the heuristic answer by proxy model, we conduct query rewriting, retrieval necessity judgment, and claim-based query filtering. this approach enables accurate perception for when and what to retrieve for llms. experiments across various datasets show a marked improvement in the end-to-end performance of llm question-answering, achieving or exceeding stateof-the-art results. moreover, this enhancement is attained with little additional computational cost.