Invoke Interfaces Only When Needed: Adaptive Invocation for Large Language Models in Question Answering (2025)

Abstract: The collaborative paradigm of large and small language models (LMs) effectively balances performance and cost, yet its pivotal challenge lies in precisely pinpointing the moment of invocation when hallucinations arise in small LMs. Previous optimization efforts primarily focused on post-processing techniques, which were separate from the reasoning process of LMs, resulting in high computational costs and limited effectiveness. In this paper, we propose a practical invocation evaluation metric called AttenHScore, which calculates the accumulation and propagation of hallucinations during the generation process of small LMs, continuously amplifying potential reasoning errors. By dynamically adjusting the detection threshold, we achieve more accurate real-time invocation of large LMs. Additionally, considering the limited reasoning capacity of small LMs, we leverage uncertainty-aware knowledge reorganization to assist them better capture critical information from different text chunks. Extensive experiments reveal that our AttenHScore outperforms most baselines in enhancing real-time hallucination detection capabilities across multiple QA datasets, especially when addressing complex queries. Moreover, our strategies eliminate the need for additional model training and display flexibility in adapting to various transformer-based LMs. Our code is available at https://github.com/Robot2050/AttenHScore.

## introduction with the profound study of the scaling law @xcite and the density law @xcite , the development and application of language models (lms) have exhibited a diversified pattern. in this context, the remarkable performance of large language models (llms) such as gpt-4o in reasoning tasks has attracted significant * corresponding author: qinbiao@ruc.edu.cn attention @xcite . however, due to their complex structures and massive parameter scales, these llms consume considerable computational resources during training and inference. consequently, many of these llms are only available through paid api services, undoubtedly increasing their monetary cost. meanwhile, small language models (slms), with their lightweight architectures and efficient inference capabilities @xcite , demonstrate significant advantages in specific scenarios, such as real-time responses on edge devices @xcite and rapid processing of simple tasks @xcite . nevertheless, when faced with higher-level tasks requiring complex semantic understanding, the capabilities of slms appear to be inferior compared to those of llms @xcite . to balance performance and cost while enhancing overall efficiency, a new paradigm of collaboration between large and small lms has emerged from the perspectives of cost-effectiveness and resource optimization. this paradigm aims to fully leverage the advantages of llms in handling complex tasks while exploiting the efficiency of slms in simple problem scenarios, thus achieving optimal resource allocation and efficient task processing. as illustrated in figure 1 , we conduct retrieval-based question answering (qa) experiments utilizing two lms, one large and one small, across five datasets from longbench @xcite , to evaluate the performance of both lms in scenarios without retrieval, and with top-5, top-10, top-15 retrieval results. llm exhibits overall superior performance, but the gap between it and slm is remarkably narrow on certain datasets. under these circumstances, researchers have mainly proposed two strategies: routing and cascading. the core mechanism of the former lies in accurately directing user queries to a specific model based on criteria provided by specially trained models @xcite . comparatively, the latter exhibits a more flexible and phased processing mode. according to this strategy, user queries are first sent to slms for initial processing. then, based on the output results of these models, the system determines whether further in-depth reasoning by llms is necessary @xcite . based on the aforementioned research, we find that routing strategies require the introduction of auxiliary models for decision-making during implementation, which contradicts the initial goal of simplicity and efficiency. more importantly, these auxiliary models not only require specialized training but also often rely on specific datasets @xcite , potentially limiting their versatility across different tasks. in view of this, we have chosen to adopt a cascading strategy, where the main technical challenge lies in accurately determining when hallucinations occur in slms. currently, research on hallucination detection in lms primarily focuses on the post-reasoning phase @xcite @xcite . however, such methods exhibit significant limitations when integrated into the practical llms applications. the primary issue is that these post-processing methods often incur high computational costs and notable delays. for instance, cutting-edge detection methods typically utilize llms such as chatgpt, opt, etc. @xcite , making the cost of hallucination detection comparable to or even more expensive than llms reasoning tasks. what's more, post-processing methods are independent of the reasoning process @xcite , thus they cannot delve into the origins and evolution of hallucinations within each lms. seeking to surmount the outlined restrictions, we shift the focus of optimizing lms invocations towards understanding their existing available signals, rather than training and running more auxiliary models. this paper proposes a practical invocation evaluation metric, attenhscore, designed to calculate the accumulation and propagation of hallucinations during the generation process of slms. by continuously amplifying potential error points, this metric enables more skillfully identify deviations between generated content and facts, thereby improving the detection accuracy of hallucinations. furthermore, from the perspective of retrieval-augmented generation (rag), we guide slms to evaluate the uncertainty between queries and different text chunks, optimizing the information arrangement by moving more relevant content from the retrieval to the front of the prompt, thereby further assisting slms in capturing key information and enhancing their accuracy in qa tasks. the main contributions of this work are as follows: ## optimizing the adaptive invocation interface for llms our design philosophy is to "maximize the success rate of slms and minimize the need for llm invocations", thereby embodying the idea of "invoke interfaces only when needed". we achieve this goal through a two-stage strategy that combines "pre-event optimization" with "post-event quality control": re-ranking: this module aims to reduce the likelihood of slms generating hallucinations at the source. as we pointed out in the paper, slms have limited capabilities in processing long texts and extracting key information. by re-ranking the input information and placing the most critical content at the forefront, we can significantly optimize the working environment for slms, thereby proactively minimizing scenarios that might necessitate llm invocations. hallucination detection: when "pre-event optimization" is still insufficient to completely avoid errors, this module serves as a safety net. it is responsible for real-time monitoring of slm outputs to precisely identify moments when llm invocations are truly necessary. ## conclusion amidst the drive for efficiency and resource optimization, this study delves into the challenges of hallucination detection and prompt re-ranking within the collaboration of large and small lms. we introduce a novel invocation discriminant metric, attenhscore, which quantifies the accumulation and propagation of hallucinations in slms generations, enabling more precise detection of potential reasoning errors. additionally, within a retrieval-based qa context, we steer slms to assess the uncertainty of queries relative to various text chunks, thereby achieving superior re-ranking and enhanced accuracy. extensive experiments across four datasets reveal that our proposed realtime, plug-and-play detection methodology and re-ranking strategy strike an effective balance between cost and performance, eliminating the need for domain-specific knowledge or model training. we anticipate that our insights will inspire further researches into hallucination detection and reranking, ultimately promoting the development of collaboration between large and small lms.