Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector (2024)

Abstract: Hallucination detection is a challenging task for large language models (LLMs), and existing studies heavily rely on powerful closed-source LLMs such as GPT-4. In this paper, we propose an autonomous LLM-based agent framework, called HaluAgent, which enables relatively smaller LLMs (e.g. Baichuan2-Chat 7B) to actively select suitable tools for detecting multiple hallucination types such as text, code, and mathematical expression. In HaluAgent, we integrate the LLM, multi-functional toolbox, and design a fine-grained three-stage detection framework along with memory mechanism. To facilitate the effectiveness of HaluAgent, we leverage existing Chinese and English datasets to synthesize detection trajectories for fine-tuning, which endows HaluAgent with the capability for bilingual hallucination detection. Extensive experiments demonstrate that only using 2K samples for tuning LLMs, HaluAgent can perform hallucination detection on various types of tasks and datasets, achieving performance comparable to or even higher than GPT-4 without tool enhancements on both in-domain and out-of-domain datasets.

## introduction recently, large language models (llms) @xcite have demonstrated exceptional capabilities across a variety of tasks within the field of natural language processing. however, hallucination @xcite @xcite @xcite ) in text generated by llms remains an underlying concern, impeding the application of llms in real-world scenarios @xcite . @xcite have indicated that, despite the existence of some effective hallucination mitigation strategies, the occurrence of hallucinations in llms is inevitable. therefore, reliable and effective hallucination detection methods are necessary and urgent. existing hallucination detection methods can be roughly categorized into two primary approaches. one line of work relies on the internal knowledge of llms to directly identify hallucinations via prompts @xcite or evaluating the semantic consistency among multiple responses to the same question generated by llms @xcite . however, these methods are usually constrained not only by llms' internal knowledge but also by their abilities to utilize knowledge. another line of work extends the detection ability of llms by employing external tools (e.g., search engine) to obtain supporting evidence for hallucination detection @xcite @xcite . however, these approaches mostly depend on closed-source powerful llms such as gpt-4. moreover, their detection process is usually pre-determined by human, making it difficult for the model to autonomously and effectively execute the hallucination detection. to address these issues, in this paper, we propose the haluagent, an autonomous llm-based agent framework for hallucination detection. this agent is based on smaller open-source models (i.e., baichuan2-chat 7b and 13b @xcite ) and capable of bilingual hallucination detection in chinese and english. the motivations are twofold: (1) designing autonomous detection agents that can actively make decisions and judgements, without human assistance; (2) enabling relatively smaller models to effectively perform complex detection, without reliance on close-sourced llm apis. to achieve this, we make three major technical contributions. first, we extend the llm's capability to detect a broader range of hallucination forms such as text, code, math expression, or their combination by curating a multi-functional toolbox, in contrast to previous work only focused on textual hallucination or limited tools @xcite . second, we design a finegrained three-stage detection framework along with memory mechanism, including sentence segmentation, tool selection and verification, and reflection. third, we leverage existing hallucination datasets to synthesize detection trajectories for fine-tuning the llm, where we first employ gpt-4 to execute the above three stages until obtaining detection results consistent with the ground-truth label and then synthesize the instruction data. we compare haluagent and previous work in to verify the effectiveness, we evaluate haluagent on both in-domain and out-of-domain datasets at response-and sentence-level granularities. after fine-tuning with only 2k trajectories, the detection performance of haluagent has been improved significantly (e.g., overall accuracy increases from 46.44% to 79.70% in four in-domain datasets and from 49.50% to 78.43% in two out-of-domain datasets), reaching a level comparable to or even higher than gpt-4 without tool enhancement. in the sentence-level detection experiments, haluagent also achieved substantial improvements, particularly with f1 scores on the math and science datasets increasing from 19.51% to 68.80% and from 17.54% to 94.16%, respectively. ## approach in this section, we introduce haluagent, our proposed autonomous agent for detecting hallucinations across various text types. the core of our haluagent framework is a well-instructed llm, which can autonomously leverage tools to detect a broader range of hallucination types. first, we define several tasks for hallucination detection and then design a toolbox with supporting tools to extend the llm's capability. to enable step-by-step detection, we design a three-stage detection frame-work equipped with memory mechanism. finally, we synthesize high-quality detection trajectory data to fine-tune open-source llms. we present the overall architecture of haluagent in figure 1 . ## conclusion in this work, we proposed an autonomous agent framework, haluagent, which is capable of bilingual hallucination detection in chinese and english. in our approach, we first curated a multi-functional toolbox to extend the llm's capability to detect hallucinations. next, we designed a fine-grained three-stage detection framework along with memory mechanism, including sentence segmentation, tool selection and verification, and reflection. then, we leveraged existing datasets to synthesize detection trajectories by employing gpt-4 to execute the detection process following the haluagent framework. finally, we fine-tuned baichuan2-chat 7b and 13b on the synthesized trajectories. the halu-agent models achieved notable improvements on both in-domain and out-of-domain datasets, with performance comparable to or even higher than gpt-4 without tool enhancements. due to its high flexibility and adaptability, haluagent can effectively serve as a hallucination detector when human users interact with llms in real-world scenarios. in future work, we will extend our method to deal with more types of tools and hallucinations.