OPDAI at S em E val-2024 Task 6: Small LLM s can Accelerate Hallucination Detection with Weakly Supervised Data (2024)

Abstract: This paper mainly describes a unified system for hallucination detection of LLMs, which wins the second prize in the model-agnostic track of the SemEval-2024 Task 6, and also achieves considerable results in the model-aware track. This task aims to detect hallucination with LLMs for three different text-generation tasks without labeled training data. We utilize prompt engineering and few-shot learning to verify the performance of different LLMs on the validation data. Then we select the LLMs with better performance to generate high-quality weakly supervised training data, which not only satisfies the consistency of different LLMs, but also satisfies the consistency of the optimal LLM with different sampling parameters. Furthermore, we finetune different LLMs by using the constructed training data, and finding that a relatively small LLM can achieve a competitive level of performance in hallucination detection, when compared to the large LLMs and the prompt-based approaches using GPT-4.

## introduction the emergence of large language models (llms) @xcite has sparked a significant transformation in the field of natural language processing (nlp), ushering in a new era of unparalleled advancements in text generation and comprehension. this revolutionary technology has elevated the capabilities of ai systems, enabling them to perform complex reasoning and problemsolving tasks with remarkable proficiency @xcite . at the heart of this transformation lies the llms' ability to compress vast amounts of knowledge into neural networks, effectively turning them into versatile agents capable of engaging in natural language conversations with humans @xcite . this has broadened the scope of ai applications beyond traditional domains such as chatbots and virtual assistants, into areas previously thought to be the exclusive domain of humans, particularly those involving creativity and expertise. llms are not only limited to languagerelated tasks but can also function as generalist agents, collaborating with external systems, tools, and models to achieve a wide range of objectives set by humans @xcite . however, recent advancements in research have uncovered a concerning weakness: their proneness to hallucinate content across a range of applications @xcite . hallucination is defined as the generation of information that either conflicts with established sources or cannot be substantiated by available knowledge. the occurrence of hallucination in llms poses a significant threat to their practical deployment. while prior works have delved into the roots of hallucination within specific, smaller-scale language models and tasks, there is still a notable gap in understanding the exact nature and prevalence of content that llms are likely to hallucinate @xcite . to address this challenge, we implements a unified system for hallucination detection of llms, when there is no labeled training data. this system comprises five parts: base model selection, prompt engineering, weakly-supervised data generation, sft and ensemble learning. we first verify the performance of different base llms on this task. then we select the best llms and prompt is optimized to improve the performance. and weakly-supervised dataset is generated by using the selected llms. for further improvement, sft is done based on the constructed dataset and ensemble learning is adopted. mixture of experts(moe) @xcite is an ai technique that involves a group of specialized models (experts) being coordinated by a gating mechanism to address various aspects of the input space, in order to optimize performance and efficiency. this approach capitalizes on the idea that an ensemble of weaker language models, each focusing on specific tasks, can yield more precise results, similar to traditional ml ensemble methods. however, it introduces a novel concept of dynamically routing the input during the generation process. in the subsequent methods, we will conduct comparative experiments using the intelligent model base of moe. this paper @xcite integrates pos information and word semantic representation using an moe approach. model ensembling combines the predictions from multiple models together. traditionally this is done by running each model on some inputs separately and then combining the predictions. however, if the candidate models have the same architecture, they can be combined together to create a new model for prediction. many surveys systematically elucidate the basic concepts of ensemble learning and various methods, including model fusion and model voting @xcite @xcite . lora (low-rank adaptation of large language models) @xcite is a widely used and lightweight training technique that markedly reduces the number of trainable parameters. it functions by adding a smaller set of new weights to the model and training only these. as a result, training with lora is notably faster, more memoryefficient, and yields smaller model weights (a few hundred mbs), which are more manageable for storage and sharing. some other methods @xcite @xcite have been developed to improve lora. chain-of-thought prompting(cot) @xcite is an emerging application of language model technology. the core idea of this method is to encourage the model to not only generate the final answer but also gradually demonstrate its reasoning and the process of reaching conclusions. subsequent work @xcite has applied the idea of cot, breaking down problems into a series of sub-problems, allowing the model to reason step by step and ultimately provide the correct answer. and another work @xcite introduces a method called "self-consistency" to further enhance the effectiveness of cot prompting. by generating multiple reasoning paths and selecting the most consistent answer, the model can reduce errors and improve the accuracy of reasoning. ## methodology figure 1 shows the overview of our approach. our method consists of five main steps. first of all, multiple llms are compared on the hallucination detection validation dataset and among which we select the best base model. the llms selected in the first step will be utilized in the subsequent steps 2, 3, and 4. in the second step, we designed a prompt engineering module consisting of three sub-modules: few-shot prompting, instruction optimization, and the utilization of chain-of-thought. the subsequent experiments will demonstrate that prompt engineering significantly enhances the capabilities of the base model. then moving on to the label generation step, we apply the prompt engineering module to the selected best llms. we make predictions on the unlabeled training set and ensure the inference consistency among multiple llms as well as the inference consistency under different inference parameters of individual llm. we also ensure label balance in this process. the following steps are model training and ensemble learning. we utilize sft based on the constructed dataset using the lora @xcite method. finally, we select a few top-performing models and perform fusion at both the model weight level(model fusion) and the model prediction probability level(model voting) in order to seek better performance. ## results and analysis in this section, we will present a series of experiments to illustrate the effectiveness of our method. ## conclusion in this paper, we present a unified system for hallucination detection with llms when there is no labeled dataset, which wins the 2nd place with an accuracy score of 0.836 in the model-agnostic track and the 4th place with an accuracy score of 0.8053 in the model-aware track. to begin with, we generate high-quality weakly-supervised dataset by using large-sized llms with prompt engineering and few-shot learning. then we perform weaklysupervised fine-tuning based on the constructed dataset with different llms. our experiments yield several noteworthy findings: as for the dm task, the instruction is the same as the naive version.