H allu S afe at S em E val-2024 Task 6: An NLI -based Approach to Make LLM s Safer by Better Detecting Hallucinations and Overgeneration Mistakes (2024)

Abstract: The advancement of large language models (LLMs), their ability to produce eloquent and fluent content, and their vast knowledge have resulted in their usage in various tasks and applications. Despite generating fluent content, this content can contain fabricated or false information. This problem is known as hallucination and has reduced the confidence in the output of LLMs. In this work, we have used Natural Language Inference to train classifiers for hallucination detection to tackle SemEval-2024 Task 6-SHROOM (Mickus et al., 2024) which is defined in three sub-tasks: Paraphrase Generation, Machine Translation, and Definition Modeling. We have also conducted experiments on LLMs to evaluate their ability to detect hallucinated outputs. We have achieved 75.93% and 78.33% accuracy for the modelaware and model-agnostic tracks, respectively. The shared links of our models and the codes are available on GitHub.

## introduction large language models are compelling in content generation. the ability of these models has led to their widespread use in various applications. some of the use cases of these models are in sensitive fields, such as consulting in medicine and law. the eloquence of llms makes their content appear very acceptable, and these models respond with high confidence. an important shortcoming of these models is hallucination. hallucination is the production of fabricated or false content @xcite . hallucination detection and mitigation are necessary to avoid the dangers of spreading false and harmful information. according to @xcite , hallucinations can be divided into input hallucinations, context hallucinations, and factual hallucinations. through this task, we have gained knowledge about hallucinations, their causes, and the various approaches to deal with them. language model responses can be so fluent that it becomes difficult even for a human agent to detect hallucinations. therefore, it is essential to train these models to recognize the limits of their knowledge. if they lack sufficient understanding of a subject, they should search for reliable sources and inform the human user if they are unsure of their answer. our team ranked 19th and 30th in the model-aware and model-agnostic tracks, respectively, with a difference of 2.93% and 8.4% compared to the topranked team. we found that the decision boundary for detecting hallucinations can be very narrow in some cases. while our system has shown relatively good performance, there is still room for improvement. ## background as mentioned earlier, there are three types of hallucinations. the types of hallucinations considered in this task are "factual" and "input". the "factual" type occurs in the definition modeling task, where the definition of a word or phrase must be provided, and the "input" type appears in the paraphrase generation and machine translation tasks. the hallucination detection track has two sub-tracks: modelaware and model-agnostic. in the model-aware subtrack, the model that generated the data is specified, and participants can use model parameters for diagnosis or analysis. however, our approach assumes the models are black-box and can be used for situations where we do not have access to the internal states and parameters of the model. it is important to note that overgeneration is another issue in llm outputs. samples with this issue should also be labeled as one, indicating the presence of hallucinations. hallucination is not specific to llms, and before the emergence of these models, it has been investigated in nlp tasks such as summarization and machine translation @xcite . to deal with the hallucination problem in llms, it is essential to find the causes of the problem first. two probable causes of hallucination, stated in @xcite , are the model focusing on producing one token each time and random sampling to increase diversity in text production. some believe overfitting to training data may lead to hallucination @xcite . in contrast to this point of view, in @xcite , they have shown that prompts consisting of only random meaningless tokens can also elicit hallucinations in llms. they believe that hallucinations are beyond training data and consider them as adversarial features. they have observed in their experiments that a slight change in the original prompt can produce a completely different claim by the llm, which indicates that llms are very non-robust. in rawte et al. (2023), they measure the relationship between linguistic factors such as readability, formality, and concreteness of prompts and hallucinations. their results show that more concrete and formal prompts lead to fewer hallucinations, but no definite conclusion can be drawn regarding the effect of readability on hallucinations. according to this article, prompt engineering can be effective in reducing the problem of hallucinations. lengthy prompts can hurt the understanding of the llm. in some experiments, it has been observed that the llm performs better when the critical information is placed at the beginning or end of the prompt. the performance quality decreases when the model needs to access the middle parts of the prompt for information. hallucination can be mitigated in different stages of an llm's life cycle. as we know, the life cycle of an llm consists of pre-training, sft (supervised fine-tuning), rlhf (reinforcement learning with human feedback), and inference @xcite . the datasets with which llms are pre-trained are collected without human supervision. these data can include false or outdated information, which may cause hallucinations. the training in the sft phase should also consider the knowledge of the model, and the model should not be fine-tuned for an application that has not acquired sufficient knowledge during the pre-training. one way to reduce hallucinations in both the sft and rlhf phases is to teach the model to be honest. the language model should be trained to avoid commenting on a subject if it does not have enough information @xcite . the methods investigated in this work are related to detecting and mitigating hallucination in the inference phase. the related previous works can be categorized as white box, gray box, and black box depending on the level of access to internal parameters of the llm. the methods that use the internal state of the language model for diagnosis are white-box approaches. gray box approaches are methods that access the output distribution of the model, such as detecting hallucinations at the token level. finally, blackbox approaches only have access to the textual output of the model. ## system overview in this section, we introduce our proposed system. the general system sketch is presented in figure 1 . additionally, detailed statistics regarding the dataset are outlined in examples of data samples for pg, mt, and dm tasks are presented in in addition to training classifier models, we have conducted tests to evaluate the performance of src the budget cannot be adopted against the will of the european parliament. hyp the european parliament does not approve the budget. pg label not hallucination src doonii fayyadamuun meeshaa geejibuun namootabaay'ee fi meeshaalee galaanarra cesisuuf karaa baayee si'aataa dha. tgt using ships to transport goods is by far the most efficient way to move large amounts of people and goods across oceans. hyp using a gas-fired device is a way to stop people from using natural gas and other equipment. mt label hallucination src communistic birds. what is the meaning of communistic? tgt living or having their nests in common. hyp of or pertaining to communism. dm label hallucination we also thought we might find a meaningful connection between token probabilities in the output sequence and hallucination. for this, we took the top token probabilities of the output sequence of the llm (pg, mt, and dm llms) with their labels. we fed them as input to an rnn model, such as lstm, to predict hallucination based on model uncertainty of token probabilities. unfortunately, we found out that when outputting hallucinated output, the model is as confident as non-hallucinated ones, and the classifier model could learn absolutely nothing from the token probabilities, no matter how we change the model complexity or hyperparameters. ## experimental setup to provide enough labeled data to train our models, a total of 3000 of the model-aware and modelagnostic training samples were labeled. different data splits were tested to get the best accuracy on each task. the details of the split used to train the model with the best accuracy for each task are specified in colab notebooks on github all three models are trained for ten epochs with a learning rate equal to 2e-5 and batch size equal to eight samples. the base model for all three tasks is deberta-v3-large @xcite , trained on the nli task with two classes of contradiction and entailment. we have used the hugging-face transformers library @xcite to train de-berta models implemented with pytorch. for instruction fine-tuning the falcon-7b model, we also used the hugging-face library and lora method @xcite . the prompt used for fine-tuning falcon and inference from chatgpt is similar to that used in the selfcheckgpt @xcite al., src a five, six, seven, eight. tgt and 5, 6, 7, 8. hyp a number between five and eight. pg gold label hallucination src est-ce que tu l'aimes? tgt do you love him? hyp do you love her? mt gold label not hallucination src haul away, keeping strain on both parts of the halyard so that @xmath0. tgt (nautical) a staff that carries a flag or pennant above the mast of a sailboat. hyp (nautical) a halyard. dm gold label not hallucination ## results we have achieved 75.93% and 78.33% accuracy for the model-aware and model-agnostic tracks of hallucination detection on final test data. we have ranked 19th and 30th in model-aware and modelagnostic tracks with a 2.93% and 8.4% difference with respect to the first-ranked team in the competition. the accuracies of the best model for each task, along with the accuracy of the base nli model, are provided in ## conclusion in this work, we have trained classifiers based on natural language inference to detect hallucinated outputs for the two model-aware and modelagnostic subtasks of the semeval-2024 task-6-shroom @xcite . we have also conducted experiments to evaluate llms' ability to perform this task. the fluency of the output of llms makes it difficult even for a human evaluator to recognize the hallucinated output. to train the classifiers, we labeled 3000 training data. labels may be a little affected by the subjectivity of the annotator, and for future work, it is better to have more than one person label each data sample. our hallusafe classifiers have achieved 75.93% and 78.33% accuracy for the model-aware and modelagnostic tracks of hallucination detection on final test data and have outperformed official baselines. regarding future work, enhancing the quality of training data in the pre-training and fine-tuning stages can effectively reduce hallucinations. given the potential limitations of storing all necessary information within the memory of models, coupled with the need for regular updates to certain information, it may be beneficial to equip models with search tools rather than relying solely on memory. it is important to train llms during the fine-tuning and instruction-tuning stages to refrain from answering questions if they lack sufficient knowledge on a particular subject, which needs a mechanism to be incorporated into these models to enable them to identify the boundaries of their knowledge.