UMUT eam at S em E val-2024 Task 6: Leveraging Zero-Shot Learning for Detecting Hallucinations and Related Observable Overgeneration Mistakes (2024)

Abstract: In these working notes we describe the UMUTeam’s participation in SemEval-2024 shared task 6, which aims at detecting grammatically correct output of Natural Language Generation with incorrect semantic information in two different setups: model-aware and model-agnostic tracks. The task is consists of three subtasks with different model setups. Our approach is based on exploiting the zero-shot classification capability of the Large Language Models LLaMa-2, Tulu and Mistral, through prompt engineering. Our system ranked eighteenth in the model-aware setup with an accuracy of 78.4% and 29th in the model-agnostic setup with an accuracy of 76.9333%.

## introduction recently, the emergence of large language models (llms) has brought about a paradigm shift in natural language processing (nlp), leading to unprecedented advances in natural language understanding (nlu) @xcite and reasoning @xcite . in general, llms refer to a set of general-purpose models based on the transformer architecture and pre-trained on large text corpora, such as gpt-3 @xcite , llama @xcite , palm @xcite and gpt-4 @xcite . by scaling the amount of data and model capacity, llms demonstrate incredible emergent capabilities, typically including in-context learning (icl) @xcite , chain-of-thought prompting @xcite , and instruction following @xcite . natural language generation (nlg) faces two related challenges. first, current models often produce output that is fluent but inaccurate. second, the metrics used to evaluate the llms performance prioritize fluency over correctness, exacerbating the problem of "hallucination", in which llms produce fluent but incorrect output. consequently, significant research in underway to automatically detect such errors. in many nlg applications, output correctness is paramount, as in cases such as machine translation, where producing a plausible but inconsistent translation compromises the utility of the system. thus, the shroom shared-task focuses on identifying grammatically correct outputs that contain incorrect semantic information, regardless of whether the model producing the output is accessible or not @xcite . to this end, the organizers have adapted a post-hoc environment in which the models have already been trained, and the outputs have already been produced. the participants' task is a binary classification problem to identify cases of hallucinations, i.e. to detect grammatically correct outputs that contain incorrect or unsupported semantic content, in two different setups: model-aware and model-agnostic tracks. to address the shroom challenge, our team used a zero-shot learning (zsl) approach with llama-2 @xcite , and tulu @xcite llms to detect grammatically correct output that contains incorrect semantic information through the prompt. the zsl technique refers to the ability of llms to perform tasks without being explicitly trained on them, meaning that the model can generate responses or make predictions on topics or domains that were not part of its explicit training. this is achieved by exploiting the general knowledge that the models have acquired during their massive pretraining on large text corpora. during our experiments, we observed that these llms were able to identify hallucinations. in particular, tulu is the one best suited for this task. the rest of this paper is organized as follows. section 2 provides a summary of important details about the task setup. section 2 gives an overview of our system for two subtasks. section 4 presents the specific details of our systems. section 5 dis-1 675 cusses the results of the experiments, and finally, the conclusions are presented in section 6. in general, hallucinations in nlg tasks can be divided into two main types @xcite : intrinsic hallucinations and extrinsic hallucinations. on the one hand, intrinsic hallucinations refer to the output of llms that conflict with the source content. on the other hand, extrinsic hallucinations refer to llm generations that cannot be verified from the source content. the shroom task aims to automatically detect hallucinations and related observable overgeneration errors. to achieve this, the organizers have provided a collection of checkpoints, inputs, references and outputs from systems covering three different nlg tasks: (1) definition modeling (dm), (2) machine translation (mt), and (3) paraphrase generation (pg), trained with different levels of accuracy. the development set includes binary annotations from at least five different annotators and a majority vote gold label. the generalizability of llms is very attractive because it allows us to adapt state-of-the-art methods to specific goals. for example, an llm trained on multilingual texts can perform translations without being explicitly trained to do so (known as zero-shot capability, zsl). another possibility is to guide models by providing them with examples of the input and the desired output (known as fewshot learning, fsl). for example, in @xcite , llms have shown good performance in a zsl scenario for identifying hate speech in spanish and english. in this sense, it is possible to ask for a sentence and its translation before ask-ing it to translate another sentence. this additional information helps to improve the quality of the output. for text classification tasks, the ability to make such predictions with little or no training makes these models particularly promising for empirical research, as they have the potential to perform accurately without the need for costly and time-consuming annotation procedures. therefore, we took advantage of this zsl classification capability of llms to detect hallucinations and related observable overgeneration errors. the following models are evaluated: figure 1 shows the architecture of our system. we can see that we have introduced a specific prompt for each llm to generate a response with the desired structure. then we have a module called "filter" that extracts a binary response based on the response and the correlation value. ## experimental setup in this section, we explain the dataset used, the hyperparameters used in the llms to generate re-3 677 sponses, and details of the metrics used by the organizers for evaluation. in this task, the organizers provided participants with unlabeled train data, trial data, and validation data for both the model-aware and model-agnostic setups. we have only used the validation data to evaluate the performance of different models using a zsl approach. figure 3 displays the distribution of the validation set. the hyperparameters used in the llms to generate the response are 0.95 for top_p, 0 for top_k, 256 for max_new_tokens and the default temperature for each llm. the evaluation metrics used are accuracy for binary classification and rho to evaluate correlation. the rho metric, commonly known as the rho correlation coefficient (ρ), is a statistical measure that evaluates the relationship between two ordinal variables. it is particularly useful when the variables are not continuous but are divided into ordered categories. ## results ## conclusion here we describe the umuteam's participation in shroom (semeval 2024), concerning the development of models for detecting grammatically correct output from nlgs, but with incorrect semantic information in two different setups: model-aware and model-agnostic tracks. we have used the zsl approach with llama-2 @xcite , and tulu @xcite llms to detect output that contains incorrect semantic information through the prompt. tulu performed best in the evaluation set. using this model, we ranked eighteenth in the model-aware setup with an accuracy of 78.4% and nineteenth in the model-agnostic setup with an accuracy of 76.9333%. as further work, we propose to investigate hallucination detection in the political domain. in politics, automated content generation can help politicians to generate text on a variety of political topics, which can help political campaigns, think tanks, and government agencies quickly produce tailored content. hallucination detection can help to mitigate misleading or fabricated content. in this sense, we propose to generate political discourse that imitates politicians from different political wings @xcite and to identify the generated hallucinations by different llms.