A lpha I ntellect at S em E val-2024 Task 6: Detection of Hallucinations in Generated Text (2024)

Abstract: One major issue in natural language generation (NLG) models is detecting hallucinations (semantically inaccurate outputs). This study investigates a hallucination detection system designed for three distinct NLG tasks: definition modeling, paraphrase generation, and machine translation. The system uses feedforward neural networks for classification and SentenceTransformer models for similarity scores and sentence embeddings. Even though the SemEval-2024 benchmark shows good results, there is still room for improvement. Promising paths toward improving performance include considering multi-task learning methods, including strategies for handling out-of-domain data minimizing bias, and investigating sophisticated architectures.

## introduction ai hallucination refers to a phenomenon where a large language model (llm) -usually generative ai or a computer vision tool produces nonsensical and inaccurate outputs @xcite . thus, this leads to fluent but inaccurate generations. the term 'hallucination' is usually associated with human or animal brains but from the standpoint of machines, hallucinations refer to these inaccurately produced outputs. hallucinations in generative models may arise due to multiple factors such as overfitting during model training, complexity of the model, and bias in training data. according to multiple surveys @xcite , hallucinations in natural language models may arise primarily due to two reasons -hallucinations due to data and hallucinations during modeling. hallucinations in ai models may prove to be a threat in multiple scenarios such as healthcare where a model may not be able to predict the existence of the exact condition that needs to be treated. hallucinations in fluent over generations may also lead to the spread of misinformation. the most pressing problem in the modern-day natural language generation landscape is that the existing metrics @xcite can mostly detect fluency in generation rather than accuracy. to deal with this issue, several studies have explored different techniques, such as knowledge graph integration, bias detection, and mitigation @xcite . building upon this prior research, our work proposes training tailormade deep learning models and using transformer @xcite based architectures to identify cases of hallucinations. to deal with this scenario, a task @xcite was proposed to build a system to detect instances of hallucinations in generated text. ## task the primary task was to build a hallucination detection system capable of detecting outputs that are grammatically sound but are semantically inaccurate concerning the provided source inputboth with or without access to the model used to generate the outputs. this is essentially a binary classification task and we were provided with two tracks -model agnostic and model aware. model agnostic refers to the track where one would have no access to the model used to generate the outputs and model aware refers to the track where the model was provided in the dataset. ## related work recent advances in natural language processing (nlp) have resulted in the development of transformer based models such as bert and its specialized variations that have introduced efficiency and accuracy in several nlp tasks. ## datasets since the task was divided into two tracks -modelaware and model-agnostic, we were provided with sets of two datasets. the model-aware dataset had a separate column for the generative model used to produce the outputs. the training dataset consisted of three natural language generation (nlg) tasks -definition modelling, paraphrase generation, and machine translation. each entry in the dataset comprises three text columns -hyp, src and tgt. ## methodology 6.1 experimental setup since the generated texts have been divided into three tasks, we have divided the process into three branches. for definition modeling, we used 'all-mpnetbase-v2' foot_0 and for paraphrase generation, we have used 'paraphrase-minilm-l6-v2' foot_1 , both sentence-transformer models to encode the sentences and calculate two sets of cosine similarity values. one represents the similarity score between the hypothesis (hyp) and source (src) text and the other represents the similarity score between the source (src) and target (tgt) text. these similarity scores are converted into a numpy array using numpy's column stack for input into two sequential models respectively. for machine translation, we used 'all-minilm-l6-v2' foot_2 , also a sentencetransformers model to encode the sentences and produce embeddings. the spearman correlation between the two sentences in the hypothesis and target columns is calculated. we chose these columns specifically as 'hyp' contains the english translation produced by the generative model and 'tgt' contains the correct translation. we used the scipy library of python for this metric. finally, the correlation coefficients are pushed into a numpy array. in this case, we have used a different sequential model for training. the input array is split into an 80:20 ratio for training and validation respectively and the test dataset was entirely used for producing the outputs. we have used a common pipeline for processing the entire dataset and then branched the input array according to the task label -if the task is 'dm' it was fed into the model prepared for definition modeling. ## evaluation the predicted probability values generated by the model were translated into binary labels using a threshold approach. data points with a predicted probability of 0.5 or higher were assigned the label "hallucination," while those below 0.5 were labeled "not hallucination." these labels were then saved in a json file conforming to the specified format. the semeval-2024 task had two measures to evaluate the performance: it is given by: also, the submissions were divided into model agnostic and model aware tracks. our model placed 31st out of all entries in the model-aware track and 41st out of all entries in the model-agnostic track. ## conclusion and future work this paper explored detecting hallucinations in natural language generation (nlg) outputs using specialized models for definition modeling, paraphrase generation, and machine translation tasks -both while having access to the models used to generate the sentences and without. we used transformerbased models for calculating the similarity scores as they outperform other models such as universal sentence encoder (use) @xcite and doc2vec (lau and baldwin, 2016). while there have been previous studies on hallucination detection, our approach offers several key novelties that have contributed to its effectiveness. several avenues exist for further development of our hallucination detection system. to enhance the performance of our model, we advise investigating data augmentation techniques, as transformerbased models have a large thirst for data. to increase the model's robustness and durability, we also suggest using adversarial training and exploring more advanced deep learning architectures. this project has been possible due to the contributions of sohan choudhury, who developed the architecture for the definition modelling task, priyam saha, who created the paraphrase generation model, and subharthi ray, who built the machine translation model. we are also deeply grateful for the insightful guidance and mentorship provided by shankha shubhra das and dr. dipankar das throughout this journey.