O ctavian B at S em E val-2024 Task 6: An exploration of humanlike qualities of hallucinated LLM texts (2024)

Abstract: The tested method for detection involves utilizing models, trained for differentiating machine-generated text, in order to distinguish between regular and hallucinated sequences. The hypothesis under investigation is that the patterns learned in pretraining will be transferable to the task at hand. The rationale is as follows: the training data of the model is human-written text, therefore deviations from the training set could be detected in this manner.A second method has been added post competition as a further exploration of the dataset involving using the loss of the generation as determined by a pretrained LLM.

## introduction the shroom shared task, as described by @xcite , has as its objective 'detecting grammatically sound output that contains incorrect semantic information (i.e. unsupported or inconsistent with the source input), with or without having access to the model that produced the output'. this type of output is encompassed by generations often referred to as "hallucinations". according to varshney et al.,in the context of language models, hallucinations refer to the generation of text or responses that seem syntactically sound, fluent, and natural but are factually incorrect, nonsensical, or unfaithful to the provided source input. @xcite @xcite with the advent of mainstream large language models brought upon by openais chatgpt @xcite , it is a relatively new and important topic in this context. apart from the possible spread of misinformation, being able to make this distinction is crucial for the adoption of large language models in domains highly sensitive to misinformation, such as the medical, jurisdictional or financial fields. possible repercussions include medical misdiagnosis, fictitious financial or legal advice, or exploitation by bad actors in order to deceive users. in our approach to solving this task, we employ two methods. the first method involves utilizing models trained for detecting machine-generated text in order to distinguish between regular and hallucinated sequences. the other involves using looking at the loss of the hypothesis as scored by an llm(large language model), in the hope that generations with low probabilities can be properly tagged as hallucinations. this method was introduced by @xcite as gptscore as a way to get a numerical assessment of an aspect in a given text. when it comes to the first method, the hypothesis to be tested is that patterns which help differentiate machine generated texts will be transferable to the task at hand. the rationale is as follows: the training data of the model is human-written text, therefore deviations from the training set could be detected in this manner. from the experimental results on the model aware track, the performance of this method yielded a score of 0.483. this is below the baseline achieved using an llm to label the generations. these results could stem from the hypothesis itself, or the fact that the model is not able to differentiate the texts of newer llms. the second method was employed after the end of the competition, as a way to further explore the dataset and its characteristics. it was based on the success of ji et al., who used a similar approach in a reprompting system meant to reduce hallucinations. on the validation data, it yielded an accuracy of 0.686 without the target reference and 0.702 when it was included in the prompt on the model aware track. ## background and dataset the objective of the shroom shared task is to detect hallucinations in two distinct datasets: one that is model agnostic and one that is model aware. both of the datasets consist of text pertaining to 3 tasks: dm -'definition modeling' -which involves providing the definition of a word given surrounding context, pg -'paraphrase generation' -in which the generated text is meant to be a paraphrase of the input, and mt -'machine translation' -in which the task is to translate a given sequence. the text provided for the definition modeling and the paraphrase generation task types is in english. for the machine translation task, the prompt is provided in the native language and the hypothesis and target are both in english. the model-aware validation dataset consists of 499 datapoints, while the model-agnostic version has 501 datapoints. the test sets both have 1500 samples. the main difference between the datasets is that the 'model' column is not present in the model agnostic version. this distinction is not relevant to the experiments presented in this paper, as no data is used apart from the 'hyp' -generated sequence column. the datasets are split by the organizers in train, validation and test sets respectively, with the validation and test sets containing human-annotated labels. the probability of hallucination is defined as the average of the label given by each annotator, and the final label is chosen by majority vote from said labels. the validation and test set have 5 such labels per entry. an example datapoint consists of the input 'resembling or characteristic of a weasel.' -corresponding to this input, the output is structured as per ## related work due to the importance and the relative novelty of the llm hallucination detection task, there are using a community sourced body of knowledge, for example wikipedia, in order to greatly enhance context @xcite . perturbations to the input to check for model self consistency @xcite . segmenting the generations and reprompting to check for consistency also appears to have lead to good results. @xcite @xcite looking at the log probabilities of the output words to detect lowconfidence generations @xcite has also been proposed, an approach very similar to one of the two methods used. ## experimental setup the first method is fully unsupervised, and therefore does not require calibration on the training set. the second method requires us to have a subset of labeled data to determine the score threshold. for evaluating our models, we used the metrics proposed by the organizers: accuracy, based on the labels and spearman's rho, based on the probabilities assigned to each entry. ## results the outputs on the test set model-aware track yielded a score of 0.483. the results on the validation dataset were the following: ## conclusions we have proposed two methods, the first based on using models pretrained on generated text detection, and the second based on looking at the confidence displayed by the llm under the form of logits. reviewing the results, we can assess that the tasks of generated text detection and hallucination detection showcase too large of a divergence for the approaches to generally be used interchangeably. however, the results on the paraphrase generation task may warrant further investigation into the use of models pretrained for text generation detection for the hallucination detection task. concerning the second method we have explored, it has showcased promising results on specific tasks in each track, which may warrant use in an ensemble method.