Can LLM s Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation? (2025)

Abstract: A frequently observed problem with LLMs is their tendency to generate output that is nonsensical, illogical, or factually incorrect, often referred to broadly as “hallucination”. Building on the recently proposed HalluciGen task for hallucination detection and generation, we evaluate a suite of open-access LLMs on their ability to detect intrinsic hallucinations in two conditional generation tasks: translation and paraphrasing. We study how model performance varies across tasks and languages and we investigate the impact of model size, instruction-tuning, and prompt choice. We find that performance varies across models but is consistent across prompts. Finally, we find that NLI models perform comparably well, suggesting that LLM-based detectors are not the only viable option for this specific task.

## introduction the introduction of large language models (llms) has revolutionised the field of natural language processing (nlp). state-of-the-art llms have demonstrated excellent language generation capabilities. in conversational ai @xcite , as well as strong performance on more specific nlp tasks like summarisation @xcite , open-domain question answering @xcite , sentiment analysis @xcite , and machine translation @xcite . despite this success, llms are prone to producing output that is fluent and grammatical, but semantically inadequate or factually incorrect, a phenomenon broadly referred to within the nlp community as "hallucination". the impact of hallucinations by llms may be severe in downstream applications where accurate output is mission critical, or where hallucination leads to erroneous decisions with negative consequences that directly impact humans e.g. in the medical or legal domain. in many cases, it may be infeasible to have a human in the loop, or it may be difficult for humans to identify hallucinations, which motivates the need for automated methods for detection and evaluation. in this paper, we aim to discover whether llms can be used to detect hallucinated content, focusing on a special case of what @xcite call intrinsic hallucinations, that is, cases where the output is deficient with respect to a particular input and where the deficiency can be detected given only the input and output. 1 more precisely, for the tasks of paraphrasing and machine translation, we define a hallucination to be an output, or hypothesis, that is not entailed by the input, or source. we build upon our previous work from the elo-quent lab at clef 2024 @xcite , specifically the hallucigen task, where we asked participants to apply llms to the task of detecting and generating hallucinations. we extend the work from the shared task with a series of experiments in prompting open-access llms to detect hallucinations, framing it as a contrastive challenge task: given a source sentence, and a pair of hypotheses, the model should detect which one contains a hallucination. we evaluate the llms on hallucination detection in paraphrase generation and translation, as defined in the hallucigen task @xcite . through a systematic investigation of model performance on the hallucination detection task, we address the following questions: two concepts that are often used to characterise different types of hallucinations are faithfulness and factuality. faithfulness means being consistent with a given source or input and has long been used as an evaluation criterion in conditional generation tasks like machine translation; a faithfulness hallucination is therefore any output that lacks such consistency, regardless of whether it is factually correct. by contrast, factuality means corresponding to real-world knowledge, and a factuality hallucination is therefore any output that makes a false claim, regardless of context and input. a related distinction is made between intrinsic and extrinsic hallucinations, where the former can be detected from the input and output of a system alone, while the latter requires more information @xcite . prior work has mostly focused on building systems to detect factuality hallucinations. for example, @xcite introduce a benchmark targeting cases of factual hallucinations in the context of question-answering, knowledge-grounded dialogue, and summarisation. aside from the halluci-gen task, the closest work to ours is the shroom shared task @xcite ) from semeval 2024. shroom defines hallucinations as cases when the hypothesis cannot be inferred from its semantic reference. despite the similarity with our definition, there is a significant difference in how the hallucinations are constructed. in shroom they are generated by models prompted to solve the specific task scenario, whilst we mostly construct hallucinations manually based on specific categories of errors; by switching gender, negation, or tense, replacing words with their antonyms, by substituting named entities, numbers, dates, and currencies, and by making superfluous additions. the two tasks also differ in terms of their coverage of nlp tasks and target languages. shroom includes the additional task of definition modeling; hallucigen covers an extra language for paraphrase but has limited coverage for machine translation. there is limited evidence so far on the effectiveness of using llms for detecting hallucinations. @xcite find that llms, including llama2 and chatgpt, perform poorly on the task of identifying hallucinations that have been generated by llms to be factually incorrect, in english questionanswering and summarisation. according to the hallucigen task results @xcite , gpt-4 and llm majority voting approaches outperform smaller english-centric models such as llama3-8b and gemma-7b. similar conclusions emerge from shroom, where submissions based on gpt-4 or model ensembling exhibit the strongest performance. model fine-tuning on shroom training data is another successful approach. conversely, textual entailment classifiers have been utilised for detecting faithfulness hallucinations. @xcite argue that textual entailment classifiers correlate with the faithfulness of summarised texts, making nli models a suitable candidate for automatic evaluation. textual entailment has also been applied to the evaluation of translations. @xcite address the issue of robustness in mt evaluation and propose a metric based on features motivated by textual entailment for "assessing the meaning equivalence between reference and hypothesis". similarly, @xcite developed a metric directly incorporating a textual entailment system, where a perfect translation pair would score highly in entailment in both directions (noting that omissions and additions can adversely affect entailment). @xcite compare the performance of both approaches by introducing selfcheck-gpt, which detects sentence-level hallucinations using generative llm prompting, llm probabilities, and nli models. interestingly, their experimental results show that llm prompting outperforms the nli-based method only by a small margin, and both outperform all other selfcheck-gpt methods and baselines. likewise, @xcite demonstrate that classifiers trained on mnli @xcite can perform well on factuality hallucination detection tasks. however, they are outperformed by similar classification models trained on a set of synthetically generated hallucinations (through sentence negation, entity swapping, and noise insertion), with the objective of classifying a source document and claim sentence as either "consistent" or "inconsistent". additionally, nli-based methods yield promising results for high-resource languages in multilingual setups, often outperforming other lexical metrics (like rouge), especially for intrinsic hallucinations where the hypothesis would clearly contradict the source @xcite . the ability of nli models to detect intrinsic hallucinations is arguably unsurprising as they must "handle phenomena like lexical entailment, quantification, coreference, tense, belief, modality, and lexical and syntactic ambiguity" @xcite to successfully predict entailment, contradiction, and neutral relations between sentence pairs. ## dataset description the hallucigen detection task @xcite covers the two following scenarios: each hallucinated hypothesis belongs to one of eleven categories, defined by the type of error or addition that breaks the entailment relation: addition, named-entity, number, conversion, date, gender, pronoun, antonym, tense, negation, natural. the last category refers to hallucinated responses by llms that did not fit into any of the other above categories. examples of each hallucination category for the paraphrase task can be found in ## results tables 2 and 3 present model scores for different prompt formulations and prompt languages in the paraphrase and translation scenarios. overall, we observe that performance varies considerably between models. we also note that the nli baseline is hard to beat, especially in the paraphrase scenario and for translation from french to english. this corroborates the findings of @xcite . ## discussion the results presented in section 5 support the use of llms, and also nli models, for the hallucination detection task. we now discuss the differences in performance across target languages as well as the effects of model size, instruction tuning, and the language and formulation of the prompts. ## conclusion we have presented a suite of experiments to investigate the capabilities of open-access llms for detecting hallucinations, as defined in the hallu-cigen task @xcite . the strongest models, mixtral-8x7b-instruct and meta-llama-3-70b-instruct, perform consistently well across all languages and scenarios, suggesting that llms are appropriate for 3 in swedish: "det är svårt att säga utan mer sammanhang." this task. the strong performance of the considerably smaller nli models suggests that llm-based detectors are not the only viable option. we analyse the effect of four different factors: target language, model size, instruction-tuning and prompt -and find that none of them can be used as a straightforward predictor of model performance on this task. our controlled experiments indicate that: (i) models perform consistently across languages, with a slight advantage for english; (ii) the impact of model size differs between model families; (iii) instruction-tuning has a clear positive effect only for the largest model; (iv) english prompts generally yield the best overall performance, while including the term "hallucination" in the prompt has a partially negative impact; and (v) for some models, a high error rate can be traced to the proportion of invalid outputs. we acknowledge the need for further investigation of these effects by systematically varying one factor at a time across different models. in future work, we aim to explore whether llms may be used to generate datasets for training and evaluating hallucination detectors and apply these in a cross-model evaluation setting. in addition, given the relatively strong performance of nli models in our experiments, it may be worth investigating whether other pre-existing techniques and metrics can be useful for detecting intrinsic hallucinations, including standard evaluation metrics for translation, paraphrasing and summarisation.