Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models (2024)

Abstract: Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates sentence-level hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs.

## introduction text generation models have drastically improved in recent years especially with the capabilities of llms in producing realistic and fluent output. however, hallucination continues to undermine user trust, as it generates and propagates misinformation and sometimes nonsensical outputs @xcite @xcite . this issue is especially critical in highstakes domains like medicine and law, where hallucinations in medical texts can result in harmful 1 data and code are available on github. misunderstandings about diagnoses or treatment instructions, and inaccuracies in legal contract translations can lead to severe financial or legal consequences. one practical way of reducing hallucination in mt is by building more robust models, especially for lrl which tend to exhibit significantly higher hallucination rates. there are several efforts on scaling mt models to lrls, such as m2m-100 @xcite , @xcite , madlad-400 @xcite etc. despite initiatives to minimize hallucinations during the mt process, issues still persists. therefore, detecting hallucinations post-translation remains a critical alternative approach to ensure the reliability and trustworthiness of the translated content. previous work on post-translation evaluation has primarily focused on general translation errors, with evaluation scores often under-representing the impact of hallucinations due to their relatively low frequency compared to less severe errors like omissions @xcite . studies on hallucinations have mainly concentrated on english-centric (en) to hrl translation direction, while research involving non-english lrls remains limited @xcite . for instance, sentence similarity measures between source and translated texts using crosslingual embeddings, such as laser @xcite and labse @xcite , have proven effective at identifying severe hallucinations @xcite , though their limitations with lrls are often overlooked @xcite . recent studies have highlighted the capabilities of llms in multilingual mt evaluation, demonstrating strong performances across various languages, although discrepancies remain for lrls @xcite . @xcite demonstrated that, when properly prompted, llms can assess the quality of machine-generated translations, achieving state-of-the-art results in system-level evaluation for hrls. furthermore, @xcite pioneered the use of llms for mt tasks in lrls with the introduction of the automqm prompting technique, though this approach primarily targets broader translation errors rather than focusing on hallucination detection. recently, @xcite introduced halomia benchmark dataset for detecting hallucination in mt that includes en↔hrls (ten directions) and en↔lrls (six directions), as well as two non-english directions hrl↔lrl, including different scripts. blaser-qe @xcite , the state-of-the-art (sota) hallucination detector, is reported as the top performer on the halomi benchmark. it calculates a translation quality score by evaluating the similarity between encoded source texts and machine-translated texts within the sonar embedding space @xcite . in this paper, we evaluate the performance of llms and embedding based methods as hallucination detectors, aiming to enhance performance in both hrls and lrls. to this end, we use the halomi benchmark dataset with a binary sentencelevel hallucination detection approach. for our evaluation, we include 14 methods: eight llms with different prompt variations, and four embedding spaces by computing the cosine similarity between source and translated texts. we find that llms are highly effective for hallucination detection across both high and low resource languages, although the optimal model selection depends on specific contexts. for hrls, on average across directions, the llama3-70b model significantly surpasses the previous sota method, blaser-qe, by 16 points. moreover, embeddingbased methods have also demonstrated superior performance over the current sota in high resource contexts. however, for lrls, claude sonnet is the best performing model, improving previous methods by a smaller difference. more precisely, llms outperformed blaser-qe in five out of eight lrl translation directions, including the non-english-centric ones. finally, our research makes the following primary contributions: first, we evaluate a wide range of llms for mt hallucination detection and establish that llms, despite not being explicitly trained for the task, are competitive and greatly outperform even the previous sota for hrls. second, large multilingual embedding spaces improve upon previously proposed methods and show that they remain competitive for hrls, but struggle for lrls. third, we establish a new sota for 13 of the 16 languages that we evaluate on, including high and low resource languages. surpassing the previous sota, which was explicitly trained for the task, on average by 2 mcc points. ## results llms are the new sota for hallucination detection the results in figure 2 and figure 15 demonstrate that llms have the best overall performance across languages for binary hallucination detection. specifically, llama3-70b surpasses the previous best performing model, blaser-qe, by +5 points, with an mcc of 0.43. for hrls, 10 out of 12 evaluated methods outperform blaser-qe (0.46), with llama3-70b greatly improving over the baseline by 16 points (0.63). notably, the results show that the choice of llm should rely on the resource level; as for lrls, claude sonnet achieves the highest average mcc. however, gpt4o was the more robust llm across all languages, with the lowest standard deviation. finally, for 13 out of the 16 evaluated translation directions, the evaluated methods outperform blaser-qe, with the exception of @xmath0, @xmath1. our findings on llms' superior hallucination detection capabilities align with prior research on their effectiveness in mt quality assessment @xcite . embedding-based hallucination detectors remain competitive for hrls for hrls, simple embedding-based methods display competitive capabilities, outperforming more sophisticated models in five out of eight translation directions. for instance, although blaser-qe is a more advanced model based on sonar, sonar exhibits comparable or superior performances in most hrls directions. this suggests that the effectiveness of these methods may be highly sensitive on their training data, and hence to the resource level, as we observe sota performances for hrls and subop-timal results for lrls. additionally, the embeddings' performance may be highly dependent on the threshold chosen using the en↔de validation set, generalizing well for hrls but not for lrls. llms' contrastive performances across lrls first, while llama3-70b obtains the best performance overall, it was outperformed in most translation directions, especially in lrl. this result reveals a hrls-centric approach of the model but also concludes that there is not one-llm fits all resource levels. secondly, for lrls, models such as sonnet, opus, gpt4o, and mistral -in order of decreasing performances, achieve higher scores, supporting the feasibility of employing llms in settings encompassing a wide range of languages. these results should be contrasted with a wide difference of hallucination distribution across resource levels, for example with the @xmath2. more precisely, sonnet and blaser-qe perform on par for lrl, with the particularity that blaser-qe has a significantly higher rate of false negatives, while sonnet maintains a more balanced ratio of false positives to negatives. moreover, blaser-qe performs well in translations from english and comparably to sonnet in translations to english, but falls short in non-english-centric translations, which follows the same trends as previously reported models in @xcite . figure 2 and figure 15 provide a more detailed view of these performance metrics. embeddings are high performers for non-latin scripts, while llms can generalise to non-english centric translations for @xmath3,ru,zh), embeddings are the best performers, suggesting high capabilities with cross-script transfer learning. these observations align with the findings of @xcite , who report decreased performance for non-latin scripts in llm-based evaluators. in the two non-english centric translation directions (es↔yo), opus outperforms by far both blaser-qe (0.11) and the best embedding mistral (0.12), with a score of 0.28. unlike the overall lrls trends, opus outperforms sonnet for this direction pair: this can suggest that the advanced analytical capabilities of llms can generate improved results even in scenarios with limited relevant training data. remarkably, in the @xmath4, six out of our fourteen methods and blaser-qe exhibit scores close to random guessing (within the [-1, +1] range). this observation underscores the pressing need for enhanced capabilities in detecting hallucinations in non-english-centric translation settings. figure 1 presents two examples that highlight the challenges faced by llms when dealing with non-latin scripts, with the exception of llama3-70b. additionally, it illustrates how embeddings may struggle with reasoning capabilities in non-english centric contexts. ## conclusion in this work, we demonstrates that llms and embedding semantic similarity are highly effective for hallucination detection in machine translation, with llms establishing a new state-of-the-art performance across both high and low-resource languages. our findings suggest that the optimal model selection depends on specific contexts, such as resource level, script, and translation direction. our study highlights the practical advantage of reference-free models like llms, which allow realtime hallucination detection without relying on external knowledge @xcite . however, there remains a critical need for further research to improve hallucination detection, particularly in low-resource and non-englishcentric translation settings.