OTTAWA : Optimal T ranspor T Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection (2024)

Abstract: Recently, there has been considerable attention on detecting hallucinations and omissions in Machine Translation (MT) systems. The two dominant approaches to tackle this task involve analyzing the MT system’s internal states or relying on the output of external tools, such as sentence similarity or MT quality estimators. In this work, we introduce OTTAWA, a novel Optimal Transport (OT)-based word aligner specifically designed to enhance the detection of hallucinations and omissions in MT systems. Our approach explicitly models the missing alignments by introducing a “null” vector, for which we propose a novel one-side constrained OT setting to allow an adaptive null alignment. Our approach yields competitive results compared to state-of-the-art methods across 18 language pairs on the HalOmi benchmark. In addition, it shows promising features, such as the ability to distinguish between both error types and perform word-level detection without accessing the MT system’s internal states.

## introduction concerns regarding hallucination (as known as fabrication) of machine translation (mt) systems @xcite @xcite have led to considerable efforts towards the creation of diagnostic datasets @xcite @xcite and developing detection methods @xcite . in addition to hallucination errors, which occur when words in the translation are detached from the source sequence, addressing omission errors is also crucial-these are cases where words from the source sequence are absent in the translation. figure 1 : a hallucinatory german-english translation was detected by correctly identifying the null alignments with our optimal transport-based method. here, if a target word is mapping to "null", it is likely to be hallucinated, as no source word supports its translation. existing methods for detecting hallucinations and omissions in mt either focus on analyzing the model's internal states (e.g. cross-attention) or rely on the output of external tools, such as crosslingual similarity estimators @xcite or mt quality estimators @xcite . on one hand, internal methods are limited to scenarios where white-box access to the mt system is assumed, and are further limited by dependencies on specific model architectures (e.g. attention-based). on the other hand, external methods heavily rely on scalar outputs from models optimized for related tasks, such as mt quality or sentence similarity, but not directly optimized for hallucination detection. while hallucinations and omissions constitute low-quality translations, the opposite does not always hold. furthermore, despite their effectiveness in detecting hallucinations and omissions, existing internal and external methods cannot distinguish between these two error types. this is because they frame the task as anomaly detection, relying on a single outlier score to detect both error types. cross-lingual word alignment @xcite aligns source to target tokens in parallel sentence pairs, and it provides unique insight for translation errors like hallucination and omission. following the definition by @xcite , hallucinated translation happens when the generated content is largely detached from the source text; therefore, we can measure hallucination by the amount of "misaligned" words in the translated text. similarly, an omission error is likely if a significant amount of source words are not found in the translation. despite the strong relevance between word alignments and translation errors, alignment tools have not yet been widely applied for hallucination and omission detection. in particular, we observe that existing alignment approaches @xcite @xcite are often based on the assumption that the translation is correct, such that most of the source or target words can be aligned with each other. this assumption, however, does not hold for scenarios where translation errors have occurred; hence, the direct application of existing word alignment tools does not work well for detecting translation errors. in this paper, we utilize optimal transport problem @xcite and propose ot-tawa, an optimal transport adaptive word aligner that is specifically tailored to aligning words for the pathological cases where significant hallucination and omission errors are present. our approach explicitly models the missing alignments by introducing a "null" vector. specifically, we propose a novel one-side constrained optimal transport (ot) problem, which allows the null vector to be adaptively aligned to any number of words. moreover, we set the null vector as a centric point, which is not biased during the computation of the ot problem. we show an example of our ot-based method in figure 1 . we conduct extensive experiments using the newly proposed halomi benchmark @xcite , designed for hallucination and omission detection, across a total of 18 language pairs. results show that our word aligner-based approach, equipped with ottawa, performs on par with state-of-the-art internal and external detection methods @xcite . in contrast to previous approaches, we show that our method can distinguish between hallucination and omission errors, while ottawa being crucial for performances compared to using existing word aligners. ## related work hallucination remains a persistent concern in machine translation systems, which have recently gained increased attention in the field. @xcite studied token-level hallucinations in machine translation on a chinese-english dataset, which they constructed by randomly corrupting some tokens in the translation sentences. guerreiro et al. (2023c ) developed a dataset of 3.4k naturally occurring german-english translations, manually annotated at both the sentence and token levels for hallucinations and omissions detection. halomi @xcite expanded the language coverage for this task by manually annotating hundreds of translation pairs across 18 language pairs. methods for detecting hallucinations and omissions in machine translation can be broadly categorized into two types: internal, which analyzes the translation model's own outputs and states, and external, which relies on additional tools or data outside the model itself @xcite . @xcite developed a method that employs various optimal transport-based distances @xcite to evaluate the abnormality in a machine translation (mt) model's internal cross-attention distribution of a given translation, by making comparisons with those from a collection of high-quality translations. this method not only requires access to the internal states of the mt model but also necessitates a large collection of cross-attention maps from ground truth parallel sequences for the targeted language pair. @xcite investigated the use of other internal mt model components, such as length-normalized sequence log-probability (seq-logprob) and the token-token layer-wise interaction framework (alti) foot_0 , as alternative abnormality estimators that do not require external translation data. in addition, the authors suggested directly using scores from external mt quality estimator models, such as comet-qe @xcite , or employing the cosine similarity between the source and translation as generated by cross-lingual sentence similarity models, including laser @xcite , labse @xcite . word alignment @xcite ) has been extensively studied over the years in both cross-lingual @xcite and monolingual @xcite settings. the task is approached with supervised learning methods @xcite , which require sentence-level parallel data, as well as unsupervised methods that do not rely on such data. the latter approach involves leveraging token-token attention matrices @xcite @xcite or pairwise similarity matrices between tokens' embedding vectors @xcite , both extracted from a multilingual pretrained language models @xcite . the application of optimal transport (ot, monge, 1781; @xcite @xcite in natural language processing tasks was mainly focused on similarity estimation between textual segments @xcite @xcite , and more recently for word alignment @xcite . ## method we formulate the unsupervised word alignment task in § 3.1, and then provide an overview on how standard and partial optimal transport are employed to address the task in § 3.2 and § 3.3, respectively. in § 3.4, we describe our newly proposed optimal transport word aligner, ottawa, specifically adapted to enhance the detection of mt hallucinations and omissions. ## conclusion and future work in this work, we propose a new word alignmentbased approach for detecting hallucinations and omissions in mt systems. we develop ottawa, an innovative word aligner designed specifically for this purpose. while experiments show promise in mt hallucination and omission detection, this area remains an intriguing direction for future research exploration. we plan to focus on one-tomany alignments for pathological translations.