H al O mi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation (2023)

Abstract: Hallucinations in machine translation are translations that contain information completely unrelated to the input. Omissions are translations that do not include some of the input information. While both cases tend to be catastrophic errors undermining user trust, annotated data with these types of pathologies is extremely scarce and is limited to a few high-resource languages. In this work, we release an annotated dataset for the hallucination and omission phenomena covering 18 translation directions with varying resource levels and scripts. Our annotation covers different levels of partial and full hallucinations as well as omissions both at the sentence and at the word level. Additionally, we revisit previous methods for hallucination and omission detection, show that conclusions made based on a single language pair largely do not hold for a large-scale evaluation, and establish new solid baselines.

## introduction with neural machine translation systems reaching an overall satisfactory quality, alleviating those rare but severe translation pathologies that undermine user trust becomes very important. these pathologies include hallucinations (translations containing information completely unrelated to the input) and omissions (translations that do not include some of the information of the input). while understanding hallucinations is receiving increasing attention @xcite @xcite @xcite , progress in this direction is hampered by the lack of annotated data. to the best of our knowledge, previous datasets are limited to german-english data with sentence-level annotations of hallucinations and omissions @xcite and chinese-english data with token-level hallucination labels @xcite . previously available general-purpose quality assessments, such as direct assessment (da) graham et al. (2013), mqm @xcite , or xsts @xcite do not seem suitable since they do not distinguish between hallucinations, omissions and other translation errors. in this work, we aim to address this limitation. ideally, an evaluation dataset for hallucination/omission detection should satisfy several conditions: (i) data has to cover a broad range of languages with varying resource levels and scripts, (ii) translations should be generated naturally, (iii) the models that produced the translations have to be available, and (iv) considered modeling paradigms have to cover several approaches (i.e., encoder-decoder vs decoder-only, single language pair vs multilingual, etc.). the first point is important because the best-performing detectors are different for high-and low-resource settings, and general conclusions cannot be made based on a single language pair (section 4). secondly, translations have to be generated naturally as opposed to using specifically developed perturbations of models and/or data because conclusions for the latter might not transfer for detection of natural hallucinations (section 6). thirdly, the corresponding model should be released along with the translations to allow evaluating "internal" detection methods. fi-nally, in the most ideal setting, various models are needed to test whether the detection methods transfer between modeling approaches. while satisfying all the desiderata is very challenging, we can satisfy all but last by focusing on the state-of-the-art multilingual nllb-200 model @xcite . in addition to covering a broad range of languages and being publicly available along with its training data, nllb is widely recognized foot_0 and is likely to stay the stateof-the-art for the foreseeable future. for this model, we choose 18 language pairs that include highand low-resource languages, as well as a zero-shot pair (figure 1 ). we develop rigorous annotation guidelines for identifying full and partial hallucinations and omissions and use these guidelines for manual annotation of translations in all 18 directions. the resulting dataset contains fine-grained sentence-level and token-level annotations. we highlight the importance of our dataset by making several valuable observations that would not be possible otherwise. for example, we find that for low-resource directions, internal methods perform much better than external methods that substantially fail. when analyzing performance of a range of recently introduced pathology detection methods, we see that some of the previous results do not transfer across languages. as another example, we show that relying on attention to make conclusions about translation quality is very fragile. finally, we introduce some detection tasks (e.g., token-level omission detection) that became possible only with our data. we believe our work opens the door for reliable and accessible research on detecting and analyzing translation pathologies as well as understanding their causes. overall, we: ## dataset creation the steps to create the dataset were (i) choosing the language pairs, (ii) gathering data for annotation, (iii) developing annotation guidelines and qualification sets, (iv) manual annotation, (v) postprocessing. here, we explain these steps. ## dataset description annotation format. the resulting data contains the source text and its translation, along with the word-level and sentence-level annotations of omissions and hallucinations. figure 3 shows examples of annotated translations from our dataset. overall statistics. figure 1 shows the proportions of hallucinations and omissions in the data (translations with both hallucinations and omissions are referred to as hallucinations). overall, all directions have at least 3% translations with hallucinations (1% full) and 17% with omissions (5% full). most of the full hallucinations are also labelled as full omissions, and vice versa. differences between resource levels. from figure 1 we see that, not surprisingly, high-resource language pairs hallucinate less than low-resource. a less anticipated difference between high-and low-resource settings is seen when looking within each language pair. in high-resource settings, translating to english leads to more hallucinations than translating from english. differently, for lowresource pairs, translation from english has higher hallucinatory rates than translation to english for the same language pair. this might inspire future work to analyze the role of english data in the multilingual nllb model. finally, while for the zeroshot pair one might expect more pathologies, this is not what we see: results for the zero-shot pair are comparable to those for low-resource languages. ## sentence-level detection detecting pathologies at the sentence level is the task of flagging a whole translation as pathological or not. this is the standard definition of e.g. the hallucination detection task @xcite @xcite @xcite @xcite . such sentence-level pathology detection (instead of flagging individual erroneous tokens) is an integral part of hybrid pipelines when a machine-generated translation is first passed to a quality estimation system and then, if needed, is corrected by human translators. detection tasks. for our dataset, we define three sentence-level detection tasks: evaluation methodology. we evaluate the ability of a detector to rank more severe pathologies higher (e.g., full hallucinations higher than partial, any hallucinations higher than non-hallucinations, etc). for this, we use an adaptation of the binary roc auc score for several classes. formally, we subtract from the perfect score, i.e. 1, the percentage of incorrectly ranked pairs of sentences with different labels. for two classes, this metric is equivalent to the roc auc score. we compute the metrics for each translation direction separately. ## word-level detection in contrast to sentence-level detection, detecting pathologies at the word level received much less attention. in terms of both available data and detectors, previous attempts were rather limited @xcite . here, we want to facilitate future research in this direction. detection tasks. we define two detection tasks: segmentation. we segment texts using sacre-bleu tokenizer @xcite . for chinese, it interprets each chinese character as an individual word. for other languages, it applies regex-based tokenization based on spaces and punctuation. evaluation methodology. for these binary classification tasks, we use the roc auc score. since models operate at the token level, we make predictions for tokens and not words. if a word is segmented into several tokens, we assign the worst score among its tokens (i.e., if one of a word's tokens is hallucinated, the entire word is hallucinated). ## natural vs "artificial" pathologies one of the difficulties when dealing with hallucinations (and, to a lesser extent, omissions) is that this is a rare phenomenon. therefore, previous work often resorted to artificially amplifying the problem by applying various perturbations @xcite ; raunak et al. ( 2021 ), among others). however, it is not clear whether conclusions made based on this synthetic data would transfer to pathologies generated by a model in a natural setting. in appendix d, we compare performance of detection methods between two datasets: (i) our dataset with natural translations and (ii) translations generated with perturbed model. we find that data with translations generated under perturbation has to be used with caution, especially when evaluating pathology detection methods: the conclusions are likely to not transfer to the natural setting. ## discussion we saw that some of the internal and external methods can detect hallucinations and omissions with the quality that is much better than nothing, much worse than perfect. but what are the cases in which methods perform well and what can be improved? ## additional related work except for mentioned above work, previous work on hallucinations in machine translation largely avoided human annotation. to judge whether a translation is hallucinated, they relied on various heuristics or string-based automatic evaluation metrics @xcite ; @xcite ; @xcite ; @xcite ). these, however, were shown not to be indicative of hallucinations @xcite , which highlights the importance of our human-annotated data. for omissions, previous work mostly focused on empty translations @xcite with some work using artificially created undertranslations @xcite . as we saw in section 6, the latter is unlikely to be helpful when evaluating detection methods. ## conclusions we present the first dataset with human-annotated hallucinations and omissions that satisfies several conditions: (i) it covers a broad range of languages with varying resource levels and scripts, (ii) the translations are generated naturally (i.e., without artificial perturbations), (iii) the model producing the translations is publicly available. through our extensive experiments, we illustrate why each of these conditions is important. additionally, we make several observations of individual importance. for example, for low-resource directions internal pathology detection methods perform better than most of the external ones, attention is very fragile when used to judge translation quality, among others. we believe our work opens the door for a reliable and accessible research on detecting and analyzing translation pathologies.