Translation Errors Significantly Impact Low-Resource Languages in Cross-Lingual Learning (2024)

Abstract: Popular benchmarks (e.g., XNLI) used to evaluate cross-lingual language understanding consist of parallel versions of English evaluation sets in multiple target languages created with the help of professional translators. When creating such parallel data, it is critical to ensure high-quality translations for all target languages for an accurate characterization of cross-lingual transfer. In this work, we find that translation inconsistencies do exist and interestingly they disproportionally impact low-resource languages in XNLI. To identify such inconsistencies, we propose measuring the gap in performance between zero-shot evaluations on the human-translated and machine-translated target text across multiple target languages; relatively large gaps are indicative of translation errors. We also corroborate that translation errors exist for two target languages, namely Hindi and Urdu, by doing a manual reannotation of human-translated test instances in these two languages and finding poor agreement with the original English labels these instances were supposed to inherit.

## introduction multilingual benchmarks, such as xnli, xtreme, play a vital role in assessing the cross-lingual generalization of multilingual pretrained models @xcite . typically, these benchmarks involve translating development and test sets from english into different target languages using professional human translators. however, such a translation process is susceptible to human errors and could lead to incorrect estimates of cross-lingual transfer to target languages. we find translation errors do emerge and they disproportionately affect human translations from english to target language, translated back to english machine translations from english to target language, translated back to english translations in certain low-resource languages such as hindi and urdu. 2 consider the well-known cross-lingual natural language inference (xnli) benchmark @xcite that contains human translations of english premise-hypothesis pairs (with the labels reproduced from english) into 14 typologicallydiverse target languages. prior work raised concerns about whether the semantic relationships between premise and hypothesis are preserved in such human translations, but did not probe into this issue further @xcite . we find that there are indeed errors introduced in the human translations leading to label inconsistencies and that this issue disproportionately affects lowresource languages. to visualize the impact of low-quality translations on low-resource languages, figure 1 compares zero-shot xnli performance on all 14 target languages using the xlmr model @xcite finetuned on english nli with the following two input types: 1. human translations of the original english nli instances to the target language from xnli, translated back to english. 2. machine translations of the original english nli instances to the target language, translated back to english. we see a clear differential trend with larger gaps between the (scores over the) two input types for low-resource languages such as swahili, urdu and turkish (appearing on the right) and smaller gaps for high-resource languages such as spanish, german and french (appearing on the left). we also observe that the cross-lingual transfer gap when comparing the performance of human-translations for each target language with that of english (the latter shown as a dotted line) is largely overestimated for low-resource languages. to summarize, our main contributions are: ## human evaluation for two low-resource languages hindi and urdu, we reannotate a subset of the human-translations with nli labels and check how well they match the labels inherited from the original english text. we pick random, non-overlapping sets of 200 instances each in english, hindi and urdu and get them relabelled by native speakers. (appendix f provides more annotation details.) the new labels matched the original labels 90.5%, 66.5% and 60% of the time for english, hindi and urdu, respectively. this clearly highlights the large drop in label agreement for hindi and urdu compared to english, with relative reductions of 24% and 30.5% for hindi and urdu, respectively. in @xcite , the same experiment was conducted using english and french and the original labels were recovered 85% and 83% of the time, respectively. the authors concluded there was no loss of information in the translations. however, we find there to be a significant loss of information in translations for languages such as hindi and urdu. to verify if machine translations (te) (rather than xnli's human translations (orig)) align better with the labels from the original english, we relabel 200 instances translated from english to hindi and urdu (via google translate). the annotators recovered the ground-truth labels 80% and 71% of the time for hindi and urdu, respectively, highlighting that label inconsistencies in hindi/urdu human translations (orig) are significantly worse than with machine translations (te). ## attention-based analysis we assess how the attention distributions learned for xnli over the english test instances correlate with the attention distributions learned for humanannotated hindi/urdu/swahili test instances and google-translated (english to) hindi/urdu/swahili test instances. for each correctly predicted english instance, we consider both human-translated (ht) and machine-translated (mt) target language translations and compute word alignments between english and these translations using awesomealign @xcite . aligned words whose attention score is greater than the mean attention score for the sequence are counted and normalized by the total number of such words in a sequence. finally, we compute an average over all these overlap fractions across instances in the dataset. these mean overlap scores shown in ta- ## impact of using translations for multilingual datasets while he said they were done speaking. ## related work there is growing interest in building multilingual benchmarks for the evaluation of crosslingual transfer. e.g., xtreme @xcite covering a wide range of languages and tasks including xnli @xcite , xquad @xcite , paws-x (yang et al., 2019) and mlqa @xcite . recently, many extensions of xtreme: indx-treme @xcite focusing on 18 indian languages, xtreme-r @xcite and xtreme-up @xcite have also been released. translation artifacts have only been studied in select prior works. @xcite study how translations can alter sentiment labels in arabic text. in very recent work, @xcite advocate for the use of english-only finetuning using machine-translation systems. however, this relies on high-quality human translations in the target languages which we highlight needs to be carefully examined especially for low-resource languages. ## conclusions this work studies the problem of translation irregularities in evaluation sets of multilingual benchmarks like xnli that are created by translating english into multiple target languages. we find that the translation sets of low-resource languages like urdu, swahili exhibit most inconsistencies while translations of high-resource languages like french, german are more immune to this problem. we suggest an effective way to check the quality of human translations by comparing performance with machine translations, and show how the cross-lingual transfer estimates can significantly vary with improved translations.