CODET : A Benchmark for Contrastive Dialectal Evaluation of Machine Translation (2024)

Abstract: Neural machine translation (NMT) systems exhibit limited robustness in handling source-side linguistic variations. Their performance tends to degrade when faced with even slight deviations in language usage, such as different domains or variations introduced by second-language speakers. It is intuitive to extend this observation to encompass dialectal variations as well, but the work allowing the community to evaluate MT systems on this dimension is limited. To alleviate this issue, we compile and release CODET, a contrastive dialectal benchmark encompassing 891 different variations from twelve different languages. We also quantitatively demonstrate the challenges large MT models face in effectively translating dialectal variants. All the data and code have been released.

## introduction progress in natural language processing (nlp) and other varieties of human language technology throughout the 2010s has been undeniably swift. however, such advances are limited to a set of languages with largely available resources @xcite ; they have focused solely on dominant, "standard" language varieties. but no language is a monolith; languages vary richly across countries, regions, social classes, and other factors 2 . for modern linguae francae such as english, spanish, or french, some commercial systems apply coarse localization, e.g., google assistant supports speech recognition for english in at least seven locales. 3 this, however, is not the case for standard italian variant: source: hanno rubato il quadro gtranslate: they stole the painting ✓ alassio variant: source: i han rubbau u quaddru gtranslate: i han rubbau u quaddru ✗ the majority of the world's languages, even if they exhibit large variations across dialects and regions, often corresponding to millions of speakers. as a result, we have a limited understanding of how well modern nlp systems can handle (or not) such data. it is crucial that we first quantify such disparities in as many languages as possible before we explore ways of mitigating any performance imbalances we identify. language variants can vary along several dimensions. in this work, we focus on the robust understanding of lexical and morphosyntactic variations, which show up in the written form of languages and hence can be evaluated through a downstream task like text-based machine translation. if one wanted to capture phonological variation additionally, one should work directly on audio and tasks like automatic speech recognition or speech translation; we leave this vein of work for the future. consider the case study presented in to properly evaluate such inadequacies in the context of machine translation, one needs contrastive examples between varieties so that the evaluation metrics are comparable. our work attempts to fill this gap. in summary, our contributions are: • we extract contrastive data from previous dialectology studies in three languages: italian (439 locales), basque (39 locales), and swiss german (368 locales); • we re-purpose contrastive data from various sources in seven languages: arabic (25 vernaculars), occitan (2 varieties), tigrinya (2 varieties), farsi (2 varieties), malay-indonesian (2 varieties), swahili (2 varieties), and greek (1 variety); • we create a limited amount of contrastive data in additional languages: bengali (5 varieties) and central kurdish (4 varieties). ## related work mt is one of the most studied and pioneering tasks in the nlp realm. many previous studies have focused on proposing more efficient methods, particularly with recent advances in sequence-to-sequence models @xcite , attention mechanism @xcite , and transformers @xcite that have left their impact on other tasks in nlp as well. although creating mt models for languages around the globe has received much attention, as in flores-200 benchmark and no language left behind (nllb) models @xcite , we have a considerable stretch remaining to create models that can translate dialects and varieties efficiently. most of the previous work on developing mt technologies for dialects and varieties address arabic @xcite , swiss german @xcite , @xcite , portuguese @xcite and @xcite . in this regard, one of the main challenges is finding possible translation sources and creat-ing corpora and datasets for the translation of varieties and dialects @xcite . in the same vein, exploring the translation of varieties in a few-short or zero-shot setting has received attention @xcite . similarly, fine-tuning translation models trained on closely related languages has been proposed as a remedy @xcite . given that there is currently no benchmark for the existing data on mt of dialects and varieties, our paper aims to provide one with the sole objective of evaluating varieties and the performance and resilience of mt models to dialectal variations. we also believe this work will increase awareness of this task and motivate future efforts. ## the codet benchmark given a sentence in one dialectal variant and another in the standard variant of the same language as in given that little has been done in this vein, we focus on creating constructive datasets following three approaches, namely repurposing previous dialectological work on syntactic variations for basque, italian, swiss german, and central occita; manual translation by native dialect speakers for bengali, modern greek, central kurdish; and finally, exploiting some existing resources for arabic, farsi, malay-indonesian, tigrinya, and swahili. new data creation for a handful of languages, namely central kurdish, bengali, griko, and occitan, we did not find any existing dialectal contrastive data, but we were able to construct small evaluation benchmarks by online data scraping (occitan) and by reaching out to native speakers and translators of these varieties (for the others). ## evaluation to assess the quality of any mt system on dialectal variations, it is crucial to compare its outputs with a reference standard. one approach is to have a gold, human-created translation representing the desired translation in a standard setting. among the twelve languages considered, we only have gold translations for basque, bengali, farsi, central kurdish, malay-indonesian, swahili, tigrinya, and aranese. for the rest, we will need to be able to evaluate mt robustness without references. evaluating without references our goal is to evaluate the robustness of mt systems concerning dialectal variation. while access to human-created gold translations can certainly reveal a complete picture of the model's performance, thankfully, it is not a hard requirement. in this work, we adapt the ideas of michel and neubig (2018) and @xcite which presented frameworks for evaluating the robustness of mt systems to adversarial or non-native noisy inputs. concretely, consider the following notation: ## results and analysis preliminaries for all language varieties, we benchmark mt systems in the x-to-english direction. the choice of english as a target language is a pragmatic one. still, a more comprehensive evaluation should consider many other target languages for future work, especially since we do not require gold references to perform our analyses. we present baseline results in all languages using four different-sized nllb-200 (nllb team et al., 2022) models using the huggingface @xcite the comet evaluation framework relies on xlm-roberta (conneau et al., 2020), a multilingual language model, to generate embeddings for each token in the input source, machine-translated (mt) sentence, and reference sentence. however, since xlm-roberta was trained on texts of the standard dialect, the quality of the embeddings created for source sentences in different dialectal variants may be compromised. to investigate this, an ablation study was conducted with and without the source sentence as input to the comet scorer. figure 1 presents the results of this ablation study for 13 basque dialects. the dialectal sentences were translated to english using the nllb-200-dis-600m model. the blue bars represent comet scores when the source sentences were replaced with blank sentences, while the orange lines represent comet scores when the source sentences were included. in all cases, the comet scores decrease when the source sentences are introduced, supporting the initial hypothesis. the general trends are very similar with and without using the source sentence. based on these findings, the source sentence will not be used to ensure more reliable evaluations for all subsequent comet calculations in this paper. ## conclusion this study compiles a benchmark of contrastive examples between standard and dialectal variants of twelve languages to facilitate the evaluation of mt systems' robustness along this variation. our findings demonstrate that mt systems excel at handling standard variants, but as the dialectal varieties start differing from the standard, the quality of the translations declines. this work emphasizes the need for further research and development in dialectal mt. excluding a significant portion of the population from the benefits of language translation cannot be considered a satisfactory solution, underscoring the importance of addressing dialectal variations within mt systems. future work this study highlights the unequal support for different language dialects in mt systems. some dialects exhibit impressive comet scores due to their close relationship with the standard variant. however, this work primarily focuses on creating a dataset to assess the performance of various dialects rather than conducting experiments to enhance the mt system's robustness. this limitation primarily stems from the scarcity of training data. the datasets created for this study are relatively small and mainly serve as test data. for future research, the mt community needs to prioritize the development of training datasets for dialects. several strategies can be explored with an adequate dataset, such as dialect-specific adaptation through fine-tuning or adapter approaches.