A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography (2023)

Abstract: For sensible progress in natural language processing, it is important that we are aware of the limitations of the evaluation metrics we use. In this work, we evaluate how robust metrics are to non-standardized dialects, i.e. spelling differences in language varieties that do not have a standard orthography. To investigate this, we collect a dataset of human translations and human judgments for automatic machine translations from English to two Swiss German dialects. We further create a challenge set for dialect variation and benchmark existing metrics’ performances. Our results show that existing metrics cannot reliably evaluate Swiss German text generation outputs, especially on segment level. We propose initial design adaptations that increase robustness in the face of non-standardized dialects, although there remains much room for further improvement. The dataset, code, and models are available here: https://github.com/textshuttle/dialect_eval

## introduction as multilingual nlp models include more and more languages, the community's focus on lowresource languages has also grown. this not only includes languages for which we have "little data" but also language varieties and dialects which often pose additional challenges, especially if they do not have a standardized orthography. recent work has shown some progress in classification tasks (e.g. @xcite @xcite as well as generation tasks where such language varieties appear on the input side only (e.g. @xcite @xcite . for these scenarios, we can use established evaluation schemes. however, for research towards nlp models generating language varieties, @xcite have shown that current evaluation metrics are not robust to translations into different dialects. gsw ... ufere webs ii te aa glueg e t w ä rd e . gsw ... ufere webs i te ah gluegt w e rd ä . de ... auf einer webseite angeschaut werden. en ... viewed on a website. what their evaluation does not consider is that language varieties often lack a standardized orthography and do not adhere to consistent spelling rules. this implies that even within a single dialect, notable orthographic variations can be observed, as illustrated in the swiss german example in figure 1 . the same utterance with a similar but different spelling would result in a high word error rate of 3 4 . many languages have multiple regional variants, such as spanish (mexican, argentinean, etc.), french (canadian, belgian, etc.), or english (british, american, australian, indian, etc.), among others. such language varieties exhibit various lexical, grammatical, and orthographical distinctions. importantly, these differences are standardized, meaning that they adhere to specific spelling rules and conventions, albeit with variations specific to each variant. this suggests that if a neural metric is exposed to a sufficient amount of data encompassing various language varieties, it should be able to develop similar representations and provide comparable scores for a given sentence in different varieties. @xcite show that pre-training a metric on data from multiple dialects indeed makes metrics more inter-dialect robust. however, for a substantial number of languages and language varieties, there exists no established standard orthography. many regions exhibit a dialect continuum where language varieties lack precise boundaries, and each dialect displays a significant range of diversity within itself. furthermore, when speakers write in their dialect, they follow their individual writing styles. such kinds of variabilities, as can be observed in the example in figure 1 , are much less consistent and localized and will differ significantly between different writers. a metric designed to handle these kinds of varieties must be capable of addressing frequent spelling differences, which is considerably more challenging to learn solely from data compared to the standardized language variation differences mentioned in the previous paragraph. in recent years, embedding-based metrics have gained increasing popularity @xcite which -in theory -could be more appropriate for assessing non-standardized language varieties than string-based mt metrics like bleu @xcite or chrf @xcite . however, these neural metrics are often not trained on the language varieties in question. additionally, recent work showed that reference-based learned metrics still rely too much on subword overlap with the reference @xcite . in this work, we follow @xcite and analyze the dialect robustness of machine translation metrics but specifically focus on non-standardized language varieties that were not seen during pretraining. our contributions are: ## related work there is a substantial amount of research on mt into language varieties @xcite @xcite @xcite @xcite @xcite . most of these works exclusively evaluate with surface-level metrics like bleu @xcite but some voice their concerns over a lack of reliable evaluation metrics @xcite . @xcite confirm that existing machine translation evaluation metrics are not dialect-robust. they show that it is possible to train more robust metrics by including a language and dialect identification task in a second language model pre-training phase. while they focus on inter-dialect robustness between well-defined dialects, i.e. brazilian and iberian portuguese, our study focuses on a setting where dialects lack standardized orthography. this absence of standardization introduces additional variability, resulting in distinct challenges and necessitating different solutions for mt systems, which need to generalize to often limited data; mt metrics, which need to be robust to spelling differences; and also meta-evaluation, which has its own challenges when collecting human assessments for dialects without standardized orthography as we outline in section 3.1. to investigate how reliable mt metrics are for nonstandardized varieties, we collect a new dataset with human translations and human judgments for mt outputs from english to two swiss german dialects. while other works also evaluate mt metrics on language varieties and dialects, @xcite is closest to our work: alam et al. ( 2023 ) only look at language varieties on the source side and @xcite only evaluate language varieties for which a standard was included in the language model pretraining. both studies also conclude that existing metrics are not robust to dialects. @xcite further propose a new automated lexical accuracy metric based on term dictionaries, similar to metrics used for automatic speech recognition (asr) @xcite which allow for more flexible string matching by using a look-up table of acceptable spellings. riley et al.'s approach may work well if there is a limited set of term differences between dialects. however, such a metric is difficult to employ for language varieties without standardized spelling rules. instead, we experiment with increasing dialect robustness by introducing character-level noise during metric training which has been shown to be useful for cross-lingual transfer to language varieties without standardized orthography @xcite @xcite . ## evaluation data for swiss german dialects while we focus on swiss german because there are enough different mt systems that can be eval-uated, swiss german is by no means the only language where its varieties do not have standardized spelling. many medium to high-resource languages like arabic @xcite or italian @xcite include dialectal varieties that lack a standardized orthography. additionally, this phenomenon extends to numerous low-resource settings @xcite , encompassing a wide array of language varieties across africa @xcite , asia @xcite , oceania @xcite and the americas @xcite . historically, even many language varieties that now have a standardized orthography did not always have one, including english @xcite . this makes our work on robust metrics for nonstandardized dialects also relevant for nlp for historical texts. to measure robustness against non-standardized dialects, we design two new datasets. with the first, we investigate how metrics behave in a realistic setup where we compare them against human judgments. the second is a challenge set that allows us to investigate score changes between different spellings and compare them to score changes when meaning is changed. this is inspired by similar experiments in @xcite . ## open questions we hope that our benchmark inspires more work on robust evaluation metrics for language varieties in the future. in this section, we list several directions we think are worthwhile exploring: expanding the benchmark: we were not able to include additional language varieties in our benchmark at the time because we could not find enough different machine translation systems that translate into these varieties. while we recognize that without reliable metrics this is a "chicken-and-egg" problem, we still advocate for more mt research that focuses on translating into language varieties. expanding our benchmark would not only allow us to draw more general conclusions but would also help with sample size for the pairwise accuracy analysis @xcite since we find that a large number of systems are required for confident results. more focus on segment level: segment-level metric scores tend to be much less correlated with human judgments when contrasted with systemlevel correlations @xcite and have also been shown to be unreliable in downstream tasks @xcite . we hope that future work aimed at enhancing metric performance on our challenge set will also contribute to greater metric reliability on segment level in general, as over-reliance on reference overlap is also a problem for languages with standardized spelling @xcite . training neural metrics that model characterlevel similarities: a segment in a dialect often resembles a reference in certain characters only rather than in full words (see figure 1 as an example). as the underlying language models of neural evaluation metrics use a fixed tokenization scheme that was learned on text that likely does not include many examples of language varieties, these similarities might be hard to account for by the neural metric. thus, we believe that character-based language models, such as canine @xcite , could provide a better basis for neural evaluation metrics to model character-level similarities. ## conclusion we evaluated the reliability of machine translation metrics when evaluating dialects without standard orthographies. as part of this work, we collected a new dataset consisting of human translations, human judgments, and a challenge set from english to two swiss german dialects. we benchmark several existing metrics and find that they are not robust to variation featured by non-standardized dialects. based on this finding, we explore several modifications that allow us to train metrics that are more robust towards spelling variation. our results show that there is still a lot of room for improvement and we offer a set of recommendations for future work on dialect robust metrics.