Expanding the WMT 24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader (2025)

Abstract: The Romansh language, spoken in Switzerland, has limited resources for machine translation evaluation. In this paper, we present a benchmark for six varieties of Romansh: Rumantsch Grischun, a supra-regional variety, and five regional varieties: Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader. Our reference translations were created by human translators based on the WMT24++ benchmark, which ensures parallelism with more than 55 other languages. An automatic evaluation of existing MT systems and LLMs shows that translation out of Romansh into German is handled relatively well for all the varieties, but translation into Romansh is still challenging.

## introduction the automatic evaluation of machine translation (mt) has been widened in recent years to cover more languages and language varieties. while massively multilingual benchmarks such as flo-res @xcite or ntrex @xcite include reference translations in hundreds of languages, no dedicated reference translations for the romansh language have been available so far. in this paper, we close this gap by extending the recent wmt24++ benchmark @xcite with reference translations for six varieties of romansh, using german as the source language. there are several reasons why romansh, which is a language from the romance family spoken in switzerland (iso 639-1: rm; iso 639-2/3: roh), has had limited resources for mt evaluation. first, romansh is considered a minority language, with 40,000-60,000 speakers @xcite . secondly, multiple varieties of romansh need to be considered for a comprehensive evaluation. rumantsch grischun is a supra-regional variety of the language, often used in official contexts. however, the five regional varieties of ro- puter sursilvan vallader puter surmiran sutsilvan s w i t z e rl an d ita ly figure 1 : distribution of romansh idioms (regional varieties) within south-eastern switzerland. the map shows municipalities where an idiom is officially used in public administration. we extend the wmt24++ benchmark with sets of reference translations for these five idioms, as well as rumantsch grischun, a supraregional variety of romansh. mansh (figure 1 ), usually referred to as idioms, are more widely spoken in everyday life, with limited mutual intelligibility @xcite . prior work on mt for romansh @xcite has leveraged multilingual government press releases @xcite , blog posts or federal laws, all of which cover only the rumantsch grischun variety. our benchmark based on wmt24++ enables a more systematic evaluation setup that includes the five idioms and a broader range of domains, such as social media and transcripts of youtube videos. we release our benchmark under the apache 2.0 license. 1 in addition, we use the benchmark to perform a systematic evaluation of mt systems and llms on german-romansh and romansh-german translation. results based on automatic evaluation metrics indicate that translation into german achieves reasonable quality for all romansh ## validation experiments we perform two automatic validation experiments to confirm that the reference translations are suitable for variety-specific evaluation: language classification we use a fasttext classifier @xcite trained on a corpus of romansh newspaper articles that were manually labeled with their variety. ## evaluation of mt systems and llms on translation from and into romansh we use our benchmark to evaluate the following machine translation systems and llms: rumantsch grischun. foot_3 ## conclusion this work fills a long-standing gap in the evaluation of machine translation for the romansh language: the creation of a benchmark for the six main varieties of romansh, and the provision of baseline results for existing mt systems and llms that cover romansh.