R o CS - MT v2 at WMT 2025: Robust Challenge Set for Machine Translation (2025)

Abstract: RoCS-MT (Robust Challenge Set for Machine Translation) was initially proposed at the test suites track of WMT 2023. Designed to challenge MT systems’ translation performance on user-generated content (UGC), it contains examples sourced from English Reddit, with manually normalised versions, aligned labelled annotation spans and reference translations in five languages. In this article, we describe version 2 of RoCS-MT in the context of the 2025 WMT test suites track. This new version contains several improvements on the initial version including (i) minor corrections of normalisation, (ii) corrections to reference translations and addition of alternative references to accommodate for different possible genders (e.g. of speakers) and (iii) a redesign and re-annotation of normalisation spans for further analysis of different non-standard UGC phenomena. We describe these changes and provide results and preliminary analysis of the MT submissions to the 2025 general translation task.

## introduction large language models (llms) have truly arrived in the field of machine translation (mt); their performance often rivals that of dedicated mt systems across various domains @xcite @xcite . however, while they have opened up new possibilities for translation, enabling fine-grained control of output formats, styles and formalities, they are also characterised by new types of errors that were less present with dedicated models, such as translating in the wrong language, inappropriate copying of the source text, generation of outputs that are not translations of the source text, etc. evaluation continues to be a highly important aspect of the field, and as mt technologies progress, so does the way in which we evaluate. the wmt shared tasks are a good example of this, with an evolution a few years ago to general translation instead of news translation @xcite , in order to challenge systems to translate a wider range of domains. one of the selected domains is social media content, known for covering a wide range of topics and containing non-standard language typical of user-generated content (ugc) @xcite @xcite @xcite . the translation of ugc has been a topic for a number of years @xcite @xcite @xcite . in particular, a shared task was organised on the matter at wmt in 2023 @xcite , designed to target non-standard language from reddit forums. several parallel test sets of ugc texts exist @xcite @xcite @xcite @xcite , for a range of languages, although they differ according to the language pairs covered and the degree of non-standardness present. in the 2023 edition of the test suite track at wmt, we proposed the rocs-mt test suite (robust challenge set for mt) @xcite , designed to contain particularly challenging sentences with respect to their non-standard nature (e.g. with spelling mistakes, use of acronyms, marks of expressiveness, devowelling, contractions, etc.). sourced from english reddit, we aimed to cover a range of these phenomena in the texts selected, which we manually segmented into sentences, normalised into standard english (according to guidelines that normalised as reasonably possible whilst preserving fluency and meaning) and then translated by professional translators into french, german, czech, ukrainian and russian. for this 2025 edition of wmt @xcite , we resubmit rocs-mt in an improved version (v2), after (i) some minor corrections to the source-side normalisations, (ii) corrections to the existing references and addition of multiple references to account for different genders and (iii) improvements to the annotations of the non-standard phenomena for additional analysis. we release this version in huggingface's datasets @xcite . 1 in this paper we describe those changes and provide results and analysis for the wmt2025 mt systems, with a major difference being that all systems this year were applied at the document level (at the level of reddit post text in out case). we compare different segmentations of the texts and the performance of systems when applied to the original and normalised source texts. changes with respect to the first version several changes were carried out in this second version of the test suite, namely: the new annotation types can be found below. detailed descriptions including examples can be found in appendix a. ## wmt 2025 submissions there were 56 submissions to the 2025 general task (including variants of the same systems and systems run by the general mt task organisers) that translated the test suite. a range of architectures were used, with a majority using llms. in a bid to encourage document-level translation, one of the important factors to be taken into account in mt evaluation @xcite , this year's general mt task focused on document-level mt, where documents were typically paragraphs of text. for the test suites, individual segments as provided in the test suite were concatenated to form source documents to be translated in one go. rocs-mt was provided in two different formats: (i) manual segmentation and (ii) segmentation purely based on newlines within posts. the language directions of the 2025 shared task overlap somewhat with the 2023 language directions (e.g. english to czech, chinese, japanese, ukrainian and russian), although not all target languages for which we have reference translations are present (e.g. french and german), and many of the language directions are new and therefore do not have references (e.g. arabic, bhojpuri, estonian, icelandic, italian, korean, massai, serbian). we therefore choose in our initial results and analysis (see section 4) to concentrate on quality estimation (qe) (i.e. without using the references for automatic analysis). we use two different metrics to calculate the scores (at the document level) used for the rankings: (i) cometkiwi (rei et al., 2022), foot_0 and (ii) metricx @xcite . 3 this is to avoid bias towards a single metric, especially as many systems optimise for a particular qe metric. we acknowledge however that (i) these metrics may well have issues handling certain languages, particularly those not explicitly included in the training data of the underlying models, and (ii) the scores may well favour models that optimise for qe in general, even if the same qe model is not used. ## results and analysis in this section, we present results for the submitted systems along with several brief analyses. in order to compute rankings, we took into account the fact that not all systems took part in every language pair, and adopted the ranking algorithm commonly used for nations in the olympic games, which addresses a comparable situation. we observed that if systems are ranked based on their absolute scores for each language pair, a few systems are consistently ranked first, meaning that most systems never achieve a rank of 1 or 2, even if they have relatively high overall scores. the result of this is that an overall weaker system that has low scores in most languages but happens to be ranked highly for a single language pair can be ranked higher than a system that has high but not the best scores across all language pairs. we therefore choose to apply the "gold first" algorithm on quartile-based rankings instead of raw rankings; for a given language pair, all systems within the top 10% of scores are assigned to rank 1 and treated as such by the "gold first" algorithm, the next 10% to rank 2, and so on. in other words, systems are then ordered according to the number of language pairs in which they achieve rank 1 (i.e. belong to the top 10%). in the event of a tie, the number of rank 2 placements (top 10-20%) is considered, followed, if necessary, by the number of rank 3 placements (top 20-30%), and so on. we applied this approach to get rankings from each of the two metrics used. these rankings are based on the original inputs (i.e. before normalisation) that have been manually segmented into sentences. we then computed overall rankings based on the average of the two rankings. results highlight a small group of systems that dominate performance. yolu occupies the top position with nine first-place results across all twelve language pairs on both metrics, followed closely by shy-hunyuan-mt and commanda-wmt. laniqo and salamandrata follow, despite the fact that laniqo participated in only seven pairs. among the organiser-run systems, gpt-4.1 ranks 7th another surprising observation is that several systems are positioned very differently in our two metric-specific rankings. the most extreme case is kazemt, ranked 52nd using cometkiwi but 9th using metricx. another example of a large ∆rank is gemini 2.5 pro, ranked 38th using cometkiwi but 7th using metricx. several single-pair submissions also display large discrepancies, such as yandex and systran, which both achieve first place according to one metric (metricx for yandex and cometkiwi for systran), but do not perform as well according to the other metric. such discrepancies could be explained, at least in some cases, in the way these models were trained or fine-tuned, for instance by optimising for qe, as mentioned above. ## conclusion we have presented a new version (v2) of the rocs-mt challenge set, first presented at the wmt 2023 test suites shared task track. this 2025 edition has several improvements, with minor corrections to source texts, some corrections to references and improved categorisation of non-standard phenomena. we describe these changes and also use the challenge set to compare systems submitted to this year's shared task, comparing translation from the original ugc inputs and their manually normalised versions. a major difference with previous years of the shared task is a switch to document-level mt, so whole chunks of posts were submitted to systems for translation. we nevertheless compare two different segmentation types (to see if initially manually segmenting into sentences and then concatenating the sentences with newlines could help translation) and discuss preliminary insights into the shortcomings of popular metrics such as cometkiwi and metricx when applied on nonstandard text mt. chhablani, bhavitvya malik, simon brandeis, teven le scao, victor sanh, canwen xu, nicolas patry, and 13 others. 2021. datasets: a community library for natural language processing. in proceedings of the 2021 conference on empirical methods in natural language processing: system demonstrations, pages 175-184, online and punta cana, dominican republic. association for computational linguistics. wang ling, guang xiang, chris dyer, alan black, and isabel trancoso. 2013. microblogs as parallel corpora. in proceedings of the 51st annual meeting of the association for computational linguistics (volume 1: long papers), pages 176-186, sofia, bulgaria. association for computational linguistics. paul mcnamee and kevin duh. 2022. the multilingual microblog translation corpus: improving and evaluating translation of user-generated text. in proceedings of the thirteenth language resources and evaluation conference, pages 910-918, marseille, france. european language resources association. paul michel and graham neubig. 2018. mtnt: a testbed for machine translation of noisy text. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 543-553, brussels, belgium. association for computational linguistics. hamdy mubarak, sabit hassan, and ahmed abdelali. 2020. constructing a bilingual corpus of parallel tweets. in proceedings of the 13th workshop on building and using comparable corpora, pages 14-21, marseille, france. european language resources association. lydia nishimwe, benoît sagot, and rachel bawden. 2024. making sentence embeddings robust to usergenerated content. in proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), pages 10984-10998, torino, italia. elra and iccl. jungsoo park, mujeen sung, jinhyuk lee, and jaewoo kang. 2020. adversarial subword regularization for robust neural machine translation. in findings of the association for computational linguistics: emnlp 2020, pages 1945-1953, online. association for computational linguistics. ben peters and andre martins. 2025. did translation models get more robust without anyone even noticing? in proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: long papers), pages 2445-2458, vienna, austria. association for computational linguistics. ricardo rei, josé g. c. de souza, duarte alves, chrysoula zerva, ana c farinha, taisiya glushkova, alon lavie, luisa coheur, and andré f. t. martins. 2022. comet-22: unbabel-ist 2022 submission for the metrics shared task. in proceedings of the seventh conference on machine translation (wmt), pages 578-585, abu dhabi, united arab emirates (hybrid). association for computational linguistics. josé carlos rosales núñez, djamé seddah, and guillaume wisniewski. 2019. comparison between nmt and pbsmt performance for translating noisy user-generated content. in proceedings of the 22nd nordic conference on computational linguistics, pages 2-14, turku, finland. linköping university electronic press. djamé seddah, benoit sagot, marie candito, virginie mouilleron, and vanessa combet. 2012. the french social media bank: a treebank of noisy user generated content. in proceedings of coling 2012, pages 2441-2458, mumbai, india. the coling 2012 organizing committee. henny sluyter-gäthje, pintu lohar, haithem afli, and andy way. 2018. footweets: a bilingual parallel corpus of world cup tweets. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018), miyazaki, japan. european language resources association (elra). vaibhav vaibhav, sumeet singh, craig stewart, and graham neubig. 2019. improving robustness of machine translation with synthetic noise. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1916-1920, minneapolis, minnesota. association for computational linguistics. rob van der goot, rik van noord, and gertjan van noord. 2018. a taxonomy for in-depth evaluation of normalization for user generated content. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018), pages 684-688, miyazaki, japan. european language resources association (elra). iñaki san vicente, iñaki alegría, cristina españa-bonet, pablo gamallo, hugo gonçalo oliveira, eva martínez garcia, antonio toral, arkaitz zubiaga, and nora aranberri. 2016. tweetmt: a parallel microblog corpus. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 2936-2941, portorož, slovenia. european language resources association (elra). haoran xu, amr sharaf, yunmo chen, weiting tan, lingfeng shen, benjamin van durme, kenton murray, and young jin kim. 2024. contrastive preference optimization: pushing the boundaries of llm performance in machine translation. in proceedings of the forty-first international conference on machine learning.