R o CS - MT : Robustness Challenge Set for Machine Translation (2023)

Abstract: RoCS-MT, a Robust Challenge Set for Machine Translation (MT), is designed to test MT systems’ ability to translate user-generated content (UGC) that displays non-standard characteristics, such as spelling errors, devowelling, acronymisation, etc. RoCS-MT is composed of English comments from Reddit, selected for their non-standard nature, which have been manually normalised and professionally translated into five languages: French, German, Czech, Ukrainian and Russian. In the context of the WMT23 test suite shared task, we analyse the models submitted to the general MT task for all from-English language pairs, offering some insights into the types of problems faced by state-of-the-art MT models when dealing with non-standard UGC texts. We compare automatic metrics for MT quality, including quality estimation to see if the same conclusions can be drawn without references. In terms of robustness, we find that many of the systems struggle with non-standard variants of words (e.g. due to phonetically inspired spellings, contraction, truncations, etc.), but that this depends on the system and the amount of training data, with the best overall systems performing better across all phenomena. GPT4 is the clear front-runner. However we caution against drawing conclusions about generalisation capacity as it and other systems could be trained on the source side of RoCS and also on similar data.

## introduction as the quality of state-of-the-art machine translation (mt) systems is becoming indistinguishable in certain scenarios and domains from that of human translators @xcite , the task of tackling the translation non-standard texts is becoming an increasingly realisable aim. a considerable proportion of texts produced today are done so online in informal, unedited settings, e.g. on forums such as twitter and reddit, and mt is frequently to make posts accessible to a global audience. however, it has been shown that mt still struggles with usergenerated content (ugc) @xcite , as the type of language can differ considerably from the edited texts that have traditionally been used to train and evaluate mt models. the rocs-mt challenge set (robust challenge set for machine translation) is designed to provide a test bed for the automatic translation of nonstandard ugc phenomena. it contains approximately 2k sentences from the online forum reddit that have been manually normalised and professionally translated into five languages: french, german, czech, ukrainian and russian. the sentences were selected specifically for the presence of non-standard phenomena, of which we provide manual annotations (e.g. spelling errors, devowelling, capitalisations, acronymisms, etc.). inspired by other datasets such as the french social media bank @xcite and its parallel component @xcite , our aim is to provide an evaluation set that is more challenging than certain previous efforts, such as the commonly used mtnt dataset @xcite . we also make different choices from most previous efforts concerning the guidelines for normalisation and translation of the source sentences. we choose to first normalise the source sentences before translation in order to optimise the quality of the translation and to reduce the arbitrariness that may be introduced when transferring non-standard variation to the target language (e.g. on which characters to apply spelling errors, how many characters to duplicate when elongating words). for normalisation, we aim to strike a balance between normalisation as much as possible while making sure that the normalised text remains natural. 1 in this paper, we describe the creation of the challenge set, and in the context of the wmt23 test suite shared task, we analyse the models submitted to the general mt shared task for the from-english shared task language pairs: @xmath0, german, hebrew, japanese, russian, ukrainian, chi-nese} (@xmath1, @xcite . through automatic and manual analysis of system outputs, we find that many of the phenomena remain challenging for most systems (in particular those that create potential out-of-vocabulary or rare words such as phonetically inspired spellings, contractions, devowelling and truncation). however, the difficulty varies depending on the phenomenon, the particular instance (notably how frequent the nonstandard word is) and the system, especially with respect to the quantity of training data. the highest performing systems overall generally do better across the board on all phenomena, whereas the weaker systems struggle in particular with certain phenomena. gpt4 has a clear lead over other systems, correctly translating even some of the most challenging examples and sometimes (although inconsistently) reproducing non-standardness in its outputs. however conclusions are limited given that the training data is unknown (as is the case of other unconstrained systems). we make the challenge set, system outputs, evaluation code and guidelines (for the normalisation, annotation and translation) openly available for research purposes. 2 2 related work several parallel ugc datasets exist across different language pairs. while some are extracted automatically from crawled data @xcite @xcite , a majority are based on monolingual sentences that are then translated into the target language @xcite @xcite @xcite . the closest to our rocs-mt dataset are @xcite and @xcite , which were designed to contain challenging non-standard phenomena, whereas many of the existing datasets do not apply any such filter. like rocs-mt, the mtnt dataset @xcite ) contains texts from reddit. to target non-standard language, they select sentences that have a low probability using a language model trained on standard data. in practice, and as shown by rosales @xcite , the amount of non-standard language remains limited with this method. rosales @xcite base their parallel dataset on the french social media bank dataset @xcite , which targets nonstandard language by searching for specific nonstandard keywords. they show that this leads to a higher level of non-standard language, although the method is by nature more biased towards the keywords and phenomena used for data selection. an error analysis of the dataset was conducted in @xcite , showing mt quality (using bleu) for different ugc phenomena. despite significant effort to describe and classify ugc phenomena @xcite , there is no consensus as to how texts should be normalised (and indeed translated). one extreme is to normalise all phenomena to standard forms, as is often done in lexical normalisation tasks @xcite , but which in several cases would lead to unnatural outputs (e.g. if lol and lmao, were systematically normalised to laughing out loud and laughing my ass off ). this makes translation difficult too, as the translations would also be unnatural. at the other end of the spectrum is the choice to not normalise source texts and in addition to attempt to translate the phenomena into the target language, with the disadvantage that some phenomena are language-specific 3 and others would result in arbitrary decisions being made such as to which characters to apply spelling errors. the current datasets targeting particularly non-standard phenomena choose to at least in part transfer some phenomena to the target language, whereas we adopt a higher degree of normalisation (see section 3.1.1 for more details), producing standard but naturalsounding translations. ## translation systems in this article, we evaluate the systems submitted to the general translation task at wmt2023. there are both constrained and unconstrained systems, the two settings presenting significant differences in training data that should be taken into account when comparing systems. constrained systems constrained systems followed similar strategies, with many systems doing data filtering/cleaning and data augmentation, using either bilingual or multilingual models and reranking. the constrained systems submitted were airc @xcite , anvita, cuni-transformer and cuni-doctransformer (popel, 2020) (we refer to these system as cuni-trans and cuni-doctrans to save space in the results tables), cuni-ga @xcite , hw-tsc @xcite , iol_research @xcite , naist-nict @xcite , sam-sung_research_philippines (cruz, 2023) (hereafter samsung_rp), skim @xcite and uva-ltl @xcite . unconstrained systems as in previous years of the shared task, translations were produced from anonymised online systems, corresponding in this addition to online-{a,b,g,m,w,y} submissions. this year, translations from gpt4 were also produced using 5 few-shot examples (gpt4-5shot). 11 note that caution should be taken when comparing results from gpt4, given that it is very possible that source sentences from rocs-mt are included in gpt4's training data. two systems based on nllb @xcite were also submitted in the context of the metrics shared task: nllb_greedy and nllb_mbr_bleu (hereafter nllb_mbr), which both rely on the same model but differ by the decoding strategy, either standard (greedy) or based on the minimum bayes risk strategy @xcite . a number of unconstrained systems were also submitted by participants, namely lan-bridgemt (wu and hu, 2023), kyb , gtcom (zong, 2023), @xcite , promt @xcite , yishu @xcite and zenghuimt @xcite . ## evaluation and analysis evaluation of ugc translation is more challenging than standard text; a correct translation can either be standard or non-standard in the target language, and there may be multiple ways of being non-standard that may not all be covered by available references. in our case, we chose to produce standard reference translations (see section 3.1.2). any system that produces non-standard language may therefore be underestimated using referencebased metrics. we test three different metrics (bleu, comet and comet-qe) to evaluate the systems' translations of rocs-mt, looking at how coherent they are between each other, and whether it is possible to use quality estimation to evaluate mt robustness in order to remove the need for reference translations (section 5.1). we also look at the mt quality of each system per phenomenon by calculating comet scores over subsets of the data. finally, we perform a qualitative analysis, manually looking at how the different systems handle ugc phenomena, and confirming some of the trends using some simple automatic analyses (section 5.2). ## conclusion we have presented a new resource, rocs-mt, a robustness challenge set for mt, designed to test mt systems on non-standard ugc. our automatic and manual analysis show that non-standard texts are still a problem for many of the systems, including the unconstrained ones, and that certain phenomena such as phonetically inspired spellings pose a problem in particular. the comparison of comet and comet-qe metrics suggest that it may be possible to draw similar conclusions from automatic scoring without using references, although future work could go into more depth into analysing what is captured by the different metrics.