Understanding the Impact of UGC Specificities on Translation Quality (2021)

Abstract: This work takes a critical look at the evaluation of user-generated content automatic translation, the well-known specificities of which raise many challenges for MT. Our analyses show that measuring the average-case performance using a standard metric on a UGC test set falls far short of giving a reliable image of the UGC translation quality. That is why we introduce a new data set for the evaluation of UGC translation in which UGC specificities have been manually annotated using a fine-grained typology. Using this data set, we conduct several experiments to measure the impact of different kinds of UGC specificities on translation quality, more precisely than previously possible.

## introduction this work takes a critical look at the evaluation of user-generated content (ugc) automatic translation. the well-known specificities of ugc (high rate of oovs, rare, grammatical constructs, ...) raise many challenges for machine translation and has been the topic of many recent works @xcite . several ugc parallel corpora @xcite have been introduced to evaluate the robustness of mt, some of which, such as @xcite , are specially annotated to identify ugc idiosyncrasies allowing to measure the impact of a given specificity. our analyses ( §2), indeed, show that measuring the average-case performance using a standard metric on a ugc test set falls far short of giving a reliable image of the ugc translation quality: explaining the observed performance gap requires a particular evaluation framework made of tailored metrics and specific test sets in which ugc idiosyncrasies have been precisely annotated. that is why, following this line of works, we introduce pmumt, a new parallel data set for the evaluation of ugc translation between french and english in which ugc specificities have been manually annotated using a fine-grained typology. pmumt is larger, relies on a more refined error typology and, more importantly, its annotations are more detailed than existing noisy parallel corpora. its annotation scheme enables us to generate automatically parallel corpora in which the kind and number of ugc specificities are precisely controlled. contrary to many works studying the robustness of nmt systems by adding artificial noise to canonical corpora, pmumt is made of attested ugc examples. using this framework, we conduct several experiments on three out-of-the-box nmt architectures in a zero-shot scenario, to measure more precisely than what was possible before the impact of the different kinds of ugc specificities on translation quality. surprisingly enough, our experiments ( §3) on natural data show that out-of-the-box models exhibit unexpected strong robustness against several kinds of noise, questioning several results reported in the literature @xcite . we believe that this data set and its associated evaluation framework will pave the way for a better understanding of the interactions at play in neural machine translation of noisy user-generated content contexts. @xcite with english translations. these two data sets raise many challenges for mt systems: they notably contain characters that have not been seen in the training data (e.g. emojis), rare character sequences (e.g. inconsistent casing or usernames) as well as many oovs denoting url, mentions, hashtags or more generally named entities (ne). most of the time, oovs are exactly the same in the source and target sentences. ## analyzing the impact of ugc on translation quality in order to understand the impact of ugc specificities on translation quality, we have annotated a new corpus in which ugc peculiarities are identified in each source sentence and 'normalized' to a canonical form. ## conclusions this work introduces pmumt a new corpus of ugc translation in which ugc specificities are manually annotated using a fine-grained typology. thanks to our detailed annotation process, we were able to build a new framework that allows us to automatically generate several parallel corpora in which the number and kind of ugc specificities is precisely controlled. our experiments show that, contrary to what was previously believed, out-of-the-box nmt models are robust to many different kind of ugc specificities and that the different architectures we tested are complementary, in the sense that they are not sensitive to the same specificities. in our future work, we plan to explore the intricacies of the robustness that seem to be linked to specific ugc idiosyncrasies. we make this data set and its associated evaluation framework public foot_2 as we believe it can pave the way for a better understanding of the interactions at play in neural machine translation of noisy user-generated content contexts. as some of the data sets contain as few as 40 sentences, we have also computed the 95% confidence interval for all bleu scores in table a .3 using the bootstrapping method described in @xcite . the width of all intervals is smaller than 0.30 for the bleu scores (roughly 1% of the corresponding score) and than 0.006 for the ratios, which shows that we can trustfully compare their values. similarly, we have included results for the chrf @xcite and multi-bleu-detok.perl evaluation metrics since sacrebleu showed a ratio between performances on noisy and clean text versions, indicating that the noisy version could be easier to translate than its normalization. in this regard, we can see in table a .3 that at least one of the three reported metrics gives a ratio value equal or smaller than 1.0 within the 95% confidence interval error (ci err.), suggesting that our normalization introduce limited artificial noise, comparable to the difference between correlated evaluation metrics.