Evaluating Machine Translation for Emotion-loaded User Generated Content ( T rans E val4 E mo- UGC ) (2024)

Abstract: This paper presents a dataset for evaluating the machine translation of emotion-loaded user generated content. It contains human-annotated quality evaluation data and post-edited reference translations. The dataset is available at our GitHub repository.

## introduction machine translation (mt) technology has developed so rapidly in recent years that some claimed to have achieved human parity in chinese-english news translation @xcite . different from news translation, automatically translating user generated content (ugc) has revealed additional challenges for mt systems including handling slang, emotion, literary devices such as irony and sarcasm @xcite . this is particularly prominent in chinese social media texts, as various homophones are used to replace offensive words to avoid censorship @xcite . to evaluate how mt systems perform on emotion-loaded ugc, we collected chinese microblog texts, and employed google translate 2 (gt) to translate them to english. trained annotators were recruited to directly evaluate translation quality in terms of emotion preservation (through error annotation). professional translators were hired to post-edit the gt outputs to produce reference translations. post-edited translations can be used to compare with the mt outputs and to showcase the high-quality translations achievable by human translators. the human evaluation process © 2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. 1 https://github.com/surrey-nlp/hadqaet for data & licence 2 https://translate.google.co.uk/ on the 30th of may, 2022 was funded by the university of surrey. the postediting activity was funded by the european association for machine translation (eamt) through its 2022 sponsorship of student activities. ## data description this project delivered a dataset comprising 5538 instances of chinese microblog texts (source), their machine-translated english versions, information on human-annotated errors and quality evaluation scores (qeval information), and postedited translations (reference). ## conclusion to our best knowledge, this dataset is the first open-sourced chinese-english resource in the mt area that includes human-annotated translation er-rors, words that cause the errors, quality evaluation scores in terms of emotion preservation, postedited reference translations, and emotion labels. we believe it is valuable for the evaluation of translation quality for emotion-loaded ugc, and for the training of new mt systems.