QR ev: Machine Translation of User Reviews: What Influences the Translation Quality? (2020)

Abstract: This project aims to identify the important aspects of translation quality of user reviews which will represent a starting point for developing better automatic MT metrics and challenge test sets, and will be also helpful for developing MT systems for this genre. We work on two types of reviews: Amazon products and IMDb movies, written in English and translated into two closely related target languages, Croatian and Serbian.

## description data sets used for mt research include mainly "formal written text" (such as news) and "formal speech" (such as ted talks). recently, there has been an increase of interest in the translation of "informal written text" which focuses on very noisy texts originating from sources like what-sapp, twitter and reddit. on the other hand, other types of "informal written text" such as user reviews have not been investigated thoroughly, although they are important both from commercial and from a user perspective -user reviews of products have become an important feature that many customers expect to find. this project focusses on user reviews in order to investigate which new challenges this "mid-way" kind of text poses for current mt systems. the main goal is to identify important quality aspects for mt of user reviews which will enable: although the focus of the project are user reviews translated into serbian and croatian (as a case involving mid-size less-resourced morphologically rich european languages), the proposed evaluation strategy is completely genre/domain/language independent, so it can be applied to any genre, domain and language pair. ## data sets we are working with two types of publicly available user reviews: ## mt systems the main goal of the project is to find the common aspects important for the translation quality, and not to evaluate or compare particular mt systems. we are currently analysing mt outputs foot_2 of three on-line systems: google translate foot_3 , bing foot_4 and amazon translate foot_5 . we are also developing our own system using publicly available data, which will be analysed in the later stages of the project. ## evaluation procedure our evaluation procedure is based on comprehensibility and fidelity (adequacy) @xcite , and it is being carried out on the review level (not on the sentence level). it should be noted that comprehensibility is not fluency -a fluent text can be incomprehensible, and vice versa. the novelty of our procedure is asking the annotators to concentrate on problematic parts of the text and to mark them, without assigning any scores or classifying errors. the procedure can also be guided by other evaluation criteria, not only comprehensibility and adequacy. the annotators were computational linguistics students and researchers fluent in the source language and native speakers of the target language. the annotation consisted of two independent subsequent tasks with the following guidelines: comprehensibility a monolingual task without access to the original source language text. which parts of the translated review are not understandable? distinguish two levels: "completely incomprehensible" and "not fully clear due to grammatic or stylistic errors". ## first results: inter-annotator agreement and percentage of issues inter-annotator agreement (iaa) is shown in