Comparative Quality Assessment of Human and Machine Translation with Best-Worst Scaling (2024)

Abstract: Translation quality and its assessment are of great importance in the context of human as well as machine translation. Methods range from human annotation and assessment to quality metrics and estimation, where the former are rather time-consuming. Furthermore, assessing translation quality is a subjective process. Best-Worst Scaling (BWS) represents a time-efficient annotation method to obtain subjective preferences, the best and the worst in a given set and their ratings. In this paper, we propose to use BWS for a comparative translation quality assessment of one human and three machine translations to German of the same source text in English. As a result, ten participants with a translation background selected the human translation most frequently and rated it overall as best closely followed by DeepL. Participants showed an overall positive attitude towards this assessment method.

## introduction human and machine translation quality and their assessment have been of importance in research and industry alike @xcite . different concepts in the field of translation studies include those focusing on preserving the purpose of the source text in the translation, such as the skopos theory @xcite , on the target text as central point in the analysis of quality as @xcite , or on pragmatic aspects of translation as @xcite . quality assessment (qa) approaches in the field of machine translation (mt) include qa frameworks for assessment by humans and by machines. very well-known automated metrics that compare candidate translations to reference translations are, for example, bleu @xcite , nist @xcite , and meteor @xcite . mt quality estimation @xcite represents a fairly new approach that instead of using reference translations trains machine learning models to predict the output quality of a specific mt system. human assessment of machine translation consists of human ranking @xcite , overall assessment @xcite or error classification @xcite and is generally considered subjective, time-consuming and therefore expensive. best-worst scaling (bws) @xcite ) is an annotation method that addresses these limitations, since it allows for subjective and time-efficient annotations. annotators are provided with n items in a set and are asked to select the best and the worst items from the set. with a set of four items, this simultaneously leads to a ranking with one clear best item, two that are better than the fourth, and a fourth worst item. bws has successfully been applied to annotating emotion intensities @xcite , evaluating stakeholder priorities in health matters @xcite , assessing consumer preferences in wine attributes @xcite , among many other application scenarios. bws has also been applied to assess genderfair language strategy preferences in translation @xcite . however, to the best of our knowledge, comparative translation quality assessment with bws has not been proposed before. in this paper, we propose a comparative analysis of five sets of four german translations of the same english source text, one human and three machine translations from google translate, deepl and microsoft bing translator. ten master students of translation studies or multilingual technologies selected the best and the worst option from the set and rated the best from +4 to 0 and the worst from 0 to -4 in an online survey. this rating provides an overall score for each translation method, but also allows for a more detailed analysis on how high or low each method is assessed. in contrast to ranking, e.g. @xcite , not each translation needs to be annotated with a rank label for each set, but only the best and the worst. furthermore, the agreement between choices and ratings can be directly assessed without having to calculate an inter-annotator agreement. the translations were selected across domains and consisted of one paragraph from non-fiction books, which required a comparatively low level of domain expertise from participants. in addition, participants were invited to leave comments on each set in free text fields and evaluate the overall method at the end of the survey. the results showed an overall positive attitude towards bws. since translation quality assessment by humans in itself is rather subjective, we believe that bws provides a viable, time-efficient and easy to implement alternative for comparing translations, which can be a comparison of mt systems, of human translations, or, as in this case study, to compare both. ## preliminaries as a basis for the study presented below, we provide an exemplary overview of selected work on mt quality assessment as well as combined assessment of human and machine translation, with no claims regarding completeness. in addition, we will briefly introduce the concept of bws and typical use cases. ## method in order to assess the translation quality of different machine translation systems as well as human translation, a combination of two methods for measuring subjective assessment was used: best-worst scaling (bws) and a likert scale. bws @xcite was used to select the subjectively best and worst translation, whereas the likert scale @xcite was used to rate the quality of the selected best and worst translations. these methods were used jointly in order to not only rate whether a specific translation was perceived to be the best or worst of a set, but also how high or low the selected translations are rated. ## results in this section, the participants' profiles, their bws ratings for the five sets of translations as well as the corresponding likert scale ratings will be presented, followed by an analysis of the free text answers and experience with the bws method. in total, the overall completion time for the entire survey ranged between 20 and 35 minutes. ## discussion the major objective of the proposed study was to evaluate whether the method of bws can effectively be applied to a comparative analysis of translation by humans and/or machines. the purpose of this case study was also to show that this method directly leads to a comparative result without the need of any further inter-annotator agreement calculations or scoring methods of the individual participating translation modes. overall, it can be stated that the results show a very clear preference for human translations and deepl from the set of selected mt systems. while a particularly unpopular result for a single set can influence the overall rating, the trends for the participating translation modes are still clearly visible from the final overall results. the one extremely negative rating for google translate and microsoft bing translator might have contributed to the overall negative rating, however, the fact that both were not selected as often as the other modes contributed just as much. this statement can be particularly reinforced by the fact that also the human translations ranged among the worst for particular sets. thus, even though individual sets might influence the final result, the overall tendency of being a viable or less preferable translation option can be deduced from the results. the decision to indicate to participants that there is a direct comparison between different translation modes, i.e., human and machine translation, is entirely open for the proposed method and could easily be adapted. human translations were overall selected and rated as the best option, however, it should be noted that each set contained a translation by a different professional translator. this can particularly be noticed by the strong differences in ratings across the sets, which, however, could have been the case with the same human translators for each set. nevertheless, in future work, it would be interesting to repeat the experiment with human translations from a single professional translator or maybe even two human translations in the individual sets, restricting the number of domains to specific sets of expertise. bws is considered a perfectly equipped annota-tion or prioritization method of subjective nature, which means each person can take a subjective decision. nevertheless, the overall results return tendencies, especially for translation quality, where at times the selection corresponds to a 100%. as indicated by the comments of the participants, a slight variation in wording or a divergence in the selection of just one word can already influence the decision on whether to select a translation option or not and as best or worst. the advantage of bws is that strong variations in one set still allow for a tendency and trend in the overall results in the end. without any further context on the topic, participants selected translations that are more faithful to the source text, which in many cases was one of the mt options. as a matter of fact, the lack of context is the most substantial limitation of this case study, limited source and target texts to less than 50 words. thus, it should be considered for future surveys how bws can be provided with more context without risking a cognitive and temporal overload of participants. as a method, it still provides a viable alternative to direct assessment with reference translations and ranking methods, especially considering the number of helpful comments left for this case study. in terms of limitations, it has to be acknowledged that the number and especially scope of evaluated source texts and translations is strongly limited. only five sets of individual paragraphs were evaluated in this study. in addition, only the language combination of translating from english to german was considered, which is in favor of training settings of major mt systems. furthermore, the number of participants was limited to 10. while this is a small number, it, however, shows that bws is adequate for different sizes of participation numbers. in this study, the objective was less to reach a wide audience but rather to make sure participants have a translation background and experience with quality assessment, in order to test this novel method and to obtain feedback on its efficacy and user-friendliness. in this regard, it is within human nature that it is easier to exclude an option we clearly like least, i.e., select the worst option, rather than identifying the best among four options, which is indicated by the ratings and comments of the participants at the end of the survey. while in this study students with translation backgrounds participated, it would be interesting to repeat this study with professional and experienced translators, in which case the domain should be limited to their respective expertise. nevertheless, in this case study, the level of required expertise and technical vocabulary was intentionally kept at a low level to facilitate participation by language rather than domain experts. ## conclusion the study showed that assessing translation quality with bws is an easy to implement and understand method, which can be successfully administered without lengthy explanations and returns interesting results. the two major benefits to be expected from bws for translation quality assessment are time efficiency and subjective decisions. even though the selected number and sizes of translations was small, the survey also only took between 20 and 35 minutes to be completed. while each participant made individual choices for each set, the subjective decisions still provide an overall tendency on which translation method and origin might be preferable for these domains and text genres. it is interesting that in the comments participants remarked on the fact that this is a highly subjective exercise, which, however, when evaluating the overall results is not negative. quite to the contrary, with bws and the rating of each selected best and worst option the results show that an effective and consistent comparison of translation quality can be achieved with this method. for future research we suggest using longer texts in order to provide more context for the mt systems, as well as to perform studies on a larger scale and with professional translators of more experience. in addition, it would be interesting to directly compare this quality assessment method with previously, state-of-the-art methods related to ranking and rating the overall quality of translation, be it machine and/or human, which is part of our future endeavours. in addition, it is interesting to see how this method can be applied to different types of methods related to machine and/or human translation, such as pre-editing, post-editing, and specific translation strategies. then be asked to rate the chosen options as well as to motivate your choice. please do not select the same translation for best and worst option and make sure to only select one option as best and exactly one as worst, which means two options will remain unrated. please find below an example of how it should be done (if there is one box selected in each of the columns, this is correct).