Assessing Human-Parity in Machine Translation on the Segment Level (2020)

Abstract: Recent machine translation shared tasks have shown top-performing systems to tie or in some cases even outperform human translation. Such conclusions about system and human performance are, however, based on estimates aggregated from scores collected over large test sets of translations and unfortunately leave some remaining questions unanswered. For instance, simply because a system significantly outperforms the human translator on average may not necessarily mean that it has done so for every translation in the test set. Firstly, are there remaining source segments present in evaluation test sets that cause significant challenges for top-performing systems and can such challenging segments go unnoticed due to the opacity of current human evaluation procedures? To provide insight into these issues we carefully inspect the outputs of top-performing systems in the most recent WMT-19 news translation shared task for all language pairs in which a system either tied or outperformed human translation. Our analysis provides a new method of identifying the remaining segments for which either machine or human perform poorly. For example, in our close inspection of WMT-19 English to German and German to English we discover the segments that disjointly proved a challenge for human and machine. For English to Russian, there were no segments included in our sample of translations that caused a significant challenge for the human translator, while we again identify the set of segments that caused issues for the top-performing system.

## introduction recent results of machine translation evaluation shared tasks indicate that state-of-the-art is now achieving and possibly even surpassing human performance, with the most recent annual conference on machine translation (wmt) news task provid-ing extensive human evaluation of systems, concluding that several systems performed on average as well as human for english to russian, english to german and german to english translation and a top system even surpassed human performance for the last two language pairs. since 2017 the official results of the wmt news tasks have been based on the human evaluation methodology known as direct assessment (da) @xcite , due to its many advantages over older technologies. da, for example, includes quality control mechanisms that allow data collected anonymously from crowd-sourced workers to be filtered according to reliability. 1 although wmt news task results are admittedly based on substantially more valid methodology than those usually found in general in system comparisons using automatic metrics such as bleu, results in wmt human evaluations still leave some questions unanswered. for example, da scores are based on average ratings attributed to translations sampled from large test sets, and although such methodology does allow application of statistical significance testing to identify potentially meaningful differences in system performance, they do not provide any insight into the reasons behind a significantly higher score or the degree to which systems perform better when translating individual segments. furthermore, da score distributions produced in the human evaluation of the news task are based on individual da scores that alone cannot be relied upon to reflect the quality of individual segments @xcite . past work, has however provided a means of running a da human evaluation in such a way that da scores accurately reflect the performance of a system on a given individual segment @xcite . this method comes with the trade-off of requiring substantially more repeat assessments per segment than the test set level evaluation generally run, for example, to evaluate all primary submissions in the wmt news task. in this work we demonstrate how this method has the potential to be employed as a secondary method of evaluation in wmt tasks for a smaller subset of systems to provide segment-level insight into why the top-performing systems outperform one another or indeed to investigate the degree to which human and machine performance differs for individual segments. ## related work over the past number of years, machine translation has been biting at the heels of human translation for a small number of language pairs. beginning with the first claims that machines have surpassed human quality of translation for chinese to english news text, conclusions received with some skepticism and even controversy @xcite , as claims of human performance resulted in re-evaluations that scrutinized the methodology applied, highlighting the influence of reverse-created test data and lack of wider document context in evaluations @xcite . despite re-evaluations taking somewhat more care to eliminate such sources of inaccuracies, they additionally included some potential issues of their own, such as employing somewhat outdated human evaluation methodologies, non-standard methods of statistical significance testing and lack of planning evaluations in terms of statistical power. @xcite , on the other hand re-run the evaluation, identify and fix remaining causes of error, and subsequently confirm that, on the overall level of the test set, with increased scrutiny on evaluation procedures, conclusions of human parity were still overly ambitious at that time. it was not long before results were shown to have reached human performance however, according to more scrutinous human evaluation procedures, as one year later at wmt 2019, mt system performance for some language pairs reached human performance and even surpassed it for two language pairs @xcite . although the admittedly rigorous human evaluation employed in wmt evaluations provides valid conclusions about systems significantly outperforming human translation, it nonetheless em-ploys the somewhat opaque average direct assessment scores computed over large test sets of segments that subsequently leave some important questions unanswered in terms of human parity. for example, even if a system performs better on average than a given human translator, this does not necessarily mean that the system translates every sentence better than the human translator. when a tie occurs between human and machine translation, it would be useful to know how performance compares between the two on individual segments. the current wmt human evaluation methodology does not allow for this, however. in this paper, we carry out fine-grained segmentlevel comparison of system and human translations using human evaluation and provide a comparison on the segment-level of the top-performing mt systems from wmt-19 news task and the human translator for all language pairs in which a system was shown to either tie (english to russian) or surpass human performance (english to german; german to english). human evaluation is required, as opposed to segment-level bleu, for example, because metrics such as bleu are not sufficiently accurate to identify fine-grained segment-level differences in quality, as can be seen from low correlations with human assessment @xcite . we make all code and data collected in this work publicly available to aid future research. foot_2 ## segment-level direct assessment segment-level direct assessment requires running human evaluation with sampling of translations carefully structured to ensure that repeat assessment of the same set of translations occurs a minimum of 15 times for both the translations produced by the systems of interest @xcite . for example, this can be carried out for a reduced number of translations and for a reduced number of systems than the entire test set, since collecting 15 repeat assessments makes exhaustive segment-level evaluation for every participating system likely to be overly costly. it is reasonable to focus the segment-level evaluation on a sample of approximately 500 translations selected at random for the two top-performing systems or indeed, as we do now, the top-performing system and the human translator. an important consideration however, is that regardless of which systems may be selected for fine-grained segment-level analysis, segment-level evaluation should be run for precisely the same set of segments for all systems of interest so that a comparison of the performance of systems on the same segments will ultimately be possible. the desired number of source language segments should therefore be sampled at random from the test set before pooling target side translations for the systems of interest, shuffling and arranging them within human intelligence tasks (hits). we construct hits of 100 translated segments, subsequently evaluated by humans blind to which system has produced each translation, or as in our case, blind to whether a human or machine produced the translated segment. the configuration of da we employ is a source-based segment-level evaluation in which human assessors are (i) shown the source language input segment; (ii) the translated text (either human or machine-produced); and (iii) asked to rate the adequacy of the translation on a 0-100 rating scale. source-based da has the advantage of freeing up reference translations so that they can be included in the evaluation as if they had been produced by a system. source-based da comes with the trade-off, however of requiring bilingual human assessors. ## experiments in order to investigate the degree to which human and machine perform differently on individual test set segments, we run segment-level da on translations of the same random sample of 540 segments by the top-performing system and the human translator. we do this for each language pair in which there was a tie with human performance wmt-19 (english to russian) or where machine translation performance had surpassed human translation quality (german to english; english to german). 3 in order to access the bilingual speakers required for the source-based da configuration we run all source-based da hits on an in-house crowdsourcing platform. in total 108,829 assessments were collected via the in-house platform. after removing quality controls, we ended up with 87,211 assessments for which we are confident of worker reliability, and employ all those assessments in our final analysis. 3 for german to english translation, although human and the top-performing system, facebook-fair, are ranked in the same cluster, the system significantly outperforms human translation in head-to-head significance test results. english to german 0.000 0.025 0.050 0.075 40 60 80 100 adequacy density system facebook-fair human figures 1, 2 and 3 include density plots for human translation and the top-performing facebook-fair system @xcite for the same 540 translated segments from wmt-19 for the three language pairs we investigate. for german to english and english to german translation in figures 2 and 3 a similar pattern emerges in terms of comparison of human and machine-translated segments, as for both a slightly larger proportion of facebook-fair translations are scored high compared to the human translator -as can be seen from the higher red peak close to the extreme right of both plots indicating that the machine produces a marginally higher number of translations with higher levels of adequacy. for english to russian translation, however, a different pattern occurs, as shown in figure 1 , as it appears that there are locations lower down on the adequacy scale in which the facebook-fair system performs worse than the human translator in three noticeable locations within its score distribution. however, these differences between language pairs are somewhat unsurprising considering that human and system were tied for english to russian but system beat human in terms of statistical significance for both english to german and german to english to russian + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 20 40 60 80 100 0 20 40 60 80 100 facebook-fair (src) human a (src) figure 4: scatter plot of accurate segment-level da scores for top-performing system, facebook-fair in wmt-19 versus the human translator where in the official results the system tied with human performance; human a denotes evaluation of segments translated by the creator of the standard wmt reference translations; src denotes a source-based configuration of direct assessment was employed to collect scores; segmentlevel scores for human and machine are the average of a minimum of 15 human assessment scores ## conclusions the question we ask in this work is highly relevant -what are the differences between human translations and the top mt translations on the segment level when "human parity" is reached. for the english-russian system our analysis makes it clear that there are a number of segments where the human did better than the machine, but on close inspection of these sentences, there appears to be no generalizable difference that clearly characterizes these kinds of sentences. for english to/from german, the situation between human and machine is more finely balanced, and segment-level analysis has shown only a small number of random errors on each side, revealing only minor differences are present even on the segment level when we compare human and machine translations.