Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for E nglish-to- C hinese (2020)

Abstract: This research presents a fine-grained human evaluation to compare the Transformer and recurrent approaches to neural machine translation (MT), on the translation direction English-to-Chinese. To this end, we develop an error taxonomy compliant with the Multidimensional Quality Metrics (MQM) framework that is customised to the relevant phenomena of this translation direction. We then conduct an error annotation using this customised error taxonomy on the output of state-of-the-art recurrent- and Transformer-based MT systems on a subset of WMT2019’s news test set. The resulting annotation shows that, compared to the best recurrent system, the best Transformer system results in a 31% reduction of the total number of errors and it produced significantly less errors in 10 out of 22 error categories. We also note that two of the systems evaluated do not produce any error for a category that was relevant for this translation direction prior to the advent of NMT systems: Chinese classifiers.

## introduction the field of machine translation (mt) has been revolutionised in the past few years by the emergence of a new approach: neural mt (nmt). nmt is a dynamic research area and we have witnessed two mainstream architectures already, the first of which is based on recurrent neural c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. networks (rnn) with attention @xcite while the second, referred to as transformer, makes use of the self-attention mechanism in nonrecurrent networks @xcite . several studies have analysed in depth, using both automatic and human evaluation methods, the resulting translations of nmt systems under the recurrent architecture and compared them to the translations of the previous mainstream approach to mt: statistical mt @xcite , e.g. @xcite @xcite @xcite . however, while the transformer architecture has brought, at least when trained with sufficient data, considerable gains over the recurrent architecture @xcite , the research conducted to date that analyses the resulting translations of these two neural approaches is, to the best of our knowledge, limited to automatic approaches @xcite @xcite @xcite . in this paper we conduct a detailed human analysis of the outputs produced by state-of-the-art recurrent and transformer nmt systems. namely, we manually annotate the errors found according to a detailed error taxonomy which is compliant with the hierarchical listing of issue types defined as part of the multidimensional quality metrics (mqm) framework @xcite . we carry out this analysis for the news domain in the english-to-chinese translation direction. to this end, we define an error taxonomy that is relevant to the problematic linguistic phenomena of this translation direction. this taxonomy is then used to annotate errors produced by nmt systems that fall under the recurrent and transformer architectures. the main contributions of this paper can then be summarised as follows: the rest of the paper is arranged in the following way. section 2 presents a brief review of related work. next, section 3 outlines the recurrent-and transformer-based nmt systems and the dataset used in our experiments. subsequently, section 4 presents the methodology for error annotation and the definition of the error taxonomy, followed by results and statistical analysis of the annotation. finally, section 5 gives a conclusion and suggestions for future work. ## related work this section provides an overview of related research on the two topics that correspond to our main contributions: human error analysis of mt outputs for the language pair english-chinese (section 2.1) and analyses of mt systems based on the recurrent and transformer architectures (section 2.2). ## machine translation systems this section reports on the mt systems and the dataset used in our experiments. we have used output from systems that fall under the recurrent and transformer architectures and were top-ranked at the news translation shared task at the conference on machine translation (wmt). we chose the university of edinburgh's mt system @xcite as our recurrent nmt system due to the fact that this system had the highest bleu score (36.3) for the translation direction @xmath0. as for the transformer-based mt system used in our research, we have taken the patech submission to wmt2019. 3 we conducted our experiments before the human evaluation of wmt2019 was available, and therefore we chose the pat-ech's system based on the automatic evaluation of wmt2019, in which this system was the best performing one. 4 however, patech's system was not included in the human evaluation of wmt2019. therefore we carried out an additional annotation on the top-performing system from that human evaluation: the transformer system developed by kingsoft ai lab @xcite , hereafter referred to as ksai. before our human error analysis, we would like to compare the recurrent and transformer mt systems in terms of an automatic evaluation metric. this is not possible from their outputs since they correspond to two different test sets (newstest2017 and newstest2019). in order to be able to compare them, we asked the developer of the recurrent system to provide us with the output from their system for newstest2019. as shown in ## error annotation this section details the annotation setup (section 4.1), explains how we defined our mqmcompliant error taxonomy adapted to the relevant characteristics of translating from english into chinese and the challenges faced by nmt systems in this translation direction (section 4.2) and presents the results of the annotation, as well as analysis and discussion thereof (section 4.3). ## conclusion this paper presented a fine-grained manual evaluation for @xmath1: rnn and transformer. the evaluation was approached in the form of a human error annotation based on a customised mqm error taxonomy. the error taxonomy was developed from the mqm core taxonomy for mt evaluation. chinese linguistic features and issues emerged in the calibration set were taken into account by including customised error types, such as extraneous function word, classifier and typography. the error type extraneous function word underpins investigating westernised chinese phenomena of extraneous function words by specifying it into three word classes: preposition, adverb and particle. from our analysis, it is clear that transformerbased systems generate significantly more accurate, fluent and comprehensible translation with less westernised chinese expressions. however, transformer systems do not handle typography as well as rnn. we also note that none of the mt systems did produce any errors related to unpairedmarks and only one system produced errors related to classifiers, which were very unfrequent (0.16% of the tokens). we can conclude that transformer systems produce an overall better translation compared to rnn when translating from english to chinese, which corroborates findings of prior studies on other language pairs. a limitation worth mentioning is that our annotation was conducted by only two annotators on a limited amount of data. our taxonomy could be of use for further error analysis on chinese mt quality. future research could include a larger annotation sample to investigate if punctuation is a a common issue in nmt systems based on transformer and to verify that nmt is able to produce correct classifiers. also, as transformer still shows a major problem in mistranslation, the error taxonomy can be extended with more specific categories to explore this issue in more detail. the annotations for the three mt systems and the code used for the analysis thereof are publicly available. 8