Fine-Grained Error Analysis on E nglish-to- J apanese Machine Translation in the Medical Domain (2020)

Abstract: We performed a detailed error analysis in domain-specific neural machine translation (NMT) for the English and Japanese language pair with fine-grained manual annotation. Despite its importance for advancing NMT technologies, research on the performance of domain-specific NMT and non-European languages has been limited. In this study, we designed an error typology based on the error types that were typically generated by NMT systems and might cause significant impact in technical translations: “Addition,” “Omission,” “Mistranslation,” “Grammar,” and “Terminology.” The error annotation was targeted to the medical domain and was performed by experienced professional translators specialized in medicine under careful quality control. The annotation detected 4,912 errors on 2,480 sentences, and the frequency and distribution of errors were analyzed. We found that the major errors in NMT were “Mistranslation” and “Terminology” rather than “Addition” and “Omission,” which have been reported as typical problems of NMT. Interestingly, more errors occurred in documents for professionals compared with those for the general public. The results of our annotation work will be published as a parallel corpus with error labels, which are expected to contribute to developing better NMT models, automatic evaluation metrics, and quality estimation models.

## introduction we performed a manual annotation of translation errors using fine-grained error typology in domain-specific neural machine translation (nmt) of japanese and english language pairs. although several approaches have been proposed to evaluate the performance of nmt, it has been commonly presented as scores of automatic evaluation, and detailed analysis of problems in nmt is limited. previous studies @xcite annotated errors in mt outputs; however, they targeted only on a general domain and european languages. detailed error detection is essential, especially in the domain-specific settings, where tiny mistakes, such as incorrect translation of a technical term, leads to significant misunderstanding. to tackle this problem, we performed an annotation-based analysis of errors that occurred in nmt for a specific technical domain. professional translators annotated types and positions of errors that occurred in translation from english to japanese. the error typology was designed based on an existing framework, multidimensional quality metrics (mqm) @xcite , which was customized to our study. we selected medicine as the domain field because medical translation is in growing demand in the society to enrich healthcare information, which requires highly specific domain expertise. recent issues regarding public health, such as the pandemic of coronavirus disease 2019, highlight demands on sharing correct and understandable information throughout the world including asian countries. we prepared five medical contents with english-to-japanese translation data using state-of-the-art nmt systems. as a result, 4, 912 errors in five types were annotated on 2, 480 sentences. we also analyzed the annotation results in detail to reveal distributions and characteristics of errors produced by current nmt systems. the results of annotation will be published as a parallel corpus with error labels. this is the first corpus of error annotation (1) on domain-specific and (2) on english-to-japanese nmt outputs. such corpora annotating errors in machine translation (mt) are valuable resources to understand problems in nmt models, develop automatic evaluation metrics, and estimate the quality of machine translation @xcite . ## related work our annotation corpus is based on the error typology that conforms to structured categories of quality metrics for translation quality. previous studies employed a few different typologies, such as mqm and scate (smart computer-aided translation environment) @xcite . among them, mqm is one of the most common frameworks for quality assessment of human translation. the framework of the typology in our study also refers to the mqm. qt21 consortium has published post edited and error annotated data for machine translations in four languages: czech, english, german, and latvian @xcite based on mqm. this data just included languages in europe, and prior studies that used the mqm have evaluated translation of european languages @xcite . our corpus in english to japanese will add a useful resource of annotation. the shared task of quality estimation in the conference on machine translation (wmt) has also employed the mqm for document-level quality estimation since 2018. approaches of quality estimation tasks with mqm include word-level annotation @xcite and the estimation of mqm score with prediction models @xcite . nonetheless, there has been a limited resource for domain-specific translation @xcite , which is indispensable to develop an evaluation strategy for appropriateness of word choice in the technical context. ## error typology & development of annotation guidelines in this study, we developed customized errortypology criteria for the evaluation of domainspecific nmt. our typology was based on mqm. the major error categories in mqm are "accuracy," "fluency," "design," "locale convention," "style," "terminology," and "verity," of which subcategories are defined for a specific type of incorrectness. we selected and customized several error subtypes in the original mqm for annotation that were applicable to translations by nmt systems. in this paper, we focused on subtypes that annotation results confirmed as the major problems of the current nmt systems, namely, "addition," "omission," and "mistranslation" from "accuracy;" "terminology;" and "grammar" from "fluency;" as summarized in we customized these error subtypes to handle domain specificity and the japanese language due to different systems of grammar and sociolinguistic register from western languages. the following sections describe these error types and guidelines given to annotators to identify each error. foot_0 ## annotation setup in this section, we describe the annotation procedure and resources used to perform the annotation. ## quality control of annotation this kind of error annotation is inevitably subjective, because the ability to detect errors in translation depends on the level of expertise. in addition, determination of the type and span of errors should be contingent on the preference of each annotator, which may cause the variation of the annotation work. 5 in this study, to collect reliable annotations alleviating such subjectivity, we conducted a pilot study and reconciliation of annotated labels. ## analysis of annotation results we conducted an in-depth analysis of annotation results from four perspectives: ## discussion and future work we performed the error analysis of nmt for the english and japanese language pair in the medical domain, based on fine-grained and quality-controlled manual annotation. in the analysis of detected 4, 912 errors on 2, 480 sentences, we found that the major errors in nmt were "mistranslation" and "terminology," rather than "addition" and "omission." the errors of "addition" and "omission" have been deemed typical in nmt as over-generation and under-generation, respectively; however, our results revealed that the semantic and terminology errors were more common in domain-specific technical documents. interestingly, these errors were often observed in quantitative and polysemous words. this finding suggests future challenges in machine translation research targeting in the representation of numeric and multi-sense words. we found more errors in documents for health-care professionals compared with those for the general public, specifically in terms of errors in "grammar" and "terminology." this finding encourages further research to improve the performance of nmt in documents that include sentences with complex syntax and highly-specialized technical terms. the results of annotation will be published as a parallel corpus with detailed error labels, which is expected to be a valuable resource to improve nmt models, develop automatic evaluation metrics, and estimate qualities of machine translation. the limitations in current automatic evaluation metrics are partly attributable to insufficient understanding of the real performance of nmt systems. furthermore, the dependence on the reference translation is problematic. the similarity to the reference does not necessarily represent the seman-tic accordance of the translation to the source sentence. natural language is characterized by its ambiguity, such as multiple meanings and contextual implications, and thus translation should not have the unique correct answer. while verbatim similarly to the reference enforces a strict constraint, it does not ensure the actual quality of translation. better estimation of translation quality should incorporate features reflecting the actual quality of translation, such as semantic accuracy and linguistic fluency. we believe our corpus contributes to research on evaluation or estimation models of nmt performance to overcome these limitations. essentially, it is a valuable resource for assessing the domain-specificity of translation outputs. as future works, we will develop quality estimation models using the corpus to allow fine-grained and domain-specific evaluation. also, we will extend the annotation corpus in other domains and language pairs.