Quality Analysis of Multilingual Neural Machine Translation Systems and Reference Test Translations for the E nglish- R omanian language pair in the Medical Domain (2023)

Abstract: Multilingual Neural Machine Translation (MNMT) models allow to translate across multiple languages based on only one system. We study the quality of a domain-adapted MNMT model in the medical domain for English-Romanian with automatic metrics and a human error typology annotation based on the Multidimensional Quality Metrics (MQM). We further expand the MQM typology to include terminology-specific error categories. We compare the out-of-domain MNMT with the in-domain adapted MNMT on a standard test dataset of abstracts from medical publications. The in-domain MNMT model outperforms the out-of-domain MNMT in all measured automatic metrics and produces fewer errors. In addition, we perform the manual annotation over the reference test dataset to study the quality of the reference translations. We identify a high number of omissions, additions, and mistranslations in the reference dataset, and comment on the assumed accuracy of existing datasets. Finally, we compare the correlation between the COMET, BERTScore, and chrF automatic metrics with the MQM annotated translations. COMET shows a better correlation with the MQM scores compared to the other metrics.

## introduction neural machine translation (nmt) models have achieved competitive performance on low-resource language pairs, particularly for non-specialised domains @xcite . however, in a high-risk and low-resource domain, like the medical domain, the accurate translation of terminology, alongside the absence of hallucinations and mistranslations are crucial for exchanging information across international healthcare providers or users @xcite . multilingual nmt (mnmt) models leverage many language pairs and millions of segments @xcite within one system. the inclusion of many language pairs helps to improve the translation quality for low-resource languages by transferring knowledge from highresource languages via similar cross-lingual word representations. moreover, domain adaptation techniques are used to adapt mnmt models into new domains @xcite . however, evaluation studies of mnmt models are focused on automatic metrics without providing insights into the quality of the translation of a specialised domain. these automatic metrics require high-quality reference translations which reflect the specialised terminology and style of a given domain, but such translations are difficult to find. also, given that translation processes and expertise vary among translators and other text producers, the quality of datasets in different language pairs can differ considerably. in addition, justifiable differences between source and target sentence content are caused by legitimate pragmatic translation strategies. overall, automatic or even semi-automatic translation data gathering processes are not sophisticated enough yet to improve the quality of the source and/or target content, or filter out content mismatches between source and target sentences before aligning them. in this paper, we study the quality of a pre-trained mnmt model in the medical domain for a lowresource language pair (english-romanian). our goal is to compare an out-of-domain mnmt with a fine-tuned in-domain mnmt in terms of automatic metrics and a manual error typology annotation. we use a pre-trained model based on mbart @xcite and fine-tune it with a medical in-domain parallel corpus. in addition, we analyse the quality of the reference test dataset, because errors present in the reference translations can bias the findings of automatic metrics. we test the models on the english-romanian language pair with a corpus of medical paper abstracts @xcite . we evaluate the quality of both models with automatic metrics and an error typology annotation based on the multidimensional quality metrics (mqm) framework @xcite , to which we added terminology-based categories from @xcite . the terminology categories provide a fine-grained discrimination of errors. finally, we analyse the segment-level correlation between automatic metrics (chrf @xcite , comet @xcite , and bertscore @xcite ) and the mqm error annotations based on the reference translations. the fine-tuned mbart model outperforms mbart on the automatic metrics. in addition, the error analysis based on the terminology-enhanced-mqm shows that the fine-tuned model also produces fewer errors than the mbart model. the comet score shows the highest correlation with the mqm scores. however, it is also important to mention that we identified a total of 157 translation errors in 66 of the 75 reference translation segments, as detailed in section 4.1 below; this questions current assumptions regarding the quality of the reference datasets for our chosen language pair and domain. ## background and related work mnmt models are based on transferring parameters or information across multiple languages, where low-resource languages benefit from the high-resource languages. the mnmt model shares a common word representation (i.e., word embeddings) across language pairs. during training, the mnmt model clusters word representations with similar contexts from the high-and low-resource segments @xcite . the low-resource pairs learn meaningful word representations given the access to a large number of similar contexts from the high-resource language pairs. multiple languages are processed jointly by indicating the target translation direction on each segment of the multilingual corpora in the input training data by using an artificial token (@xmath0. for ex-ample, an english-romanian segment pair would be labelled as follows: mnmt models outperform standard bilingual baselines on translation quality for low-resource languages @xcite . mbart is an example of a sequence-to-sequence model pre-trained on monolingual data from 25 languages based on a text reconstruction learning objective for mnmt @xcite . mbart incorporates a monolingual training step before the multilingual mt training for a better initialisation of the translation model. in other words, mbart first learns an improved representation of each language with monolingual data. after that, mbart continues with the multilingual translation training based on parallel data. mbart shows a better translation quality compared to previous mnmt models. however, most mnmt models are generalpurpose systems trained with web-crawled corpora @xcite , and as such they struggle with specialised domains (e.g., medical). domain adaptation aims to improve the translation performance in specialised domains, where fine-tuning is a low-cost and common technique. fine-tuning consists of resuming the training of an out-of-domain resource-rich mt model with a poor-resourced indomain corpus @xcite . the resulting model is adapted to work with an in-domain language pair, instead of re-training the mnmt model from scratch @xcite . mt models are usually evaluated with automatic metrics that take into account fluency and adequacy, by comparing the machine translation output against one or more human reference translations @xcite . metrics produce a corpuslevel score or a segment-level score for a given mt model @xcite . however, automatic metrics are not designed to identify translation errors in mt outputs, such as errors in terminology, for example @xcite . on the other hand, error typology evaluation frameworks, such as the multidimensional quality metrics (mqm) @xcite , are based on manually classifying and annotating errors using predefined categories. the mqm error typology covers high-level error categories, such as: accuracy, style, terminology, linguistic conventions, lo-cale conventions, audience appropriateness, and design and markup. each high-level category can be further expanded into fine-grained categories; for example, accuracy can be further sub-categorised into mis-translation, over-translation, omission, etc. expert evaluators identify an error in the mt output, label it with a category from the typology, and also assign a severity score to it. @xcite propose a fine-grained error typology with a focus on terminology. they use a legal domain corpus and develop a gold-standard terminology resource of identified terms based on the previous error typology. given the terminological richness within the medical domain, we found it relevant to supplement mqm @xcite with this terminology-specific error typology. @xcite compare the quality of phrasebased mt, factored phrase-based mt, and nmt with a manual error annotation of 100 segments with mqm for the english-croatian language pair. the nmt system was the best performing, with fewer errors produced. @xcite perform a large-scale study based on mqm annotation of systems from the workshop on machine translation, and they use mqm error-based scores for evaluation. their error annotation shows a preference for human translations over mt outputs, and the automatic metrics correlate positively with the mqm scores. we use bleu @xcite , chrf @xcite , bertscore @xcite , and comet @xcite for automatic evaluation. for human evaluation, we use mqm core extended with the terminology categories from @xcite , which contain eight terminology-related error categories -partial error, source term copied, inflectional error, reorder error, disambiguation issue in target, incorrect lexical selection, term drop, and other er-ror -, and three severity levels with corresponding weights -minor (1), major (5) and critical (10). ## manual evaluation analysis to gain insights into the translation errors produced by the two models, we show a sample of 12 abstracts with a total of 75 segments to three annotators working collaboratively @xcite ; the motivation for this joint in-person annotation approach was to increase agreement for identifying terms and errors. the annotators are native romanian speakers with in-house and freelance translation experience; moreover, one of the annotators also has in-house and freelance translation experience in the medical domain. the annotators had access to the source, the reference, and the output of the two mnmt systems in order to annotate the reference translation, as well as each mt segment, with error categories @xcite mqm @xcite and the fine-grained terminology typology @xcite . to perform the annotation, we set up a translation project in trados studio 2021 4 and import the source texts, reference translations, and mt output files as bilingual .xlsx files. we install the freely-available qualitivity foot_4 plug-in for studio; this serves as the environment in which the annotators record any identified errors, their severity level and proposed corrections, along with explanatory comments. at the end of the annotation process, we export a report from qualitivity containing the full annotation data for the reference segments, as well as mbart and fine-tuned mbart outputs. ## conclusions and future work we quantified the impact of domain adaptation on mbart in the medical domain for english-romanian. the fine-tuned mbart outperforms the general model with automatic metrics and produces fewer errors in the relatively small sample (75 segments belonging to the 12 medical publications abstracts) we annotated. we show that the gold-standard reference translations provided in the datasest contain a high number of errors. blindly assuming good quality of the reference translations when performing evaluations can be problematic and the community should be more open about the shortcomings of existing data gathering methods, and incorporate translators' contributions to improving test and training datasets to a greater extent. while fewer terminology errors were recorded in the partial error, source term copied, disambiguation issue in target, incorrect lexical selection, and term drop categories, in the three remaining ones (inflectional error, reorder error, and other error), the fine-tuned mbart output actually contained more errors than the general mbart output. of these three categories, the inflectional error and other error items present in the fine-tuned mbart output are related to the bpe vocabulary. in future work, we plan to extend the bpe vocabulary in mbart @xcite to cope with in-domain terminology. comet shows a higher correlation with mqm scores compared to other automatic metrics. comet can be an option for evaluating nmt systems for the medical domain, and in particular for scientific abstracts. at the same time, reference translation datasets need to be prepared much more carefully, keeping in mind shortcomings in the translation output produced by nmt models. finally, it is essential to raise awareness among machine translation post-editors, as well as clients, that errors may persist in mt output even after fine-tuning. errors in nmt output remain difficult to identify due to the apparent fluency of the output, and can thus be overlooked even by subjectmatter experts. it is for these reasons that translators should be able to work in post-editing interfaces which stimulate their attention to such errors. it is also why synchronous collaborative translation, revision, and post-editing workflows which use newer, more ergonomic and interactive technologies should be promoted and adopted to a much greater extent.