Examining Large Pre-Trained Language Models for Machine Translation: What You Don’t Know about It (2022)

Abstract: Pre-trained language models (PLMs) often take advantage of the monolingual and multilingual dataset that is freely available online to acquire general or mixed domain knowledge before deployment into specific tasks. Extra-large PLMs (xLPLMs) are proposed very recently to claim supreme performances over smaller-sized PLMs such as in machine translation (MT) tasks. These xLPLMs include Meta-AI’s wmt21-dense-24-wide-en-X (2021) and NLLB (2022). In this work, we examine if xLPLMs are absolutely superior to smaller-sized PLMs in fine-tuning toward domain-specific MTs. We use two different in-domain data of different sizes: commercial automotive in-house data and clinical shared task data from the ClinSpEn2022 challenge at WMT2022. We choose the popular Marian Helsinki as smaller sized PLM and two massive-sized Mega-Transformers from Meta-AI as xLPLMs.Our experimental investigation shows that 1) on smaller-sized in-domain commercial automotive data, xLPLM wmt21-dense-24-wide-en-X indeed shows much better evaluation scores using SacreBLEU and hLEPOR metrics than smaller-sized Marian, even though its score increase rate is lower than Marian after fine-tuning; 2) on relatively larger-size well prepared clinical data fine-tuning, the xLPLM NLLB tends to lose its advantage over smaller-sized Marian on two sub-tasks (clinical terms and ontology concepts) using ClinSpEn offered metrics METEOR, COMET, and ROUGE-L, and totally lost to Marian on Task-1 (clinical cases) on all official metrics including SacreBLEU and BLEU; 3) metrics do not always agree with each other on the same tasks using the same model outputs; 4) clinic-Marian ranked No.2 on Task- 1 (via SacreBLEU/BLEU) and Task-3 (via METEOR and ROUGE) among all submissions.

## introduction owing to the recent development of neural machine translations (nmts) @xcite @xcite @xcite , especially the self-attention based transformer learning structures @xcite , pretrained language models (plms) have been dominant in natural language understanding (nlu) and natural language processing (nlp) tasks. these applications include long-short term memory (lstm) and bert (pre-training of deep bidirectional transformers) based models to text mining @xcite , question-answering @xcite , reading comprehension @xcite , and summarisation (perez-beltrachini and lapata, 2021), etc., in addition to mt @xcite @xcite . plms often have a large amount of trainable parameters for downstream applications. for instance, in translation task, the popular marian nmt @xcite pre-trained by microsoft translator team foot_0 on opus foot_1 @xcite multilingual corpus has 7.6 million parameters, which can still be fine-tuned on google's colab or aws at virtually no cost. however, very recent work has shown much larger plms that have much more parameters than smaller models, e.g. the multi-lingual transformer model submitted to wmt2021 shared task by meta-ai research group "wmt21-dense-24-wide-en-x"(wmt21fb) @xcite , which has 4.7 billion parameters, i.e. 618 times bigger than marian, and does not fit into regular gpus. in this year, meta-ai published another model nllb @xcite that has 54.5 billion parameters and covers 200 languages in the full model. from now on, we name both "wmt21-dense-24-wide-en-x" and nllb as meta-ai's mega-transformer models. meta-ai's mega-transformer (wmt21fb) has claimed the best performing system on 10 out of 14 language pairs in wmt2021 shared task including winning bilingual-trained models. in this work, we raise the question whether extralarge plms (xlplms) such as meta-ai's mega-transformers have absolute superiority in nmt tasks on domain-specific fine-tuning. we prepare experimental investigation on two different data set to answer this question. one is our specific automotive domain in-house commercial data and the other is clinical domain data from clinspen2022 challenge task we attended which is affiliated with wmt2022 foot_2 . we set up the following hypothesis and research questions. our hypothesis is: xlplms do not absolutely demonstrate superiority over smaller sized plms in nmt fine-tuning and it shall depends on specific tasks deployed including domain topic, size of available in-domain data, and performancecost trad-off. from this hypothesis we derive two research questions (rqs): 1) do xlplms always demonstrate better performances in nmt over smaller sized plms for domain fine-tuning? 2) if not, in what situations? to the best of our knowledge, this is the first published work that has been carried out in the field on fine-tuning meta-ai's extra-large multilingual plm maga-transformers, and in translating specialised automotive and clinical data. the rest of the paper is organised as below: section 2 introduces more details on related work to ours including plms and fine-tuning in automotive and clinical domains, section 3 describes our initial model settings including deployed baseline models, section 4 presents our experimental evaluation carried out on our in-house commercial automotive domain data, section 5 describes our system submission to clinspen biomedical-mt challenge task at wmt2022 on clinical data, and section 6 gives our conclusion and future work plan. regarding plm applications in automotive domain, the only recent work we found is from @xcite , who tested the distil-bert and xlm-roberta plms for text classification task using swedish truck manufacturer data, instead of mt. there are also researchers working on the overview of model comparability, bench-marking, and fine-tuning methodologies regarding larger scale plms, e.g., from @xcite ; @xcite . overall, none of the work mentioned before has investigated into extra-large mega-transformer level plms (xlplms) for nmt in automotive and clinical domains, especially their comparisons to smaller sized plms. ## initial model settings to investigate into plms with fine-tuning for specialised domain nmt from different scales, we firstly deploy two of such models in different sizes from a multilingual setting. the first one is the popular marian nmt model developed in c++ since 2018 using deep @xcite . it is mostly maintained by the microsoft translator team and features with efficiency, fast training, and state-of-the-art nmt architectures 4 . this plm has a smaller sized 7.6 million trainable parameters. the second one we use for fine-tuning is one of the extra-large plms (xlplms) meta-ai's mega-transformer "wmt21.dense-24-wide.en-x" @xcite developed for wmt2021 shared task on multilingual mt, which was submitted to 14 language pairs and claimed the best on 10 of them 5 . it has 4.7 billion trainable parameters, which is super large in comparison to marian model. in the later section (5), we will explain another mega-transformer model nllb developed in this year by meta-ai and deploy it for our clinspen2022 shared task submission on clinical domain. ## submission to clinspen at wmt22 in this section, we introduce our system submissions to biomedical-mt task in wmt2022. in this task, we attended the affiliated clinical domain machine translation on spanish-english language pair (clinspen) task 9 , which is hosted in codalab @xcite 10 . the aim of this task is to promote the development of mt models on medical domain via three sub-tasks: 1) clinical cases (cc): on 202 covid-19 clinical case reports; 2) clinical terms (ct): using more than 19k parallel terms extracted from biomedical literature and electric health records (ehrs); 3) ontology concepts (oc): using more than 2k parallel concepts from biomedical ontology. the translation direction on these three subtasks are @xmath0, en←es, and @xmath1. ## discussion and future work in this work, we carried out experimental investigations on if extra-large pre-trained language models (plms) always demonstrate superiority over much smaller-sized plms using two domain specific data. the first experimental results using marian vs "wmt21.dense-24-wide.en-x" shows that even though xlplm still perform better evaluation scores in comparison to much smaller sized marian, their score difference is much smaller after fine-tuning and the xlplm costs more than smaller plm from performance-cost trade-off point of view in practical applications, e.g. for language service providers (lsps). the second experimental results using clinical data show that with carefully prepared certain amount of fine-tuning data (250k sentence pairs), the xlplm nllb even loses with its evaluation score in comparison to smaller plm marian in task 1 "clinical cases" over all automatic metrics used, and in task 2 "clinical terms" and 3 "ontology concepts" on partial of the automatic evaluation metrics officially used by clinspen2022. finally, our system submission clinic-marian ranked the second place using sacrebleu/bleu for task-1, and using meteor/rouge for task-3 among all teams who submitted on-time before the shared task deadline. we looked into the translation outputs from clinic-nllb for error analysis, and it shows that some of the translation errors come from very literal translation, and others come from gender related mistakes. in conclusion, our two stage experimental investigations verify our hypothesis and rqs from different aspects. we also doubt if the official automatic metrics used for clinspen challenge can correctly distinguish the nmt systems because mostly they do not really measure the translation output quality but the similarity to the gold standard single reference. therefore, domain specific automatic evaluation metrics or metrics better measuring semantic similarities might be needed. in the future work, we plan to carry out more ex-perimental investigations from qualitative aspects looking into translation errors using human experts and classifying them into possible categories with examples and statistics, especially from clinical domain. this will allow us to validate automatic metrics with professional human judgements for this domain. we will continue to fine-tune our models towards different domains and languages and use more of the available corpus for current clinical domain challenge task. we also plan to try different stateof-the-art pre-trained language models for evaluation.