Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning (2023)

Abstract: Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating superpowers and the pre-knowledge they acquire for downstream tasks. This work investigates whether MMPLMs can be applied to clinical domain machine translation (MT) towards entirely unseen languages via transfer learning. We carry out an experimental investigation using Meta-AI’s MMPLMs “wmt21-dense-24-wide-en-X and X-en (WMT21fb)” which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and the opposite direction. We fine-tune these MMPLMs towards English- Spanish language pair which did not exist at all in their original pre-trained corpora both implicitly and explicitly.We prepare carefully aligned clinical domain data for this fine-tuning, which is different from their original mixed domain knowledge.Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain EN-ES segments for three sub-task translation testings: clinical cases, clinical terms, and ontology concepts. It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a high-resource setting in the pre-training.To the best of our knowledge, this is the first work on using MMPLMs towards clinical domain transfer-learning NMT successfully for totally unseen languages during pre-training.

## introduction multilingual neural machine translation (mnmt) has its root from the beginning of nmt era @xcite but only made its first milestone when google's end-to-end mnmt arrived @xcite where the artificial token was introduced for the first time for translation task at the beginning of the input source sentence to indicate the specified target language, e.g. "2en" as translating into english. this model used a shared word-piece vocabulary and enabled multilingual nmt through a single encoder-decoder model training. google's mnmt also demonstrated the possibility of "zero-shot" translation as long as the languages to be translated from or to have been seen during the training stage, even though not explicitly. however, as the authors mentioned, google's mnmt only allows translating between languages that have been seen individually as "source and target languages during some point, not for entirely new ones" in their many-to-many model, which was tested using the wmt14 and wmt15 data @xcite . this set an obstacle to translating freshly new languages that do not exist in their pre-training stage. then using the later developed nmt structure transformer and bert @xcite , facebook ai extended the coverage of multilingual translation into 50, 100, and 200+ languages via mbert-50 @xcite , m2m-100 @xcite , and nllb @xcite models. however, these models never address the issue of translating entirely new languages that do not exist in their pre-training stage, which sets an obstacle for mt applications in serving an even broader community. in this work, we move one step forward towards domain-specific transfer-learning @xcite for nmt via fine-tuning an entirely new language pair that does not exist in the deployed multilingual pre-trained language models (mplms). the mplms we used are from facebook ai (meta-ai)' s submission to the wmt21 news translation task, i.e. "wmt21-dense-24-wide-en-x" and "wmt21-dense-24-wide-x-en" which were pretrained for 7 languages hausa (ha), icelandic (is), japanese (ja), czech (cs), russian (ru), chinese (zh), german (de) to english (en), and backward @xcite . we use a well-prepared 250k pairs of english-spanish (en-es) clinical domain corpus and demonstrate that not only it is possible to achieve successful transfer-learning on this explicit new language pair, i.e. the spanish language is totally unseen among the languages in the mplm, but also the domain knowledge transfer from general and mixed domain to the clinical domain is very successful. in comparison to the massively mplm (mmplm) nllb which covers spanish as a high-resource language at its pre-training stage, our transfer-learning model achieves very close evaluation scores in most sub-tasks (clinical cases and clinical terms translation) and even wins nllb in ontology concept translation task by the metric comet @xcite using clinspen2022 testing data at wmt22. this is a follow-up work reporting further findings based on our previous shared task participation @xcite and pre-print @xcite . ## related work regarding the early usage of special tokens in nmt, @xcite designed the token t from latin tu and v from latin vos for familiar and polite indicators attached to the source sentences towards englishto-german nmt. @xcite designed @xmath0, <all-passive>, <@xmath1; either they are active, passive, reference aware or prediction guided. subsequently, google's mnmt system designed target language indicators, e.g. <2en> @xmath2. google's mnmt also designed mixed target language translation control, e.g. (1-α)<2ko> + α<2jp> tells a mixed language translation into korean and japanese with a weighting mechanism. we take one step further to use an existing language controller token from a mplm as a pseudo code to fine-tune an external language translation model, which was entirely not seen during the pre-training stage. regarding transfer-learning applications for downstream nlp tasks other than mt, @xcite applied transfer learning from mplms towards unseen languages of different typologies on dependency parsing (dep), named entity recognition (ner), and partof-speech (pos) tagging. @xcite carried out zero-shot transfer learning for natural language inference (nli) tasks such as question answering. in this paper, we ask this research question (rq): can massive multilingual pre-trained language models create a knowledge space transferring to entirely new language (pairs) and new (clinical) domains for machine translation task via fine-tuning? ## model settings to investigate into our rq, we take meta-ai's mnmt submission to wmt21 shared task on news translation, i.e. the mmplm "wmt21-dense-24-wide-en-x" and "wmt21-dense-24-wide-x-en" as our test-base, and we name them as wmt21fb models @xcite foot_0 . they are conditional generation models from the same structure of massive m2m-100 @xcite having a total number of 4.7 billion parameters which demand high computational cost for fine-tuning. wmt21fb models were trained on mixed domain data using "all available resources" they had, for instances, from historical wmt challenges, large-scale data mining, and their in-domain back-translation. then these models were finetuned in news domain for 7 languages including hausa, icelandic, japanese, czech, russian, chinese, german from and to english. the challenging language we choose is spanish, which did not appear in the training stage of wmt21fb models. the fine-tuning corpus we use is extracted from mespen @xcite clinical domain data, of which we managed to extract 250k pairs of english-spanish segments after data cleaning. they are from ibecs-descriptions, ibecs-titles, medlineplus-health_topicstitles, medlineplus-health_topics-descriptions, ## conclusion and future work we investigated if real transfer-learning nmt is possible using massive multilingual pretrained lms (mmplms) to translate external languages that are unseen at all in the training phase. we used meta-ai's mixed domain multilingual plms (wmt21fb) as our test base, 250k well-prepared en-es clinical data as fine-tuning corpus, @xmath3. we tested the fine-tuned model on clinspen2022 clinical domain shared task data, and the results show that this fine-tuning is successful, which achieves very comparable scores to meta-ai's mmplm nllb model, which had spanish in the training phase as a high-resource setting. we think this demonstrates that the hyper-transformer model from wmt21fb does build a language-independent "semantic space" that allows one to understand a different language and correctly construct a totally different language model when fine-tuned on the language which was absent and different from the languages it was trained upon. this finding can be very useful for future clinical knowledge transformation, e.g. from existing high-resource languages to low-resource languages, such that clinicians from low-resource language speakers can also benefit from ai-supported decisionmaking. the well-trained clinical models based on properly translated resources can also potentially support patients' self-diagnoses and self-care in originally scarce resource settings. there are many future works that can be carried out based on the findings from this work. firstly, we plan to carry out an extensive human-expert-based evaluation, e.g. using hope metric @xcite , looking into the differences between the outputs of these two mmplms, such as on translating multi-word expressions in the clinical domain @xcite . we also designed corresponding measurements on the evaluation of uncertainty and interrater reliability (irr) levels @xcite . secondly, we think it is valuable to integrate more high-performance automatic metrics into the comparison such as hlepor @xcite . finally, we will try more external languages from different typologies in future work. limitations 1) on plm capability for transferring to new language, in this work, we used meta-ai's wmt21 multilingual pre-trained language models as our test-base for the knowledge transfer into an external language fine-tuning and translation. this new-language ability is much dependent on the mplms we used, such as wmt21fb @xcite as a huge size model, a conditional generation from meta-ai's massive m2m-100 model @xcite . if we try to fine-tune a bilingual model on an external language that the plm did not see, it will not be that good because for smaller-sized models such fine-tuning would be too much of a change, and the model will lose generalisation which leads to problems. for huge multilingual plm models, the 250k of fine-tuning data is a small set of numbers, and that's why the model does not lose generalisation and captures new data well without losing linguistic knowledge of other languages that it was trained on. 2) on the impact of language families, the mmplm wmt21fb we deployed has both alphabetic languages and cjk (chinese, japanese, koran) character languages, as well as slavic language (russian). this might make it easier to transfer to a new language, e.g. alphabetic language. however, in situations when the mplms did not include any of the language scripts that belong to the language family of the target one, it can be much harder for it to transfer to the new target language. this needs further investigation. one possible extension of this work is using the dynamic vocabulary method proposed by @xcite . research projects meta-ai's wmt21.dense-24wide.en-x (2021) and nllb (2022) we used. this work has been partially supported by grant ep/v047949/1 "integrating hospital outpatient letters into the healthcare data space"(funder: ukri/epsrc).