A Multilingual Neural Machine Translation Model for Biomedical Data (2020)

Abstract: We release a multilingual neural machine translation model, which can be used to translate text in the biomedical domain. The model can translate from 5 languages (French, German, Italian, Korean and Spanish) into English. It is trained with large amounts of generic and biomedical data, using domain tags. Our benchmarks show that it performs near state-of-the-art both on news (generic domain) and biomedical test sets, and that it outperforms the existing publicly released models. We believe that this release will help the large-scale multilingual analysis of the digital content of the COVID-19 crisis and of its effects on society, economy, and healthcare policies. We also release a test set of biomedical text for Korean-English. It consists of 758 sentences from official guidelines and recent papers, all about COVID-19.

## motivation the 2019-2020 coronavirus pandemic has disrupted lives, societies and economies across the globe. its classification as a pandemic highlights its global impact, touching people of all languages. digital content of all types (social media, news articles, videos) have focused for many weeks predominantly on the sanitary crisis and its effects on infected people, their families, healthcare workers and the society and economy at large. this calls not only for a large set of tools to help during the pandemic (as evidenced by the submissions to this workshop), but also for tools to help digest and analyze this data after it ends. by analyzing the representation and reaction across countries with different guidelines or global trends, it might be possible to inform policies in prevention of and reaction to future epidemics. several institutions and groups have already started to take snapshots of the digital content shared during these weeks @xcite . however, because of its global scale, all this digital content is accessible in a variety of different languages, and most existing nlp tools remain english-centric @xcite . in this paper we describe the release of a multilingual neural machine translation model (mnmt) that can be used to translate biomedical text. the model is both multi-domain and multilingual, covering translation from french, german, spanish, italian and korean to english. our contributions consist in the release of: this paper is structured as follows: in section 2 we overview previous work upon which we build; section 3 details the model and data settings, and the released test set; and section 4 compares our model to other public models and to state-of-the-art results in academic competitions. the model can be downloaded at https:// github.com/naver/covid19-nmt . the repository contains a checkpoint that is compatible with fairseq @xcite , a script to preprocess the input text, and our released korean-english test set. ## related work in order to serve its purpose, our model should be able to process multilingual input sentences, and generate tailored translations for covid-19related sentences. as far as nmt models are concerned, both multilingual and domain-specific sentences are just sequences of plain tokens that should be distinguished internally and handled in a separate manner depending on the multiple languages or domains. due to this commonality in both fields of mnmt and domain adaptation of nmt models, they can be broadly categorized into two groups: 1) data-centric and 2) model-centric @xcite . the former focuses on the preparation of the training data such as handling and selecting from multi-domain @xcite or multilingual parallel corpora @xcite ; and generating synthetic parallel data from monolingual corpora @xcite . the model-centric approaches, on the other hand, center on adjusting the training objectives @xcite ; modifying the model architectures @xcite ; and tweaking the decoding procedure @xcite . while the two types of approaches are orthogonal and can be utilized in tandem, our released model is trained using data-centric approaches. one of the frequently used data-centric methods for handling sentences of multiple languages and domains is simply prepending a special token that indicates the target language or domain that the sentence should be translated into @xcite . by feeding the taskspecific meta-information via the reserved tags, we signal the model to treat the following input tokens accordingly. recent works show that this method is also applicable to generating diverse translations @xcite and translations in specific styles @xcite . in addition, back-translation of target monolingual or domain-specific sentences is often conducted in order to augment the low-resource data @xcite . the back-translated data (and existing parallel data) can be filtered @xcite ; treated with varying amount of importance @xcite using data selection methods; and tagged to achieve even better results @xcite . while myriads of research works on mnmt and domain adaptation exist, the number of publicly available pre-trained nmt models is still low. for example, fairseq, a popular sequence-tosequence toolkit maintained by facebook ai research, has released ten uni-directional models for translating english, french, german, and russian sentences. 1 for its widespread usage, we trained our model using this toolkit. a large number of public mt models are available thanks to opus-mt, foot_1 created by the helsinki-nlp group. utilizing the opus corpora (tiedemann, 2012), more than a thousand mt models are trained and released, including several multilingual models which we use to compare with our model. to the best of our knowledge, we release the first public mnmt model that is capable of producing tailored translations for the biomedical domain. the covid-19 pandemic has shown the need for multilingual access to hygiene and safety guidelines and policies @xcite . as an example of crowd-sourced translation, we point out "the covid translate project" foot_2 which allowed the translation of 75 pages of guidelines for public agents and healthcare workers, from korean into english in a matter of days. although our model could assist in furthering such initiatives, we do not recommend relying solely on our model for translating such guidelines, where quality is of uttermost importance. however, the huge amount of digital content created in the last months around the pandemic makes such professional translations of all that content not only infeasible, but sometimes unnecessary depending on the objective. for instance, we believe that the release of this model can unlock the possibility of large-scale translation with the aim of conducting data analysis on the reaction of the media and society on the matter. ## model settings and training data the model uses a variant of the transformer big architecture @xcite with a shallower decoder: 16 attention heads, 6 encoder layers, 3 decoder layers, an embedding size of 1024, and a feed-forward dimension of 8192 in the encoder and 4096 in the decoder. as all language pairs have english as their target language, no special token for target language was used (language detection can be performed internally by the model). as the model performs many-to-english translation, its encoder should be able to hold most of the complexity. thus, we increase the capacity of the encoder by doubling the default size of the feed-forward layer as in @xcite . md on the other hand, previous works @xcite have shown that it is possible to reduce the number of decoder layers without sacrificing much performance, allowing both faster inference, and smaller network size. during training, regularization was done with a dropout of 0.1 and label smoothing of 0.1. for optimization, we used adam (kingma and ba, 2014) with warm-up, and maximum learning rate of 0.001. the model was trained for 10 epochs and the best checkpoint was selected based on perplexity on the validation set. as training data, we used the standard openaccessible datasets, including biomedical data whenever available, for example, the "corona crisis corpora" (taus, 2020). following our past success in domain adaptation @xcite , we used domain tokens @xcite to differentiate between domains, allowing multi-domain translation with a single model. we initially experimented with more tags, and combinations of tags (e.g., @xmath0. the results however were not very conclusive, and often underperformed. an exception worth noting was the case of transcribed data such as ted talks, and opensubtitles, which are not the main targets of this work. therefore, for simplicity, we used only two tags: medical and back-translation. no tag was used with training data that does not belong to one of these two categories. in addition to biomedical data, we also used back-translated data, although only for korean, the language with the smallest amount of training data (13.8m sentences). like arivazhagan et al. (2019) , we used a temperature parameter of 5, to give more chance to korean. additionally, the biomedical data was oversampled by a factor of 2. as for pre-processing, we cleaned the available data by conducting white-space normalization and nfkc normalization. we filtered noisy sentence pairs based on length (min. 1 token, max. 200), and automatic language identification with langid.py @xcite . we trained a lower-cased shared bpe model using sentencepiece @xcite by using 6m random lines for each language (including english). we filtered out single characters with fewer than 20 occurrences from the vocabulary. this results in a shared vocabulary of size 76k. we reduced the english vocabulary size to speed up training and inference, by setting a bpe frequency threshold of 20, which gives a target vocabulary of size 38k. to get the benefits of a shared vocabulary (i.e., tied source/target embeddings) we sorted the source fairseq dictionary to put the 38k english tokens at the beginning, which lets us easily share the embedding matrix between the encoder and the decoder. 4 the bpe segmentation is followed by inlinecasing @xcite , where each token is lower-cased and directly followed by a special token specifying its case (<@xmath1, <@xmath2, no token for lower-case). word-pieces whose original case is undefined (e.g., "macdonalds") are split again into word-pieces with defined case ("mac" and "donalds"). ## benchmarks we benchmark the released multilingual models against: 1) reported numbers in the literature, and 2) other publicly released models. we use opus-mt, a large collection (1000+) of pre-trained models released by the nlp group at university of helsinki. note that these models were trained with much smaller amounts of training data. our single model obtains competitive results on "generic" test sets (news and iwslt), on par with the state of the art. 9 we also obtain state-of-the-art results on the wmt19 biomedical test sets. note the sota models were trained to maximize performance training on the medline data. while we included this data in our tagged biomedical data, we did not fine-tune aggressively over it. as shown in ## conclusion we describe the release of a multilingual translation model that supports translation in both the general and biomedical domains. our model is trained on more than 350m sentences, covering french, spanish, german, italian and korean (into english). benchmarks