FJWU Participation for the WMT 21 Biomedical Translation Task (2021)

Abstract: In this paper we present the FJWU’s system submitted to the biomedical shared task at WMT21. We prepared state-of-the-art multilingual neural machine translation systems for three languages (i.e. German, Spanish and French) with English as target language. Our NMT systems based on Transformer architecture, were trained on combination of in-domain and out-domain parallel corpora developed using Information Retrieval (IR) and domain adaptation techniques.

## introduction due to vast availability of multilingual information, neural machine translation (nmt) systems have achieved remarkable growth over statistical machine translation (smt) systems. although the amount of training resources has significantly increased in the past few years but availability of large in-domain parallel data is still a challenging task. performance of nmt system may quickly degrade as soon as the application domain deviates from training domain. domain adaptation @xcite is a promising active research topic to enhance the translation quality when faced with data scarcity issues. in domain adaptation, initially large amount of parallel out-ofdomain corpora is utilized for training nmt models and then fine-tuning is performed on small indomain data for adapting to novel domains @xcite . fine-tuning does not require building system from scratch, instead it is fast and efficient method of integrating in-domain data. an nmt model already trained on general domain data is further fine-tuned on in-domain data with less time and effort @xcite . training mt systems on back-translated data is a proven domain adaptation method @xcite , where synthetic parallel data is combined with original data to generate large indomain training corpus. in addition, information retrieval (ir) technique to extract relevant sentences from out-of-domain corpus has shown promising results to overcome data scarcity @xcite . nmt system incorporating multiple languages into single model is known as multilingual nmt (mnmt) @xcite . multilingual nmt systems are gaining popularity due to effective use of available resources and boosting translation quality with translation knowledge transfer @xcite . in this paper, we present study on adapting mnmt systems (multiway many-to-one) for translating english (en) language from french (fr), german (de) and spanish (es) using fairseq @xcite implementation of transformer model. our main focus is to investigate the effect on en translation in biomedical domain using multilingual nmt systems. we have also explored the domain adaptation for fine-tuning of bilingual nmt models into multilingual nmt models using outdomain and in-domain corpora. furthermore, we show the effectiveness of utilizing in-domain data generated through ir techniques @xcite by training a nmt system on combined parallel in-domain data. we also compare in-domain multilingual and bilingual models. the remainder of this paper is organized as follows. section 2 introduces the literature review followed by corpus processing in section 3. section 4 presents experiments and results. in section 5, we conclude the findings of our work. ## literature review mnmt models tend to acquire knowledge from more than one language which helps in generalization and in building systems for low resource languages. mnmt models may help in miti- corpus de/en es/en fr/en in-domain training data ufal 2.6 m 631 k 2.6 m scielo health -124 k 9 k scielo biological -581 k -edp --3 k medline titles -285 k 612 k medline abstracts 18 k 66 k 46 k emea 1.10 m 1.09 m 1.09 m in-domain ir training data news commentary-ir2 --65 k wikipedia-ir2 --84 k out-domain training data ufal 30.9 m 74.8 m 73.3 m ufal dictionary 733 k 544 k 744 k scielo -433 k -un -21.9 m 25.8 k development data medline18 321 239 311 medline19 439 437 400 test data medline20 409 466 479 ## corpus pre-processing this section describes parallel corpora used in training and evaluation of our models. statistics of train, development and test data are presented in main sources of data were provided by wmt21 biomedical translation task. data sources include: from experiments conducted by @xcite , corpora with top-2 best sentences gave good results in training nmt models for biomedical domain. medline18 and 19 testsets are used as development set. we used medline20 testset provided by wmt20 @xcite as initial test sets to determine quality of our translation models. preprocessing of data include tokenization and learning joint byte pair encoding (bpe) @xcite using sentencepiece foot_3 with a vocabulary size of 32k over in-domain corpus and encoding all available corpora with learned bpe. ## experiments and results in this section we present details of experimentation along with training configurations. ## conclusion in this paper we have described our system submissions at wmt21 biomedical shared translation task under fjwu's submission. for our submission we trained multilingual nmt systems for german, spanish and french languages with english as target language. we focused on utilizing in-domain and out-domain parallel corpora and domain adaptation techniques for training multilingual nmt systems. we showed that, domain adaptation using fine-tuning of multilingual nmt model can be a reasonable alternative to achieve good translation quality for novel domains.