FJWU participation for the WMT 20 Biomedical Translation Task (2020)

Abstract: This paper reports system descriptions for FJWU-NRPU team for participation in the WMT20 Biomedical shared translation task. We focused our submission on exploring the effects of adding in-domain corpora extracted from various out-of-domain sources. Systems were built for French to English using in-domain corpora through fine tuning and selective data training. We further explored BERT based models specifically with focus on effect of domain adaptive subword units.

## introduction in this paper, we present neural machine translation (nmt) systems developed by fatima jinnah women university for participation in wmt20, biomedical shared translation task. the systems are developed for translating english/french (en/fr) in both directions for biomedical domain using fairseq @xcite and bert @xcite . to tackle in-domain corpus shortage challenge, selective data training and fine tuning are explored. we focused our submission on investigating the effects of adding in-domain corpora extracted from out-of-domain sources of various domains, objective was to study the effect of domain non-relatedness in schemes involving data selection through information retrieval or any sentence selection method. we further explored bert based models specifically with focus on effect of domain adaptive subword units. neural machine translation systems have shown substantial growth with the ongoing introduction of new tool kits and training techniques to support developers in training models @xcite . but the availability and cleaning of domain related corpora to achieve terminology advantage and fluency is still a challenge for many researchers as the accessible corpora is relatively small in size and comparatively noisy. in order to improve in-domain nmt systems, out-of-domain data is used and the most common method is to fine tune pre trained nmt models on in-domain data and selective data training @xcite . our last years submission presented promising results using selective data training incorporating data retrieved from news commentary corpus by building two layered rnn systems. we extend our framework to study the quality of retrieved sentences from 3 more parallel corpora. we did not restrict to parallel data for mining biomedical sentences, rather this year we included monolingual data in our framework and studied the effect of using back translations (bt) in our framework. for building nmt models we explored subword units and report the results on using pre-trained bert fused embedding. ## data selection architecture improving translation quality is a challenging task especially for domains where enough in-domain parallel corpus is not available to train a good translation system. to overcome data scarcity problem, several data selection techniques have been proposed over the years including information retrieval (ir) @xcite , edit distances @xcite , cross entropy measures @xcite and several others. we used the approach of relative query sentences using information retrieval to retrieve matching sentences from general domain corpora. french-english is not a resource scarce language pair and has numerous parallel corpora available for various domains. there exist sizable corpus for the biomedical domain to train the initial systems, but the great difference of terminologies and language jargon in various sub domains makes it challenging as the results of previous years bio medical tasks indicate. parallel corpora extracted from "other" easily available corpora, like comparable corpora and monolingual corpora do help improve mt performance @xcite . but, what is the effect of the domain of the corpus used to find the related sentences, is the question we focus on in our data selection design. our aim is to study the improvements achieved by using the sentences from different genre/domain of corpora. however, to be able to extract sizeable amount of biomedical sentences, the corpora should not be very unrelated, for example, the europarl corpus @xcite which is composed of parliament proceedings would not be a good choice foot_0 . our intent was to do a comparative study of quality of extracted sentences from varied but yet not too far off domain corpora. thus, for mining related sentences from general domain corpora we used books foot_1 , news commentary foot_2 and wikipedia foot_3 corpus obtained from open parallel corpus (opus) @xcite . french wikipedia foot_4 (frwikipediamono) was also used which was available as monolingual corpus and was translated to english. our data selection strategy is graphically presented in figure 1 . we followed the data selection approach based on ir as proposed by @xcite . the choice of corpus to use as queries was a critical one: queries should have maximum biomedical terminologies to enable targeting and choosing domain specific sentences from the general domain corpora. we chose medline titles as queries hypothesising on the fact that the title essentially contains the specific domain terminology. we used english side of medline titles as queries when retrieving similar sentences from english side of the corpora and french side of medline titles as queries for ir from french side. we retrieved 10-best sentences and experimented with top-1, top-2 and top-3 sentences as shown in section 4. ## corpora in this section, we present details of corpora used to train our systems, pre-processing and training parameters. we used the in-domain corpora provided by the organizers along with our mined in-domain sentences from the general domain corpora. the in-domain corpora included were: ## related work numerous challenges arise when dealing with biomedical data used for translation due to limited size of corpus and unstructured alignments. various approaches have been adopted by researchers in wmt biomedical translation. @xcite submitted a nmt system that combined in-domain data set and used transfer learning approach to train the model along with ensemble learning. @xcite trained by using transformer architecture using biomedical and news domain and employed cascaded word segmentation along with bpe. @xcite ) emphasize on using multi-source approach like romance languages with in-domain data by implementing transformer architecture using opennmt in pytorch. @xcite created terminology list for biomedical words using babelnet api, inserted the information at a token level and trained nmt system using transformer model @xcite . @xcite used selective learning for building additional corpus from out-of-domain data and incorporated transfer learning approach by using recurrent encoder decoder nn model for training of in-domain biomedical data. @xcite trained their transformer model on in-domain and out-of-domain data for six translations using transfer learning methods. the model used attention mechanism along with relu activation function yielding better results for in-domain biomedical data. @xcite used transfer learning using bayesian interpolation for multi-domain data for ensemble weighting. @xcite participated in wmt19 with four translation directions by creating concatenating corpora from umls, out-of-domain and in-domain data and trained the systems using transformer model. ## conclusion in this paper, we present our submission for wmt20 biomedical tasks. our model trained for english to french language direction ranked third in official scores provided by wmt20. we trained different models to investigate the performance of bert-fused nmt over transformer model and to explore the effect of selective data training in bert-fused nmt for french to english language direction. results show decline in performance of bert-fused nmt models over transformer architecture as general domain bert suppressed the learned parameters from in-domain training corpus. bert-fused models yielded better results when fine tuned on in-domain corpus and trained with ir data.