Optum’s Submission to WMT 22 Biomedical Translation Tasks (2022)

Abstract: This paper describes Optum’s submission to the Biomedical Translation task of the seventh conference on Machine Translation (WMT22). The task aims at promoting the development and evaluation of machine translation systems in their ability to handle challenging domain-specific biomedical data. We made submissions to two sub-tracks of ClinSpEn 2022, namely, ClinSpEn-CC (clinical cases) and ClinSpEn-OC (ontology concepts). These sub-tasks aim to test translation from English to Spanish. Our approach involves fine-tuning a pre-trained transformer model using in-house clinical domain data and the biomedical data provided by WMT. The fine-tuned model results in a test BLEU score of 38.12 in the ClinSpEn-CC (clinical cases) subtask, which is a gain of 1.23 BLEU compared to the pre-trained model.

## introduction the quality of neural machine translation (nmt) was boosted by the use of recurrent neural networks (rnn) for machine translation. in this approach, the source sentence is fed to an encoder which outputs a context vector. this context vector is fed to the decoder to output the target language text @xcite . some approaches also use long short term memory @xcite for this task @xcite . machine translation (mt) systems after seeing great progress in recent years have been found to be sensitive to synthetic and natural noise in input, distributional shift, and adversarial examples @xcite @xcite @xcite . fine-tuning has proven to be a successful technique to carry out this task. one of the most prominent variations is described in @xcite , which trains an nmt model on out-of-domain corpora until model convergence and then resumes training from step 1 on a mix of in-domain and out-of-domain data. a fine-grained human evaluation research of the transformer based systems and state-of-the-art recurrent systems was carried out on the translation from english to chinese. the evalution results shows reduction in errors by 31 percent and significantly less errors in 10 out of 22 error categories when using transformer based mt systems. @xcite . another research has shown that improved efficiency and accuracy can be obtained by converting a pre-trained transformer into its efficient recurrent counterpart. a swap procedure is implemented which replaces softmax attention of a pertained transformer with its linear-complexity recurrent alternative followed by fine-tuning. finetuning has proven to help reduce the training cost and improve efficiency and accuracy @xcite . we took part in wmt 2022 biomedical translation task from english to spanish using the finetuning approach on the transformer based models and we describe our efforts in this paper. the paper is structured as follows. the data sets and their preparation is outlined in section 3 and section 4, followed by details of the experiments carried out and their results in section 5. we then present the summary of our findings and conclusion in section 6. ## related work machine translation systems out of domain performance has been negatively impacted to the extent that they completely sacrifice adequacy for the sake of fluency. hence, the presence of domain inconsistency is considered a key challenge in machine translation @xcite . the common approach to tackle this challenge is firstly to train an mt system on a (generic) source domain and secondly to fine-tune it on a (specific) target domain @xcite @xcite , followed by continuous fine-tuning of data sets which are similar to the target domain @xcite , or to dynamically change the balance of data towards the target domain @xcite . an alternative approach is to train systems on multiple domains simultaneously, while adding domain-specific tags to the input examples @xcite . other methods include the works around dual contextual (dc) module, which is an extension of the conventional self-attention unit, to effectively make use of both, local and global contextual information. this work aims to further improve the sentence representation ability of the encoder and decoder sub-networks, thus enhancing the overall performance of the translation model ( ampomah @xcite . domain adaptation methods include instance weighing, data selection @xcite and incorporating a domain classifier @xcite . some language pairs do not have enough parallel text for training. hence, to counter the data sparsity problem of the nmt training some have used various strategies like augmenting training data, exploiting training data from other languages, alternative learning strategies that use only monolingual data @xcite . some of the researchers have made use of monolingual data available either in the target domain, for example, by training the decoder on these data sets @xcite , or by back-translating @xcite , or in the source domain, using similar techniques @xcite . ## data in the experiments described in this paper, we use data sets from both the general and clinical domains. paracrawl, emea, and wmt are available in the public domain, while, m&r letters is a data set internal to optum. the m&r in-domain data set comprises of medical claim correspondence letters sent to the insurance customers which have been manually translated to spanish. among the public data sets, paracrawl is the largest publicly available parallel corpora for european languages. emea is a multi-lingual parallel corpus made out of pdf documents from the european medicines agency. we have used data from all three subtracks namely, clinical cases, clinical terminology, and ontology concepts of the clinspen data set provided by wmt. ## data preparation the data preparation very closely follows the steps outlined in @xcite . the additional steps are listed below. ## experiments and results as described in the data section (3), we are using data sets from both the general domain and medical/clinical domain. to fine-tune the model, we have a 2-gpu setup with a docker container deployed on on-premise machines containing all the required packages to fine-tune the opus en-es translation model foot_0 . we use huggingface transformers library @xcite for all our experiments. the following fine-tuning experiments are done on the transformer model used by the helsinki-nlp opus-mt-en-es model. as evident from its model card, this model was trained on general-purpose english to spanish training corpus and in these experiments, we will try to fine-tune the model to the clinical domain. since the data provided by the sub-task was limited, we used the entire wmt 2022 data as training data and used train-test splits on other clinical domain data sets to test the success of fine-tuning. one of our key observation while doing the experiments and serving these models on production systems were that they regularly need to be checked for over-fitting and hallucination errors. in addition to evaluation by bleu scores, we do a "sanity check" by running an inference with source language strings of various lengths to mimic handwritten text and check if the translation is not adding extra tokens. ## conclusion we fine-tuned a publicly available model in multiple ways using different combinations of data from various sources. we showed how fine-tuning is sensitive to new domains and can show promising results if done diligently. this paper shows the results of fine-tuning on a single domain but we think that fine-tuning on any new domain would provide gains in the translation quality. the scale of this gain, however, can depend on the amount of training data available in that particular domain.