Y ereva NN ’s Systems for WMT 20 Biomedical Translation Task: The Effect of Fixing Misaligned Sentence Pairs (2020)

Abstract: This report describes YerevaNN’s neural machine translation systems and data processing pipelines developed for WMT20 biomedical translation task. We provide systems for English-Russian and English-German language pairs. For the English-Russian pair, our submissions achieve the best BLEU scores, with en \rightarrow ru direction outperforming the other systems by a significant margin. We explain most of the improvements by our heavy data preprocessing pipeline which attempts to fix poorly aligned sentences in the parallel data.

## introduction biomedical machine translation is a perfect playground to develop narrow domain neural machine translation models. in such tasks, the available parallel in-domain data is usually limited and noisy which creates many challenges. in the previous works @xcite , researchers focused on transfer learning methods @xcite or attempted to mix the training data with other sources @xcite to address the issue of data scarcity. in this work, we show that the transfer performance is very dependent on the quality of the training data, and with a little effort, it is possible to improve the given medline training data and gain a significant performance boost. we have manually created a much higher quality subset of the original medline training data for local evaluation purposes. the insights collected during this manual analysis was then used to fix the most common issues within the training data. in particular, we noticed that the original dataset contained paper abstracts in two languages without sentence-level alignments, and the training corpus provided by the organizers was created using an automated sentence segmentation and alignment process, which was not perfect. we built a data pipeline foot_0 that handles 1) cleanup, 2) sentence segmentation, 3) alignment of translation sentence pairs and 4) preprocessing. in our experiments, we did not use any data source other than medline. we chose our baseline model and two other models with the highest bleu scores on a local test set as our three submissions. the best ones got 35.2% bleu on english-german and 41.3% on german-english test sets. for english-russian and russian-english directions we reached bleu scores of 37.9% and 43.2% respectively, which are the best scores among all submissions of wmt20 biomedical translation task. moreover, our models are cheap to train: the average training time of our best models is approximately 30 minutes on a single nvidia titan v gpu. the paper is organized as follows: section 2 presents fine-tuning details and evaluation methods for our nmt systems, section 3 describes the data used in the experiments and the data processing pipeline, section 4 presents our novel method of monotonic alignment based on multilingual language models. section 5 discusses the results. ## monotonic alignments the problems of the training set described in section 3.1 can be caused by poor 1) xml parsing, 2) sentence segmentation, or 3) monotonic segment alignment method. here we describe a novel method for monotonic sentence alignment using multilingual language models and discuss the contribution of its hyperparameter choices. multilingual language models have been previously shown to be effective in parallel data mining @xcite . we also compare our approach to the baseline data pipeline by the shared task organizers which is based on syntok segmentation system and gma @xcite . our method of monotonic sentence alignments is as follows: we calculate a similarity matrix of all source-target candidate pairs and decode pairs to maximize the similarity of the resulting sentence pairs. we consider two approaches for the decoding step: greedy and dynamic. ## results for wmt20 biomedical translation task we prepared three submissions: run1 for all directions was the baseline model, while for run2 and run3 we chose the best models according to their bleu score on the local test set at the time of the submission. in run2 and run3, all the models besides @xmath0. the official bleu scores on samples with "ok" aligned sentences alongside with our local test set are presented in for @xmath1, backtranslation data was collected with beam search (size of 8), in case of @xmath2, we had noise added similar to @xcite algorithm 2: one-to-many (k) dynamic decoding s n , t m ← source and target sentences for run2 and run3, we used v2 preprocessing, the sentence splitting was done with scispacy (for english and german) and a slightly modified version of razdel (for russian). after our submissions, we further improved our data pipeline. fixing the issues of the standard preprocessing (v2 vs. v1) gives a significant boost, especially when decoding to russian (@xmath3. the effect of training with bigger batch sizes gives only a slight improvement, while the absolute training duration reduces drastically. as mentioned previously, there were issues with section names and titles in the provided parsed documents. after addressing these issues, our greedy approach gives better alignments. the total similarity optimization using dynamic programming is not always better than the greedy method, but the performance improves for @xmath4.1% bleu score. overall, the new data pipeline gives an enhancement in nmt performance: +1.6% bleu for @xmath5.5% bleu score for @xmath6. although we observe consistent performance improvement for both directions en↔ru, the effect for @xmath7. we could not determine the reason for such assymmetry. ## conclusion this work presents the systems our team developed for english-german and english-russian language pair tracks of wmt20 biomedical translation task. we achieve the best results on the official test set for english↔russian language pair, outperforming competitors by a significant margin on @xmath8. we show that it is possible to improve the performance of neural machine translation models by simply improving the quality of the in-domain parallel data. the suggested method for monotonic sentence segment alignment based on pretrained multilingual language models demonstrated promising results. we explored how different components of our data processing pipeline contributed to the quality of the resulting translation systems. in future work, we plan to investigate the applicability of this pipeline to a wider set of language pairs and domains.