Addressing Exposure Bias With Document Minimum Risk Training: C ambridge at the WMT 20 Biomedical Translation Task (2020)

Abstract: The 2020 WMT Biomedical translation task evaluated Medline abstract translations. This is a small-domain translation task, meaning limited relevant training data with very distinct style and vocabulary. Models trained on such data are susceptible to exposure bias effects, particularly when training sentence pairs are imperfect translations of each other. This can result in poor behaviour during inference if the model learns to neglect the source sentence. The UNICAM entry addresses this problem during fine-tuning using a robust variant on Minimum Risk Training. We contrast this approach with data-filtering to remove ‘problem’ training examples. Under MRT fine-tuning we obtain good results for both directions of English-German and English-Spanish biomedical translation. In particular we achieve the best English-to-Spanish translation result and second-best Spanish-to-English result, despite using only single models with no ensembling.

## introduction neural machine translation (nmt) in the biomedical domain presents challenges in addition to general domain translation. text often contains specialist vocabulary and follows specific stylistic conventions. for this task fine-tuning generic pretrained models on smaller amounts of biomedicalspecific data can lead to strong performance, as we found in our 2019 biomedical submission @xcite . for our wmt 2020 submission we start with strong single models from that 2019 submission and fine-tune them exclusively on the small medline abstracts training sets @xcite . this allows fast training on very relevant training data, since the test set is also made up of medline abstracts. however, fine-tuning on relevant but small corpora has pitfalls. the small number of training examples exacerbates the effect of any noisy or poorly aligned sentence pairs. we treat this as a form of exposure bias, in that model overconfidence in training data results in poor translation hypotheses at test time. our contributions in this system paper are: ## conclusions our wmt20 biomedical submission investigates improvements on the english-german and english-spanish language pairs under a single strong model. in particular, we focus on the behaviour of models trained on sentences with some predictable irregularities. we find that aggressively filtering target sentences can help overall performance, but that aggressively filtering source sentence tends to hurt performance. we also find that minimum risk training can benefit from imperfectly aligned training examples while reducing the effects of exposure bias.