The Highs and Lows of Simple Lexical Domain Adaptation Approaches for Neural Machine Translation (2021)

Abstract: Machine translation systems are vulnerable to domain mismatch, especially in a low-resource scenario. Out-of-domain translations are often of poor quality and prone to hallucinations, due to exposure bias and the decoder acting as a language model. We adopt two approaches to alleviate this problem: lexical shortlisting restricted by IBM statistical alignments, and hypothesis reranking based on similarity. The methods are computationally cheap and show success on low-resource out-of-domain test sets. However, the methods lose advantage when there is sufficient data or too great domain mismatch. This is due to both the IBM model losing its advantage over the implicitly learned neural alignment, and issues with subword segmentation of unseen words.

## introduction neural machine translation (nmt) has achieved state-of-the-art performance in a variety of language pairs and settings @xcite , but it is vulnerable to domain mismatch, where the test set differs significantly from the training data in terms of vocabulary, genre, length, etc. this issue is exacerbated in a lowresource condition @xcite . teacher forcing is used during traditional maximum likelihood neural network training, leading to a strong exposure bias, and model confusion when presented with unexpected sequences. this typically results in hallucinations in the output @xcite , because the overly zealous language model component prefers a fluent translation, as opposed to an adequate one. a number of methods have been proposed in order to tackle the issue: exposing the model to its predictions during training @xcite @xcite 2019; @xcite ; tuning directly towards bleu @xcite or using minimum bayes risk decoding @xcite . a common weakness of such methods is that they are computationally expensive. in this paper, we adopt and experiment with two approaches inspired by previous research. we use lexical shortlisting to interpolate a statistical alignment model with nmt; on top of it, we perform n-best list re-ranking by hypothesis agreement. our aim is to constrain the lexical choice of the decoder, to prevent hallucinations from being generated. the methods are computationally simpler, as they require no change to the model or training. we analyse the effectiveness of these methods in different scenarios of domain adaptation. we show bleu gains on a variety of out-of-domain datasets in a low-resource english-german setting. however, the methods show no improvements once the datasets are large, or the domains are too distant. ## methodology we purse and analyse two separate strategies for improving neural machine translation's performance in low-resource domain mismatched settings: lexical shortlisting and n-best list re-ranking based on inter hypothesis agreement. ## results and analysis we present in we see that the shortlisting method is always superior to the baseline method on all out-of-domain datasets, although the results vary with the data preprocessing. this is true even more so of reranking, which is much better when the bpe vocabulary is learned on all domains. combining both shortlisting and re-ranking always brings in a slight improvement over just re-ranking on the outof-domain datasets, but its effectiveness is again preprocessing-dependent. this clearly shows that the ibm model implemented by fast-align can learn information complementary to nmt, and interpolating them is beneficial for achieving higher bleu scores in out-ofdomain settings. the lower bleu scores on indomain test sets are related to the aggressive shortlist we used: by increasing the output layer limit from 10 to 50 most probable tokens, we achieve identical bleu scores as the baseline. ## limitations although our methods show promising gains on the low-resource domains, we found that they have limited application when the domain mismatch is too great or there is sufficient resource available. ## conclusion we explore computationally cheap methods to improve neural machine translation performance in out-of-domain settings. we suggest that adding a lexical shortlist trained on the same data is always beneficial. while re-ranking also improves bleu, it targets the bleu brevity penalty, and does not produce better word choices. although our results are promising in a low-resource condition, they do not transfer well to a scenario with very distant domains or sufficient resources. our analysis shows that this is due to little vocabulary overlap, and the limited contribution from the ibm model under out-of-domain bpe segmentation.