Look It Up: Bilingual and Monolingual Dictionaries Improve Neural Machine Translation (2020)

Abstract: Despite advances in neural machine translation (NMT) quality, rare words continue to be problematic. For humans, the solution to the rare-word problem has long been dictionaries, but dictionaries cannot be straightforwardly incorporated into NMT. In this paper, we describe a new method for “attaching” dictionary definitions to rare words so that the network can learn the best way to use them. We demonstrate improvements of up to 3.1 BLEU using bilingual dictionaries and up to 0.7 BLEU using monolingual source-language dictionaries.

## introduction despite its successes, neural machine translation (nmt) still has unresolved problems. among them is the problem of rare words, which are paradoxically very common because of zipf's law. in part, this is a problem intrinsic to data-driven machine translation because the system will inevitably encounter words not seen in the training data. in part, however, nmt systems seem particularly challenged by rare words, compared with older statistical models. one reason is that nmt systems have a fixedsize vocabulary, typically 10k-100k words; words outside this vocabulary are represented using a special symbol like unk. byte pair encoding (bpe) breaks rare words into smaller, more frequent subwords, at least allowing nmt to see them instead of unk (sennrich et al., 2016). but this by no means solves the problem; even with subwords, nmt seems to have difficulty learning translations of very rare words, possibly an instance of catastrophic forgetting (mccloskey and cohen, 1989). humans deal with rare words by looking them up in a dictionary, and the idea of using dictionaries to assist machine translation is extremely old. from a statistical perspective, dictionaries are a useful complement to running text because the uniform distribution of dictionary headwords can smooth out the long-tailed distribution of running text. in pre-neural statistical machine translation systems, the typical way to incorporate bilingual dictionaries is simply to include them as parallel sentences in the training data. but (as we show), this does not work well for nmt systems. we are aware of only a few previous attempts to find better ways to incorporate bilingual dictionaries in nmt. some methods use dictionaries to synthesize new training examples (zhang and zong, 2016; qi et al., 2018; hämäläinen and alnajjar, 2019). arthur et al. (2016) extend the model to encourage it to generate translations from the (automatically extracted) dictionary. post and vilar (2018) constrain the decoder to generate translations from the dictionary. what these approaches have in common is that they all treat dictionary definitions as target-language text, when, in fact, they often have properties very different from ordinary text. for example, cedict defines 此致 (cǐzhì) as "(used at the end of a letter to introduce a polite salutation)" which cannot be used as a translation. in the case of a monolingual source-language dictionary, the definitions are, of course, not written in the target language at all. in this paper, we present an extension of the transformer (vaswani et al., 2017) that "attaches" the dictionary definitions of rare words to their occurrences in source sentences. we introduce new position encodings to represent the nonlinear structure of a source sentence with its attachments. then the unmodified translation model can learn how to make use of this attached information. we show that this additional information yields improvements in translation accuracy of up to 3.1 bleu. because our method does not force dictionary definitions to be treated as target-language text, it is generalizable to other kinds of information, such as monolingual source-language dictionaries, which yield smaller improvements, but still as much as 0.7 bleu. ## methods our method is built on top of the transformer (vaswani et al., 2017). for each unknown source word with an entry in the dictionary, we attach the first 50 tokens of the definition (discarding the rest of the definition) to the source sentence. as described below, we encode the definition so as to differentiate it from the source sentence proper and to record which source word the definition is attached to. we leave the task of deciding whether and how to use the definition up to the translation model, which we use without any modifications. ## experiments in this section, we describe our experiments on chinese-english and german-english translation, comparing our method (which we call attach) against two baselines. one baseline is the standard transformer without any dictionary information (which we call baseline). the other baseline is the standard transformer with the bilingual dictionaries included as parallel sentences in the training data (which we call append). ## analysis to further examine how our methods improve translation, we looked at some examples in our um-spoken dev set, shown in definitions indicates that the word is outside of the system's vocabulary. in the first example, 对称性 (duìchènxìng, symmetry) is unknown to the word-based systems. adding the definition to the parallel training data (append) does not help word-based translation because the word remains unknown, whereas our model correctly generates the translation symmetry. with bpe, the word is broken into three pieces, so that the append system can correctly generate the word symmetry. but the third character (性, xìng) can also mean "sex," and together with the following character (性感, xìnggaň) can mean "sexy." this explains why the append system incorrectly adds the words of sex. ## discussion in section 1, we mentioned several other methods for using dictionaries in nmt, all of which treat dictionary definitions as target-language text. an alternative approach to handling rare words, which avoids dictionaries altogether, is to use word embeddings trained on large amounts of monolingual data, like fasttext embeddings (bojanowski et al., 2017). qi et al. (2018) find that fasttext embeddings can improve nmt, but there is a sweet spot (likely between 5k and 200k lines) where they have the most impact. they also find that pre-trained embeddings are more effective when the source and target languages are similar. we, too, experimented with using fasttext word embeddings in our nmt system, but have not seen any improvements over the baseline -perhaps because our datasets are somewhat larger than those used by qi et al. (2018). we also experimented with using dictionaries to improve word embeddings and found that the present approach, which gives the model direct access to dictionary definitions, is far more effective. the most significant limitation of our method is runtime: because it increases the length of the source sentences, training and decoding take 2-3 times longer. another limitation is that the effectiveness of this method depends on the quality and coverage of the dictionaries. in the future, we plan to experiment with additional resources, like thesauruses, gazetteers, or bilingual dictionaries with a different target language. second, from our examples, we see that our model is able to select a snippet of the definition and adapt it to the target context (for example, by inflecting words), but further analysis is required to understand how much the model is able to do this. finally, our method currently requires an exact match between a dictionary headword and a source word; we plan to extend the model to enable matching of headwords with inflected forms. ## conclusion in this paper, we presented a simple yet effective way to incorporate dictionaries into a transformer nmt system, by attaching definitions to source sentences to form a nonlinear structure that the transformer can learn how to use. we showed that our method can beat baselines significantly, by up to 3.1 bleu. we also analyzed our system's outputs and found that our model is learning to select and adapt parts of the definition, which it does not learn to do when the dictionary is simply appended to the training data. we also found that our method has some potential to work with monolingual dictionaries. source 1. ich hoffe , dass diese auslassung(unk) korrigiert werden kann . 2. wäre das nicht eine alternativlösung(unk) ? definitions 1. auslassung: omission(unk) 2. alternativlösung: alternative solution reference 1. i hope that this omission can be corrected. 2. would this not be an alternative solution? baseline 1. i hope that this unk can be corrected. 2. would this not be a unk? append 1. i hope that this unk can be corrected. 2. would this not be a unk? attach 1. i hope that this unk can be corrected. 2. would this not be an alternative solution? in the second example, adding the dictionary definition of alternativlösung to the parallel data does not help, but our model is able to incorporate it. bpe source 1. ich hoffe , dass diese aus@@ l@@ assung korrigi@@ ert werden kann . 2. w@@ äre das nicht eine altern@@ ativ@@ lösung ? fused source 1. ich hoffe , dass diese auslassung(unk) korrigi@@ ert werden kann . 2. w@@ äre das nicht eine alternativlösung(unk) ? definitions 1. auslassung: om@@ is@@ sion 2. alternativlösung: alternative solution reference 1. i hope that this omission can be corrected. 2. would this not be an alternative solution? baseline 1. i hope that this approval can be corrected. 2. would this not be a alternative solution? append 1. i hope that this interpretation can be corrected. 2. would this not be a alternative solution? fuse 1. i hope that these pieces can be corrected. 2. would this not be a pronounce? attach 1. i hope that this omission can be corrected. 2. would this not be an alternative solution?