Improving Rare Word Translation With Dictionaries and Attention Masking (2024)

Abstract: In machine translation, rare words continue to be a problem for the dominant encoder-decoder architecture, especially in low-resource and out-of-domain translation settings. Human translators solve this problem with monolingual or bilingual dictionaries. In this paper, we propose appending definitions from a bilingual dictionary to source sentences and using attention masking to link together rare words with their definitions. We find that including definitions for rare words improves performance by up to 1.0 BLEU and 1.6 MacroF1.

## introduction the current state-of-the-art for machine translation (mt) is still the transformer encoder-decoder architecture @xcite . while large language models such as llama and gpt-4 have achieved great success on various nlp tasks, they still fall behind dedicated encoder-decoders for mt @xcite . a major drawback of encoder-decoder models, however, is that they continue to struggle with rare word translation @xcite . dictionaries, both monolingual and bilingual, are an indispensable resource for human translators, and in pre-neural statistical mt systems, it was common to use bilingual dictionaries to improve translation of rare words @xcite . however, the use of dictionaries in neural mt is not straightforward, as there is a strong dependence on the surrounding context and word frequency in the training data @xcite . in this paper, we explore a new approach for incorporating dictionaries into neural mt systems. we hypothesize that dictionaries could be useful both for low-resource translation, where the target language has limited training data, and out-ofdomain translation, where the testing domain differs significantly from the training domain(s). in addition, dictionaries could facilitate continual learning by enabling zero-shot adaptation of mt systems. however, the morphology of both the source and target language poses a major challenge for the use of dictionaries with mt compared to other nlp tasks that incorporate methods of retrieval-augmented generation @xcite . mt systems that incorporate dictionaries must be capable of inflecting definitions for the target language and the context in which those definitions must appear, as dictionary entries and their definitions are often base forms. in figure 1 , we see the adjective kühner is declined for the genitive case, but only the lemma kühn would be in a german dictionary. moreover, if the target language has adjective declension, then the mt system must also decline the dictionary form of the definition. our approach is to retrieve dictionary definitions for low-frequency words, append the definitions to source sentences containing rare words, and use attention masking to link together rare words with their definitions. we find that appending definitions for rare words improves mt performance by up to 1.0 bleu and 1.6 macrof1. ## related work previous work on dictionaries for neural mt can be divided into two broad categories, which we call dictionaries-as-translators and dictionaries-as-text. in the dictionaries-as-translators approach, the dictionary is assumed to contain high-quality translations of words, and the technical challenge is to get the mt system to use the dictionary's translations when appropriate. in the dictionaries-as-text approach, dictionary entries are added somehow to the source sentence, and it is up to the mt system to learn how to use them. in this approach, the dictionary can contain definitions that are not necessarily translations (e.g., one definition for german halt is: "indicating that something is generally known, or cannot be changed, or the like; often untranslatable"). this approach could, in principle, use other resources like monolingual dictionaries, grammars, and so on. ## methodology our approach falls squarely into the dictionaries-astext category: given a source sentence, we retrieve relevant entries from a bilingual dictionary and include them in the source sentence. to decide which entries to include, we use a source-language lemmatizer, which should be more reliable and faster than fuzzy matching. to represent the input, we use attention masking instead of positional encodings since we suspect that attention is a more natural mechanism by which an encoder-decoder model can associate definitions (keys) with rare words (queries). in this section, we break down our approach for using a bilingual dictionary for machine translation with a transformer-based, encoder-decoder model into the following steps: (1) headword selection, (2) definition retrieval, and (3) attention masking. ## experiments in this section, we describe our translation model, the source-side lemmatizer, and the bilingual dictionary used hereafter throughout the paper. ## results in this section, we report and analyze the results of our experiments described in the previous section. we found that appending definitions for rare words and using attention masking (masking) improved translation performance over the baseline models: baseline, parallel, and dpe. furthermore, we observed that using a lower frequency threshold during training and increasing that threshold during inference resulted in the largest performance improvement. we speculate that this figure 3 : the attention scores of the masking model for the german sentence: "mit seiner tarnkappe schlich sich siegfried aus der burg" with the definition string "invisibility cloak crept slunk tiptoed sigurd." the attention scores are summed for all encoder layers and attention heads. we observe both attention masks being utilized by the model. in the low-resource setting, dpe struggles to improve over the baseline models, while masking has the largest bleu improvement, demonstrating a boost in low-resource translation performance. in the high-resource setting, although dpe and masking are not significantly different in bleu, they are significantly different in macrof1. since rare words are less frequent in the high-resource setting, the bleu improvement of dpe and masking over baseline and parallel is not as large. however, masking has the largest macrof1 improvement, demonstrating a boost in rare word translation performance. in baseline and parallel contain several incorrect translations of rare words. in particular, we observe that tarnkappe and todfeind were translated overliterally, with the first noun in the compound being dropped all together. even the apple translate app translated tarnkappe over-literally as camouflage cap. the dpe model, instead of dropping the first noun like baseline/parallel, used a seemingly random noun and translated the second over-literally. only the masking model correctly translated tarnkappe as invisibility cloak. in fact, masking used at least one definition for every rare word, getting the closest to the reference. in in figure 3 , we use an attention heat map to visualize the attention scores for a german sentence. the sentence shown is a trimmed version of the example in ## discussion rare word classification as mentioned previously, compound words that do not occur in the training data may still be accurately translated as a result of subword segmentation, suggesting that frequency is not an ideal or reliable metric for classifying rare words. in the future, frequency could be replaced with a source-side estimation of model confidence in the translation of rare words. ## conclusion in this paper, we proposed using bilingual dictionaries and attention masking to improve translation performance for rare words, a problem that encoderdecoder models continue to struggle with in mt. our method was to append definitions to source sentences for low-frequency words and use attention masking to associate rare words with their definitions. we found that our method improved mt performance by up to 1.0 bleu and 1.6 macrof1. in the future, we are interested in incorporating other external knowledge sources, such as monolingual dictionaries and knowledge graphs, to reduce translation ambiguity and further improve the translation of rare words.