Continuous Learning in Neural Machine Translation using Bilingual Dictionaries (2021)

Abstract: While recent advances in deep learning led to significant improvements in machine translation, neural machine translation is often still not able to continuously adapt to the environment. For humans, as well as for machine translation, bilingual dictionaries are a promising knowledge source to continuously integrate new knowledge. However, their exploitation poses several challenges: The system needs to be able to perform one-shot learning as well as model the morphology of source and target language. In this work, we proposed an evaluation framework to assess the ability of neural machine translation to continuously learn new phrases. We integrate one-shot learning methods for neural machine translation with different word representations and show that it is important to address both in order to successfully make use of bilingual dictionaries. By addressing both challenges we are able to improve the ability to translate new, rare words and phrases from 30% to up to 70%. The correct lemma is even generated by more than 90%.

## introduction recent advances in neural machine translation (nmt) have led to astonishing translation quality of research systems in evaluation campaigns as well as for commercial systems. these improvements even led to discussions whether automatic machine translation is already on par with human translation @xcite . one challenge that has raised less attention is the ability of these systems to continuously learn over time. in contrast, humans are continuously improving their skills and adapting to an ever-changing environment. there are several reasons why this is necessary: first, nobody is fluent in all possible domains. even professional translators need to adapt to the specific vocabulary of different domains. secondly, language is not static but developing over time and translators need to learn new terms, meanings and expressions. for humans, one successful approach to adapt to the environment is the usage of a dictionary foot_0 . learning translations from a dictionary has several advantages: dictionaries contain minimal examples. we do not need to collect full sentences, but can directly learn translations from a single phrase. furthermore, this can even be generalized to other inflected forms of the same lexem. secondly, it enables the system to directly integrate correction. if a user sees a specific problem, the user can interact with the system by adding a specific dictionary entry. this is very important if a specific terminology should be used. motivated by the success for human translators, in this work we will enable nmt to also successfully integrate knowledge from bilingual dictionaries. thereby, we will focus on learning translations that could not be learned from parallel data. this poses several interesting research challenges as shown in the example in the system needs to be able to continuously learn new dictionary entries and then should directly be able to translate all occurrences of this phrase. secondly, the model must be aware of the morphology of the source and target language. in a dictionary only the base form of a word is given. in the example only the lemma giraffe is in the dictionary, but not the plural form giraffes. therefore, we must enable the system to translate different lexemes of a lemma by knowing only the translation of the base form. this involves analysing the mor-source: tell us, what have you got against giraffes ? dictionary: @xmath0: in order to assess the approaches on this challenging condition, it is essential to define an appropriate evaluation scheme. while the ability to continuously learn new translations is essential in many practical applications, the newly learned terminology will only occur rarely. therefore, standard methods for evaluating machine translation are not able to measure the effect appropriately. in order to address these challenges, we develop the following contributions: ## evaluation scenario the first important research question that needs to be addressed in the targeted continuous learning scenario is the evaluation approach. while the evaluation of machine translation is well-established (e.g. using bleu @xcite ), new learned words are typically rare words and therefore their influence on a bleu score calculated on all words is very limited. in order to have a valid evaluation approach, the evaluation should focus on phrases that cannot be learned from the parallel data. these are typically very rare phrases. furthermore, we want to translate them in a real world situation. therefore, the evaluation data should not be synthetic sentences. finally, the approach should be using the standard parallel data without the need of collecting additional parallel data. a first attempt would be to use existing test data and select sentences where dictionary entries are needed as e.g. done in @xcite . however, if we limit ourselves to phrases that do not occur in the parallel data or only a few times, the number of occurring words in the test sets are too low to draw any conclusions. therefore, we evaluate our approach by proposing a new test-train split of existing parallel data. in a first step, we filter a large background dictionary for entries that help to translate phrases that only occur a few times in the existing parallel data. in a second step, we select some of the sentences with their matching dictionaries entries as the new test sets. an overview of the process is shown in figure 1 . finally, we specifically evaluate the ability of the translation system to translate the dictionary entries. in addition, it is important to ensure that the proposed methods do not have negative side effects on the overall translation quality. therefore, we also evaluate the model using standard evaluation metrics on well-established test sets and on the proposed test set. due to the weakness of these metrics to measure improvements in rare words, we do not expect that the proposed methods improve on these metrics, but it is important that the performance measured in these metrics does not decrease significantly. ## nmt dictionary integration to successfully integrate the dictionary into the nmt system, we need to address two challenges: first, we need to enable the system to perform oneshot learning. it should be able to translate a phrase after seeing it only once in the bilingual dictionary. furthermore, it needs to be possible to continuously add new translations. secondly, we need to model the morphology of the dictionary entries. we need to use the dictionary for different inflected forms of the word and also generate various inflected forms of the target phrase. ## experiments we evaluate the approaches on three different data sizes and on two different language pairs (english-german and english-czech). since we are focusing on the generation of different morphological forms, we always use the morphologically rich language as the target language. ## related work in recent years, several different approaches to integrate additional data into neural machine translation have been suggested. if this is parallel data, fine-tuning on the additional, better matching data @xcite is often successful. if the additional data is provided in other forms, different techniques have been investigated. for human feedback, @xcite suggested to use fine-tuning on the human generated post edits. @xcite used phrase pairs extracted by statistical machine translation to annotate translations of rare phrases. in the similar scenario @xcite used a neural network to store the external phrase pairs. even more work has been done to integrate dictionaries into neural machine translation. a first work by @xcite used the additional dictionary to influence the softmax probabilities of the neural machine translation. another possibility is to include the dictionary as an additional knowledge source during training using posterior regularization @xcite . a different approach is chosen by @xcite using the dictionary as additional training sentences or generating synthetic sentences. in contrast to this work, these do not allow the integration of new words after training the nmt system. several authors investigate the integration of the dictionary as an additional constraint during the coding process @xcite @xcite . this leads to a larger complexity in decoding that has been addressed by @xcite . however, the dictionary is typically a hard constraint which makes it difficult to learn words forms that do not occur in the dictionary. most similar to this work is the approach by @xcite , which like this work @xcite annotates the source sentence with possible translations. they showed that state-of-the-art models no longer need architecture changes, but can directly learn to copy form the source sentences. in this work, we additionally focus on generating new word morphological forms not occurring in the dictionary. we investigated different word representations and analysed their influence on the ability to copy the dictionary entries. ## conclusion by introducing the new continuous learning test set using a different train-test split for existing corpora we could highlight the challenges of state-ofthe-art neural machine translation systems. while they achieve very good performance, they are still challenged by new emerging terms. the baseline system was only able to correctly translate 20 to 30 percent of these phrases. our integration of bilingual dictionaries into the systems improves the translation performance to correctly translate the words by up to 70%. in 90% of the cases at least the lemma of the word is predicted correctly. furthermore, in this case, we see no difference in accuracy between words only seen in the dictionary and words also seen a few times in the parallel data. however this is only possible by modelling both: enabling the model to perform one-shot learning and modeling the different morphological forms of the rare phrases. the first one is addressed by annotating the source sentence with dictionary translation while the second one is addressed by using character-based models. by combining character-based and word-based representations we are able to model the different morphological variants of a word as well as enabling the system for fast training. as mentioned before, this work concentrates on the morphological variants of the dictionary entries and ignores ambiguities due to different possible translation. in the future, we intend to address this by including word sense disambiguation into the translation process.