Entropy-Based Vocabulary Substitution for Incremental Learning in Multilingual Neural Machine Translation (2022)

Abstract: In a practical real-world scenario, the longstanding goal is that a universal multilingual translation model can be incrementally updated when new language pairs arrive. Specifically, the initial vocabulary only covers some of the words in new languages, which hurts the translation quality for incremental learning. Although existing approaches attempt to address this issue by replacing the original vocabulary with a rebuilt vocabulary or constructing independent language-specific vocabularies, these methods can not meet the following three demands simultaneously: (1) High translation quality for original and incremental languages, (2) low cost for model training, (3) low time overhead for preprocessing. In this work, we propose an entropy-based vocabulary substitution (EVS) method that just needs to walk through new language pairs for incremental learning in a large-scale multilingual data updating while remaining the size of the vocabulary. Our method has access to learn new knowledge from updated training samples incrementally while keeping high translation quality for original language pairs, alleviating the issue of catastrophic forgetting. Results of experiments show that EVS can achieve better performance and save excess overhead for incremental learning in the multilingual machine translation task.

## introduction multilingual neural machine translation (nmt) aims at performing multi-directional translation with a single model. due to its effectiveness and efficiency, it has attracted intensive attention in recent years @xcite @xcite @xcite . typically, a multilingual nmt system is trained on a multilingual parallel corpus covering all the interested language pairs. as a result, the work of preparing and optimizing the characteristics of the training corpus makes it cumbersome and timeconsuming to extend an existing multilingual nmt system to support new language pairs @xcite . the stream of data evolves over time by adding new language pairs in the real-world scenario. due to the high-cost of gpu resources, an efficient way is to incrementally train the initial translation model, instead of training the model from scratch every time when the new language pairs arrive @xcite @xcite . incremental learning is a potential solution, which allows neural models to learn new knowledge from updated training samples while inheriting the original knowledge @xcite @xcite . therefore, the highcost of incorporating new languages in multilingual nmt models can be alleviated with the incremental training paradigm. the main challenge in incremental learning is catastrophic forgetting @xcite . moreover, an inevitable part of the incremental learning paradigm for multilingual nmt is how to deal with vocabulary @xcite . a certain amount of the "out-of-vocabulary (oov)" tokens (<@xmath0. this situation will hurt the translation performance naturally @xcite . previous approaches @xcite attempt to address this problem by replacing the initial vocabulary with a new vocabulary. however, these methods suffer from the following two challenges in a further complicated scenario: (1) excessive time cost in preprocessing, and (2) more oov tokens on original language pairs. in particular, the former is due to the rebuilt processing of vocabularies with the standard byte-pair encoding (bpe) @xcite or sentencepiece model (spm) @xcite procedures on the sum of the initial and incremental training data. and the latter is due to the diversity between the incremental and original languages @xcite . another intuitive and simple scheme for vocabularies is to expand the embedding layers of original nmt models directly @xcite . however, growing tokens lead to representation sparsity, which may hurt neural translation model learning @xcite . more importantly, as the data continues to be updated rapidly, the embedding size will grow uncontrollably. both the memory and time overhead increase relentlessly and it is not a sustainable strategy. in this work, considering the diversity of languages, we construct a further complicated and comprehensive setting for incremental learning in multilingual nmt. due to the dissimilar scripts and diverse language branches, there is a little token overlap (about a quarter) between the original and rebuilt vocabularies. to alleviate this issue, we propose an entropy-based vocabulary substitution (evs) method to retain the learned knowledge from the original translation model with minimizing the case @xmath1. moreover, our proposed method maintains the same embedding size with the original vocabulary for incremental learning, and does not expand the size of the initial model to keep the memory and time costs of nmt models. to sum up, our contributions are as follows: ## related work past works develop a universal translation model to provide high-quality translation service between any pair of languages @xcite @xcite . they leverage knowledge transfer techniques to train neural translation models on a set of languages. the shared knowledge enables the set of languages to help each other @xcite @xcite . while these approaches are trained on initially selecting a set of languages. the multilingual models need to be retrained when incorporating new languages or data. previous approaches study how to adapt a machine translation model to new languages from an updated stream of data timely @xcite by incremental learning. @xcite presents two strategies that can rapidly adapt the translation model to new low-resourced languages. and some approaches attempt to improve the translation qualities of pretrained multilingual machine translation models, which incorporate new data @xcite . @xcite leverages the language-specific encoders and decoders to incrementally extend a neural translation model from bilingualism to multilingualism. however, the situation is further complicated, in which the vocabulary needs to be updated in order to avoid the issue of oov tokens. @xcite extends the initial model to adapt new languages by a dynamic vocabulary. @xcite rebuilds a vocabulary from the sum of initial and updated training data with the standard bpe or spm procedures. @xcite constructs an additional vocabulary to replace the initial vocabulary with the same setting of the initial procedure from a new language. however, the translation quality will decrease dramatically for original languages if the rate of token overlap is low between the initial and additional vocabulary. it is not suitable for the situation where the incremental languages are not related to the original languages. different from previous works, our proposed method utilizes an entropy-based vocabulary substitution strategy with the minimum-cost reconstruction to alleviate the issue of low token overlap. the method can effectively adapt to large-scale updated training data by incremental learning, without the superfluous time and memory overhead. ## background multilingual nmt utilizes a single encoderdecoder model to handle different translation directions by jointly training on the multilingual parallel dataset. to achieve better translation performance in multilingual training, we share the embeddings for the encoder and decoder embedding layers. to indicate the target language, a prepending language token is appended to each source sentence @xcite . formally, given the source sentence x ′ = (x 1 , x 2 , ..., x i ), the modified source sentence is represented as @xmath2, x 1 , x 2 , ..., x i ). l i represents the target language. and its target sentence is represented as @xmath3, y 2 , ..., y j ). a sequence of word embeddings e(x) is fed into the encoder component. the probability of a target sentence is given by: ## approach in this work, we aim to leverage vocabulary substitution strategies to incrementally update the initial model when new language pairs arrive. the previously-learned knowledge can be retained with the token overlap between the initial vocabulary and a new vocabulary, alleviating the issue of catastrophic forgetting. as a result, we present a variant scheme for incremental learning in multilingual machine translation, as shown in figure 1 , where the substitution method does not change the size of the initial vocabulary. moreover, to improve the efficiency and practicability of vocabulary substitution methods in the real-world scenario, our method alleviates the issue of excessive time cost in preprocessing. the subsequent subsections introduce the definition of incremental learning in the multilingual machine translation task and discuss how to alleviate the new challenge in the complicated setting. ## conclusion in this work, we propose an entropy-based vocabulary substitution (evs) method for incremental learning in multilingual machine translation. and we adopt the incremental learning scheme to learn new knowledge from updated training samples while keeping high translation quality for original language pairs, alleviating the issue of catastrophic forgetting. it is more efficient to utilize incremental learning based on the proposed method than the model trained from scratch. experimental results demonstrate that the proposed method can also outperform several stronger baselines without the excess time and memory overhead.