Token-level Adaptive Training for Neural Machine Translation (2020)

Abstract: There exists a token imbalance phenomenon in natural language as different tokens appear with different frequencies, which leads to different learning difficulties for tokens in Neural Machine Translation (NMT). The vanilla NMT model usually adopts trivial equal-weighted objectives for target tokens with different frequencies and tends to generate more high-frequency tokens and less low-frequency tokens compared with the golden token distribution. However, low-frequency tokens may carry critical semantic information that will affect the translation quality once they are neglected. In this paper, we explored target token-level adaptive objectives based on token frequencies to assign appropriate weights for each target token during training. We aimed that those meaningful but relatively low-frequency words could be assigned with larger weights in objectives to encourage the model to pay more attention to these tokens. Our method yields consistent improvements in translation quality on ZH-EN, EN-RO, and EN-DE translation tasks, especially on sentences that contain more low-frequency tokens where we can get 1.68, 1.02, and 0.52 BLEU increases compared with baseline, respectively. Further analyses show that our method can also improve the lexical diversity of translation.

## introduction neural machine translation (nmt) systems @xcite ; all the target tokens (bpe sub-words with 30k merge operations ) of the training set are ranked by their frequencies in descending order. the 'token order' column represents the frequency interval ([10%, 30%) means the frequency of token is between top 10% and 30%). the 'average frequency' column represents the average frequencies of the tokens in each interval, which show the token imbalance phenomenon in natural language. the last two columns show the vanilla nmt model tends to generate more high-frequency tokens and less low-frequency tokens than reference. @xcite @xcite are data driven models, which highly depend on the training corpus. nmt models have a tendency towards over-fitting to frequent observations (e.g. words, word co-occurrences) while neglecting those lowfrequency observations. unfortunately, there exists a token imbalance phenomenon in natural languages as different tokens appear with different frequencies, which roughly obey the zipf's law @xcite . some work tries to improve the rare word translation by maintaining phrase tables or back-off vocabulary @xcite @xcite or adding extra components @xcite , which bring in extra training complexity and computing expense. some nmt techniques which are based on smaller translation granularity can alleviate this issue, such as hybrid word-character-based model @xcite , bpe-based model @xcite and word-piece-based model @xcite . these effective work alleviate the token imbalance phenomenon to a certain extent and become the de-facto standard in most nmt models. although sub-word based nmt models have achieved significant improvements, they still face the token-level frequency imbalance phenomenon, as furthermore, current nmt models generally assign equal training weights to target tokens without considering their frequencies. it is very likely for nmt models to ignore the loss produced by the low-frequency tokens because of their small proportion in the training sets. the parameters related to them can not be adequately trained, which will, in turn, make nmt models tend to prioritize output fluency over translation adequacy, and ignore the generation of low-frequency tokens during decoding, which is illustrated in to address the above issue, we proposed tokenlevel adaptive training objectives based on target token frequencies. we aimed that those meaningful but relatively low-frequency tokens could be assigned with larger loss weights during training so that the model will learn more about them. to explore suitable adaptive objectives for nmt, we first applied existing adaptive objectives from other tasks to nmt and analyzed their performance. we found that though they could bring modest improvement on the translation of low-frequency tokens, they did much damage to the translation of highfrequency tokens, which led to an obvious degradation on the overall performance. this implies that the objective should ensure the training of highfrequency tokens first. then, based on our observations, we proposed two heuristic criteria for design-ing the token-level adaptive objectives based on the target token frequencies. last, we presented two specific forms for different application scenarios according to the criteria. our method yields consistent improvements in translation quality on zh-en, en-ro, and en-de translation tasks, especially on sentences that contain more low-frequency tokens where we can get 1.68, 1.02, and 0.52 bleu increases compared with baseline, respectively. further analyses show that our method can also improve the lexical diversity of translation. our contributions can be summarized as follows: ## background in our work, we apply our method in the framework of transformer @xcite which will be briefly introduced here. we denote the input sequence of symbols as @xmath0, . . . , x j ), the ground-truth sequence as y * = (y * 1 , . . . , y * k ) and the translation as @xmath1, . . . , y k ). ## method our work aims to explore suitable adaptive objectives that can not only improve the learning of low-frequency tokens but also avoid harming the translation quality of high-frequency tokens. we first investigated two existing adaptive objectives, which were proposed for solving the token imbalance problems in other tasks, and analyzed their performance. then, based on our observations, we introduced two heuristic criteria for designing the adaptive objective. based on the proposed criteria, we put forward two simple but effective functional forms from different perspectives, which can be adapted to various application scenarios in nmt. ## related work rare word translation. rare word translation is one of the key challenges for nmt. for word-level nmt models, nmt has its limitation in handling a larger vocabulary because of the training complexity and computing expense. some work tries to solve this problem by maintaining phrase tables or back-off vocabulary @xcite @xcite . the subword-based nmt @xcite @xcite reduces the size of vocabulary greatly and become the mainstream technology gradually. gowda and may (2020) gave a detailed analysis about the effects of the bpe size on the data distribution and translation quality. some recent work tried to further improve the translation of the rare words with the help of the memory network or the pointer network @xcite . in contrast, our methods can improve the translation performance without extra cost and can be combined with other techniques. ## conclusion in this work, we focus on the token imbalance problem of nmt. we show that the output of vanilla nmt contains more high-frequency tokens and has lower lexical diversity. to alleviate this problem, we investigated existing adaptive objectives for other tasks and then proposed two heuristic criteria based on the observations. next, we gave two simple but effective forms based on the criteria, which can assign appropriate training weights to target tokens. the final results show that our methods can achieve significant improvement in performance, especially on sentences that contain more low-frequency tokens. further analyses show that our method can also improve the lexical diversity.