Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation (2021)

Abstract: Recently, token-level adaptive training has achieved promising improvement in machine translation, where the cross-entropy loss function is adjusted by assigning different training weights to different tokens, in order to alleviate the token imbalance problem. However, previous approaches only use static word frequency information in the target language without considering the source language, which is insufficient for bilingual tasks like machine translation. In this paper, we propose a novel bilingual mutual information (BMI) based adaptive objective, which measures the learning difficulty for each target token from the perspective of bilingualism, and assigns an adaptive weight accordingly to improve token-level adaptive training. This method assigns larger training weights to tokens with higher BMI, so that easy tokens are updated with coarse granularity while difficult tokens are updated with fine granularity. Experimental results on WMT14 English-to-German and WMT19 Chinese-to-English demonstrate the superiority of our approach compared with the Transformer baseline and previous token-level adaptive training approaches. Further analyses confirm that our method can improve the lexical diversity.

## introduction neural machine translation (nmt) @xcite @xcite @xcite @xcite @xcite has achieved remarkable success. as a data-driven model, the performance of nmt depends on training corpus. balanced training data is a crucial factor in building a superior model. however, natural languages conform to the zipf's law @xcite , the frequencies of words exhibit the long tail characteristics, which brings an imbalance in the distribution of words in training corpora. some studies @xcite assign different training weights to target tokens according to their frequencies. these approaches alleviate the token imbalance problem and indicate that tokens should be treated differently during training. however, there are two issues in existing approaches. first, these approaches believe that lowfrequency words are not sufficiently trained and thus amplify the weight of them. nevertheless, low-frequency tokens are not always difficult as the model competence increases @xcite . second, previous studies only use monolingual word frequency information in the target language without considering the source language, which is insufficient for bilingual tasks, e.g., machine translation. the mapping between bilingualism is a more appropriate indicator. as shown in to tackle the above issues, we propose bilingual mutual information (bmi), which has two characteristics: 1) bmi measures the learning difficulty for each target token by considering the strength of association between it and the source sentence; 2) for each target token, bmi can dynamically adjust according to the context. bmi-based adaptive training can dynamically adjust the learning granularity on tokens. easy tokens are updated with coarse granularity while difficult tokens are updated with pleasing (847) gāoxìng (81); yúkuài (74); xǐyuè (63); qǔyuè (49) ... bearings (847) zhóuchéng (671) ... ## bmi-based adaptive training in this section, we start with the definition of the bilingual mutual information (bmi). then we analyze the relationship between bmi and translation difficulty. based on this, we introduce our bmibased token-level adaptive training objective. ## experiments we evaluate our method on the transformer @xcite and conduct experiments on two widely-studied nmt tasks, wmt14 english-to-german (en-de) and wmt19 chineseto-english (zh-en). ## conclusion we propose a novel bilingual mutual information based adaptive training objective, which can measure the learning difficulty for each target token from the perspective of bilingualism, and adjust the learning granularity dynamically to improve token-level adaptive training. experimental results on two translation tasks show that our method can bring a significant improvement in translation quality, especially on sentences that are difficult to learn by the model. further analyses confirm that our method can also improve the lexical diversity.