Dual-teacher Knowledge Distillation for Low-frequency Word Translation (2024)

Abstract: Neural Machine Translation (NMT) models are trained on parallel corpora with unbalanced word frequency distribution. As a result, NMT models are likely to prefer high-frequency words than low-frequency ones despite low-frequency word may carry the crucial semantic information, which may hamper the translation quality once they are neglected. The objective of this study is to enhance the translation of meaningful but low-frequency words. Our general idea is to optimize the translation of low-frequency words through knowledge distillation. Specifically, we employ a low-frequency teacher model that excels in translating low-frequency words to guide the learning of the student model. To remain the translation quality of high-frequency words, we further introduce a dual-teacher distillation framework, leveraging both the low-frequency and high-frequency teacher models to guide the student model’s training. Our single-teacher distillation method already achieves a +0.64 BLEU improvements over the state-of-the-art method on the WMT 16 English-to-German translation task on the low-frequency test set. While our dual-teacher framework leads to +0.87, +1.24, +0.47, +0.87 and +0.86 BLEU improvements on the IWSLT 14 German-to-English, WMT 16 English-to-German, WMT 15 English-to-Czech, WMT 14 English-to-French and WMT 18 Chinese-to-English tasks respectively compared to the baseline, while maintaining the translation performance of high-frequency words.

## introduction neural machine translation models typically require large amounts of parallel corpora @xcite @xcite @xcite . while such data normally have an unbalanced word distribution, the * corresponding author: hongfei @xcite translation models trained on the data usually tend to favor high-frequency words while ignoring lowfrequency words. @xcite point out the severe imbalance issue between high-frequency and low-frequency words, and that translation models rarely have the opportunity to learn the true labels of low-frequency words during training. as a result, nmt models rarely have the opportunity to learn and generate those ground truth low-frequency tokens, even though these low-frequency words often carry important semantic information, typically representing specific concepts or emotions found in certain domains, literary work, or dialects. to improve the translation of rare words, @xcite ; @xcite ; @xcite ; @xcite maintain a phrase table or lowfrequency word table, and @xcite ; @xcite introduce additional components to the model. however, these approaches brought additional inference complexity and computational costs. the imbalance word distribution issue can be alleviated by segmenting low-frequency sub-words into high-frequency ones while applying byte pair encoding (bpe) @xcite , but the problem remains. @xcite explore target token-level adaptive objectives based on token frequencies to assign larger weights to meaningful but relatively low-frequency words. @xcite have shown that knowledge distillation is effective for long-tailed visual recognition. in this paper, we utilize knowledge distillation to optimize the translation of low-frequency words. we obtain a low-frequency teacher model by finetuning on the low-frequency part of the training set. then we use knowledge distillation to guide the learning of the student model for low-frequency word translation while retaining its performance on high-frequency words. furthermore, we propose using dual teacher models to guide the student model in learning both high-frequency and lowfrequency words, to further ensure the translation performance of high-frequency words of the student model. our main contributions are as follows: ## conclusion in this study, we investigate the low-frequency word translation problem, which may make the nmt model neglect low-frequency tokens carrying critical semantic information and affect the translation quality. we leverage knowledge distillation to transfer low-frequency word translation knowledge from low-frequency teacher model to the student model. we also present a dual-teacher knowledge distillation framework to ensure the performance with high-frequency words in challenging settings with very large training sets. experiment results show that our single-teacher distillation method can already obtain +0.64 bleu improvements over the state-of-the-art method on the wmt 16 @xmath0. while our dual-teacher framework leads to +0.87, +1.24, +0.47, +0.87 and +0.86 bleu improvements on the iwslt 14 @xmath1, wmt 16 @xmath2, wmt 15 @xmath3, wmt 14 @xmath4, while maintaining the performance on the high-frequency test set even on very large datasets. these results prove the effectiveness of our approach even in very challenging settings.