D ata R ejuvenation: E xploiting I nactive T raining E xamples for N eural M achine T ranslation (2020)

Abstract: Large-scale training datasets lie at the core of the recent success of neural machine translation (NMT) models. However, the complex patterns and potential noises in the large-scale data make training NMT models difficult. In this work, we explore to identify the inactive training examples which contribute less to the model performance, and show that the existence of inactive examples depends on the data distribution. We further introduce data rejuvenation to improve the training of NMT models on large-scale datasets by exploiting inactive examples. The proposed framework consists of three phases. First, we train an identification model on the original training data, and use it to distinguish inactive examples and active examples by their sentence-level output probabilities. Then, we train a rejuvenation model on the active examples, which is used to re-label the inactive examples with forward- translation. Finally, the rejuvenated examples and the active examples are combined to train the final NMT model. Experimental results on WMT14 English-German and English-French datasets show that the proposed data rejuvenation consistently and significantly improves performance for several strong NMT models. Extensive analyses reveal that our approach stabilizes and accelerates the training process of NMT models, resulting in final models with better generalization capability.

## introduction neural machine translation (nmt) is a data-hungry approach, which requires a large amount of data to train a well-performing nmt model @xcite . however, the complex patterns and potential noises in the large-scale data make training nmt models difficult. to relieve this problem, several approaches have been proposed to better exploit the training data, such as curriculum learning @xcite , data diversification @xcite , and data denoising @xcite . in this paper, we explore an interesting alternative which is to reactivate the inactive examples in the training data for nmt models. by definition, inactive examples are the training examples that only marginally contribute to or even inversely harm the performance of nmt models. concretely, we use sentence-level output probability (kumar and sarawagi, 2019) assigned by a trained nmt model to measure the activeness level of training examples, and regard the examples with the least probabilities as inactive examples ( §3.1). experimental results show that removing 10% most inactive examples can marginally improve translation performance. in addition, we observe a high overlapping ratio (e.g., around 80%) of the most inactive and active examples across random seeds, model capacity, and model architectures ( §4.2). these results provide empirical support for our hypothesis of the existence of inactive examples in large-scale datasets, which is invariant to specific nmt models and depends on the data distribution itself. we further propose data rejuvenation to rejuvenate the inactive examples to improve the performance of nmt models. specifically, we train an nmt model on the active examples as the rejuvenation model to re-label the inactive examples, resulting in the rejuvenated examples ( §3.2). the final nmt model is trained on the combination of the active examples and rejuvenated examples. experimental results show that the data rejuvenation approach consistently and significantly improves performance on sota nmt models (e.g., lstm @xcite , transformer @xcite , and dynamicconv @xcite ) on the benchmark wmt14 english-german and english-french datasets ( §4.4). encouragingly, our approach is also complementary to existing data manipulation methods (e.g., data diversification @xcite and data denoising @xcite ), and combining them can further improve performance. finally, we conduct extensive analyses to better understand the inactive examples and the proposed data rejuvenation approach. quantitative analyses reveal that the inactive examples are more difficult to learn than active ones, and rejuvenation can reduce the learning difficulty ( §5.1). the rejuvenated examples stabilize and accelerate the training process of nmt models ( §5.2), resulting in final models with better generalization capability ( §5.3). our contributions of this work are as follows: ## methodology figure 1 shows the framework of the data rejuvenation approach, in which we introduce two models: an identification model and a rejuvenation model. the identification model distinguishes the inactive examples from the active ones. the rejuvenation model, which is trained on the active examples, rejuvenates the inactive examples. the rejuvenated examples and the active examples are combined to train the final nmt model. there are many possible ways to implement the general idea of data rejuvenation. the aim of this paper is not to explore this whole space but simply to show that one fairly straightforward implementation works well and that data rejuvenation helps. ## analysis and discussion in this section, we performed an extensive study to understand inactive examples and data rejuvenation in terms of linguistic properties ( §5.1), learning stability ( §5.2) and generalization capacity ( §5.3). we also investigated the strategy to speed up the pipeline of data rejuvenation ( §5.4). unless otherwise stated, all experiments were conducted on the en⇒de dataset with transformer-base. ## conclusion in this study, we propose data rejuvenation to exploit the inactive training examples for neural machine translation on large-scale datasets. the proposed data rejuvenation scheme is a general framework where one can freely define, for instance, the identification and rejuvenation models. experimental results on different model architectures and language pairs demonstrate the effectiveness and universality of the data rejuvenation approach. future directions include exploring advanced identification and rejuvenation models that can better reflect the learning abilities of nmt models, as well as validating on other nlp tasks such as dialogue and summarization. they contain more rare words, which make them more difficult to be learned by nmt models than the active examples. coverage. coverage measures the ratio of source words being aligned by any target words @xcite . firstly, we train an alignment model on the training data by fast-align foot_3 @xcite , and force-align the source and target sentences of each subset. then, we calculate the coverage of each source sentence, and report the averaged coverage of each subset. the lower coverage of inactive examples indicates that they are not very well aligned as the active examples, which also make them more difficult for nmt models to learn.