Token-Level Self-Evolution Training for Sequence-to-Sequence Learning (2023)

Abstract: Adaptive training approaches, widely used in sequence-to-sequence models, commonly reweigh the losses of different target tokens based on priors, e.g. word frequency. However, most of them do not consider the variation of learning difficulty in different training steps, and overly emphasize the learning of difficult one-hot labels, making the learning deterministic and sub-optimal. In response, we present Token-Level Self-Evolution Training (SE), a simple and effective dynamic training method to fully and wisely exploit the knowledge from data. SE focuses on dynamically learning the under-explored tokens for each forward pass and adaptively regularizes the training by introducing a novel token-specific label smoothing approach. Empirically, SE yields consistent and significant improvements in three tasks, i.e. machine translation, summarization, and grammatical error correction. Encouragingly, we achieve averaging +0.93 BLEU improvement on three machine translation tasks. Analyses confirm that, besides improving lexical accuracy, SE enhances generation diversity and model generalization.

## introduction sequence-to-sequence learning (seq2seq) with neural networks @xcite has advanced the state-of-the-art in various nlp tasks, e.g. translation @xcite , summarization @xcite , and grammatical error correction @xcite . generally, seq2seq models are trained with the cross-entropy loss, which equally weighs the training losses of different target tokens. however, due to the token imbalance nature @xcite and the truth that different tokens contribute differently to the sentence meaning @xcite , figure 1 : an example to illustrate the changing token difficulties in different training steps in wmt'14 en-de. the token "abschließen/ sache" is hard/ easy to learn at 50k while the trend is totally reversed at 100k. several works are developed to reweigh the tokenlevel training loss according to explicit (e.g. frequency) or implicit (uncertainty estimated by offthe-shelf language models) priors @xcite @xcite . for example, @xcite proposed two heuristic criteria based on word frequency to encourage the model to learn from larger-weight low-frequency tokens. @xcite introduce target-context-aware metric based on an additional target-side language model to adjust the weight of each target token. despite some success, there are still limitations in these adaptive training approaches. first, most of them predetermine the difficult tokens and fix such prior to guiding the training. however, in our preliminary study, we find the hard-to-learn tokens are dynamically changing during training, rather than statically fixed. as shown in figure 1 , as the training progress goes, although the sentence-level loss is nicely converging, the difficult token is changing from "abschließen" to "sache" in terms of the token-level loss. second, these adaptive training methods overly emphasize fitting the difficult tokens' one-hot labels by reweighing the loss, which empirically may cause overfitting and limit the generalization @xcite @xcite . also, a more recent study @xcite provides theoretical evidence to support that reweighting is not that effective to improve the generalization. correspondingly, we design a simple and effective token-level self-evolution training (se) strategy to encourage seq2seq models to learn from difficult words that are dynamically selected by the model itself. specifically, se contains two stages: ❶self-questioning and ❷self-evolution training. in the first stage, the seq2seq models dynamically select the hard-to-learn tokens based on the tokenlevel losses, then we encourage the seq2seq models to learn from them in the second stage, where, rather than adopting reweighing, we introduce a novel token-specific label smoothing approach to generate easily digestible soft label, which considers both the ground truth and model's prediction. experiments across tasks, language pairs, data scales, and model sizes show that se consistently and significantly outperforms both the vanilla seq2seq model and the re-implemented advanced baselines. analyses confirm that besides improved lexical accuracy, se generates diverse and humanlike generations with better model generalization. ## methodology preliminary sequence-to-sequence (seq2seq) learning aims to maximize the cross-entropy (ce) loss of the log-likelihood of each target word in @xmath0, . . . , y n }, conditioned on source x, where the optimization treats all tokens equally: however, due to the different learning difficulties of each token, it is sub-optimal to treat all tokens equally @xcite . to address this limitation, a series of token-level adaptive training objectives were adopted to re-weight the losses of different target tokens @xcite . the common goal of these methods is to facilitate the model training by fully exploiting the informative but underexplored tokens. however, our preliminary study shows that the hard tokens are dynamically changing (see figure 1 ) in different training steps (or model structures), thus it is sub-optimal to employ static token priors (e.g. frequency) during training. also, recent studies @xcite in the ml community theoretically show that reweighting is not that effective to improve the generalization. based on the above evidence, we present the self-evolution learning (se) mechanism to encourage the model to adaptively and wisely learn from the informative yet under-explored tokens dynamically determined by the model itself (stage❶ in §2.1), with an easy-tolearn label distribution (stage❷ in §2.1). a similar work to ours is @xcite . however, their method mainly considers the situation where the predicted answer is incorrect but close to the golden answer, while our method focuses on all dynamic hard tokens. ## evaluation machine translation on three widely-used benchmarks @xcite @xcite : smallscale wmt16 english-romanian (en-ro; 0.6m), medium-scale wmt14 english-german (en-de; 4.5m), and large-scale wmt14 english-french (en-fr; 36.0m). we implement the baselines and our approach under transformer-base settings. we follow the previous adaptive training approach @xcite to pretrain with the cross-entropy loss with n steps, and further finetune the same steps with different adaptive training objectives, including freq-exponential @xcite , freq-chi-square @xcite , d2gpo @xcite , bmi-adaptive @xcite , mixcrossentropy @xcite , cbmi-adaptive @xcite , and spl @xcite . for n , we adopt 100k and 30k for larger datasets, e.g. en-de and en-fr, and small dataset, i.e. en-ro, respectively. we empirically adopt 32k tokens per batch for large datasets, the learning rate warms up to 1e-7 for 10k steps, and then decays 90k, while for small dataset en-ro, the learning rate warms up to 1e-7 for 4k steps, and then decays 26k steps. all the experiments are conducted on 4 nvidia tesla a100 gpus. the sacrebleu @xcite was used for evaluation. besides translation, we also follow previous works @xcite @xcite to validate the universality of our method on more sequenceto-sequence learning tasks, e.g., summarization and grammatical error correction. text summarization on xsum corpus (0.2m). we follow fairseq @xcite to preprocess the data and train the model, then finetune them for the same steps. we evaluated with the rouge @xcite , i.e. r-1, r-2, and r-l. ## conclusion in this paper, we propose a self-evolution learning mechanism to improve seq2seq learning, by exploiting the informative-yet-underexplored tokens dynamically. se follows two stages, i.e. selfquestioning and self-evolution training, and can be used to evolve any pretrained models with a sim-2 http://www.delfdalf.fr/dalf-c2-en.html automatic eval. human eval.