In-sample Curriculum Learning by Sequence Completion for Natural Language Generation (2023)

Abstract: Curriculum learning has shown promising improvements in multiple domains by training machine learning models from easy samples to hard ones. Previous works which either design rules or train models for scoring the difficulty highly rely on task-specific expertise, and cannot generalize. Inspired by the “easy-to-hard” intuition, we propose to do in-sample curriculum learning for natural language generation tasks. Our learning strategy starts training the model to generate the last few words, i.e., do sequence completion, and gradually extends to generate the whole output sequence. Comprehensive experiments show that it generalizes well to different tasks and achieves significant improvements over strong baselines.

## introduction curriculum learning (cl) proposed by @xcite provides performance improvements on a number of machine learning tasks. it mimics the learning process of humans by training models with samples in a more meaningful order, i.e., from the easy ones to the hard ones. therefore, ranking training samples by difficulty lies in the core of cl, which is also the key challenge when it's applied to natural language generation (nlg) tasks. previous work on cl for nlg focuses on measuring the difficulty of training samples in two ways. one is to resort to human-crafted rules based on various linguistic features and human observations @xcite . the other uses models either trained from outside data or the same data but in previous epochs/steps @xcite @xcite . either way seeks to produce a numeric score for each training sample relying on domain expertise so that it can be ranked, making it difficult to generalize to different tasks. for example, summarization focuses more on generating concise outputs while style transfer emphasizes style changes. so the former should pay attention to the ratio between the lengths of the output and the input (the more compressed the more difficult), while the latter should focus on differences in style between the input and output (the more different the more difficult). designing a comprehensive or universal scoring function is difficult or even impossible under this definition of cl. in this paper, we propose an alternative to sample-wise cl, which we call in-sample cl (icl). icl re-orders the learning sequence within the sample. one particular icl re-ordering strategy which we find effective is to predict the last few tokens given a long prefix first from the original output, and then gradually increase the number of tokens at the end while shortening the prefix, to create an easy-to-hard training order. such a curriculum learning strategy focuses more on the difficulty of language generation itself, leading to a better generalization ability among tasks. actually, we are not the first to propose the idea of icl. @xcite introduced the notion of "token-wise curriculum learning(tcl)". illustrations of tcl, icl and the traditional cl are shown in figure 1 . their work considers generating the first few tokens in the output sequence to be easier than generating a longer sequence in the output. based on this idea, they proposed a "hard" version of tcl that creates training samples of increasing output length by cutting the sentences short. in this way, tcl is similar to data augmentation with incomplete and even "incorrect" samples, while our icl considers each training sample in full length. a "soft" version of tcl that places decaying weights on the end tokens instead of cutting short is introduced as a mitigation to avoid incomplete samples, which was proved to uniformly outperform the "hard" version. sample sample sample sample sample sample training set _ _ _ _ _ _ _ _ _ _ _ _ t t t t _ _ t t _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ le le le raining set a training sample a training sample (a) sample-wise curriculum learning sample sample ## approach we present an icl strategy in the context of the vanilla sequence-to-sequence (seq2seq) training objective with a detailed learning algorithm. ## analysis for a better understanding of icl-sc, we did comprehensive ablation studies and combined it with the traditional cl. the experiments in this section are done on dialogue summarization, which is representative due to the medium output length. ## related work natural language generation has received great attention with deep neural networks, especially pre-trained language models. it refers to the task where expected outputs for different purposes are in natural language @xcite . the inherent characteristic of having more than one correct output given the same input is the core challenge of solving this kind of task, especially for evaluation @xcite . curriculum learning @xcite ) boost models' performances in a range of machine learning areas @xcite by reordering the training samples. it meets great obstacles when applying to nlg tasks as it's hard to evaluate the difficulties of training samples. different rules are developed for different tasks @xcite . for example, @xcite measures the complexity of question-answering pairs from the view of frequency and grammar simply for answers. @xcite focuses more on pos features and the length of translation pairs. other works utilize additional models or targeting models in the previous training step @xcite . @xcite reorder samples by the accuracy from an independent emotion classifier for response generation. however, such salient features do not always exist or can be well classified. there is also work @xcite using either the reference perplexity or generations evaluated by corresponding metrics for ranking during training, while these scores are not ideal due to the one-tomany characteristic of nlg. thus, designing a cl strategy generalizing well for nlg is difficult. instead of figuring out the oracle scoring function for training samples, we propose to measure the language generation difficulty within a sample. @xcite did something similar though their approach amounts to data augmentation by doing sub-sequence generation, which is not exactly curriculum learning. we, on the other hand, train on the original sample with a decreasing prefix length and thus learn from easy to hard. ## conclusion this paper defines a kind of curriculum learning strategy for nlg tasks called in-sample curriculum learning (icl) by manipulating the difficulty of training within a training sample instead of ranking among samples. we propose the icl algorithm with the sequence completion curriculum which boosts the performance of strong baselines on a wide range of tasks, showing the effectiveness and strong generalization ability of our approach. more training strategies under icl digging the inherent difficulties of generating a language sequence are expected in the future.