An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models (2022)

Abstract: We propose a framework for training non-autoregressive sequence-to-sequence models for editing tasks, where the original input sequence is iteratively edited to produce the output. We show that the imitation learning algorithms designed to train such models for machine translation introduces mismatches between training and inference that lead to undertraining and poor generalization in editing scenarios. We address this issue with two complementary strategies: 1) a roll-in policy that exposes the model to intermediate training sequences that it is more likely to encounter during inference, 2) a curriculum that presents easy-to-learn edit operations first, gradually increasing the difficulty of training samples as the model becomes competent. We show the efficacy of these strategies on two challenging English editing tasks: controllable text simplification and abstractive summarization. Our approach significantly improves output quality on both tasks and controls output complexity better on the simplification task.

## introduction neural sequence-to-sequence (seq2seq) models primarily developed and tested for machine translation (mt) @xcite ; @xcite ; @xcite are increasingly used for other sequence transduction tasks. this paper focuses on editing tasks, such as post-editing of mt output @xcite , style transfer @xcite , or text simplification @xcite , where systems directly edit the input sequence, instead of generating the output from scratch as in mt. as illustrated in while dedicated architectures have been designed for these editing tasks, based on e.g., a original: the mauritshuis museum is staging an exhibition focusing on the 17th century selfportraits, highlighting the similarities and the differences between modern-day snapshots and historic works of art. simplified: the mauritshuis museum is now set to open an exhibit on the 17th century selfportraits. it shows the similarities and differences between modern photos and artworks. this work is centered on the hypothesis that training nar models for editing tasks using the same strategy as for mt leads to a mismatch between train and test settings that limits their generalization ability and output quality. specifically, the learning algorithms designed for mt are aligned with inference strategies that generate output from an empty initial sequence. by contrast, in sequence editing tasks, the inference step is initialized instead with the original input sequence. in addition, since editing samples might range from limited lexical substitutions to more thorough rewrites, training samples cover a wide range of edit distances. during training, the loss can thus be dominated by the more distant samples leading to undertrained models and poor generalization. by contrast, the distance between input and output samples in mt is more uniform, since it always involves at least lexical translation of the input tokens. to address these issues, we introduce a new training framework, editing curriculum, which dynamically exposes the model to more relevant edit actions during training and exploits the full spectrum of available training samples more effectively. first, we design a new roll-in strategy, editing roll-in, that exposes the model to intermediate sequences that it is more likely to encounter during inference. second, we introduce a training curriculum to expose the model to training samples in order of increasing edit distance, thus gradually increasing the complexity of oracle edit operations that the model learns to imitate. we show that our approach improves the quality of outputs on two challenging english text editing tasks: controllable text simplification (ts) and abstractive summarization. it also improves the degree of ts control by generating simplified outputs that match the target reading grade level better than the baselines. we conduct an extensive analysis which supports our hypothesis, and show that the sequences generated by our training policy improve exploration during training and are easier to learn from, leading to better generalization across samples with varying edit distances. training with curriculum further improves output quality. models differ based on the nature of edit actions used and support different operations such as insertion, deletion, reposition and substitution. we select the operations from the editor model based on its competitive performance on constrained de-coding tasks that require editing non-empty initial sequences @xcite . it is a transformer model that uses two types of actions or edits on sequences, y: at each decoding iteration, the model applies an action a that consists of a reposition and an insertion operation. this refinement process is repeated until two consecutive decoding iterations return the same output @xcite , or a preset maximum number of them is reached @xcite . from. a roll-out policy is then used to estimate the cost-to-go from the generated roll-in sequences to the desired output sequences. the cost-to-go is calculated by comparing the model actions to oracle demonstrations. we summarize the policies of various nar models proposed for mt in for editor, the roll-in sequences for the reposition (or insertion) module are stochastic mixtures (parameterized by α or β) of the output of the insertion (or reposition) module or a noised version of the output sequence. the oracle is the levenshtein edit distance @xcite . the noisy sequence is generated by applying random word dropping @xcite and random word shuffle @xcite with a probability of 0.5 and maximum shuffle distance of 3. figure 1 shows an example instantiation of the edit actions generated by the levenshtein edit distance to transform the original input sequence ("a b c d e") to the output sequence ("c a t"). in this example, the oracle action is to delete the tokens ["b", "d", "e"], reposition "a" and "c" and insert "t" at the appropriate position. the reposition and the insertion modules are trained in a supervised fashion to predict these oracle operations during training. ## our approach: editing curriculum to tailor training to editing tasks, we propose to modify the roll-in policy to better match the intermediate sequences encountered at inference, and introduce a curriculum to increase the difficulty of oracle actions learned throughout training. editing roll-in sequences generated using the roll-in policy control the search space explored during training. those sequences should therefore be representative of the intermediate sequences generated at inference time @xcite . while typically, the roll-in policy is a stochastic mixture of the model and the expert demonstrations as described above, the noise incurred early on due to the large difference between the expert demonstration and the learner's policy actions may hurt overall performance @xcite @xcite . as we will see ( §5), this is what happens on editing tasks when training the model to imitate experts using learned roll-in sequences. at the same time, rolling in with expert demonstrations raises its own issues, as it can limit the exploration of the search space. motivated by these observations, we propose a new policy, editing, that allows exploration by injecting noise to the input sequence to generate new intermediate sequences for training. this lets the model learn to fix errors without deviating from learning the task at hand. figure 2 create training dataset from by selecting all samples, b t using s @xmath0, such that d(s i ) ≤ c(t). 12 return best π rps and π ins evaluated on validation set. ent reposition and insertion edit operations starting from the same input sequence, hence enabling exploration. we modify the roll-in policies to be aligned with the editing inference process, where the reposition operation is followed by insertion on the original input sequence: curriculum controlled roll-out to prevent undertraining when samples with large edit distances overwhelm the loss, we use a curriculum to expose the model to easy-to-learn actions first, then gradually increase the difficulty of the edit-operations performed as the learner becomes more competent. prior work on curriculum learning (cl) does not agree on standard measures of sample difficulty for seq2seq tasks @xcite @xcite or apply cl for the different problem of shifting the training of a transformer model from ar to nar regimes @xcite . by contrast, in our settings, the levenshtein distance provides a measure of difficulty that directly aligns with the model design and the training oracle. resulting algorithm given a training dataset @xmath1, y * } m @xmath2, the difficulty score d(s i ) for each sample s @xmath3, y * i } ∈ d is measured by the levenshtein distance between the input and the output sequence. the cumulative density function (cdf) of the difficulty scores results in one difficulty cdf score per sample, d(s i ). at each training step t, we estimate the progress made by the learner by computing the competence of the model c(t) ∈ (0, 1] as follows: based on this competence value c(t), the model is then trained on all the samples whose difficulty as measured by the levenshtein distance between the input and the output sequence is lower than that competence value, i.e. d(s i ) ≤ c(t). the resulting algorithm is also shown in algorithm 1. ## experimental settings we evaluate our approach on controllable simplification and abstractive summarization, two challenging sequence editing tasks that are motivated by real world information access needs. they are challenging because they require learning to perform a wide range of rewrites (from local substitution to sentence restructuring). ## analysis we conduct further experiments to better understand the factors that help our training strategies improve editing quality. ## conclusion this paper introduced two complementary strategies to address undertraining and poor generalization when adapting nar models to editing tasks: 1) a new roll-in policy that generates intermediate sequences that the model is likely to encounter during inference and 2) a curriculum to control the difficulty of the roll-out policy which estimates the cost-to-go from the roll-in sequences to the desired output sequences, throughout training. together, these strategies improve output quality consistently on controllable simplification and abstractive summarization. these results open space for further research to evaluate the potential of this approach for other editing tasks (e.g., post editing, style transfer), and to further tailor imitation learning policies and curriculum design to these tasks.