Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting (2021)

Abstract: In this paper, we propose S equence S pan R ewriting (SSR), a self-supervised task for sequence-to-sequence (Seq2Seq) pre-training. SSR learns to refine the machine-generated imperfect text spans into ground truth text. SSR provides more fine-grained and informative supervision in addition to the original text-infilling objective. Compared to the prevalent text infilling objectives for Seq2Seq pre-training, SSR is naturally more consistent with many downstream generation tasks that require sentence rewriting (e.g., text summarization, question generation, grammatical error correction, and paraphrase generation). We conduct extensive experiments by using SSR to improve the typical Seq2Seq pre-trained model T5 in a continual pre-training setting and show substantial improvements over T5 on various natural language generation tasks.

## introduction text infilling (e.g., masked language modeling) has become a prevalent learning objective for pretrained language models (ptlms) @xcite @xcite @xcite @xcite . it provides self-supervision by masking out tokens or spans in text, and trains a model to infill the masked content based on the contexts, accordingly guiding the model for representation learning, as figure 1(a) shows. in this paper, we propose to extend the conventional text infilling to a novel sequence-to-sequence (seq2seq) pre-training objective, namely sequence span rewriting (ssr). we train a model to rewrite machine-generated imperfect text spans into the ground truth text, as illustrated in figure 1(b) . ssr has two advantages over text infilling: (1) ssr provides better supervision signals, as ssr trains the model with diverse and fine-grained rewriting patterns beyond filling the blanks; (2) ssr bridges the gap between pre-training and fine-tuning, because many downstream seq2seq tasks like summarization and paraphrase generation are naturally sequence span rewriting tasks where a source sentence is mapped to the target sentence following specific rewriting patterns. the key element in implementing ssr is how to generate imperfect text spans that are both diverse and informative. inspired by electra @xcite , we use a powerful pre-trained text infilling model -t5-large @xcite -as the imperfect span generator. compared with random or rule-based noising approaches, the t5-based imperfect span generator can derive various informative text spans that benefit the model to learn meaningful and diverse rewriting patterns including paraphrasing and enhancing the fluency and contextual consistency through correcting grammatical, commonsense and factual errors, to improve a text sequence. these rewriting patterns resemble the goal of various nlg tasks and thus strengthen the ability of pre-trained model for downstream applications. in our experiments, we apply ssr to the typical seq2seq pre-trained model -t5 @xcite in a continual learning fashion. we show ssr outperforms both the original pre-trained t5 models and their continual training counterparts with the conventional text infilling objective on various seq2seq tasks, including text summarization, question generation, and grammatical error correction, with a small number of optimization steps with moderate amount of machine-generated data, which confirms the potential of ssr to serve as a plug-and-play method to improve various existing pre-trained seq2seq models. notably, we find ssr transformer @xmath0, <@xmath1. <@xmath2, with the help of a powerful imperfect span generator. this observation sheds light on a new approach for knowledge transfer from large models to smaller ones. the key difference is that our approach focuses on span-level texts and trains the model to correct the mistakes instead of simply detecting them, which includes more diverse and informative signals and enables the model to perform text generation tasks in a seq2seq fashion. to enable mask language models for natural language generation tasks, @xcite used a decoder to generate the masked tokens autoregressively. unilm @xcite multitasks mlm and language modeling objectives. more recently, bart @xcite and t5 @xcite pre-train seq2seq models with the text span infilling objective, which removes text spans in the input texts and train the models to recover the original texts in an auto-regressive fashion. more recently, calm @xcite introduces concept-to-sentence generation and concept order recovery as two self-supervised objectives that encourage seq2seq ptlms to acquire generative commonsense reasoning ability. marge @xcite pre-trains a seq2seq model with an unsupervised multi-lingual crossdocument paraphrasing objective. their approach is related to our text rewriting objective. however, marge requires multi-lingual paraphrase documents and needs to train a separate retrieval model while our method can simply used an off-the-shelf model pre-trained with text infilling to generate training data. also, marge is pre-trained to generate a paraphrase-like document in another language, thus mainly helpful for translation tasks and multi-lingual tasks. in contrast, ssr focus on monolingual text rewriting and improve general text generation tasks. ssr departs significantly from the aforementioned methods for seq2seq pre-training as it employ machine-generated noises instead of rulebased ones, thus introducing more diverse training signals. also, ssr receives complete inputs without artificial masks during pre-training relying solely on monolingual corpus. ## methodology the key idea of ssr is to train a seq2seq model to rewrite machine-generated text spans that may contain a variety of noise such as paraphrase, grammatical and factual errors, into ground truth that are correct and appropriate in the context. as illustrated by figure 1 (b), ssr involves three steps: ## discussion pre-training via rewriting we discuss several key advantages of ssr over the conventional text infilling objectives here. (1) ssr is closer to the downstream sequence transduction tasks. this is because the model's prediction is not only based on its bidirectional context but also conditioned on the imperfect spans. in this way, the gap between pre-training and fine-tuning stages, which is introduced by the masked tokens or spans in conventional pre-training objectives, is alleviated. indeed, many nlg tasks can be viewed as sequence span rewriting problems that rewrite the input text into another language, more compact format, grammatically correct sentences, or another style. distillation via rewriting ssr sheds light on a new perspective of exploiting the knowledge of a large pre-trained model to improve smaller models. similar to knowledge distillation (kd), this can be achieved by using a large-size teacher model pre-trained with the text infilling objective as the imperfect span generator, and pre-train or refine a small-size student model with the generated data using ssr. different from conventional kd @xcite or sequence-level kd @xcite , ssr enables the student model to exploit both teacher outputs and the ground truth at the same time. it is also related to boost learning @xcite and residual learning @xcite in a sense that the model only needs to learn the prediction error of the teacher model, instead of the original task, text infilling, which may be too difficult for smaller-size models. ## conclusion we present sequence span rewriting (ssr), a novel self-supervised objective for improving sequenceto-sequence transformers pre-trained with conventional text infilling objectives. ssr introduces more diverse and fine-grained learning signals and also bridges the gap between self-supervised pretraining and task-specific fine-tuning on common nlg datasets. our experiments on continual t5 pre-training confirm the effectiveness of ssr on improving pre-trained t5 models of different sizes across different tasks and datasets. also, the large improvements achieved on small models with a larger imperfect span generator indicates a new perspective of exploiting the knowledge of a large pre-trained model to help train smaller ones.