Noisy Self-Knowledge Distillation for Text Summarization (2021)

Abstract: In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summarization model is trained with guidance from a teacher which generates smoothed labels to help regularize training. Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.

## introduction automatic summarization has enjoyed renewed interest in recent years, thanks to the popularity of neural network models and their ability to learn continuous representations without recourse to preprocessing tools or linguistic annotations. the availability of large-scale datasets @xcite @xcite containing hundreds of thousands of document-summary pairs has driven the development of neural architectures for summarization. several approaches have been proposed, in the vast majority sequence-to-sequence models which are trained in an end-to-end fashion with a maximum likelihood estimation loss @xcite @xcite . despite promising results, there are specific characteristics of the summarization task which render it ill-suited to standard sequence-to-sequence training. for instance, maximum-likelihood training on single reference datasets might not be optimal for summarization which is subject to a great deal of human variation @xcite . in the context of extractive summarization, different people select different sentences to include in a summary @xcite , and when writing abstracts, disagreement exists both in terms of writing style and the specific content deemed important for the summary @xcite . although summarization models would naturally benefit from multiple target references, it is unrealistic to expect that multi-reference datasets can be created at scale for neural network training. in fact, most popular benchmarks are collated opportunistically, based on summaries which only loosely correspond to the source input. for example, @xcite create a dataset by pairing the first sentence of a news article with the rest of the document under the assumption that the introductory sentence expresses the gist of the article. @xcite pair articles with metadata available in html pages under the assumption that html tags (e.g., description) denote summary-like content. in other work @xcite , multidocument summarization datasets are created by viewing lead sections in wikipedia articles as summaries of documents cited therein. the inherent noise in the data collection process further hampers training with models often being prone to hallucination @xcite , and struggling to identify which content units are salient @xcite . in this paper, we propose to alleviate these problems by turning to knowledge distillation @xcite @xcite . knowledge distillation transfers knowledge from a larger "teacher" network to a smaller "student" model by training the student to imitate the teacher's outputs (in addition to learning from the training data set). in "born-again networks", @xcite the teacher and student have the same neural archi-tecture and model size, and yet surprisingly the student is able to surpass the teacher's accuracy. intuitively, such self-knowledge distillation is effective because the teacher's output distribution provides a richer training signal capturing additional information about training examples. in the context of summarization, the teacher can benefit student training in two ways. it provides a softened distribution over reference summaries thereby enriching the single reference setting. moreover, the teacher's distribution is (to a certain extent) denoised enabling the student to circumvent inaccuracies in the training data. we further capitalize on the idea that both the teacher and the student should be robust to noise and introduce several noise injection techniques which together with knowledge distillation improve model generalization and performance. we present experiments on several summarization benchmarks @xcite @xcite covering single-and multi-document summarization settings as well as different types of summaries (e.g., verbose or more telegraphic). across datasets, the proposed framework boosts the performance of pretrained and non-pretrained abstractive summarizers, achieving new state-of-the-art results. ## self-knowledge distillation for text summarization self-knowledge distillation refers to the special case where the teacher and student have identical neural network architectures. surprisingly, perhaps, it has been consistently observed @xcite @xcite ) that students trained with self-knowledge distillation outperform their teachers by significant margins in several computer vision and language modeling tasks. recent efforts have also focused on understanding why this happens, e.g., by observing that knowledge transferred by the teacher is localized mainly in higher layers and does not affect early (feature extraction) layers much (gotmare et al., 2019), by interpreting the teacher's knowledge as importance weighting @xcite , by showing that early-stopping is crucial @xcite , and by studying how selfdistillation modifies regularization @xcite . for text summarization, we argue that selfknowledge distillation can potentially alleviate problems in conventional maximum likelihood training. summarization models are typically trained on single reference document-summary pairs, however considering a single summary as the only correct reference during maximum likelihood training can harm model generalization @xcite and is counter-intuitive. there can be multiple valid summaries for a source input @xcite and even the single reference summaries available are not entirely goldstandard due to the inherent noise in the automatic construction of large-scale summarization datasets @xcite . with self-knowledge distillation, teacher outputs provide softened distributions of the reference summaries, which can be viewed as an enrichment of the single reference setting and a reweighting of gold summaries to prevent the student from becoming overconfident in its predictions. the standard objective for an abstractive summarization model is negative log likelihood: it is common practice to compensate for no direct access to the training data (see equation ( 3 )) by interpolating between the two losses in equations (3) and (2). so, the final objective for training the student becomes: noisy teacher to inject noise into the distillation signals, we incorporate a teacher dropout mechanism @xcite , where dropout is kept active while generating teacher predictions for training the student. in this manner, the teacher generates variable supervision labels for the student with some degree of uncertainty, alleviating the problem of overfitting to the teacher predictions. meanwhile, it can also be considered as approximating an average ensemble from many neural networks @xcite . the knowledge distillation loss now becomes: noisy student to inject noise into the training data, we propose various mechanisms to perturb the source input. random perturbation is effective in enforcing local smoothness for training text generation models under the assumption that semantically similar inputs can be mapped to the same or similar targets. a related approach has been shown to improve the performance of machine translation models in self-training settings @xcite . for text summarization, where the input is usually a long document, we design the following perturbation policies: ## experimental setup in this section, we describe the summarization datasets used in our experiments and discuss various implementation details. ## conclusions in this paper we advocated the use of selfknowledge distillation for abstractive summarization, as a means to alleviate problems associated with maximum-likelihood training for this task. we also introduced several noise functions (in the training signal and training data) which help regularize training and further boost performance. experiments on three benchmark datasets demonstrate that our framework can improve both nonpretrained and pretrained summarizers. in the future we would like to investigate more thoroughly which aspects of pretrained models improve and how self-knowledge distillation can be enhanced with more sophisticated noise functions.