Enhancing Abstractiveness of Summarization Models through Calibrated Distillation (2023)

Abstract: In this paper, we propose a novel approach named DisCal to enhance the level of abstractiveness (measured by n-gram overlap) without sacrificing the informativeness (measured by ROUGE) of generated summaries. DisCal exposes diverse pseudo summaries with two supervision to the student model. Firstly, the best pseudo summary is identified in terms of abstractiveness and informativeness and used for sequence-level distillation. Secondly, their ranks are used to ensure the student model to assign higher prediction scores to summaries with higher ranks. Our experiments show that DisCal outperforms prior methods in abstractive summarization distillation, producing highly abstractive and informative summaries.

## introduction text summarization is the task of generating a concise and condensed summary of a source document while preserving its most important information @xcite . unlike extractive summarization, which involves selecting and concatenating sentences from the original document @xcite , abstractive summarization is a sequence-to-sequence (seq2seq) problem that can generate novel phrases and sentences that were not present in the original document @xcite @xcite . recent advances in large pre-trained language models have greatly accelerated summarization modeling progress @xcite , but there are still significant concerns for the large language model's real use cases due to the slow inference speed under a production-level environment. input: jose mourinho has lauded chelsea's consistency, with a hint of caution, as his side bid to wrap up a wire-to-wire premier league victory. (...) chelsea have topped the premier league since the opening day but jose mourinho (left) will remain focused. blues captain john terry has been a pivotal (…) seq-level distil: chelsea have topped the premier league since the opening day. jose mourinho has praised chelsea's consistency, with a hint of caution, as his side bid to wrap up a wire-to-wire premier league victory. the blues have led or shared the lead since the opening round of fixtures and entered this weekend's matches seven points clear with eight matches remaining. w.o. distil: chelsea are top the premier league since opening day. chelsea have led or shared the lead since opening round of fixtures. the blues have been a key figure in keeping the side in consistent form. jose mourinho says his side will remain focused. manchester city slip up at crystal palace last week to end their title hopes. predicted summaries (rouge-1: 30.8 / novel 5-gram: 82.8) (rouge-1: 39.2 / novel 5-gram: 27.3) (rouge-1: 42.0 / novel 5-gram: 90.1) discal: chelsea won the premier league since the opening day. the blues have been sevenhave points clear with eight matches remaining. jose mourinho has praised the consistency and confidence. chelsea face qpr at loftus road. mourinho says manchester city lost 2-1 at crystal palace. figure 1 : summaries generated from the models without using knowledge distillation, "w.o. distil"; using sequence-level distillation, "seq-level distil" @xcite ; and using calibrated distillation ("dis-cal", ours) on cnndm data. fragments from the input are color-coded to indicate overlap: green, yellow, and red for over three, five, and ten tokens, respectively. knowledge distillation is a widely used technique for compressing a large model into a smaller one for faster inference with minimal performance loss @xcite @xcite . a prominent direction for abstractive summarization is known as sequencelevel distillation @xcite . this method involves generating a pseudo summary for each training document using a teacher model, and training a student model on pairs of training documents and their corresponding pseudo summaries. compared to methods that rely on word-level information, such as minimizing the cross-entropy loss between teacher and student prediction distributions @xcite , this approach enables the student to better mimic the teacher model's generation at the sequence level. despite the high rouge @xcite ) score achieved through sequence-level distillation, we argue that the pseudo summary generated by the teacher model exacerbates the student model's ten- cathay pacific was forced to cancel a scheduled flight from london to hong kong after one of the pilots was arrested after trying to (…) dynamic generator (teacher model) input text a b c cathay pacific canceled (…) pilot arrested after trying (…) cathay pacific was forced (…) pilot stopped during security checks as the flight prepared to depart (…) golden reference abstractiveness (novel n-gram) informativeness (rouge) b pilot arrested after trying to (…) b a @xmath0 ## related work large pre-trained seq2seq models have emerged as the de facto standard for abstractive summarization due to their exceptional performance and versatility. they excel in capturing the salient information from documents through various techniques. for instance, t5 @xcite predicts corrupted text spans, bart @xcite employs denoising auto-encoding, pegasus @xcite identifies the most summary-worthy sentences, and dialogled @xcite employs window-based denoising. due to the high computational cost associated with large models, there has been a surge of research focused on compressing these models @xcite . one prominent approach in this field is known as knowledge distillation, which involves training a smaller student model to mimic the predictions of a larger teacher model by minimizing the difference between the teacher and student predictions @xcite @xcite . particularly, in the context of abstractive summarization distillation with seq2seq models, @xcite proposed the sequence-level knowledge distillation approach which involves training a student model using pseudo summaries generated by the teacher model using beam search decoding. on the other hand, shleifer and rush (2020) proposed the shrink and fine-tune (sft) framework. this approach involves removing certain layers from the teacher model to create a smaller student model, which is then fine-tuned using gold summaries. in a recent study, @xcite introduced the method plate, which aims to smooth attention distributions of teacher models during pseudo summary generation and then fine-tune the shrunken student model with them. in addition, there is an interesting line of work called model calibration @xcite @xcite , where they leverage different candidate summaries to calibrate the model's predictions to overcome the problem of exposure bias @xcite . in contrast to prior research, our work focuses on the previously overlooked problem of decreased abstractiveness when distilling summarization models. we propose a solution called calibrated distillation, which achieves a high level of informativeness and abstractiveness using a smaller model. ## methodology we introduce a new distillation method dis-cal (abstractive summarization distillation with calibration) in this section. briefly speaking, the dynamic summary generator (in section 4.1) provides a list of feasible pseudo summaries for each document and the calibrated distillation (in section 4.2) tunes the student model to output the summary with high informativeness and abstractiveness. ## evaluation datasets. we evaluate discal on three widely-used abstractive summarization datasets: two news summarization datasets of cnn/dailymail @xcite and xsum @xcite ; and a dialogue summarization dataset of samsum @xcite . the detailed statistics of the datasets can be found in teacher and student models. following the literature @xcite , we consider bart large @xcite , one of the widely used seq2seq transformer architectures for abstractive summarization. the bart large model is trained on the entire dataset with gold summaries as a teacher model. then, we configure two student models with identical transformer encoder layers to the teacher, but they differ in the number of decoder layers: bart 12-6 and bart 12-3, with six and three decoding layers, respectively. referring to the sft pipeline (shleifer and rush, 2020), the student models are initialized from the 12-encoder-layer/12-decoder-layer teacher. the two student models copy the full encoder from the teacher model. but, the decoder of bart 12-6 is copied from the {0, 2, 4, 6, 8, 10} decoder layers of the teacher, while the decoder of bart 12-3 from the {0, 5, 11} decoder layers. this initialization is simple but effective since it eliminates the need for separately pre-training the two student models. in appendix d, we validate the generalizability of discal on a different state-of-the-art seq2seq model, dialogled @xcite . algorithms. we compare discal with the three prior knowledge distillation approaches, namely shrink and then fine-tune (sft) @xcite and two sequence-level knowledge distillation methods, seq-distil @xcite and plate @xcite . the sft method trains the student model on pairs of documents and their corresponding gold summaries without using pseudo summaries. on the other hand, the other two methods only rely on the pseudo summary generated by the teacher model using beam search decoding; plate is different from seq-distil in terms of scaling up the teacher's attention temperature in pseudo summary generation. we re-implement all compared methods and train them in the same environment using eight nvidia v100 gpus and pytorch 1.13.1 @xcite . implementation details. similar to recent studies @xcite , we train bart large (teacher model) using the adam optimizer (kingma and ba, 2014) with an initial learning rate of 5e-5 and a label smoothing of 0.1. the teacher model is trained for 20, 000 steps on cnndm and xsum with a weight decay of 0.001 and a batch size of 64, while 5,000 steps on sam-sum with a weight decay of 0.1 and a batch size of 16. we use the same training configuration on the two student models for seq-distil and plate. as for our hyperparameters, we tune them on validation sets. the maximum value for re-scaling in eq. ( 3 ), the balancing term for the calibration score in eq. ( 4 ), and the weight for nnl loss in eq. ( 7 ) are respectively set at 2.0, 0.2, and 0.01 on cnndm; 1.5, 0.2, and 1.0 on xsum; and 1.5, 0.2, and 0.1 on samsum. the number of pseudo summaries n per document is set at 6. the detailed implementation including hyperparameter settings for all methods are provided in appendix b.1. regarding inference, we apply beam search following the convention @xcite . we set the beam size, length penalty, minimum length, and maximum length to 4, 2.0, 55, and 142 on cnndm; 6, 2.0, 10, and 62 on xsum; 6, 2.0, 11, and 62 on samsum. for evaluation, we use rouge f1 and novel n-gram scores as the informativeness and abstractiveness metrics. refer to appendix b.2 for details. ## conclusion we propose discal, an improved knowledge distillation method that leverages diverse candidate summaries generated by teacher model. by evaluating and ranking pseudo summaries during distillation, discal chooses the best summaries in terms of informativeness and abstractiveness, and this enhances model predictions based on output calibration. experiments on three summarization datasets demonstrate that discal produces summaries with a higher level of abstractiveness as well as informativeness.