I nherit S umm: A General, Versatile and Compact Summarizer by Distilling from GPT (2023)

Abstract: While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications. Conversely, previous studies have found that although automatic metrics tend to favor smaller fine-tuned models, the quality of the summaries they generate is inferior to that of larger models like GPT-3 when assessed by human evaluators. To address this issue, we propose InheritSumm, a versatile and compact summarization model derived from GPT-3.5 through distillation. InheritSumm not only exhibits comparable zeroshot and fewshot summarization capabilities to GPT-3.5 but is also sufficiently compact for fine-tuning purposes. Experimental results demonstrate that InheritSumm achieves similar or superior performance to GPT-3.5 in zeroshot and fewshot settings. Furthermore, it outperforms the previously established best small models in both prefix-tuning and full-data fine-tuning scenarios.

## introduction recently, the development of large language models (llms) like gpt-3 @xcite and palm @xcite has largely revolutionized the text summarization community, bringing a new paradigm in the the way summaries are generated. llms have demonstrated unparalleled ability to produce highly readable summaries while requiring little to no training data, overwhelmingly prefered by human annotators than those from smaller models @xcite . human evaluators note that gpt-3 generates summaries with superior readability and coherence, sometimes even more favored than human-generated summaries @xcite . one of the key advantages of llms is their capacity for zeroshot and fewshot learning @xcite , which enables them to adapt to new domains with ease. there-fore, llms are highly attractive for a wide range of summarization applications. despite these remarkable achievements, training and deploying llms for summarization is computationally expensive. deploying an llm for inference is already impractical for many nlp practitioners, especially for summarization where long input length of the document and demonstrative examples are typical. moreover, the prohibitive cost to finetune an llm makes them hard to adapt to various custom domains, when abundant training data is available. this largely hurts the widespread adoption of llms for summarization. to this end, @xcite proposes the target of achieving all three aspects of the impossible triangle for nlp models: moderate model size, superior zeroshot / fewshot learning capability, and superior supervised learning capability. in light of these challenges, we propose inher-itsumm. the model exhibits a compact size and offers cost-effective fine-tuning and deployment capabilities. its versatility is demonstrated by its applicability in both zero-shot/few-shot settings as well as supervised settings. furthermore, the model's generalization capability is evidenced by its promising performance across a wide range of benchmarks in all mentioned settings. inheritsumm is trained using knowledge distillation from the gpt-3 model, by mimicing the gpt-generated summaries on general documents. to facilitate this, we curated the gpt-summ dataset, with over 7 million documentsummary pairs. the documents are collected from general language corpora and gpt-3 generates the corresponding summary for them. utilizing this dataset, we train a zcode++ @xcite model by following the gpt-generated summaries in both zeroshot and fewshot learning settings. one important limitation of fewshot learning for summarization is the input length limit: the in-context demonstrations consume valuable input space, making it challenging to incorporate many in-context examples within the prompt. to address this issue, we propose utilizing succinct demonstrations that aim to shorten the demonstration documents and subsequently allow for the inclusion of more in-context examples. by training on the knowledge in gptsumm and using succinct demonstrations, we show that inheritsumm is able to closely match, or sometimes even outperform gpt-3 in zeroshot and fewshot summarization. therefore, our inher-itsumm with only 398m parameters is able to achieve the impossible triangle (zhu and zeng, 2022) for the summarization task. we show that in-heritsumm has strong performance and most versatile summarization capability to date: it achieves best or close-to-best performance in all the settings including zeroshot/fewshot learning with prompts, fewshot learning with prefix tuning, as well as fully supervised learning. our key contribution are three-fold. firstly, we build the gptsumm dataset with more than 4m paragraph-summary pairs by using querying gpt-3.5. to the best of our knowledge, gptsumm is the largest corpus to date focusing on distilling from gpt-3.5, and is the first one focusing on summarization. secondly, we build inheritsumm based on the gptsumm corpus. we show that in-heritsumm exhibits versatile capabilities across zeroshot, fewshot, and supervised settings, achieving the impossible triangle for summarization @xcite . lastly, we propose new methods to include in-context examples for fewshot summarization that improves the performance. ## problem statement and overview in summarization, the goal is to summarize the input document d into a short summary y . for zeroshot and fewshot learning, we can add a prompt p to the document to familiarize the model (parameterized with θ) with the given task. a sequence-tosequence model then takes the input @xmath0; d] 1 and predicts the summary ŷ . in zeroshot learning, p is a short description of the task, e.g., "summarize the following document into a short paragraph". for fewshot learning with in-context examples, the prompt consists of several descriptive examples along with an instruction, i.e., @xmath1; y 1 , ..., x n ; y n ; i], where x i and y i are 1 our method also applies to the case where the prompt content appears after the document d. illustrative examples, and i is the task description. for prefix tuning and supervised learning, p is empty and @xmath2. we tune parameters θ to adapt the model to a given task. figure 1 describes our overall method. we distill summarization skills from gpt-3 by using it to generate summaries for documents from different domains, including general documents used for language modeling, as well as specialized documents from labeled datasets. the gpt-generated summaries and (relatively fewer) human generated summaries from labeled datasets forms our gptsumm dataset. we then use gptsumm to train a seq2seq model, and then adapt it to zeroshot/fewshot learning, prefix tuning and supervised learning summarization. in the following subsections, we first describe the method to build the gptsumm dataset in section 3, and then introduce the model training process in section 4. finally we describe our method to adapt the pretrained inheritsumm model to zeroshot, fewshot, and supervised summarization. ## distillation data collection of gptsumm we discuss the construction of gptsumm as the data for distillation in this section. ## model training we proceed to train a seq2seq transformer language model zcode++ @xcite parameterized by θ, utilizing the gptsumm data. the objective is to minimize negative log-likelihood given the inputs with prompts [p ; x]: ## experiments we detail the experiment results in this section. we first specify the implementation details and hyperparameters in sec. 5.1, and then introduce the main experiment results in sec. 5.2. we use the zcode++ model @xcite ) as the language model backbone for training inherit-summ. we choose zcode++ over other pretrained models since it achieves state-of-art performance on summarization tasks after finetuning. we experimented with both the pretrained z-code++ base and z-code++ large model. we train the model on the gptsumm corpus with seq2seq loss (1) for 300k steps, with a maximum input length of 3072. we summarize the hyperparameters in ## conclusion & future works we propose inheritsumm by distilling knowledge from gpt-3.5 using its summary on a broad range of documents. base model of inherit-summ with only 400m parameters exhibits versatile capability on zeroshot, fewshot, and fully supervised summarization, surpassing performance of previous small models and beats or rivals gpt-3.5 in the overall performance. for future works, it would be interesting to investigate the performance when distilling from other variants of gpt, like text-davinci-003 or gpt-4. another interesting direction is controllable summarization -by using proper instructions, in-heritsumm can be further trained to generate customized summarizations with style or length constraints.