T ri S um: Learning Summarization Ability from Large Language Models with Structured Rationale (2024)

Abstract: The advent of large language models (LLMs) has significantly advanced natural language processing tasks like text summarization. However, their large size and computational demands, coupled with privacy concerns in data transmission, limit their use in resource-constrained and privacy-centric settings. To overcome this, we introduce TriSum, a framework for distilling LLMs’ text summarization abilities into a compact, local model. Initially, LLMs extract a set of aspect-triple rationales and summaries, which are refined using a dual-scoring method for quality. Next, a smaller local model is trained with these tasks, employing a curriculum learning strategy that evolves from simple to complex tasks. Our method enhances local model performance on various benchmarks (CNN/DailyMail, XSum, and ClinicalTrial), outperforming baselines by 4.5%, 8.5%, and 7.4%, respectively. It also improves interpretability by providing insights into the summarization rationale.

## introduction large language models (llms), such as gpt-3 @xcite and its successors @xcite @xcite , has greatly advanced natural language processing tasks, including machine translation @xcite , question-answering (qa) systems @xcite , and text summarization @xcite . however, due to their substantial model size and computational demands, their utility can be limited in resourceconstrained environments @xcite . moreover, privacy becomes a major concern when sending proprietary data to external llm services like chatgpt. among others, text summarization is a crucial task for transforming lengthy texts into concise yet informative summaries @xcite . however, many existing methods struggle to generate structured summaries @xcite @xcite . these structured summaries need to encompass essential aspects, key entities and relationships, and a coherent final summary derived from these aspects and rationales. recent developments have seen the utilization of llms to grasp a text's topic structure and core ideas @xcite , suggesting their potential in generating structured text summaries. while rational distillation from llms has been employed for nlp tasks like qa, natural language understanding (nlu), and arithmetic reasoning @xcite @xcite , its applicability to abstractive text summarization remains unexplored. in this study, we aim to distill llms' text summarization prowess into a more compact local model. we enhance the transparency and interpretability of this local model by incorporating elicited rationales from llms' summarization process as additional guidance. to achieve this, we introduce a three-step framework trisum (as shown in figure 1 ) involving llm rationale probing, golden rationale selection, and local training: step 1: we first prompt vital aspect-triple rationales and summaries from the input text using llms. this set includes essential aspects, rele-vant triples extracted from the text, and a concise summary that's tied to these aspects and triples. step 2: next, to ensure quality, we employ a dualscoring method for selecting golden (high-quality) rationales to use in the subsequent training. this method evaluates the summary's quality based on semantic similarity and ensures coherent rationales using a topic distribution-based approach. step 3: last, we train our compact local model using a curriculum learning approach @xcite . this method progressively fine-tunes the model by starting with simpler tasks and gradually advancing to more complex ones. this process enables our model to gradually incorporate the rationalized summarization skills acquired from the llms. our research brings the following contributions. overall, our study streamlines powerful summarization models in resource-limited contexts, offering insights into harnessing llms' inherent summarization abilities. however, the resource demands of llms have limited their widespread use. concerns over privacy when using llm-as-a-service apis have also arisen, especially for sensitive data. this highlights the need for more compact local models that can still capture summarization abilities. to harness the summarization ability of llms, @xcite uses llms to augment labels for headline generation, while @xcite used summaries created by llms as benchmarks for training their local models. llms were also used to evaluate summary quality during training. however, this approach did not fully transfer the reasoning skills of llms to the local models, indicating a partial capture of llms' summarization abilities. also, the uncertainty of labels generated by deep learning models may affect reliability. rationale distillation for interpretability in llms knowledge distillation, as introduced by @xcite , refers to the concept for transferring knowledge from a large model (teacher) to a smaller one (student) to make deep learning models usable in resource-limited environments. this idea has been applied and extended across various fields @xcite @xcite @xcite . notably, @xcite focused on abstractive summarization, while @xcite emphasized extractive summarization. the complexity of deep neural networks has driven research toward making ai models interpretable @xcite . rationale generation is an emerging technique in interpretability, highlighting a model's key reasoning steps @xcite . in knowledge distillation, rationale generation enhances interpretability, offering insights into the decision-making of llms. this informs the development of better knowledge distillation methods. @xcite developed a smaller model using llm-generated rationales and questions. others @xcite @xcite used llm-produced rationales to train models, improving performance and transparency in predictions, primarily for tasks like qa, nlu, arithmetic reasoning, and extractive summarization @xcite . this has left a gap concerning abstractive text summarization. to bridge this gap, we introduce an aspect-triple rationale generation approach, aimed at distilling the summarization prowess of llms. this method consists of a procedure of extracting essential aspects, pinpointing primary relationships, and constructing a definitive summary. ## conclusion we introduced trisum, an approach aimed at distilling summarization capabilities from a large language model to a small local model. extensive experiments verified its superior performance over state-of-the-art models across diverse datasets on the abstractive summarization task. our work highlights the potential of leveraging large model insights for efficient and nuanced text summarization. in many applications. trisum not only improves summarization performance but also enhances the interpretability of the summarization process, making it more trustworthy.