LLM aided semi-supervision for efficient Extractive Dialog Summarization (2023)

Abstract: Generating high-quality summaries for chat dialogs often requires large labeled datasets. We propose a method to efficiently use unlabeled data for extractive summarization of customer-agent dialogs. In our method, we frame summarization as a question-answering problem and use state-of-the-art large language models (LLMs) to generate pseudo-labels for a dialog. We then use these pseudo-labels to fine-tune a chat summarization model, effectively transferring knowledge from the large LLM into a smaller specialized model. We demonstrate our method on the TWEETSUMM dataset, and show that using 10% of the original labelled data set we can achieve 65.9/57.0/61.0 ROUGE-1/-2/-L, whereas the current state-of-the-art trained on the entire training data set obtains 65.16/55.81/64.37 ROUGE-1/-2/-L. In other words, in the worst case (i.e., ROUGE-L) we still effectively retain 94.7% of the performance while using only 10% of the data.

## introduction customer support chats are gaining popularity as a primary medium for servicing clients in industry. summarization models that robustly capture critical information are therefore extremely beneficial for companies to support various downstream activities such as record-keeping, training. unfortunately, standard supervised training of these models rely on large labeled datasets, which can be prohibitively expensive to build. while unsupervised summarization techniques exist @xcite @xcite , enforcing the summary quality and style is still an ongoing challenge. in general terms, chat summarization can be abstractive, where the generated summary is unconstrained @xcite @xcite , or extractive, where the summary consists of parts of the original input @xcite . in this work, we propose a novel extractive dialog summarization method based on a semi-supervised learning paradigm, and demonstrate that we either improve or perform comparably to the current stateof-the-art on the tweetsumm dataset @xcite , while using only 10% of the training data. pseudolabeling @xcite is a popular and efficient method for semi-supervised learning. pre-trained llms encode a vast breadth of knowledge in various tasks and can generate human like response. they are good annotators @xcite and also ideal for transferring knowledge through distillation @xcite @xcite . hence, we decided to use llms 1) as weak labellers to generate automatic summaries for a large set of unlabelled examples (i.e., pseudo-labels); and 2) as evaluators to choose only a small number of high-quality @xcite examples to include in the next training cycle. we report strong results on the tweetsumm dataset in zero-and few-shot semi-supervised settings. our main contributions are as follows. labelled data unlabelled data pseudolabelled data filtered pseudolabelled data step1: weakly label with gpt-qa prompt step 2: evaluate (token probability) new labelled set new unlabelled set step 3: train distilbart iterate times related work chen and yang (2021) introduced conversational data augmentation (coda), a simple conversational data augmentation framework for abstractive dialog summarization with an optional extension for a semi-supervised learning setting @xcite . @xcite propose a heuristic-based weaklysupervised abstractive summarization model for the tweetsumm dataset. they first train separate models for customer and agent on a weakly-labeled set obtained by using the long and lead heuristics and then finetune such models on labelled conversation data. @xcite use pretrained encoders like bert and propose presumm, a model for abstractive and extractive summarization based on sentence classification. ## experimental setup data in our experiments, we use the tweet-summ dataset @xcite , which contains 1, 100 labelled examples d l of dialog between customers and customer support agent. we use the original splits and have 880, 110, and 110 examples for training, model selection, and testing, respectively. in semi-supervised experiments, we subsample a fraction of data as labelled and rest is used as unlabelled. we repeat these experiments our method we used the method described in section 2.2. concretely, we use distil-bart (shleifer and rush, 2020b)-a distilled version of bart @xcite with 6 encoder and 6 decoder layers-as the sequence-to-sequence model. we set the number of examples d ′ p selected by the evaluator at each cycle as 16, the total number of cycles @xmath0, and in each cycle we train the extractive summarization model for 10 epochs. vanilla method we also explore a 'vanilla' version of our method. crucially, with this method we do not transfer knowledge from a massive llm (e.g., gpt-3.5) into the summarization model but use the summarization model in a two-step noisy self-training paradigm @xcite . ## results in § 4.1, we first discuss how our method compares to simple baselines (iterative and non-iterative), clearly showing it surpasses both. in § 4.2, we examine how our approach fares in a few-shot learning setting where there is no-to-little training data available. in § 4.3, we compare our best few-shot models with the current sota using all training data available, and show that our method performs comparably to the current sota while using only ## conclusions and future work in this work, we introduce a semi-supervised approach for extractive summarization to distill knowledge from general-purpose llms into smaller specialized summarization models. we demonstrate our method on the tweetsumm dataset, and show that while training on only 10% of the data our method is competitive to presumm, the current state-of-the-art, which uses 100% of the training data. we outperform presumm by 0.3% according to rouge-1 and rouge-2, but are still 4% behind according to rouge-l. as future work, we will extend our method to other text summarization datasets. we will also explore using various open-source, tunable llms for both summarization and pseudolabelling.