Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations (2023)

Abstract: Large language models (LLMs) like ChatGPT can be expensive to train, deploy, and use for specific natural language generation tasks such as text summarization and for certain domains. A promising alternative is to fine-tune relatively smaller language models (LMs) on a particular task using high-quality, in-domain datasets. However, it can be prohibitively expensive to get such high-quality training data. This issue has been mitigated by generating weakly supervised data via knowledge distillation (KD) of LLMs. We propose a three-step approach to distill ChatGPT and fine-tune smaller LMs for summarizing forum conversations. More specifically, we design a method to selectively sample a large unannotated corpus of forum conversation using a semantic similarity metric. Then, we use the same metric to retrieve suitable prompts for ChatGPT from a small annotated validation set in the same domain. The generated dataset is then filtered to remove low-quality instances. Our proposed select-prompt-filter KD approach leads to significant improvements of up to 6.6 ROUGE-2 score by leveraging sufficient in-domain pseudo-labeled data over a standard KD approach given the same size of training data.

## introduction large language models (llms) such as the gptseries models have demonstrated great strengths in a range of natural language understanding and generation tasks with their ability to do few-shot or incontext learning @xcite . in this scenario, the input for a desired task is enhanced by incorporating a few examples (or demonstrations) of that particular task given as the prompt to the llm. this helps to customize and optimize the llm at inference time, leading to higher-quality results. many of these llms, however, are astronomically expensive and environmentally unsustainable to train and use @xcite ; wu et @xcite . an alternate strategy is to fine-tune relatively much smaller language models (lm) like bart @xcite or t5 @xcite on the target task. these fine-tuned models, however, require large annotated training datasets to achieve competitive results, preventing the adoption of this approach for a wide variety of tasks and domains. obtaining sufficient annotated data from human annotators can also be substantially expensive @xcite . an approach to address this issue of limited high-quality annotated data involves creating synthetic data that extracts and distills knowledge from icl-capable llms to smaller specialized models. this technique of knowledge distillation @xcite helps specialize smaller pre-trained lms to particular tasks using large amounts of such synthetic annotations. for example, @xcite use gpt-3 to annotate training data for nine nlp tasks, including summarization, estimating up to 96% reduction in costs over using crowdsourcing and achieving similar performance. @xcite use gpt-3 to get labels and even generate data from scratch for sequence tagging tasks. however, previous work on distillation addresses the data annotation process without careful data selection, prompt engineering, or data filtering, leading to suboptimal results. to get a better-distilled model and optimize the cost of synthetic data annotation, we propose a systematic distillation approach, named select-promptfilter knowledge distillation (spf kd), for summarization. we use chatgpt as the icl-capable llm that we distill. in this paper, we focus on summarizing online messaging conversations (e.g., forums) for which human-annotated datasets are not publicly available, though our method can be applied to other low-resource domains as well. first, a suitable subset of conversations is selected for distillation from a large unannotated raw corpus, based on similarity to a small annotated validation set. then, for each conversation in the selected subset, semantically similar conversations are retrieved from the validation set and sent to chatgpt as the prompt to get its synthetic summary. @xcite use a similar prompt-retrieval approach based on various sentence embeddings to improve gpt-3 performance on downstream tasks, although not in the context of synthetic data annotation. thereafter, we apply a filter on the generated conversationsummary pairs to remove spurious and low-quality annotations. figure 1 shows a schematic overview of our approach. our proposed method uses a semantic similarity score @xcite and a reference-less evaluation measure @xcite at different steps. we evaluate our approach on three forum and email conversation test sets. we show that the proposed kd method is able to significantly improve summarization performance of up to 6.6 rouge-2 scores compared to a standard kd approach. ## approach our method addresses the lack of high-quality annotated summarization datasets on certain domains to fine-tune smaller summarization models via sequence-level knowledge distillation @xcite . we propose a three-step approach to get the annotated training data from icl-capable llms like chatgpt, which acts as our teacher model that we distill. the three steps (figure 2 ) in our proposed select-prompt-filter (spf) knowledge distillation approach are (i) data selection (ii) prompt retrieval, and (iii) data filtering. these steps are done to get high-quality synthetic summaries for the target task of summarizing conversations. the synthetic dataset will then be used to fine-tune a smaller pretrained lm (student model). fundamental to our approach are two metrics that we use in the above steps: use-cosine: we use universal sentence encoder foot_0 (use, @xcite to get the embeddings of a conversation given as a text string. use internally uses a deep averaging network @xcite that averages the embeddings of the words and bigrams and passes it through a feed-forward network. we compute the cosine distance between the two conversation embeddings. we refer to this metric as use-cosine, which we use for computing conversation-conversation similarity in the selection and prompt retrieval steps. shannon score: shannon score foot_1 @xcite is a reference-free summary evaluation metric that measures the difference in the ability of a language model to generate the conversation c with and without the summary s. specifically, gpt-2 is used to compute the information difference, i.e., the difference between log probabilities of generating c given s as the prompt and generating c without it. the final metric, shannon score (ss), is the ratio of this value to the information difference when the c itself is used as the summary. this is under the assumption that the conversation conveys the entire information and thus the denominator becomes the upper bound of the information difference. we use shannon score to compute the conversation-summary relevance and filter the synthetic annotations generated by chatgpt that do not satisfy a certain threshold. in our proposed approach, we rely on the availability of a large unannotated training dataset of forum threads (d t ) for knowledge distillation for our target task. we use a large corpus of forum conversations for this. we also assume the availability of a small annotated validation set d v with m conversation-summary pairs. we describe the three steps used in our approach in greater detail below. ## results and analysis in ## conclusion we propose a methodical knowledge distillation approach that finetunes a smaller lm for summarizing conversations with synthetic summaries generated by chatgpt. our approach selects a suitable subset to be distilled, retrieves matching prompts for chatgpt, and filters the chatgpt-generated summaries to yield better-quality training data. our proposed method substantially improves over a standard distillation approach on three test sets covering forum and email conversations.