In-context Learning of Large Language Models for Controlled Dialogue Summarization: A Holistic Benchmark and Empirical Analysis (2023)

Abstract: Large Language Models (LLMs) have shown significant performance in numerous NLP tasks, including summarization and controlled text generation. A notable capability of LLMs is in-context learning (ICL), where the model learns new tasks using input-output pairs in the prompt without any parameter update. However, the performance of LLMs in the context of few-shot abstractive dialogue summarization remains underexplored. This study evaluates various state-of-the-art LLMs on the SAMSum dataset within a few-shot framework. We assess these models in both controlled (entity control, length control, and person-focused planning) and uncontrolled settings, establishing a comprehensive benchmark in few-shot dialogue summarization. Our findings provide insights into summary quality and model controllability, offering a crucial reference for future research in dialogue summarization.

## introduction abstractive dialogue summarization aims to distill human conversations into natural, concise, and informative text, and is a challenging and interesting task in text summarization @xcite . the major challenges come from several aspects: 1) it lacks large humanannotated datasets unlike document summarization @xcite , and 2) it requires responses to be not only fluent but also factually consistent @xcite . moreover, in practical use cases, users may impose additional constraints on system outputs, and this task is known as controlled dialogue summarization, which requires models to be capable of coherent and flexible language generation. in controlled dialogue summarization, users can specify desired attributes (i.e., control signals) to guide the response of language models. previous works have explored to incorporate control * work done during internship at i 2 r, a*star. signals during pre-training @xcite , task-specific fine-tuning @xcite , and prompt tuning @xcite . meanwhile, the advancements in llms have unveiled new paradigms. for instance, instruction tuning, which enables models to understand users' intent in natural language, is considered to be promising for conditional text generation @xcite . additionally, the emergence of in-context learning (icl) in llms has recently gained attention. the icl ability refers to learning from a few inputoutput pairs written in the natural language form (also called demonstrations) @xcite . followed by demonstrations, a query question is appended at the end to form a complete prompt. compared to the traditional supervised learning, icl requires no training and only a few annotated samples. motivated by the paradigm shift with llms and the challenges encountered in controlled dialogue summarization, this study answers the following two key questions: we comprehensively evaluate a range of recent large language models (llms) on the samsum dataset @xcite using a few-shot framework. our assessment covers several controlled scenarios, including entity control, length control, and person-focused planning, as well as uncontrolled settings. we establish a comprehensive benchmark for few-shot dialogue summarization in section 2, and elaborate on the findings in section 3. specifically, in our experiments, we observe that llms can summarize dialogues reasonably given several demonstrations, and llama and alpaca achieve a factual consistency rate exceeding 90% in the automatic evaluation. moreover, adding control ## our experimental setting of icl dialogue summarization in this section, we describe how we establish the benchmark of evaluating llms' in-context learning for abstractive dialogue summarization. ## our findings in this section, we analyze the experimental results by discussing about the two research problems. ## conclusion in this study, we have benchmarked the in-context learning performance of state-of-the-art llms in controlled and uncontrolled settings for abstractive dialogue summarization. we assessed their summarization quality, factual consistency, and controllability, while also conducting holistic evaluations and empirical analysis. we hope this study provides insights for the follow-up research about dialogue summarization using llms.