Towards Understanding Omission in Dialogue Summarization (2023)

Abstract: Dialogue summarization aims to condense the lengthy dialogue into a concise summary, and has recently achieved significant progress. However, the result of existing methods is still far from satisfactory. Previous works indicated that omission is a major factor in affecting the quality of summarization, but few of them have further explored the omission problem, such as how omission affects summarization results and how to detect omission, which is critical for reducing omission and improving summarization quality. Moreover, analyzing and detecting omission relies on summarization datasets with omission labels (i.e., which dialogue utterances are omitted in the summarization), which are not available in the current literature. In this paper, we propose the OLDS dataset, which provides high-quality omission labels for dialogue summarization. By analyzing this dataset, we find that a large improvement in summarization quality can be achieved by providing ground-truth omission labels for the summarization model to recover omission information, which demonstrates the importance of omission detection for omission mitigation in dialogue summarization. Therefore, we formulate an omission detection task and demonstrate our proposed dataset can support the training and evaluation of this task well. We also call for research action on omission detection based on our proposed datasets. Our dataset and codes are publicly available.

## introduction with the exponential increase in the volume of conversational messages from daily life, there is a growing demand for dialogue summarization @xcite @xcite @xcite , which compresses lengthy interactions into a more concise and structured piece of text while preserving the most important and relevant information. recent years have witnessed significant progress in abstractive dialogue summarization, especially using large-scale pre-trained language models @xcite . despite the advances in a high level of fluency and coherence, existing models are still prone to generate defective summaries @xcite @xcite ) that limit their practical usage. previous works @xcite @xcite have investigated the taxonomy of errors involved in output summaries, and human evaluations revealed that the majority of errors fall into the category of omission, which often leads to incomplete summaries where critical facts are lost. however, few of these works have further analyzed the omission problem, let alone addressing this problem. to reduce omission rate and improve summarization quality, a comprehensive analysis on omission problem (e.g., how omission affects summary results) and a precise omission detection (i.e., to locate which dialogue utterances are omitted in the summarization) is important. however, there are no omission related datasets in dialogue summarization literature to support such analysis and detection. hence, in this work, we construct the olds dataset, which provides high-quality omission labels for dialogue summarization. our dataset is built upon five existing benchmarks covering different domains. for each dialogue, we use different abstractive models to generate diverse candidates and propose a reference-based strategy to automatically label omissions for these candidates. the human evaluation indicates that our olds dataset presents a high quality of omission labels. based on the curated olds dataset, we comprehensively investigate the omission problem in dialogue summarization from multiple aspects. first, we analyze the proportion of candidates with omission errors and the position distribution of omitted information in dialogues. the results reveal that omission is a severe problem that frequently occurs in dialogue summarization. second, we measure the correlation between the omission rate and multiple reference-based metrics (e.g., rouge and bertscore), discovering that omission is one of the decisive factors influencing the summary evaluation results. third, we explore the potential performance improvement brought by utilizing the omission information in a post-editing manner. the analyses probe that candidate summaries could be effectively improved as long as the model is provided with the omitted dialogue utterances. hence, how to accurately locate omission information in dialogue naturally becomes a critical question. to pave the way to omission mitigation and summary improvement, we formulate the task of omission detection, which aims to identify the omitted utterance given the whole dialogue utterances and the generated summary with potential omission. in addition, we present three different frameworks as baselines for the omission detection task, including pair-wise classification, sequence labeling, and pointer network extraction. experimental analyses on the olds dataset reveal that omission detection, as a promising direction to assessment and improvement for dialogue summarization, poses significant values and challenges. the contributions of our paper are as follows: ## the olds dataset in this section, we first define what is omission. then, we introduce olds, a dataset that contains omission labels for dialogue summarization that facilitates the analysis of omission problem and dialogue: adam: i just talked to her and i must admit i worry about her. (04) karen: me too, i suggested she should see a specialist, but she wasn't very happy about it. (05) adam: no wonder... (06) karen: i know, but i think this is serious. she's saying she's depressed, like everyone around, but in her case it may be true. (07) adam: she was telling me she doesn't feel like doing anything, she's bored all the time, she never feels happy. it sounds like a real, typical depression. ...... ...... (12) adam: yes, but she doesn't want to see a specialist. basically, she doesn't want to see anyone. (13) karen: hm... i don't know... how about i call someone for advice? so we could know what to do. (14) adam: sounds rational, do you know anyone you could call? don't mention her name. (15) karen: of course i won't! i have a friend who's a psychologist, we can trust her. i'll let you know. (16) adam: thank you karen! ## understanding the omission problem in this section, we explore the omission problem in different aspects and analyze why we should pay attention to omission in dialogue summarization. we report the results of six adopted models on the test set of each dialogue domain. domains sam. dial. email. qm. tweet. rg-1 -0.563 -0.409 -0.445 -0.470 -0.574 rg-2 -0.448 -0.342 -0.394 -0.480 -0.524 rg-l -0.510 -0.398 -0.457 -0.494 -0.547 bleu -0.332 -0.289 -0.231 -0.397 -0.467 bs b -0.549 -0.502 -0.418 -0.463 -0.485 bs l -0.562 -0.504 -0.445 -0.521 -0.546 bleurt -0.567 -0.461 -0.292 -0.410 -0.525 ## the omission detection task since candidate summaries could be effectively improved given the gold omission information, how to accurately detect omission utterances in dialogue naturally becomes a critical question. in this section, we formulate the omission detection task in a reference-agnostic setting. formally, given a dialogue @xmath0, u 2 , .., u n } along with a candidate summary c, a detection model is required to extract a set of omission utterances o from d without knowing the reference summary. in this section, we introduce three typical frameworks as baselines and conduct evaluations to see how this task could benefit from them. ## conclusion in this work, we systematically study the omission problem in dialogue summarization based on the curated olds dataset, which collects candidate summaries from multiple models and domains and provides high-quality omission labels for them. we discover that omission is a significant problem that directly affects the results of dialogue summarization, and the defective candidate summary could be largely improved by leveraging the omission information properly. we further introduce an omission detection task to identify omission content, which is a challenging and valuable task that paves the way to omission mitigation and summary improvement in dialogue summarization.