Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection (2024)

Abstract: Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data and improve the performance of summarization models. While addressing label noise, previous works on semi-supervised learning primarily focus on natural language understanding tasks, assuming each sample has a unique label. However, these methods are not directly applicable to SSDS, as it is a generative task, and each dialogue can be summarized in different ways. In this work, we propose a novel scoring approach, SiCF, which encapsulates three primary dimensions of summarization model quality: Semantic invariance (indicative of model confidence), Coverage (factual recall), and Faithfulness (factual precision). Using the SiCF score, we select unlabeled dialogues with high-quality generated summaries to train summarization models. Comprehensive experiments on three public datasets demonstrate the effectiveness of SiCF scores in uncertainty estimation and semi-supervised learning for dialogue summarization tasks. Our code is available at https://github.com/amazon-science/summarization-sicf-score .

## introduction dialogue summarization generates concise summaries of dialogues, helping users quickly understand key points without navigating through complex contexts @xcite . this study prioritizes abstractive summarization, which offers more flexibility than extractive approaches @xcite . despite its wide applicability in scenarios like meetings and casual conversations, dialogue summarization faces challenges such as scarcity of annotations and high annotation costs. however, the proliferation of pretrained models and unlabeled dialogues offers a so- * the work was done during an aws ai labs internship. lution. in this paper, we explore semi-supervised dialogue summarization (ssds) @xcite , aiming to enhance dialogue summarization by a small labeled dataset alongside a large collection of unlabeled dialogues. previous ssds research @xcite ) has used data augmentation to increase the size of both labeled and unlabeled dialogue datasets, but the issue of pseudolabel noise has been largely overlooked. specifically, an initial model fine-tuned on labeled samples is used to generate pseudolabels for unlabeled samples. then, the unlabeled samples and their pseudolabels are used to train the semi-supervised model @xcite . however, the imperfections of the initial model can lead to pseudolabel noise, such as hallucination and missing key information. pseudolabel noise is a big concern in semi-supervised learning because training on pseudolabels with significant noise can deteriorate model performance. thus, we aim to address pseudolabel noise in ssds in this research. many existing solutions for pseudolabel noise estimation and mitigation are designed for understanding tasks (e.g., classification (cordeiro and carneiro, 2020)), and they are not directly applicable to ssds due to inherent diversity of ground truth summaries. specifically, these solutions, such as mix-up in mix-match @xcite , assume each sample has a unique label, representing a single attribute like a semantic class. in contrast, ssds is a generation task where each dialogue can be summarized in different ways. for instance, summaries like "the audience is happy to hear the news" and "the news makes the audience glad" convey the same message with different wording. as a result, ssds, like other generation tasks, does not have a unique label per sample. this distinction makes previous pseudolabel noise solutions unsuitable for ssds. thus, we need a new and generalized pseudolabel noise measurement solution that considers label diversity in ssds, with-out relying on ground truth summaries, as unlabeled dialogues lack them. to achieve this, we propose assessing pseudolabel quality: a predicted summary with higher quality indicates less noise in pseudolabels. we thus propose the sicf score, which measures summary quality based on common characteristics of highquality summaries, such as model confidence, information coverage, and faithfulness to the original dialogues. sicf comprises three components: "semantic invariance" assesses model confidence at the text level, "coverage" evaluates key information captured at the word level, and "faithfulness" measures alignment with the original dialogues at the sentence level. as shown in figure 1 , we then rank and select the unlabeled dialogues with high-quality pseudolabels as indicated by the sicf score. besides, since uncertainty estimation is a representative way to estimate the model prediction quality @xcite and bayesian neural network (bnn) is an effective uncertainty estimation method @xcite , we propose a variant-length multilabel bnn for our sicf score. our contributions are as below. in terms of semi-supervised extractive dialogue summarization, @xcite employ gpt 3.5 for quality assessment based on token probabilities. @xcite introduce self-supervised pre-training to enhance bert's ability to contextualize dialogue representations. regarding our focus, that is the semi-supervised abstractive dialogue summarization, coda @xcite is proposed to address ssds using data augmentation. while data augmentation can expand the size of both labeled and unlabeled data, it overlooks challenges posed by pseudolabel noise, a prevalent issue in semi-supervised learning. unlike previous ssds models that overlooked pseudolabel noise, our goal is to enhance ssds performance by measuring pseudolabel quality and effectively eliminating unreliable pseudolabels. solution of pseudolabel noise in semi-supervised learning. many methods have been proposed for label noise in natural language understanding tasks @xcite @xcite . however, most of these methods are not directly applicable to ssds, because this generation task has diverse ground truth summaries for each dialogue. while some of these methods have the potential to be applied towards ssds, like teacher-student knowledge distillation model for noisy text summarization @xcite , they do not consider the diversity of summaries within ssds. @xcite have a similar task setting to ours, but their task is multi-label image classification, which still provides a unique label for each image. @xcite focus solely on model generation without considering the interaction with context (e.g., dialogue in our task). in contrast, we consider both model prediction itself by semantic invariance and the relation between generations and context via coverage and faithfulness. as for injecting noise into the dialogues or pseudolabels @xcite , it focuses on improving the model's robustness through training with this injected noise and aims to mitigate the impact of noise. in contrast, our work aims to measure the extent of sample noise. furthermore, they need to retrain the model, and the added noise might degrade the model's performance. in contrast, our work does not require retraining the model and will not harm its performance. ## problem setting we are given a dialogue set d l with annotated summaries and a dialogue set d u without annotations. d l and d u belong to the same domain, as our focus is not domain generalization. given a pretrained dialogue summarization model g 0 , we leverage d l and some or all data of d u to fine-tune g 0 , obtaining a target dialogue summarization model ĝ. we aim to accurately evaluate the quality of pseudolabels for unlabeled samples, enabling us to select higher-quality unlabeled data for training an improved model ĝ. ## conclusion to make use of unlabeled data and measure generated summary quality for summarization model training, we benchmark ssds and uncertainty estimation on dialogue summarization. we propose the sicf score, which measures semantic invariance, coverage, and faithfulness of pseudolabels (generated summaries) at the text level, word level, and sentence level, respectively. furthermore, we extend bnn-based uncertainty estimation to a variantlength multi-label setting. our sicf score can enhance uncertainty estimation on dialogue summarization and improve ssds by up to +1-2% rouge and bertscore-f on samsum (dailychat domain), todsum (task-oriented dialogues) and dialogsum (real-life scenario).