SWING : Balancing Coverage and Faithfulness for Dialogue Summarization (2023)

Abstract: Missing information is a common issue of dialogue summarization where some information in the reference summaries is not covered in the generated summaries. To address this issue, we propose to utilize natural language inference (NLI) models to improve coverage while avoiding introducing factual inconsistencies. Specifically, we use NLI to compute fine-grained training signals to encourage the model to generate content in the reference summaries that have not been covered, as well as to distinguish between factually consistent and inconsistent generated sentences. Experiments on the DialogSum and SAMSum datasets confirm the effectiveness of the proposed approach in balancing coverage and faithfulness, validated with automatic metrics and human evaluations. Additionally, we compute the correlation between commonly used automatic metrics with human judgments in terms of three different dimensions regarding coverage and factual consistency to provide insight into the most suitable metric for evaluating dialogue summaries.

## introduction dialogue summarization is a text generation task that aims to produce a compact summary given a piece of conversation. conventional approaches to dialogue summarization rely on features of conversation data @xcite @xcite . recently, the rise of large pretrained language models (lms) has enabled coherent and fluent summaries to be generated without these features. however, low coverage and factual inconsistency remain two pressing issues as studies have shown that the summaries generated from these pre-trained lms often do not fully cover the reference @xcite 1. charlee is attending portuguese theater as a subject at university. ## method upon analyzing the dialogue summaries in sam-sum, we observe that dialogues are often summarized linearly, consistent with the findings of @xcite . therefore, we segment the summaries into sentences and use a natural language inference (nli) model to provide finer-grained training signals at the sentence level for two goals: (1) encourage generating sentences in the reference summaries that have not been covered by the generated sentences and (2) differentiate factually consistent generated sentences from inconsistent ones. to achieve these goals, we first determine the faithfulness of each sentence using an entailment-induced bipartite graph ( §2.1). then, we propose two new losses addressing each challenge in turn: an uncovered loss that encourages the model to recover missing information ( §2.2) and a contrastive loss that brings closer the representations of the reference summary and the generated sentences that algorithm 1: entailment-induced bipartite graph input: a reference summary s * = {s * 1 , ..., s * n }, a generated summary @xmath0, ..., s m }; output: the bipartite mapping ϕ between sentences in s * and s; 1 initialize ϕ as a zero matrix of size n × m where @xmath1; 2 let τ be the entailment threshold; 3 // resolve 1-to-many mappings; 4 for i ← 1 to n do 5 v ← ∅ ; 6 for j ← 1 to m do 7 if p ent (s * i , s j ) > τ and ϕ(i, j) = 0 then resolve 1-to-1 mappings; 24 for i ← 1 to n do 25 for j ← 1 to m do 26 if ϕ(i, j) = 0 and p ent (s j , s * i ) > τ and p ent (s * i , s j ) > τ then 27 ϕ(i, j) ← 1; 28 return ϕ; ## conclusion we have proposed swing, a dialogue summarization framework that generates summaries with mitigated missing information and improved faithfulness. to instruct the model to generate missing content from the reference summaries and to differentiate factually consistent generated sentences from their inconsistent counterparts, we propose two losses based on nli. experimental results on the dialogsum and samsum datasets showed that our approach achieves significantly higher faithfulness and coverage, while still maintaining conciseness, compared to prior methods. in addition, we measure the correlation between the reported automatic metrics and human judgments to provide insight into the most suitable metric for evaluating the coverage and factuality of dialogue summaries for future research.