Analyzing and Evaluating Faithfulness in Dialogue Summarization (2022)

Abstract: Dialogue summarization is abstractive in nature, making it suffer from factual errors. The factual correctness of summaries has the highest priority before practical applications. Many efforts have been made to improve faithfulness in text summarization. However, there is a lack of systematic study on dialogue summarization systems. In this work, we first perform the fine-grained human analysis on the faithfulness of dialogue summaries and observe that over 35% of generated summaries are faithfully inconsistent respective the source dialogues. Furthermore, we present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations. Experimental results show that our evaluation schema is a strong proxy for the factual correctness of summarization models. The human-annotated faithfulness samples and the evaluation toolkit are released to facilitate future research toward faithful dialogue summarization.

## introduction text summarization aims to condense a document into a short paragraph or a single sentence while conveying the core information @xcite . it can be either extractive or abstractive. extractive summarization methods identify salient sequence spans from the source document and pasting them together @xcite @xcite . abstractive summarization methods generate completely new summary in a coherent manner @xcite @xcite @xcite . previous work discovered that abstractive summarization suffers from unfaithful outputs, limiting its applicability in realworld scenarios @xcite @xcite . dialogue: freddie: nanna, are you coming to visit us soon? winnie: oh darling, nanna has broken her leg, you'll have to visit me instead. freddie: i forgott. well come soon. winnie: good, ask mummy and daddy and they will come when they can. freddie: yes love you. leg better soon? winnie: yes, quite soon. tell mummy to ring me. bye darling xxxxx summaries: human: winnie has broken her leg and will not visit any time soon. freddie will ask mummy to call winnie up. ✓ bart: nanna has broken her leg, so freddie will have to visit her instead. nanna will get better soon. ✓ mv-bart: nanna has broken her leg and freddie will have to visit winnie instead. mummy and daddy will come to visit them soon. ✗ coref-bart: freddie wants to visit winnie, but nanna has broken her leg, so he will have to visit her instead. mummy and daddy will come when they can. ✗ condigsum-bart: winnie's nanna has broken her leg and freddie will have to visit her instead. ✗ as an essential way of exchanging information, conversations usually involve multiple participants, informal language usage, repetition, and negations @xcite . therefore, dialogue summarization is vulnerable to factual issues due to its abstractive nature. in this work, we first categorize the most frequently occurred factual errors for dialogue sum-marization into 6 types. then, we collect finegrained factual annotations for human reference and the output of 4 recent dialogue summarization systems ( §3). at least two annotators are involved, and a verification process is incorporated to ensure the annotation quality. as a result, our study on human-annotated data suggests that over 35% of the generated dialogue summaries contain at least one factual error. similar observations have been made in the news summarization domain where 30%-80% of generated text are factually inconsistent @xcite . more research attention should be made toward faithful dialogue summarization. the unavailability of faithful evaluation methods hinders the development of effective dialogue summarization models. in this work, we present a model-level evaluation schema, faceval, targeting dialogue summarisation models' faithfulness ( §4). first, we synthesize a set of positive and negative summaries for each dialogue with back-translation or rule-based transformations. then, a summarization model is asked to distinguish positive and negative summaries based on conditional generation probabilities. more correct judgements indicate the model is more factually competent. to compare the model-level performance of evaluation methods, we leverage two ad-hoc training schema to synthesize a series of models with different capability ranks. then, the evaluation methods are used to predict the ranking of trained models. seven non-factual and factual evaluation methods have been examined, followed by a detailed discussion of their properties. the effectiveness of faceval is also proven by showing a strong correlation with the factual correctness of summarization models. ## fine-grained faithfulness analysis previous studies of factuality analysis in summarization mainly focus on the news domain. the typology of factual errors for dialogues can be very different. therefore, we first define a taxonomy of frequently occurred factual errors for dialogue summaries. a fine-grained analysis is then performed by measuring the factual consistency within dialogue summary pairs. ## model-level faithfulness evaluation some efforts have been made toward sample-level factual error evaluation. an example is shown in figure 3 . the sample-level evaluation methods are model-agnostic and examine a model solely based on its output sequences. most existing evaluation methods, including rouge score, human evaluation and recent factual evaluation methods, belong to this type. one ultimate goal for factuality evaluation is to discriminate better summarization models. we propose directly probing models' generation probability with a constrained search space. first, faceval generates a set of positive and negative samples with variant factual errors by rule-based transformations. then, the generation probabilities of positive and negative summaries are compared for each dialogue. a better summarization model should be more likely to generate positive summaries than negative ones. ## conclusion we believe our faithfulness analysis and evaluation method can facilitate the development of dialogue summarization systems. instead of measuring faithfulness on generated summaries, we directly assess the model's capability by multi-choice questions. we expect faceval to be effectively extended to other generation scenarios.