Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization (2023)

Abstract: A series of datasets and models have been proposed for summaries generated for well-formatted documents such as news articles. Dialogue summaries, however, have been under explored. In this paper, we present the first dataset with fine-grained factual error annotations named DIASUMFACT. We define fine-grained factual error detection as a sentence-level multi-label classification problem, and weevaluate two state-of-the-art (SOTA) models on our dataset. Both models yield sub-optimal results, with a macro-averaged F1 score of around 0.25 over 6 error classes. We further propose an unsupervised model ENDERANKER via candidate ranking using pretrained encoder-decoder models. Our model performs on par with the SOTA models while requiring fewer resources. These observations confirm the challenges in detecting factual errors from dialogue summaries, which call for further studies, for which our dataset and results offer a solid foundation.

## introduction factual inconsistency in abstractive summarization -a phenomenon where model-generated summaries contain facts that are inconsistent with the source document -is a widely known problem and has been studied extensively in the document summarization community. an example is shown in figure 1 , where the source document is a dialogue -the type of documents that this paper focuses on. existing work covers topics on factual inconsistency including error typology and factuality annotations of state-of-the-art neural summarization models @xcite @xcite @xcite @xcite , automatic factual error detectors @xcite @xcite figure 1 : example summaries that are factually consistent and inconsistent with a source dialogue. @xcite @xcite , methods to correct factual errors in summaries @xcite @xcite and methods to produce factually more consistent summaries @xcite @xcite @xcite @xcite . almost all of these works focus on news summarization based on two datasets: cnn/dailymail @xcite and xsum @xcite . dialogue summarization (cf figure 1 ), which aims to produce a condensed version of a dialogue while maintaining its salient information, is equally important due to its application to summarizing meeting transcripts @xcite @xcite , daily conversations @xcite @xcite , customer service dialogues @xcite and medical dialogues @xcite . however, factual consistency in dialogue summarization is under explored as there are currently no benchmark datasets that contain fine-grained error categories. this paper aims to fill in this gap. to investigate factual consistency in dialogue summarization, we release diasumfact with fine-grained sentence-level annotations regarding factual consistency for 475 model summaries (1,340 sentences) from six neural dialogue summarization models on two popular datasets: sam-sum @xcite and qmsum @xcite . we adopt a two-dimensional typology that considers the semantic roles and verifiability of error spans separately. we formulate factual error detection as a sentence-level multi-label classification task and use diasumfact to evaluate two state-of-the-art factual error detection models designed for document summarization. as there are no existing error detection model for fine-grained error categories, we adapt the two binary classification models to fit to our task. empirical results show that they don't work well on the task, indicating its difficulty and the domain gap between document summarization and dialogue summarization. we then propose two models: bertmulti and enderanker. bertmulti is a multi-class classification model trained on synthetic data, which is created by corrupting sentences from reference summaries @xcite . ender-anker is a simple unsupervised model that can leverage any pretrained encoder-decoder model to detect factual errors. given a model-generated summary sentence containing a span of interest for error detection, enderanker computes log likelihood scores for the sentence and its variants containing replacement spans fetched from the source dialogue. the scores are computed as bartscore @xcite , which will be explained in 4.2. we compare the scores of the sentences to determine if the span of interest and hence the summary sentence contains a factual error. we run experiments with t5 @xcite , bart @xcite and pega-sus @xcite , fine-tuned either on news summarization or dialogue summarization, as the encoder-decoder for enderanker. the results show that bertmulti and enderanker performs on par with the adapted state-of-the-art models in terms of macro-averaged f1. motivated by the strong complementarity between models, we further present two ensemble models combining the four models above. the results, while exceeding those of the individual models, are still far from indicating a practical model for factual error detection over dialogue summaries. this calls for further studies, for which our dataset and results form a solid foundation. to summarise, this paper makes the following contributions: ## related work error typology and datasets. there are a few existing datasets on factual errors. some of them use binary (factually consistent or inconsistent) labels @xcite and 5-point likert scale labels @xcite , which require lower efforts to annotate, but they do not provide information on how and where factual errors were made. to support fine-grained analysis, multi-class and multidimensional typologies are designed. pagnoni et al. (2021) propose a linguistically motivated annotation framework that covers semantic frame errors, discourse errors and content verifiability errors. goyal and durrett (2021) use a 2-dimensional typology, where content verifiability and semantic error types are considered separately. cao et al. (2022) focus on hallucinations and consider both factual and non-factual hallucination. tang et al. (2022a) unify different error types from previous works into a hierarchical taxonomy. these datasets mostly focus on news summaries. dialsummeval @xcite is another popular dataset that contains annotation on factual consistency of model-generated dialogue summaries. the core difference of our work is that we consider fine-grained error categories and the text span (i.e., starting and ending position) of an error. thus it provides a more elaborate, diagnostic assessment as to what and where goes wrong when a summary is not factually consistent. in comparison, dialsummeval only considers coarsegrained assessment of factuality using 5-point likert scale @xcite , without specifying the actual error type (e.g., entity error). factual error detection models. most popular factual error detectors are based on either textualentailment or question-answering (qa). textual-entailment-based models are generally binary classifiers that take as input the source document and a model-generated summary. for example, @xcite train binary factual error classifiers using synthetic data. @xcite use a gradient-based adversarial method to improve model accuracy. @xcite leverage dependency-level entailment achieving better performance and interpretability. qa-based models first generate questions from a model-generated summary (or source dialogue), and then answer those questions based on its source dialogue (or a model-generated summary). the factual consistency is decided by the similarity between the ground truth answer and the predicted answer. for example, @xcite ; @xcite use a precision-oriented method that generates questions from model-generated summaries and answer them using the source document. @xcite instead generate questions from a source document and answer them using the summary, making it a recall-oriented method. @xcite combine recall and precisionoriented techniques into a single framework. @xcite refine the model component design and obtain a qa-based method that outperforms textual-entailment-based methods. our unsupervised method enderanker compares a span (e.g., a person name) in a modelgenerated sentence with candidates (e.g., other people's names in the dialogue) and decide the factual consistency of the span based on its rank among candidates. it achieves comparable macro f1 with adapted sota factual error detectors for document summarization but requires no labelled resources. ## the diasumfact dataset this section presents our diasumfact dataset and procedures to construct the dataset. ## detecting factual errors in this section, we automate factual error detection in model-generated summaries. we first adapt two state-of-the-art factual error detection models from document summarization. we then propose a weakly supervised multi-class classifier and a simple yet effective unsupervised model that can utilize any pretrained encoder-decoder model to identify factual errors. finally, we present ensemble-based models combining all techniques above. problem statement. we formulate factual error detection as a sentence-level multi-label classification task, i.e., given an input dialogue and a sentence from a model-generated summary, we classify whether the sentence contains any (seman-dependency arc types error class nsubj, obj, obl:agent, iobj, dobj, nmod, vocative, appos, nummod, compound, amod, det, clf, flat tic role) factual errors as outlined in section 3.3. ## conclusions we created a fine-grained multi-faceted dataset named diasumfact on factual consistency of dialogue summarization. diasumfact offers insights into how and where current neural summarization models fail when they produce factually inconsistent details in dialogue summaries. it can also serve as a testbed for automating factual error detection. our proposed error detection method, enderanker, is shown to perform on par with state-of-the-art models even though it requires no labelled training data. that said, we ultimately found that even ensembling several error detection methods do not produce results that are good enough for practical use, indicating opportunities for future research in this area.