Revisiting text decomposition methods for NLI -based factuality scoring of summaries (2022)

Abstract: Scoring the factuality of a generated summary involves measuring the degree to which a target text contains factual information using the input document as support. Given the similarities in the problem formulation, previous work has shown that Natural Language Inference models can be effectively repurposed to perform this task. As these models are trained to score entailment at a sentence level, several recent studies have shown that decomposing either the input document or the summary into sentences helps with factuality scoring. But is fine-grained decomposition always a winning strategy? In this paper we systematically compare different granularities of decomposition - from document to sub-sentence level, and we show that the answer is no. Our results show that incorporating additional context can yield improvement, but that this does not necessarily apply to all datasets. We also show that small changes to previously proposed entailment-based scoring methods can result in better performance, highlighting the need for caution in model and methodology selection for downstream tasks.

## introduction with improvements largely driven by recent advances in pre-trained language models @xcite @xcite , modern abstractive summarization models are capable of producing summaries that are both fluent and coherent. however, they are still prone to various forms of "hallucination", generating statements that are not supported by the input text @xcite . this has lead to a growing interest in being able to accurately measure the degree to which machine-generated output is non-factual @xcite @xcite . in factuality scoring and other closely related tasks such as fact verification @xcite , the objective is to assess whether or to what degree the claims in a given text can be supported by other "evidence" texts. given this setup, previous work has drawn a parallel with the task of natural language inference (nli), which has a similar goal of determining whether the meaning of one text can be inferred (entailed) from another @xcite . as a consequence, models trained on large nli datasets @xcite @xcite have often been successfully repurposed for the task of detecting factual inconsistencies in machine-generated summaries @xcite @xcite . it is now common that highperformance nli models are trained on a combination of nli and fact verification datasets @xcite . one way to repurpose nli models for factuality scoring is to use the full text of the input and summary as the premise and hypothesis respectively, then take the factuality score to be a function of the model output distribution. however, nli models are usually trained with sentence pairs as input, and can suffer performance degradation with the longer contexts that arise in summarization @xcite . worse yet, the majority of modern nli models are based on architectures such as the transformer @xcite that use fixed-length input sizes, and it may not be possible for a full document and summary pair to fit into this context. another approach to nli-based factuality scoring is grounded in the idea of first decomposing the input text into finer levels of granularity, followed by a later score aggregation step. @xcite proposed a scoring method based on sentence level decomposition, but concluded that the nli models at the time were not robust enough for the task. however, recently both @xcite and @xcite have shown that variations on this decomposition-based strategy, in combination with the improved performance of modern nli models, can produce systems that perform well at the task of detecting factual inconsistencies in generated summaries. in this work we revisit existing studies of nlibased factuality scoring and perform a systematic comparison of input-summary decomposition methodologies at different levels of granularityfrom document to sub-sentence level. we show that contrary to previous findings, adding more context to the premise (the source document) can sometimes outperform approaches based on a more fine-grained decomposition. we also find that small changes to the factuality scoring function can lead to a substantial increase in performance, but that model performance does not necessarily generalize across benchmarks that use different metrics (even when applied to the same underlying data). our results highlight the need for caution and additional evaluation when selecting a model and methodology for downstream tasks. ## decomposition-based factuality scoring in this work we are primarily concerned with referenceless factuality scoring of document summaries. to do so, we therefore require a function from an input (document, summary) pair to a score value @xmath0. nli models typically learn a function that maps a pair of input text strings (x prem , x hyp ), commonly referred to as the premise and hypothesis, to a probability distribution over the output classes entailment, neutral, or contradiction. one simple way to repurpose nli models for factuality scoring is with (document, summary) as (x prem , x hyp ), and to take the score z to be some function f z (p e , p n , p c ) over the probability values given for entailment (p e ), neutral (p n ), or contradiction (p c ) foot_0 . we experiment with three decomposition-based scoring methods, described in the following sections. ## experiments and evaluation we evaluate the performance of our models on the summac benchmark @xcite , which comprises of six datasets for summary inconsistency detection: cogensumm (cgs) @xcite , xsumfaith (xsf) @xcite , polytope (pt) @xcite , factcc (fcc) @xcite , sum-meval (se) @xcite , and frank (fr) @xcite . evaluation is standardized by casting each task as binary classification, and then measuring performance using balanced accuracy. as the nli-based factuality scoring methods all output a scalar score value, we follow @xcite and tune thresholds separately for all methods and all datasets on the validation set, and report results using these threshold values on the test set. although the frank dataset is part of summac, we also perform a separate evaluation of it using the original metrics of pearson and spearman correlations of the model output scores with (non-binary) human scores. to assess the benefits of decomposing text for nli-based factuality scoring, we compare the per4 example scus are given in appendix d formance of the aforementioned decomposition methods with full text scoring, where either or both the source document or the summary has not been decomposed. we also test with a context length of several sentences, computed using a simplified version of the sentli rr method that we refer to as topk, as follows: to split text into sentences we use spacy @xcite . we note that @xcite used nltk @xcite for sentencesplitting, but this fails to correctly split sentences on some examples with bad punctuation (which are common in the frank dataset in particular foot_3 ). in all experiments, unless otherwise specified we use the nli model from @xcite that is fine-tuned on a combination of vitamin-c and mnli datasets foot_4 , which we refer to as vitc. for fair comparison with @xcite , we set the maximum "full document" context for the premise to be 500 tokens. ## conclusion in this work we revisited prior findings that the best way to use nli models for factuality scoring of machine-generated summaries is to first decompose the input to sentence level, score using nli, then aggregate the sentence level scores to produce a document-level score. contrary to prior work, we find that there is no single optimal level of decomposition that performs best across all tasks and evaluation metrics. we showed that in general, sentence level decomposition is preferable for the summary/hypothesis side of the nli input, but on the premise side recent models such as vitc often benefit from having longer input contexts available when scoring. we also show that for the six datasets in the summac benchmark, there is still considerable variation in the performance of our methods both across the individual datasets, and also within different metrics on the same dataset.