Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks (2024)

Abstract: We study existing approaches to leverage off-the-shelf Natural Language Inference (NLI) models for the evaluation of summary faithfulness and argue that these are sub-optimal due to the granularity level considered for premises and hypotheses. That is, the smaller content unit considered as hypothesis is a sentence and premises are made up of a fixed number of document sentences. We propose a novel approach, namely INFUSE, that uses a variable premise size and simplifies summary sentences into shorter hypotheses. Departing from previous studies which focus on single short document summarisation, we analyse NLI based faithfulness evaluation for diverse summarisation tasks. We introduce DiverSumm, a new benchmark comprising long form summarisation (long documents and summaries) and diverse summarisation tasks (e.g., meeting and multi-document summarisation). In experiments, INFUSE obtains superior performance across the different summarisation tasks.

## introduction current state-of-the-art summarisation systems are able to generate fluent summaries; however, their inability to generate factually consistent summaries remains a significant constraint in their real-world applications. as a result, the assessment of summary faithfulness, i.e., the degree to which a summary accurately represents the content of the input document, has recently received much research attention. this evaluation is key to assess progress in abstractive summarisation @xcite . existing research focuses on developing models to detect unfaithful summary content @xcite @xcite part of the work done for his msc thesis at the university of edinburgh. one way increasingly adopted to assess summary faithfulness is to use off-the-shelf natural language inference (nli; maccartney and manning 2009) models to determine whether a summary is entailed by the source document. nli models determine the semantic relationship between a pair of texts: the premise and hypothesis. if the hypothesis can be inferred from the premise, it is said to be entailed by the premise. however, existing nli models are mainly trained on relatively short texts from existing datasets @xcite . examples in these datasets often represent inference cases over a single content unit (e.g., the example at the bottom of figure 1 where inference is about the transmission event). this raises the question of how to apply them to produce entailment judgements for document-summary pairs consisting of multiple sentences aggregating several content units (e.g., the summary sentence mss in figure 1 aggregates content about the company launching a legal action, a strike event, and the consequences of the strike). producing an entailment judgement for a summary sentence with several content units is a more complex entailment reasoning task. taking summary sentences as hypotheses, existing approaches try to either identify a document sentence that acts as the premise leading to the highest possible entailment score (sentence-level nli, @xcite ) or directly measure entailment by taking the entire document as premise (document-level nli, @xcite @xcite ). however, due to content aggregation happening in summarisation, one document sentence will not contain enough content to entail a summary sentence. in figure 1 , none of the docu- lufthansa lost an appeal to a frankfurt labour court, but is making a further legal challenge that could go late into tuesday evening. 0.37 7.81 0.06 2 the pilots' strike, called over a pay dispute, will affect around 100,000 passengers, lufthansa said. 0.61 0.69 1.74 3 the industrial action is part of a long-running pay dispute at lufthansa. 0.18 0.74 0.06 4 the pilots' union vereinigung cockpit (vc) has organised 14 strikes since april 2014. 0.07 0.11 0.10 5 short and medium-haul flights from germany will be affected from 00:01 to 23:59 local time (23:01-22:59 gmt). 0.09 0.14 0.06 6 flights by lufthansa's other airlines including eurowings, swiss, austrian airlines, air dolomiti and brussels airlines are not affected by the strike, the airline said. 0.11 0.22 0.12 7 pay talks between the vereinigung union and the german airline broke down earlier this month, and lufthansa said the union had "consistently rejected the offer" of mediation. 0.20 0.41 0.06 lufthansa, which is facing increasing competition from budget rivals, offered a 2.5% increase over the six years until 2019. 0.12 0.22 0.11 10 meanwhile, a separate dispute with cabin crew at lufthansa's low-cost subsidiary, eurowings, led it to cancel more than 60 flights on tuesday. 0.27 0.47 0.32 mss german airline lufthansa has launched a fresh legal challenge against a strike by its pilots, which could lead to the cancellation of more than 1,000 flights. ss german airline lufthansa has launched a fresh legal challenge against a strike by its pilots. the strike could lead to the cancellation of more than 1,000 flights. at 8:34, the boston center controller received a third transmission from american 11 |= the boston center controller got a third transmission from american 11. figure 1 : example of input document (d) and model-generated summary sentence (mss) from the aggrefact @xcite benchmark on the xsum @xcite dataset. the example is considered unfaithful by the annotators. simplified summary (ss) is the generated summary after automatic sentence splitting. the cyan coulored text spans in the input document highlight those document content units that support the corresponding cyan spans in the summary. red spans in the summary indicate content that is not supported by the input document. the |= mss and |= ss i columns show entailment scores assigned by an off-the-shelf nli model to document sentences acting as premises and either mss or ss i sentences as hypotheses. the table in the bottom shows an example of entailment relation from the mnli dataset @xcite . entailment scores are computed by the nli model introduced in section 4 and normalised for better reading. in this work we argue that existing nli-based approaches do not operate at the right level of granularity @xcite ; even context-level nli approaches. summary sentences may convey several content units @xcite partly overlapping with different document sentences. this renders the retrieval step of document sentences based on nli scores less accurate (e.g., each document sentence in figure 1 weakly entails the complex summary sentence mss). in addition, summary sentences may aggregate content from different numbers of document sentences which makes it less accurate to have an entailing context with a fixed k number of document sentences (e.g., in figure 1 , ss 1 is entailed by two document sentences while ss 2 requires only one document sentence to show that its content is not derived from the document). 2 finally, a fine-grained assessment of summary faithfulness brings interpretability, which hugely facilitates manual inspection of model-generated summaries. we propose infuse, a faithfulness evaluation approach that incrementally reasons over a document so as to arrive at a faithfulness estimation of its summary. it aims at retrieving the best possible context to assess the faithfulness of each summary sentence (and in turn the entire summary), i.e., a context with the minimal and most relevant set of document sentences. our incremental reasoning approach approximates this via successive expansions of the context adding document sentences and evaluating whether the hypothesis is entailed by it. our approach further decomposes summary sentences for their faithfulness analysis. it does this via sentence simplification. that is, it splits long summary sentences (e.g., mss sentence in figure 1 ) into a set of shorter ones conveying the same content units (e.g., ss 1 and ss 1 in figure 1 ). most of previous work focuses on the metaevaluation of nli-based approaches on single document news summarisation @xcite . thus, the question of how nli-based evaluation works on diverse summarisation tasks is left unanswered. hence, to widen the spectrum of nli-based meta-evaluation @xcite , we analyse the performance of nli-based faithfulness evaluation approaches on long document summarisation with diverse domains and genres @xcite @xcite and multidocument summarisation @xcite . we collect human annotated model-generated summaries from previous work on these tasks @xcite @xcite . we call this new set the diversumm benchmark. we study existing nli-based approaches on ag-grefact @xcite , a benchmark for the meta-evaluation of single document summarisation, and diversumm. infuse achieves the best performance in these benchmarks. we find that the choice of an adequate level of granularity for the premise and hypothesis leads to more accurate entailment judgements when using off-the-shelf nli models. on summaries of extractive nature, retrieving a small relevant set of document sentences suffices. moreover, our results show that this is crucial for summarisation tasks with long input documents. summary sentence splitting helps to obtain better performance in all summarisation tasks. ## faithfulness annotated data for different summarisation tasks following previous work, we study faithfulness evaluation on two single document summarisation tasks, namely cnndm @xcite and xsum @xcite . for this, we take the latest introduced faithfulness benchmark, ag-grefact @xcite . it consists of a collection of document and model-generated summary pairs where summaries are annotated with faithfulness judgements by human judges. that is, each example in the benchmark is a triple (document, generated-summary, faithful/unfaithful label). aggrefact includes five annotated sets from the earlier summac @xcite bench-mark. these are xsumfaith @xcite , factcc @xcite , summeval @xcite , frank @xcite , and polytope @xcite . in addition, ag-grefact includes four sets, namely qags (wang et al., 2020) (referred as wang'20 in the benchmark), cliff @xcite , goyal'21 @xcite and cao'22 @xcite . aggrefact organises the annotated data into two major sets per summarisation task, cnndm and xsum, herein we name them cnndm ag and xsum ag . see appendix a for details on the faithfulness annotation scheme of each dataset and the standarisation criteria applied to derive aggrefact. diversumm a new benchmark to study the performance of nli-based faithfulness evaluation on diverse summarisation tasks, we propose a new benchmark, namely diversumm. it incorporates model generated summaries with human annotations about faithfulness from previous work (koh @xcite @xcite . we follow @xcite to standardise summary annotations into faithful/unfaithful labels. we discuss the summarisation task and characteristics of the annotated sets below. chemsumm @xcite embodies the task of scientific long-form summarisation in the chemistry domain. derived from academic journals, each input document contains section headers and associated paragraphs for all sections from the introduction up to the conclusion, and abstracts constitute the reference summaries. multinews @xcite is a largescale multi-document news summarisation dataset with the number of input documents per example ranging from 2 to 6 and reference summaries written by editors. qmsum @xcite is a querybased multi-domain meeting summarisation dataset. it consists of meeting transcripts and queries associated with their corresponding abstractive summaries. arxiv @xcite is a long scientific paper summarisation dataset collected from arxiv covering a wide range of topics. the main content up to the conclusion section of a paper is regarded as the document and the corresponding abstract section as the summary. diversumm contains 563 test instances with a total of 4686 summary sentences of which 3138 have sentence level annotations. error types some subsets in aggrefact and di-versumm, namely frank, arxiv, and govreport, contain sentence level and detailed error annotations for unfaithful summaries. 3 we exploit these annotations to analyse the performance of both the studied approaches and the nli model on detecting different types of faithfulness errors. concretely, unfaithful summaries are annotated with the following error types @xcite . relation error (pree) is when the predicate in a summary 3 after manual inspection of the human annotations, we filtered out some examples in arxiv and govreport with a mismatch between the sentence and summary level annotation. sentence is inconsistent with respect to the document. entity error (ente) is when the primary arguments of the predicate are incorrect. circumstance error (circe) is when the predicate's circumstantial information (i.e., name or time) is wrong. co-reference error (corefe) is when there is a pronoun or reference with an incorrect or non-existing antecedent. discourse link error (linke) is when multiple sentences are incorrectly linked. out of article error (oute) is when the piece of summary contains information not present in the document. grammatical error(grame) indicates the existence of unreadable sentences due to grammatical errors. ## infuse we denote a document as @xmath0. for a given summary sentence s n as the hypothesis, we aim to retrieve a related context r (n) , r (n) ⊆ d, to act as the premise and estimate whether s n can be entailed by r (n) (and, therefore, d) according to an nli model θ. we assume that θ predicts one of the {entailment, neutral, contradict} labels for a given premise-hypothesis pair. summary sentence faithfulness estimates, given by θ(entailment|•), are then aggregated into summary faithfulness scores with mean pooling. incremental reasoning given an nli model θ, we construct a matrix e of entailment scores via sentence level inference between document sentences d m and each summary sentence s n . we derive from e entailment ranked lists of docu-ment sentences d(n) associated to each summary sentence s n . we then incrementally select sentences from d(n) in a top-down fashion to retrieve a context r (n) for s n . starting from an empty context r (n) 0 , at each step i, we remove the top sentence from d(n) and incorporate it into the current context to obtain a new context r (n) i . we then stop adding sentences to the context when the local minimum of the neutral class probability, u i,@xmath1 êdm,@xmath2,n + e n,m 6: end for sub-sentence reasoning different document sentences d m will entail different parts of a summary sentence s n (see document sentence fusion in figure 2 ). in addition, those document sentences d m may contain irrelevant content for s n . thus, sentence level scores in e as well as final context level entailment scores for s n will be noisy (i.e., more chances of having neutral class high scores). shorter summary sentences with finer-grained content units will yield more accurate contexts and entailment estimations. figure 1 illustrates the difference in entailment scores in e when computed on the original summary sentence (mss) and when computed on its sub-sentences (ss 1 and ss 2 ). in this work, we propose to simplify each summary sentence by splitting it into multiple sub-sentences. ## experimental setup we study nli-based faithfulness evaluation approaches on aggrefact @xcite and diversumm (section 2). we adopt an albertxlarge @xcite ) model optimized on mnli @xcite and vitaminc @xcite as our nli model θ. mnli covers ten distinct genres and styles of written and spoken english data. it aims to support a broader understanding and analysis of nli across different genres and domains. vitaminc is synthetically created from wikipedia sentences. claim sentences are associated with contrastive evidence, i.e., one sentence that supports the claim and another one that does not. on mnli (vitaminc) premises are 13.23 (43.03) tokens long in average and hypotheses 13.23 (27.57). we fine-tune a t5-large @xcite ) model for sentence splitting. we use this model to simplify sentences in model generated summaries. we manually inspect several samples of split sentences and find that the performance is reasonable. details about our sentence splitting model, exam-ples, and statistics about the percentage of sentence splits are presented in appendix b. ## related work some nli-based approaches directly train document level nli models @xcite . others leverage off-the-shelf nli models @xcite @xcite @xcite . the former requires the construction of synthetic training data. in this paper, we study the latter type of approaches. these do not require additional data nor training resources. @xcite ) select a single sentence as premise while @xcite @xcite select a fixed number of document sentences, the same for all summary sentences. our approach selects a variable number of document sentences as premise for each summary sentence. recently, @xcite conduct an empirical analysis of how to use the three directions in which entailment can be computed (entailment direction implication, reverse implication, and bi-implication). however, @xcite directly use the scores from these directions in a single pass using the entire document as premise. in contrast, we apply reversed reasoning only to re-weight document sentences in the context retrieval step. @xcite propose a document-level approach with data augmentation to adapt nli models to task specific scenarios such as dialogue. furthermore, they ensemble a number of calls to the nli model via monte-carlo dropout to cope with domain shift. these ideas are orthogonal to our work and would make sense to use them in combination. the value of fine-grained assessment of summary content has been highlighted in earlier work on summarisation evaluation @xcite @xcite @xcite 2007; @xcite . this research highlights that summary sentences aggregate several content units and judgements should be initially provided for these before yielding a conclusion at summary level. however its focus is on the evaluation of content relevance. recent work in the context of summary faithfulness evaluation assesses faithfulness of summary predicates and arguments @xcite . conciliating with our results, they also show that fine-grained evaluation is beneficial. however, their approach is not based on nli; and requires syntactic analysis of summary sentences and task specific human annotated data to train a classifier. our approach is more generic and builds on existing resources. contemporary with our work, @xcite propose the evaluation of large language models (llms) generated biographies via their decomposition into smaller content units (i.e., atomic facts). their approach is applied to factual descriptive generation. in contrast, we evaluate hallucination detection in a variety of summarisation tasks. for long dialogue summarisation, @xcite propose to decompose the input into chunks, infuse could be combined with a coarse chunk selection step. finer-grained evaluation has also shown positive results in the related task of claim verification @xcite . however, in the same way as current factuality evaluation on llm generated text @xcite , they address more open-ended generation tasks where no ground truth input is assumed; their information source is either retrieved or parametric. we focus on nli-based faithfulness evaluation from given input documents. ## conclusions we study existing nli-based faithfulness evaluation approaches and propose a new one, infuse, that works at finer-grained granularity levels for computing document-summary entailment judgements. our study shows that lower granularity via premises with variable size and summary sentence splitting is key to achieve more accurate entailment judgements when using off-the-shelf nli models. we also introduce a new benchmark for long form input and diverse summarisation tasks. experimental results show that infuse achieves superior performance on evaluating faithfulness for diverse summarisation tasks.