Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model (2023)

Abstract: Despite tremendous improvements in natural language generation, summarization models still suffer from the unfaithfulness issue. Previous work evaluates faithfulness either using models trained on the other tasks or in-domain synthetic data, or prompting a large model such as ChatGPT. This paper proposes to do zero-shot faithfulness evaluation simply with a moderately-sized foundation language model. We introduce a new metric FFLM, which is a combination of probability changes based on the intuition that prefixing a piece of text that is consistent with the output will increase the probability of predicting the output. Experiments show that FFLM performs competitively with or even outperforms ChatGPT on both inconsistency detection and faithfulness rating with 24x fewer parameters. FFLM also achieves improvements over other strong baselines.

## introduction faithfulness evaluation for text summarization aims at measuring if the information in a summary is fully covered by and consistent with the source document foot_0 . although automatic text summarization has achieved remarkable improvements with pre-trained language models @xcite @xcite @xcite in recent years, especially in the aspect of fluency and informativeness. however, these neural models tend to generate unfaithful summaries. an effective faithfulness evaluation metric not only helps for implementing summarization systems in real applications but also plays a key role in developing more faithful summarization models, such as by data filtering @xcite or doing post-hoc corrections @xcite . most previous work for faithfulness evaluation either takes advantage of models trained on related tasks for zero-shot evaluation @xcite @xcite , or does weakly-supervised evaluation with synthetic in-domain data @xcite . the former requires transferring out-of-box models to the summarization domain @xcite , which lacks guarantees on the models' performance and suffers from error propagation @xcite . the latter one shows poor generalization ability @xcite as a result of the limited synthetic rules that couldn't cover various kinds of hallucinations. recently, as @xcite has shown amazing generation abilities on various tasks, researchers attempt to do human-like evaluation by designing prompts to query the model in the zero-shot manner @xcite . however, such strong language models are still sensitive to nuances, showing unstable performance with different wording of prompts @xcite . considering the above weaknesses, we think that an ideal faithfulness evaluation metric for summarization should be independent of other tasks and dataset-specific expertise, be able to generalize among different benchmarks and robust for the same document-summary pair. @xcite concludes that instruction tuning is just to teach the model to produce high-quality output while almost all of the knowledge has been learned during pre-training for large language models. based on their findings, we wonder: can we get rid of the popular prompting approaches and calculate the faithfulness score simply with a foundation language model, which meets the above expectations? in this work, we propose a metric named fflm for zero-shot faithfulness evaluation with a foundation language model. the intuition behind fflm is that the generation probability of a piece of text will increase when prefixing another piece of consistent text. following this intuition, we classify different kinds of probability changes into changes with prior probability and changes with conditional probability. the former contains a comparison between the vanilla sequence-to-sequence probabilities of the summary given document and unconditional probabilities of the summary, and a similar comparison by changing the position of the document and the summary. the latter calculates the vanilla sequence-to-sequence probability with another conditional probability by adding a piece of prefix text. similar intuition has been considered in previous works @xcite . the major differences are that their metrics were carried out on models fine-tuned by summarization data and they only consider a single kind of probability changes. our fflm is based on the foundation language model, and we hypothesize that these different probability changes capture different hallucinations (see sec. 4.4) which should be considered as a whole. on top of these three components of probability changes, we introduce a feasible design of fflm by re-weighting each token and each component to get the final faithfulness score. we did experiments in both the inconsistency detection setting and the faithfulness rating setting for summarization evaluation. the results show the favorable performance of our fflm across different settings and datasets 2 . our contributions are as follows: ## approach given a source document @xmath0, ..., x n } and the corresponding summary @xmath1, ..., y m }, the goal of this work is to design a metric fflm measuring the faithfulness of y based on the foundation model lm (•). we adopt lm (•) under the teacher-forcing strategy, which can provide a sequence of generation probabilities p of a given text with or without other conditional inputs. we first introduce three probability changes for faithfulness measurements and then propose a feasible design of our comprehensive metric fflm. scores proposed by @xcite and @xcite are in appendix a. ## experiment setup we present two evaluation settings considered by previous work for faithfulness evaluation first, with the implementation details of fflm for them later. ## results and analysis this section includes the main results for inconsistency detection and faithfulness rating, together with an ablation study, an analysis of error types, and comparisons of different model sizes of our fflm. we also discussed our metric and the prompting approach with or without instruction tuning under the same model size. ## conclusion this paper focuses on zero-shot faithfulness evaluation for summarization and introduces a novel evaluation metric fflm which is simply based on the foundation language model. experiments on both the inconsistency detection benchmark and faithfulness rating datasets show the strong generalization ability of fflm across various task settings and different datasets. it also shows favorable performance over strong baselines including chatgpt. using our proposed metric for more fine-grained consistency detection and designing more faithful summarization systems are future directions.