Evaluating Factual Consistency of Texts with Semantic Role Labeling (2023)

Abstract: Automated evaluation of text generation systems has recently seen increasing attention, particularly checking whether generated text stays truthful to input sources. Existing methods frequently rely on an evaluation using task-specific language models, which in turn allows for little interpretability of generated scores. We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. Our approach generates fact tuples constructed from Semantic Role Labels, applied to both input and summary texts.A final factuality score is computed by an adjustable scoring mechanism, which allows for easy adaption of the method across domains. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods and exhibits stable generalization across datasets without requiring further training or hyperparameter tuning. We experiment with an optional co-reference resolution step, but find that the performance boost is mostly outweighed by the additional compute required. Our metric is available online at: https://github.com/heyjing/SRLScore

## introduction one of the remaining issues that prevents productive deployments of neural text summarization systems is the low correlation of system outputs with human preferences. among those, factuality, i.e., the agreement of facts in the generated summaries with those present in the input text, is not part of the general training objectives of models, which frequently leads to hallucinated facts that are detrimental to perceived system performance @xcite . prior work has therefore introduced metrics for automated testing of factuality in generated text @xcite @xcite , * both authors contributed equally to this work. while our experiments are performed on english, we argue that the transfer of our approach to other languages is possible given only the existence of a language-specific tokenizer and a sufficiently good srl tagger. furthermore, srlscore offers the additional benefit of being an interpretable metric, due to its composition on top of fact tuples. in comparison, metrics used for factuality evaluation that are based on the intermediate presentations of language models, e.g., generation perplexity @xcite @xcite , cannot present insightful reasons why a particular score was achieved. furthermore, it has been empirically demonstrated that generationbased evaluators exhibit a self-preference of outputs generated by models similar to the factuality evaluator @xcite . this makes them a questionable choice over interpretable metrics. we empirically show that the correlation of srlscore with human ratings is on par with existing methods, and perform several ablations to study the impact of algorithmic choices within our pipeline. ## related work automated analysis of (abstractive) summaries became more relevant in recent years, with the influx of generic summarization systems becoming available @xcite @xcite . in particular, @xcite were the first to propose a reference-based estimator for factuality of generated summaries. as mentioned, their approach is based on a tuple representation of "facts" in the generated and gold summary. fact tuples are extracted based on a weakly supervised end-to-end tagger and subsequently compared on the basis of matching arguments. notably, no readily available implementation of their method currently exists. later work has proposed alternative metrics based on textual entailment @xcite and question answering (qa) @xcite , where agreement of answers to questions on the reference and summary are used for estimating factuality. however, qa-based metrics require additional task-specific fine-tuning on generic datasets, which makes the adoption to new domains fairly expensive. the only other work that to our knowledge utilizes some form of srl-based factuality estimation is presented by @xcite . in comparison to srlscore, their method aggregates "role buckets" at the document level, instead of creating sentence-specific fact tuples. empirically, their implementation has lower correlation with human ratings than compared approaches, which is contrary to our own findings. @xcite frame factuality estimation as an in-filling task, where fact statements are withheld as masked tokens in a generated summary, and a separate model is trained to predict missing facts. notably, this relies on the assumption that the majority of factual mistakes stems from noun phrases and entity mentions @xcite ). an alternative body of literature has explored the possibility to exploit language models (lms) directly for estimating factual consistency: some works, such as bertscore @xcite , use lm-generated representations to generate alignments for scoring. in comparison, prism (thompson and post, 2020) or bartscore @xcite directly use model perplexity as a factuality estimate. @xcite explore masking approaches, which fall somewhere between the works of @xcite and bartscore; their framing of counterfactual estimation still relies on model-based likelihood scores for computation. the majority of prior work expresses metric performance in terms of correlation with human factuality ratings. notably, annotations exist for subsets of the popular cnn/dailymail @xcite and xsum summarization corpora @xcite . where wang et al. (2020) collect user annotations from crowd workers, fabbri et al. (2021) additionally sample expert judgments, and find that expert ratings tend to be more representative. maynez et al. (2020) study several aspects of summarization evaluation beyond just factuality, but do not disclose the background of annotators for evaluation. generally, reliably evaluating correlation of summarization metrics with human preferences is no easy task, either: @xcite show that system-level evaluation metrics for text summarization rarely outperform simplistic metrics, such as rouge @xcite , to a statistically significant degree. partially, this can be attributed to the small number of human-annotated samples available, generally less than 1000 different instances. ## srlscore our factual consistency metric, called srlscore, is implemented as a two-stage process: first, extracting fact tuples using semantic role labeling (srl) on both the source texts and the summary texts, and then determining a factuality score based on tuple comparison. the measure outputs humaninterpretable scores between 0 and 1, where a higher score indicates greater factual consistency of a summary text. in this section, we detail the algorithmic choices and present an adaptive weighting scheme for computing the final factuality scores. ## experiments we empirically demonstrate the performance of our method through a number of experiments on two popular datasets for factual consistency evaluation, which are covered in this section. we further share implementation details and the choices for extracting srl tuples and extracting co-reference clusters. in addition to the experimental analysis, we also study the behavior of srlscore through a number of ablation experiments and a brief error analysis. ## conclusion and future directions in this work, we presented a semantically consistent metric for estimating the factual truthfulness of two pieces of text: we applied our presented metric to the problem of text summarization evaluation, and demonstrated that it performs on par with existing approaches. in fact, we find that due to the small sample sizes of evaluation datasets, there are no significant differences between any of the considered state-of-the-art factuality estimation metrics. our approach strikes with its relative simplicity and interpretability due to the intermediate representation of "fact tuples", which makes it possible for human annotators to review how or why system decisions were made. furthermore, we have demonstrated the suitability of our approach over more naive tuple-based scoring methods through a series of ablation experiments, which also show the adaptability of our method to particular unseen settings by simply adjusting a series of parameters. in our opinion, there are two key challenges concerning the effective deployment of srlscore. the current implementation still suffers from impractically long runtimes for longer input texts. notably, however, both the tuple generation and comparison stages can be parallelized and we are currently working on improving the compute effi-