Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2022)

Abstract: Current pre-trained models applied for summarization are prone to factual inconsistencies that misrepresent the source text. Evaluating the factual consistency of summaries is thus necessary to develop better models. However, the human evaluation setup for evaluating factual consistency has not been standardized. To determine the factors that affect the reliability of the human evaluation, we crowdsource evaluations for factual consistency across state-of-the-art models on two news summarization datasets using the rating-based Likert Scale and ranking-based Best-Worst Scaling. Our analysis reveals that the ranking-based Best-Worst Scaling offers a more reliable measure of summary quality across datasets and that the reliability of Likert ratings highly depends on the target dataset and the evaluation design. To improve crowdsourcing reliability, we extend the scale of the Likert rating and present a scoring algorithm for Best-Worst Scaling that we call value learning. Our crowdsourcing guidelines will be publicly available to facilitate future work on factual consistency in summarization.

## introduction pre-trained language models have achieved promising results in abstractive text summarization @xcite @xcite @xcite . a serious limitation of these models, however, is their tendency to produce text that is factually inconsistent with the input. thus, evaluating the factual consistency of the generated summaries with respect to the source is an important task @xcite @xcite @xcite . recently, metrics have been proposed for evaluating factual consistency, including applying natural language inference @xcite and question-answering models @xcite @xcite . however, current metrics still do not correlate highly with human judgments on factual consistency @xcite . to overcome the inherent limitation of automatic metrics, researchers typically crowdsource human evaluations using platforms such as amazon's mechanical turk (mturk) @xcite @xcite . however, papers often differ in their preferred evaluation protocols @xcite . these differences in the evaluation task design affect the quality of the resulting human judgments and system comparisons @xcite . two of the primary paradigms of crowdsourced evaluations are ranking-based and rating-based. best-worst scaling (louviere and woodworth, 1991) is a ranking-based method by which the annotator selects the best and worst example out of a set of examples. prior research has claimed that best-worst scaling produces higher-quality evaluations than rating scales such as the likert scale for tasks such as sentiment analysis @xcite . in the context of summarization, @xcite find that, compared to the likert scale, ranking-based protocols are more reliable for measuring summary coherence but less so for repetition. however, previous studies have not analyzed annotation reliability in the context of factual consistency for summarization. our contributions are the following: 1) we are, to the best of our knowledge, the first to study the reliability of human evaluation for summarization factual consistency. 2) we study rating and rankingbased protocols across two summarization datasets and four state-of-the-art abstractive models. we determine the factors affecting human evaluation reliability and present a novel ranking-based protocol with the highest reliability. 3) we will release our evaluation guidelines and annotations to promote future work on factual consistency evaluation. ## study design each study consists of 100 input documents randomly sampled from each dataset, and four associated model-generated summaries. ## analysis we show the average ratings across ls scales, including a modified ls scale we will later introduce, in ## conclusion in this paper, we conduct studies to understand and improve the reliability of ranking and rating-based human evaluations of summarization factual consistency. we find that best-worst scaling is largely reliable, and the likert scale also has merits, but the proper scaling and dataset characteristics must be carefully studied to ensure its reliability. we improve these two protocols based on our findings and believe that our studies advance the understanding of both models and metrics as we aim to facilitate factually consistent text generation.