How Much Annotation is Needed to Compare Summarization Models? (2024)

Abstract: Modern instruction-tuned models have become highly capable in text generation tasks such as summarization, and are expected to be released at a steady pace. In practice one may now wish to choose confidently, but with minimal effort, the best performing summarization model when applied to a new domain or purpose. In this work, we empirically investigate the test sample size necessary to select a preferred model in the context of news summarization. Empirical results reveal that comparative evaluation converges quickly for both automatic and human evaluation, with clear preferences for a system emerging from under 100 examples. The human preference data allows us to quantify how well automatic scores can reproduce preference rankings across a variety of downstream summarization tasks. We find that, while automatic metrics are stable at smaller sample sizes, only some automatic metrics are able to moderately predict model win rates according to human preference.

## introduction instruction fine-tuned language models are highly capable summarizers, and new such models are now released often. continuously comparing such models using large, reference-based benchmark assessments is a costly task, especially if one wants to use them in a new domain. here we demonstrate on (english) new summarization data that-with respect to both human and automatic evaluationspreferences toward a summarization model emerge over test sets of about 50 samples. collecting human judgements, gpt evaluations, or (if possible) manually composed references for this size dataset is reasonable. further, we evaluate gpt evaluations and two popular reference-based evaluations, rouge-1 and bertscore, in terms of their ability to predict human preferences on a set of 36 testing contexts. we collect human judgements in the * work completed while at adobe research. context of three different summarization tasks and three sources of input. for these variations, we compute the accuracy of automated scores to reproduce human preferences between pairs of systems. ## background our goal is to establish the amount of test data needed to decide which of two summarization models produces better summaries for a given distribution over inputs (i.e., different sources of text to be summarized) and different task contexts for which the summary is to be used. it is common to approach evaluation as a ratethen-compare task in which outputs from systems are rated for quality on a scale, and then average scores are used to compare systems. but it is well known that inputs may differ considerably in difficulty @xcite . paired tests for statistical significance, that evaluate the differences of scores between two systems on the same input is the basis for comparison are therefore more appropriate @xcite . most contemporary work has embraced this approach, largely abandoning scoring of outputs and instead soliciting preferences among two or more choices @xcite . given developments in llms, pairwise win rates have become the de facto standard for reporting comparisons between instruction tuned models. in this work we similarly adopt win rate to compare systems, and we empirically identify the smallest test set size that reliably reveals preferences. most closely related to our work is the study on estimating power of tests for statistical significance, i.e., the minimum test size necessary to detect statistical differences of a given size @xcite . our work is aligned with the main question of this prior work, but we present empirical estimates of differences between systems without making any assumptions of tests to be used or size of effect we 51 want to detect. our empirical findings may inform future work on power estimation. prior related work proposes ways of carrying out evaluations, either automatically or manually @xcite @xcite @xcite , and measuring the correlations between system rankings produced by human and automatic evaluations on a given benchmark @xcite . we do not propose new evaluation methods, but rather introduce a method for validating automatic evaluations that does not rely on a benchmark, and instead measures the accuracy of automatic scores in reproducing human judgements across different input distributions and intended use-cases. ## unnecessarily large benchmarks we first compare two models, flant5-xxl @xcite and stablelm @xcite via automatic scores over three news summarization benchmarks: cnn/dm @xcite , xsum @xcite , and newsroom @xcite . we use the test set splits of these datasets from huggingface. 1 cnn/daily mail and xsum contain about 10k test inputs. the newsroom test set split has over 100k samples. for efficiency, we randomly sample 10k examples from this set to scale it down to a size comparable to the other two datasets. we then 1 https://huggingface.co/docs/datasets/index generate summaries with flant5 and stablelm for all articles in the test sets, using the summarization prompts that these models have been trained on (see appendix a). for each test split we sample 1000 times with replacement smaller test set sizes ranging from [5, len(dataset)]. we evaluate the two models with the commonly used rouge-1 @xcite and bertscore @xcite ). 2 both scores compare a summary with a human-written reference summary. rouge does so using tokens, while bertscore relies on embeddings. we show score variations for flant5 and stablelm across the three datasets in figure 1 . for all three datasets, a preference for one of the models emerges early: the winning model as scored over 10k test points emerges after just 25-50 samples. given these findings, we collect human judgements on 100 samples from each of the data sources, varying the task context in which the judgement is made. we also add gpt-4 as another summarization model to be evaluated, and later report the accuracy of gpt-based evaluation against the aggregated human judgements. ## human preferences we hire three individuals on upwork (appendix f) for cnn/dm and newsroom, and one for xsum. we select 100 inputs for annotation from each dataset, which given the trends we observed in the previous section, would be sufficient to reveal human preference. 3 we also add summaries produced by gpt-4 for evaluation on the smaller dataset. flant5, stablelm, and gpt-4 represent encoder-decoder, decoder-only (open-source), and decoder-only (closed-source) models, respectively. we instruct annotators to rank the summaries for each input in order of preference. this is a typical evaluation setting in which win rates-the percentage of input for which the model was preferred over the other-provide the clearest score for each model pair. we provide three different scenarios to measure how preference may change based on context: (i) rank the summaries in order of preference; (ii) assuming you are monitoring the news for important world events, rank the summaries in order of preference; (iii) which summary best captures the main details of the event being reported on? (iv) which summary contains the fewest unnecessary details? for gpt-4, we append the summaries with the instructions and provide these as prompts to the model. ## validating automatic evaluation we presented qualitative evidence that the context in which preferences are made change the human preferences dramatically. we also provided clear examples of cases when human preference for the same two models can flip depending on the context. this judgement variability poses a novel requirement for validating automatic evaluation approaches. we cannot combine win rates across settings and compute correlations between human preferences and automatic scores because these gpt-4 flant5 stablelm come from different distributions. we do, however, have a sufficient number of pairs for comparison: 3 models evaluated on 3 sources of data, on 4 context of use. this yields 9 overall preferences and 27 contextually dependent preferences. for four automatic methods for evaluation, we compute the accuracy of the automatic score in reproducing human preferences. specifically, we compute the percentage of pairwise comparisons for which the automatic evaluation agrees with the human win rates on which system is the better one. this is a coarse requirement because it does not capture the size of the win rate. for example the win rate of one system over another in human preferences is 51% but an automatic score predicts that its win rate is 79%, the automatic score will be considered accurate. ## conclusions we presented automatic and human evaluations designed to establish the minimum amount of data necessary to choose between contemporary summarization models. comparative evaluations establish which model performs better with test sets of 50 inputs. for human evaluation, a test size of 50 is sufficient to confidently establish which of two models people prefer. human preference varies, however, depending on the intended use of the summary and on the source of data for summarization. this variation calls for new methods for validating automatic scores. we find that all four automatic evaluations predict preferences better than chance but lead to erroneous conclusions for many pairwise comparisons.