SEAHORSE : A Multilingual, Multifaceted Dataset for Summarization Evaluation (2023)

Abstract: Reliable automatic evaluation of summarization systems is challenging due to the multifaceted and subjective nature of the task. This is especially the case for languages other than English, where human evaluations are scarce. In this work, we introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness, covering 6 languages, 9 systems, and 4 datasets. As a result of its size and scope, SEAHORSE can serve both as a benchmark to evaluate learnt metrics, as well as a large-scale resource for training such metrics. We show that metrics trained with SEAHORSE achieve strong performance on the out-of-domain meta-evaluation benchmarks TRUE (Honovich et al., 2022) and mFACE (Aharoni et al., 2022). We make the SEAHORSE dataset and metrics publicly available for future research on multilingual and multifaceted summarization evaluation.

## introduction evaluating the quality of generated text is an increasingly difficult problem as large language models produce text of rapidly improving quality @xcite @xcite . in spite of the improvements, such models often generate text that includes hallucinations and other subtle errors @xcite @xcite @xcite , making reliable evaluation essential for driving progress. common n-gram metrics such as bleu @xcite and rouge @xcite are often not well correlated with human judgments 1 data and metrics are available at https://goo.gle/ seahorse for many natural language generation (nlg) tasks such as machine translation @xcite , summarization @xcite , and dialogue @xcite . consequently, human evaluation is often necessary to reliably evaluate nlg systems. however, designing human annotation pipelines and obtaining annotations is resource-intensive, time-consuming, and not easily reproducible. developing more reliable automatic evaluation metrics would make model development faster and more efficient. with this in mind, much recent work has focused on learnt metrics, i.e., neural classification or regression models that aim to directly predict scores that evaluate the quality of generated text @xcite @xcite , often trained with human ratings. as a result, large-scale collections of human evaluations serve two critical roles in nlg metric development: (1) a source of training data for learnt metrics and (2) a meta-evaluation benchmark for the performance of these learnt metrics. the large potential of such datasets is exemplified by the wmt metrics shared task, foot_0 which has enabled rapid development of learnt metrics for machine translation that exhibit considerably higher correlation to human judgment than bleu @xcite . however, outside of machine translation, the existence of such collections of human judgments is limited. human annotations collected in nlg evaluations are rarely released @xcite , and even when they are, they tend to cover a single language (typically english) and are from a single dataset or task, limiting the robustness of models and metrics trained on these annotations. moreover, such annotations are often based on the test split of existing datasets (e.g., @xcite , which can be problematic for training learnt metrics. this is because the primary advantage of reliable automatic evaluation is to help model development, e.g., hyperparameter selection on the validation set; therefore a neural metric trained on test set annotations would, in general, lead to overfitting. in this work, we propose seahorse, foot_1 a largescale dataset for multilingual summarization evaluation. our dataset consists of 96k summaries with ratings along 6 quality dimensions: comprehensibility, repetition, grammar, attribution, main ideas, and conciseness, in 6 languages, for 9 systems (8 models plus the human-authored reference summaries) across 4 summarization datasets (see examples in figure 1 ). the training and validation splits of the dataset come from the validation sets of the original summarization corpora to prevent test set contamination when training metrics. this permits us to train a learnt metric for each quality dimension that can be used for offline model evaluation. we evaluate the metrics learned from sea-horse on the seahorse test set, as well as other existing meta-evaluation benchmarks, such as mface @xcite and true @xcite . our experiments show that the metrics generalize across datasets, tasks, and languages. for example, we demonstrate that although seahorse includes data in 6 languages, the resulting learnt metrics achieve strong performance on the mface benchmark, which consists of 45 languages, exhibiting their zero-shot multi-lingual generalization potential. to summarize, the contributions of this paper are: ## the seahorse dataset the seahorse dataset consists of 96,645 summaries annotated with human ratings along 6 quality dimensions. in this section, we describe the seahorse dataset, how we generated the summaries, and how we collected the annotations. ## dataset analysis we first analyze the dataset's composition and the quality of the collected annotations. the 1-shot palm model is particularly likely to copy from the article as its output, obtaining the highest rouge-l foot_2 @xcite ) scores between the summary and the article. in 14% of cases, the beginning of the 1-shot summaries (the first 20% of the summary) exactly matched the beginning of the reference article. note that the reference summaries do not always receive the highest rate of positive responses. the way in which reference texts are collected may limit their quality along some dimensions. for example, the text that was collected as a reference summary may not have been intended to be read as a standalone replacement for the article, and therefore may not be fully attributable to the article, as rashkin et al. (2021) point out. we can use the positive response rates to inspect the quality of the dataset by verifying the presence of 3 patterns we expect to see in the data: 1) higher positive response rates for better summarization models, 2) high correlation between the responses to q4&6 and q5&6, and 3) annotator agreement across the 6 dimensions. order of model quality our first expectation is that summaries generated by better summarization models should receive more positive responses from raters. we have 3 types of model pairs where we can expect one model to generate better summaries than the other: 1) a larger model should outperform a smaller model (the xxl vs. the small model), 2) a fully trained model should outperform an under-trained model (the small vs. the small_250 model), and 3) a finetuned model should outperform a 1-shot prompted model (the finetuned vs. 1-shot palm models). we compare how often these model pairs pro-duce low-quality summaries, i.e., summaries that are unintelligible to readers. in generally, however, the seahorse ratings capture the relative differences in model quality we expect to see when evaluating two models with known differences. correlation between dimensions conciseness (q6) is related to two other dimensions in our annotation: attribution (q4) and main ideas (q5). a summary cannot be considered a "concise representation of the information in the article" if it has information that is not in the article (i.e., a "no" response for q4) or if does not represent the main points in the article (i.e., a "no" response for q5), which was a detail pointed out to evaluators in the task instructions. therefore, we expect q6 to be positively correlated with both of these dimensions if the annotators understood the task and the relationship between the dimensions of quality. @xmath0, they also say it is not concise (q6). this is also reflected in figure 3 , which shows that the strongest correlation between questions is between questions 4&6 and questions 5&6. these results show the pattern we expect to see in the data given the task definition and instructions, and it demonstrates the annotators' ability to understand and execute the annotation task. ## related work we briefly review other large-scale datasets of human evaluations of summaries that have been released and compare them to seahorse, but note that most focus on annotating the test data, which would lead to test data contamination when training metrics. summeval @xcite and real-summ @xcite are summarization meta-evaluation benchmarks with 12,800 and 7,742 annotations respectively. these benchmarks focus on a single language and single dataset: the the only other multilingual summarization evaluation dataset, to the best of our knowledge, is mface @xcite , which has annotations for 31,500 summaries covering a broader set of languages (45 languages). mface focuses on one dataset (xl-sum) and a smaller set of models than seahorse. in §4 we use mface as a comprehensive out-of-domain evaluation set, and view it as complementary to seahorse, which aims to provide large-scale and diverse training data for metrics. ## conclusion in this work, we present seahorse, a large-scale multilingual, multifaceted dataset for summarization consisting of 96k human annotations of summaries. due to its size and scope, seahorse enables the training and evaluation of learnt metrics across several quality dimensions. our results show that seahorse-trained metrics not only achieve strong performance on our own test set but also generalize to other external and out-of-domain benchmarks: mface and true. in the future, we are interested in exploring how seahorse can be used more directly to improve the quality of summarization models and metrics, and hope this paper and the public release of seahorse enables further research on these topics.