Intrinsic Evaluation of Summarization Datasets (2020)

Abstract: High quality data forms the bedrock for building meaningful statistical models in NLP. Consequently, data quality must be evaluated either during dataset construction or *post hoc*. Almost all popular summarization datasets are drawn from natural sources and do not come with inherent quality assurance guarantees. In spite of this, data quality has gone largely unquestioned for many of these recent datasets. We perform the first large-scale evaluation of summarization datasets by introducing 5 intrinsic metrics and applying them to 10 popular datasets. We find that data usage in recent summarization research is sometimes inconsistent with the underlying properties of the data. Further, we discover that our metrics can serve the additional purpose of being inexpensive heuristics for detecting generically low quality examples.

## introduction data understanding is fundamentally important in natural language processing (nlp); for data-driven learning-based methods (e.g. neural networks), the quality of the training data bounds the quality of models learned using it. therefore, understanding this data is necessary in order to ensure that models learn to perform a given task correctly. understanding data is a multidimensional problem. one line of inquiry has demonstrated why prominent datasets are insufficiently challenging: many data examples can be solved by alternative heuristics that do not encode an approach that is faithful to the task @xcite . from the perspective of datasets, several works have shown that standard datasets in areas such as visual question answering @xcite , natural language inference @xcite , and reading comprehension @xcite contain annotation artifacts that often give rise to these spurious correlations or reasoning shortcuts. data understanding can also inform scientific and ethical decision-making @xcite @xcite with recent work studying how social biases encoded in training data propagate to learned models @xcite . in this work, we extend these efforts towards the setting of summarization. we find this to be particularly timely since several summarization datasets have been released in recent years with little discussion of data quality. while prior work on evaluating nlp datasets has focused on their difficulty, transparency, or bias, we consider broadly the overall quality of the dataset -in our case, for the task of summarization. 1 our central insight is that desirable properties of a summary can be readily estimated by adapting and applying existing nlp methods. with this in mind, we present a multiaspect large-scale study of summarization datasets that dissects summarization into 5 properties that are evaluated across 10 datasets spanning multiple summarization domains. our analysis reveals that our metrics can serve as lightweight detectors of generically low quality examples. most strikingly, we show that quantifiable aspects of summarization datasets are inconsistent with their use by the nlp community in several instances. ## motivation quality assurance for data. nuanced understanding of data is requisite for drawing sound scientific conclusions. in particular, without evaluating for the quality and accuracy of data used to test models, it is impossible to be certain that progress is being made and that successive iterations of models truly make progress on the underlying task or linguistic phenomena of interest. within nlp, iconic datasets such as the penn treebank @xcite have sustained subareas such as language modelling, part-of-speech tagging, and syntactic parsing for years due to the painstaking annotation efforts put into making these high-fidelity resources. and in the context of summarization, initial datasets, such as those produced during the document understanding conference (duc) and text analysis conference (tac) evaluations, implemented fine-grained verification of data quality. 2 in part due to the emergence of data-hungry modelling techniques, the demands for larger datasets often render quality assurance procedures of this standard to be impractical and infeasible. nonetheless, several recent natural language understanding datasets @xcite @xcite institute explicit qualitycontrol procedures in crowd-sourcing dataset construction @xcite @xcite , such as using additional annotators to validate annotations (c.f. @xcite . in the sibling subfield of machine translation, which often shares similar modelling challenges and evaluation regimes as summarization due to the shared nature of being sequence-to-sequence natural language generation tasks, the annual wmt conference foot_3 consistently furnishes high quality data. in summary, ensuring data quality is both crucial and challenging. and in comparison with other subareas of nlp, we argue that summarization has lagged behind in rigorously ensuring the quality of widely-used datasets. relating data quality and model quality. for fully learning-based methods, especially those with weak/minimal inductive biases such as neural networks, i(s; a) is approximately zero. while i(s; p ) may be greater than zero (e.g. language modelling pretraining provides statistical information that may facilitate a model to avoid a priori unlikely summaries), standard pretraining regimes such as large-scale language modelling over generic text corpora @xcite are likely insufficient to meaningfully learn to summarize. under these assumptions, the mutual information between s and m is critically upper-bounded in terms of i (s; t ). we hypothesize that the quality of the training dataset t is highly correlated with its mutual information with respect to the summarization task s, i(s; t ). one size does not fit all. spärck jones (1999) famously argued that summarization systems should be understood conditional on the context in which they will be used. in recent years, the field has significantly departed from this perspective and primarily studied "general-purpose summarization" @xcite , which she denounced as ignis fatuus. with our work, we adopt the perspective that for all datasets it is strictly preferable to have all properties quantified; it is the responsibility of practitioners building summarization systems to accurately weight different metrics based on their ultimate goals and use cases. as such, we refrain from providing prescriptive domain-agnostic or context-agnostic notions of summarization. ## metrics in this work, we evaluate the quality of a dataset by aggregating scores for each example in the dataset. we conjecture that for many nlp tasks, estimating the quality of a particular data example is of similar complexity as correctly performing the task on the example. 5 nevertheless, for summarization, our insight is that various aspects of a summarization example (a document-summary pair) can be reliably estimated by re-purposing existing nlp methods. we are guided by pioneering work @xcite @xcite that defined core properties of summarization systems and influential sub-sequent work @xcite @xcite that refined and extended these properties. from this literature, we specifically study compression, topic similarity, abstractivity, redundancy, and semantic coherence as these properties are of recurring and sustained interest. 6 for each abstract property, numerous concrete methods can be proposed to quantify it. in appendix a, we describe alternatives we considered and detail how we decided which methods performed best. we restrict discussion to the bestperforming approaches in the main paper. notation. our metrics will assume indexed sets d, s such that summary s @xmath0. the length in words of a sequence s is |s| and the length in sentences is s . each metric assigns a value @xmath1, 1] to every (d i , s i ) where 1 is the maximal score and example-level scores are averaged to yield a dataset-level score. compression. we quantify compression at the word (w) and sentence (s) levels: topic similarity. we learn a topic model m on training corpus t with k topics using lda @xcite and quantify topic similarity by comparing the inferred topic distributions θ d i |m , θ s i |m for a given summary and document: redundancy. rouge @xcite implicitly penalizes redundancy but underestimates its detrimental impacts @xcite . however, we find that rouge is effective for identifying redundancy given the definitional focus on overlapping spans. we quantify redundancy as the average rouge-l f -score for all pairs of distinct sentences in the summary. ## results and analysis compression scores quantitatively disambiguate summarization tasks. concretely, we observe gw has the lowest compression scores and while gw is sometimes described as a summarization dataset @xcite , it is better seen as a headline generation dataset that is more in the style of sentence compression (as is suggested by s @xmath2. conversely, ami and movi-escript achieve the highest scores by a substantial margin and are long-document summarization datasets. classifying new summarization datasets accurately may prove useful given that successful methods from one domain often do not extend to another and this shortcoming in generalization can be attributed to the differences in compression requirements @xcite . given the goals stated in the xsum dataset paper, tl;dr may be a better choice than xsum. in particular, @xcite introduce xsum as a large dataset that legitimately requires abstraction. while xsum is more abstractive than other news datasets (barring gw) and is relatively large, tl;dr displays greater abstractivity, similar length summaries, and is 15 times larger. that said, @xcite explore topic-oriented strategies in their work and such methods may be better suited to xsum given the ts scores. cnn-dm and nyt are suboptimal for studying abstractive/extractive systems respectively. several recent works @xcite @xcite have used cnn-dm to build and evaluate abstractive systems. conversely, nyt has been used to build extractive systems @xcite . given our findings, we find both of these trends to be inconsistent with dataset properties and suboptimal given other preferable datasets for these purposes: cnn-dm is one of the least abstractive datasets and there are larger and more extractive alternatives to nyt such as nws. especially in the case of cnn-dm, we note that training learning-based systems (e.g. neural methods) using data with limited abstractivity implies the resulting summarizers will be limited in their ability to generate genuinely abstractive text. this is validated by empirical findings as both @xcite and @xcite observe limited abstractivity in abstractive systems trained on cnn-dm. in light of this, we argue systems should be characterized as abstractive or not based on their empirical behavior rather than their theoretical capability. foot_9 cnn-dm is not a representative benchmark for summarization as a whole. recent work @xcite has explicitly portrayed cnn-dm as the benchmark dataset for summarization; the field has implicitly done this for several years @xcite . while there is clear value in evaluating pretrained representations on summarization datasets, we caution against using cnn-dm as a stand-in for the entire summarization subfield. instead, we suggest using a diverse group of datasets and not reducing a highly heterogeneous subfield to a single dataset. while this adds additional overhead, this cost is necessary to draw meaningful conclusions about the impact of advances on summarization broadly given the pronounced diversity in summarization datasets ( we also observe that semantic coherence patterns similarly to redundancy. in particular, while we find the semantic coherence scores are appropriate for most examples we manually inspected, this suggests that bert relies upon word-level overlaps in making next-sentence judgments (similar to behaviors seen in other sentence-pair tasks such as natural language inference, c.f @xcite 6 detecting low quality examples to complement our quantitative dataset-level analysis, we conduct a qualitative study of individual examples by examining outliers. for each (dataset, metric) pair, we sample 10 examples from both the top and bottom 10% of examples for that metric and in that dataset. since manually considering all of the 1080 examples was not feasible, we began by examining the sampled examples for topic similarity, redundancy, and semantic coherence. our hypothesis was that example quality would positively correlate with coherence and topic similarity and negatively correlate with redundancy. we found this hypothesis to be validated by our observations as we found that examples with low coherence, low topic similarity, or high redundancy scores were generally low quality examples. every example which we judged to be low quality demonstrated at least one of the following defects: on the other hand, we did not find any systematic defects in examples with high coherence, high topic similarity, or low redundancy scores. instead, almost all of these examples were satisfactory. for the remaining two properties (compression measured by cmp w , abstractivity measured by abs 1 ), we analyzed all of the associated 400 examples. what we observed is that many of these examples tended to be generically low quality and we quantify this in ## discussion open problems and future directions. our results demonstrate that a sizeable fraction of examples in most summarization datasets are low quality. however, it remains open whether modellers should simply prune these examples, manually/automatically attempt to correct them, or model them without change. we do note that research in the machine learning and learning theory communities shows that models both theoretically and empirically do substantially worse when trained using low quality examples, even when the examples are not strictly adversarially chosen @xcite @xcite . these concerns are further compounded by the evidence of @xcite that neural models for natural language generation are not robust to naturally noisy data. our metrics may be repurposed to rank examples in designing curricula for curriculum learning ap-proaches @xcite . alternatively, they can serve as additional metrics for the (possibly unsupervised) evaluation of summarization systems, potentially mitigating deficiencies in standard metrics, such as rouge, by directly penalizing redundancy and semantic incoherence. limitations. in this work, we restrict ourselves to single-document single-reference english language summarization datasets. while the datasets we study constitute a considerable fraction of dataset usage in the summarization community, several multi-document summarization datasets have been introduced (e.g. @xcite and multi-reference summarization datasets have often been argued to be desirable due to under-constrained nature of the summarization task @xcite and the ideal evaluation paradigm for rouge @xcite . beyond english, both large summarization datasets @xcite and more general language resources/technologies @xcite are less available, which may heighten the need for data quality assurance. more broadly, the measures that we introduce are automated, and therefore non-human, judgments of the quality of summarization data. therefore, we only envision these measures to be useful as inexpensive first-order approximations of aspectlevel summary quality rather than bona fide replacements for human evaluation. additionally, since we principally envision applying these metrics to datasets, we make no efforts to make these metrics robust to adversarially-crafted data and they are likely quite susceptible to adversarial attack. ## conclusion in this work, we demonstrate that various aspects of summarization datasets can be intrinsically evaluated for. we specifically show this for 5 properties across 10 popular datasets, uncovering that dataset use is sometimes incongruous with the attributes of the underlying data. we also find that some aspectlevel estimators may be surprisingly effective at detecting low quality dataset examples. our findings suggest that more intentional and deliberate decisions should be made in selecting summarization datasets for downstream modelling research and that further scrutiny should be placed upon summarization datasets released in the future.