CDE val S umm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems (2020)

Abstract: Neural network-based models augmented with unsupervised pre-trained knowledge have achieved impressive performance on text summarization. However, most existing evaluation methods are limited to an in-domain setting, where summarizers are trained and evaluated on the same dataset. We argue that this approach can narrow our understanding of the generalization ability for different summarization systems. In this paper, we perform an in-depth analysis of characteristics of different datasets and investigate the performance of different summarization models under a cross-dataset setting, in which a summarizer trained on one corpus will be evaluated on a range of out-of-domain corpora. A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways (i.e. abstractive and extractive) on model generalization ability. Further, experimental results shed light on the limitations of existing summarizers. Brief introduction and supplementary code can be found in https://github.com/zide05/CDEvalSumm .

## introduction neural summarizers have achieved impressive performance when evaluated by rouge @xcite on in-domain setting, and the recent success of pretrained models drives the state-of-the-art results on benchmarks to a new level @xcite @xcite @xcite . however, the superior performance is not a guarantee of a perfect system since exsiting models tend to show defects when evaluated from other aspects. for example, @xcite observes that many abstractive systems tend to be near-extractive in practice. @xcite ; @xcite ; @xcite ; @xcite reveal that most generated summaries are factually incorrect. these non-mainstream evaluation methods make it easier to identify the model's weaknesses. orthogonal to above two evaluation aspects, we aim to diagnose the limitation of existing systems under cross-dataset evaluation, in which a summarization system trained on one corpus would be evaluated on a range of out-of-dataset corpora. instead of evaluating the quality of summarizers solely based on one dataset or multiple datasets individually, cross-dataset evaluation enables us to evaluate model performance from a different angle. for example, fig. 1 shows the ranking of 11 summarization systems studied in this paper under different evaluation metrics, in which the ranking list "(a) in-dataset r2" is obtained by traditional ranking criteria while other two are based on our designed cross-dataset measures. intuitively, we observe that 1) there are different definitions of a "good" system in various evaluation aspects; 2) abstractive and extractive systems exhibit diverse behaviors when evaluated under the cross-dataset setting. the above example recaps the general motivation of this work, encouraging us to rethink the generalization ability of current top-scoring summarization systems from the perspective of crossdataset evaluation. specifically, we ask two questions as follows: q1: how do different neural architectures of summarizers influence the cross-dataset generalization performances? when designing summarization systems, a plethora of neural components can be adopted @xcite @xcite @xcite . for example, will copy @xcite and coverage @xcite mechanisms improve the cross-dataset generalization ability of summarizers? is there a risk that bert-based summarizers will perform worse when adapted to new areas compared with the ones without bert? so far, the generalization ability of current summarization systems when transferring to new datasets still remains unclear, which poses a significant challenge to design a reliable system in realistic scenarios. thus, in this work, we take a closer look at the effect of model architectures on cross-dataset generalization setting. q2: do different generation ways (extractive and abstractive) of summarizers influence the crossdataset generalization ability? extractive and abstractive models, as two typical ways to summarize texts, usually follow diverse learning frameworks and favor different datasets. it would be absorbing to know their discrepancy from the perspective of cross-dataset generalization. (e.g., whether abstractive summarizers are better at generating informative or faithful summaries on a new test set?) to answer the questions above, we have conducted a comprehensive experimental analysis, which involves eleven summarization systems (including the state-of-the-art models), five benchmark datasets from different domains, and two evaluation aspects. tab. 1 illustrates the overall analysis framework. we explore the effect of different architectures and generation ways on model generalization ability in order to answer q1 and q2. semantic equivalency (e.g., rouge) and factuality are adopted to characterize the different aspects of cross-dataset generalization ability. additionally, we strengthen our analysis by presenting two views of evaluation: holistic and fine-grained views (sec. 5). our contributions can be summarized as: 1) cross-dataset evaluation is orthogonal to other evaluation aspects (e.g., semantic equivalence, factuality), which can be used to re-evaluate current summarization systems, accelerating the creation of more robust summarization systems. 2) we have design two measures stiffness and stableness, which could help us to characterize generalization ability in different views, encouraging us to diagnose the weaknesses of state-of-the-art systems. 3) we conduct dataset bias-aided analysis (sec. 4.3) and suggest that a better understanding of datasets will be helpful for us to interpret systems' behaviours. ## representative systems although it's intractable to cover all neural summarization systems, we try to include more representative models to make a comprehensive evaluation. our selection strategy follows: 1) the source codes of systems are publicly available; 2) systems with state-of-the-art performance or the top performace on benchmark datasets (e.g., cnndm @xcite ) 3) systems equipped with typical neural components (e.g., transformer, lstm) or mechanism (e.g., copy). ## datasets we explore five typical summarization datasets: cnndm, xsum, pubmed, bigpatent b and reddit tifu. cnndm @xcite and xsum @xcite are news domain summarization datasets which are various in their publications and abstractiveness. pubmed @xcite is a scientific paper dataset, which can be used to investigate the generalization ability of models on scientific domain. bigpatent b @xcite is the b category of bigpatent (a dataset consisting of patent documents from google patents public datasets). reddit tifu @xcite is a dataset with less formal posts collected from the online discussion forum reddit. detailed statistics and introduction of datasets are presented in the appendix section. ## evaluation for summarization existing summarization systems are usually evaluated on different datasets individually based on an automatic metric: @xmath0, s, m), where d, s represents a dataset (e.g., cnndm) and system (e.g., l2l) respectively. m denotes an evaluation metric (e.g., rouge). to evaluate the quality of generated summaries, metrics can be designed from diverse perspectives, which can be abstractly characterized in fig. 2 . ## cross-dataset evaluation despite recent impressive results on diverse summarization datasets, modern summarization systems mainly focus on extensive in-dataset architecture engineering while ignore the generalization ability which is indispensable when systems are required to process samples from new datasets or domains. therefore, instead of evaluating the quality of summarization system solely based on one dataset, we introduce cross-dataset evaluation (a summarizer (e.g., l2l) trained on one dataset (e.g., cnndm) will be evaluated on a range of other datasets (e.g., xsum)). methodologically, we perform cross-dataset evaluation from two views: finegrained and holistic and we will detail them below. ## experiment in what follows, we analyze different summarization systems in terms of semantic equivalence and factuality. moreover, the results are studied in holistic and fine-grained views based on the measures defined above. holistic results are showed in fig. 4 analysis and fig. 5 . on the other hand, tab. 4 and tab. 5 display the fine-grained observations. tab. 2 dispalys the in-dataset results of all models on five benchmark datasets. ## related work our work is connected to the following threads of topics of nlp research. cross-dataset generalization in nlp recently, more researchers shift their focus from individual dataset to cross-dataset evaluation, aiming to get a comprehensive understanding of system's generalization ability. fried et al. (2019) explores the generalization ability of different constituency parsers. talmor and berant (2019), on the other hand, shows the generalization ability of reading comprehension models can be improved by pretraining on one or two other reading comprehension datasets. fu et al. (2020) studies the model generalization in the field of ner. they point out the bottleneck of the existing ner systems through in-depth analyses and provide suggestions for further improvement. different from the above works, we attempt to explore generalization ability for summarization systems. diagnosing limitations of existing summarization systems beyond rouge, some recent works try to explore the weaknesses of existing systems from divese aspects. zhang et al. (2018) tries to figure out to what extent the neural abstractive summarization systems are abstractive and discovers many of abstractive systems tend to perform near-extractive. on the other hand, cao et al. (2018) and kryściński et al. (2019) study the factuality problem in modern neural summarization systems. the former puts forward one model that combining source document and preliminary extracted fact description and prove the effectiveness of this model in terms of factuality correctness. while the latter contributes to design a model-based automatic factuality evaluation metric. abstractiveness and factuality error the above works studied are orthogonal to this work and can be easily combined with cross-dataset evaluation framework in this paper as sec. 6.2 shows. moreover, wang et al. (2019); hua and wang (2017) attempt to investigate the domain shift problem on text summarization while they focus on a single generation way (either abstractive or extractive) we also investigate the generalization of summarizers when transferring to different datasets, but include more datasets and models. ## conclusion by performing a comprehensive evaluation on eleven summarization systems and five mainstream datasets, we summarize our observations below: 1) abstractive summarizers are extremely brittle compared with extractive approaches, and the maximum gap between them reaches 37% in terms of the measure stableness (rouge) defined in this paper. 2) bart (sota system) is superior over other abstractive models and even comparable with extractive models in terms of stiffness (rouge). on the other hand, it is robust when transferring between datasets as it possesses high stableness (rouge). 3) bert match (sota system) performs excellently in terms of stiffness, while still lacks stableness when transferred to bigpatent b from other datasets. 4) the robustness of models can be improved through either equipped the model with ability to copy span from source document (i.e., @xcite ) or make use of well trained sequence to sequence pre-trained model (bart). 5) simply adding bert on encoder could improve the stiffness (rouge) of model but will cause larger cross-dataset and in-dataset perfor-mance gap, a better way should be found to merge bert into abstractive model, or a better training strategy should be applied to offset the negative influence it brings. 6) existing factuality checker (factcc) is limited in predictive power of positive samples (sec.6.2). 7) out-of-domain systems can even surpass in-domain systems in terms of factuality. (sec.6.2) xsum xsum @xcite is a dataset consists of the articles and the single-sentence answers of the question "what is the article about?" as summary. it is more abstractive compared with cnn/dailymail. pubmed pubmed @xcite is drawn from scientific papers specifically medical journal articles from the pubmed open access subset. we use the introduction as source document and the abstract as summary here. bigpatent bigpatent @xcite consists of 1.3 million records of u.s. patent documents and the corresponding summaries are created by human. according to cooperative patent classification (cpc), the dataset is divided to nine categories. one of the nine categories is chosen as a dataset in difference domain in our experiment (category b: performing operations; transporting). reddit tifu reddit tifu @xcite is a dataset with less formal posts compared with datasets mentioned above which mostly use formal documents as source. it is collected from the online discussion forum reddit. they regard the body text as source, the title as short summary, and the tl;dr summary as long summary, thus making two sets of datasets: tifu-short and tifu-long. tifu-long is used in this paper.