Multi-domain Summarization from Leaderboards to Practice: Re-examining Automatic and Human Evaluation (2023)

Abstract: Existing literature does not give much guidance on how to build the best possible multi-domain summarization model from existing components. We present an extensive evaluation of popular pre-trained models on a wide range of datasets to inform the selection of both the model and the training data for robust summarization across several domains. We find that fine-tuned BART performs better than T5 and PEGASUS, both on in-domain and out-of-domain data, regardless of the dataset used for fine-tuning. While BART has the best performance, it does vary considerably across domains. A multi-domain summarizer that works well for all domains can be built by simply fine-tuning on diverse domains. It even performs better than an in-domain summarizer, even when using fewer total training examples. While the success of such a multi-domain summarization model is clear through automatic evaluation, by conducting a human evaluation, we find that there are variations that can not be captured by any of the automatic evaluation metrics and thus not reflected in standard leaderboards. Furthermore, we find that conducting reliable human evaluation can be complex as well. Even experienced summarization researchers can be inconsistent with one another in their assessment of the quality of a summary, and also with themselves when re-annotating the same summary. The findings of our study are two-fold. First, BART fine-tuned on heterogeneous domains is a great multi-domain summarizer for practical purposes. At the same time, we need to re-examine not just automatic evaluation metrics but also human evaluation methods to responsibly measure progress in summarization.

## introduction academic papers on automatic document summarization have been published since the 1950s (luhn, * research done while an intern at adobe research 1958), but broadly applicable summarizers not constrained by document type have only recently become widely available. 1 the literature contains a wealth of information on model architectures for summarization. yet, it remains hard to decide from published evaluations which are "the best" components for a good quality multi-domain summarizer. we make the idealized assumption that model size and inference cost are not an issue. we seek to find the pre-trained model and the training data from freely available resources that will produce the best multi-domain summarizer. we fine-tune and evaluate popular off-the-shelf pre-trained models ( §3.1) bart @xcite , pegasus @xcite and t5 @xcite on six datasets. we also create a mixed training dataset with a balanced representation of each of the domains ( §3.2). we find that fine-tuning on mixed-domain text, smaller in size than most of the in-domain training set, yields a robust system performing on par with in-domain models fine-tuned on the order of magnitude more data ( §4). in addition to evaluation with automatic metrics, we conduct a human evaluation ( §5). consistent with automatic evaluation, bart summaries were preferred more often than those produced by pe-gasus and t5 ( §5.2). additionally, summaries generated with bart fine-tuned on mixed-domain data are preferred over those generated with bart trained on the most popular summarization research dataset, cnn/daily mail, even though the mixed-domain dataset is the smaller of the two ( §5.3). the human evaluation also provides further insights in summary preferences that are not captured by the automatic evaluation. summaries from bart fine-tuned on mixed-domain data were even preferred over those produced by fine-tuning on indomain data matching each test sample. the model often produced summaries deemed even more informative than the human reference for the input document ( §5.4). this was not the case for models obtained by fine-tuning using data from a single source. human evaluation confirms that bart fine-tuned on diverse domains, is a good quality multi-domain summarizer for practical application. the quality of the model is even better than the expectation based on automatic evaluation. finally, we share our experience with the human evaluation process ( §5.5). the annotators were the three senior authors on this paper and found the overall experience quite frustrating, resulting in an extended adjudication phase. there were inconsistencies in ratings across annotators and also in multiple rounds with the same annotator. we expect this experience to translate and be even worse for annotators on crowdsourcing platforms. we pinpoint the difficulties we faced so that our experience can help improve the human evaluation process for longform text. at the same time, we question the reliability of crowdsourcing human ratings for such a task and using them to measure progress in summarization. ## related work some hints that domain robustness is a problem but that summarizers can to an extent, generalize across domains are found in the literature. @xcite observe catastrophic forgetting during domain adaptation via continual pre-training. this is concerning if the goal is to have a robust system that serves multiple domains. they do not explicitly measure how much systems degrade when evaluated out of domain, though it is implied by the task and results that there is degradation. there are a few direct studies of summarization cross-domain robustness. @xcite tested if meetings summarization data is useful for email summarization. they find that training on email data is best, but in the absence of such data, training on meetings is helpful. bar-haim et al. (2020) train a system for extracting key points on argumentation datasets and then evaluate the same system on municipal surveys and user reviews. the systems perform well, exhibiting robustness. in our work, we carry out a similar evaluation, but we examine the robustness of abstractive summarizers on a diverse set of datasets. these findings on cross-domain robustness are encouraging and in line with @xcite 's findings that some of the capabilities for identifying summary-worthy content are transferable between domains. they study news and opinion piece summarization for texts drawn and find that a model trained on out-of-domain data can learn to detect summary-worthy content but may not match the generation style in the target domain. stylistic markers of a domain, i.e., as in typical phrasing used to talk about certain topics, are not captured. lastly, we share our experience of the human evaluation process for summarization. some prior work @xcite also studies the efficacy of human evaluation for machine translation. ## experimental design abstractive summarizers generate a short plain text summary capturing the main points of a longer text. we work with transformer-based encoder-decoder text-to-text models: bart @xcite , pegasus @xcite and t5 @xcite . the models are pre-trained on large general-purpose corpora followed by fine-tuning on specific summarization datasets. ## automatic evaluation we first evaluate the summarizers using three automatic metrics: rouge-2 @xcite , sacrebleu @xcite and bertscore @xcite . the goal of this evaluation is to glean insights about system performance to inform the choice of specific comparisons that can be done with human evaluation. we show the average in-domain and the average cross-domain scores for each model in we also show the average automatic scores on the six test datasets with bart trained on different settings ( for a detailed view, in ## human evaluation automatic evaluations consistently indicated that (i) bart produces better summaries than t5 and pegasus across the six domains we study, and (ii) the summarizer trained on mixed domain data is the most robust to domain changes. to confirm this finding, we also conduct a manual human evaluation. we sample 10 examples from each domain, for a total of 60 documents 3 . each example has a human reference summary and five automatic summaries. the same trends for automatic scores are observed for these 60 documents as the 1500 documents in the last section. ## conclusions we study the cross-domain robustness of neural summarizers to find the recipe to build a good quality multi-domain summarizer. we find that bart is the best pre-trained model for summarization. it is especially effective when fine-tuned on mixeddomain data, even more than when fine-tuned on larger in-domain data. in the human evaluation, this summarizer is rated as producing better summaries than an in-domain summarizer and often produces summaries better than the human summary. this is not reflected in the automatic scores and will therefore, not be captured by leaderboards. we also find that human evaluation is hard to conduct, even with experienced annotators. the data is poorly formatted, hard to read, and results in superficial reading. moreover, annotators even disagree on what constitutes a good summary.