How Far are We from Robust Long Abstractive Summarization? (2022)

Abstract: Abstractive summarization has made tremendous progress in recent years. In this work, we perform fine-grained human annotations to evaluate long document abstractive summarization systems (i.e., models and metrics) with the aim of implementing them to generate reliable summaries. For long document abstractive models, we show that the constant strive for state-of-the-art ROUGE results can lead us to generate more relevant summaries but not factual ones. For long document evaluation metrics, human evaluation results show that ROUGE remains the best at evaluating the relevancy of a summary. It also reveals important limitations of factuality metrics in detecting different types of factual errors and the reasons behind the effectiveness of BARTScore. We then suggest promising directions in the endeavor of developing factual consistency metrics. Finally, we release our annotated long document dataset with the hope that it can contribute to the development of metrics across a broader range of summarization settings.

## introduction pre-trained transformers @xcite have brought tremendous progress in summarizing text in an abstract manner @xcite . unlike extractive summarization @xcite @xcite , abstractive summarization presents a blue-sky potential of generating summaries that are fluent and relevant to the source by intelligently paraphrasing salient contents rather than merely copying from source texts @xcite @xcite . nevertheless, even under a short document setting, transformer-based abstractive models often generate summaries that are repetitive @xcite , ungrammatical, and factually inconsistent with the source @xcite @xcite . furthermore, current pre-trained transformers have an input length limit that restricts them to be directly adapted to long document summarization @xcite as it would lead to a significant loss of salient information in the remaining text. these naturally bring us to a question: how far are we from building a robust abstractive summarization system for long documents? a robust abstractive summarization system should at least have (i) models that can generate high-quality summaries, and (ii) evaluation metrics that can critically assess the relevance and factuality of a summary foot_0 . however, research on analysis and critiques of models @xcite and metrics @xcite mainly focus on the shortdocument @xcite or long dialogue @xcite . consequently, our work aims to fill the gap by systematically analyzing abstractive models and evaluation metrics under the long document setting. to analyze the quality of current state-of-the-art long document abstractive models, we lack a set of model-generated summaries with sufficient diversity under long document settings. to this end, we implement bart @xcite and pe-gasus @xcite ) models under arxiv @xcite and govreport @xcite as they have been found to be the most effective pre-trained transformer in a large-scale evaluation of summarization models @xcite . however, their 1,024 token input limit would lead to a significant loss in the information required to generate a high-quality summary. hence, by closely following prior works in extending the pre-trained models using sparse attention @xcite @xcite and reduce-then-summarize mechanism @xcite , we implement different variants of longformerbased bart and pegasus to obtain a diverse set of summaries. we then perform fine-grained human analysis on the model outputs by three human annotators to qualitatively assess whether long document abstractive models can generate relevant and factually consistent summaries. effective evaluation metrics are also paramount as they can critically assess the model performance before releasing it to target users. we adapt recently proposed metrics @xcite @xcite @xcite to long document settings and thoroughly analyze their strength and weaknesses to measure the relevance and factual consistency on our annotated dataset. to our best knowledge, we are the first to assess abstractive models and evaluation metrics under the long document setting. our contributions are as follows: (1) we analyze pre-trained transformer summarizers to encourage a rethinking of architectural designs under long document settings. (2) we release human-annotated long document abstractive model outputs to further research in human-correlated evaluation metrics across a broader setting. (3) we investigate summarization metrics using our annotated long document datasets to expose the limitation of metrics and provide promising directions for the future development of evaluation metrics. ## human evaluation of metrics with high factual inconsistency rates, long abstractive summarizers remained unready for real-world implementation. it is thus paramount to ensure that the performances of future proposed models can be evaluated by metrics that are well correlated with user judgment. however, previous works on evaluation metrics have mainly focused on short document summarization research settings due to (i) the lack of human-annotated long document modelgenerated summaries and (ii) the reliance of metrics on pre-trained language models that are fine-tuned on short document datasets @xcite @xcite . relying on our annotated dataset, we adapt evaluation metrics proposed in prior works to the long document settings and correlate their metric scores with average human relevance and factual consistency scores. ## conclusion in this work, we perform human evaluations of model-generated summaries to critically analyze the relevance and factual consistency aspect of models and metrics under long document settings. for models, we highlight that the constant strive for higher rouge scores leads us to long document models with more relevant summaries but not necessarily factual ones. we also show that pe-gasus pre-training allows long document transformer to make fewer factual errors and can comprehend incoherent text better, suggesting that pe-gasus can be more beneficial than bart for reduce-then-summarize architectures that are common for long document summarizers. for metrics, we observe that rouge remains superior at assessing the relevance of summaries, while a fine-tuned bartscore can be most effective in evaluating the factuality of long document summaries. we also release the annotated dataset to encourage analysis of summarization systems across a broader range of settings. we hope that this work can provide practical insights into future research to develop long document summarization systems that can be relied upon in our daily lives.