Can LM s Generalize to Future Data? An Empirical Analysis on Text Summarization (2023)

Abstract: Recent pre-trained language models (PLMs) achieve promising results in existing abstractive summarization datasets. However, existing summarization benchmarks overlap in time with the standard pre-training corpora and finetuning datasets. Hence, the strong performance of PLMs may rely on the parametric knowledge that is memorized during pre-training and fine-tuning. Moreover, the knowledge memorized by PLMs may quickly become outdated, which affects the generalization performance of PLMs on future data. In this work, we propose TempoSum, a novel benchmark that contains data samples from 2010 to 2022, to understand the temporal generalization ability of abstractive summarization models. Through extensive human evaluation, we show that parametric knowledge stored in summarization models significantly affects the faithfulness of the generated summaries on future data. Moreover, existing faithfulness enhancement methods cannot reliably improve the faithfulness of summarization models on future data. Finally, we discuss several recommendations to the research community on how to evaluate and improve the temporal generalization capability of text summarization models.

## introduction abstractive summarization aims to generate a concise summary that contains the critical information of the source text while ensuring the generated text is fluent and faithful. in recent years, pre-trained language models (plms) have accomplished state-of-the-art outcomes across numerous summarization benchmarks @xcite . nonetheless, the test sets within these benchmarks are predominantly comprised of data samples prior to 2019, resulting in a substantial temporal overlap with both the training sets and news article from cnn in 2015: add vice president joe biden to the list of potential 2016 candidates traveling to the first-in-the-nation caucus state. biden will travel on official white house business next week to des moines, iowa where he will deliver a speech on the administration' s economic policies, his office announced friday ... ## temposum benchmark we first propose the temposum benchmark to investigate the temporal generalization capability of plms on future documents that present knowledge conflicts, i.e., the scenarios where the knowledge presented in the source document contradicts the parametric world knowledge in plms. ## result analysis we illustrate the distribution of hallucinations made by different models on the bbc and cnn datasets in figure 2a and figure 2b , respectively. from these two figures, we draw the following observations. pre-training improves the faithfulness of a summarization model on future data. we observe that pegasus produces significantly fewer hallucinations than its non-pretrained counterpart in the future test sets of both bbc and cnn datasets. the results suggest that pre-training helps a summarization model generates more faithful summaries even if the source articles contain knowledge conflicts. pre-training encourages a summarization model to hallucinate outdated information based on its parametric knowledge. we observe that on the future test set of bbc, pegasus generates substantially more outdated hallucinations than transformer (39.81% vs. 33.33%). while on the future test set of cnn, pegasus and transformer obtain a similar number of outdated hallucinations (3.94% vs 4.72%). our human evaluation results ## recommendations based on the observations from our experiments, we recommend the following four takeaways to the summarization research community. 1. we should use temporally split test sets that contain knowledge conflicts to evaluate the performance of summarization systems. this kind of test set allows us to isolate the effects of parametric world knowledge to accurately evaluate the temporal generalization performance of text summarization. ## conclusion in this paper, we propose temposum, a novel evaluation benchmark for assessing the temporal generalization capability of text summarization models. we collect samples from a broad temporal range which allow us to study the performance of plms on data from different temporal periods. through extensive experiments on temposum, we find that plms are vulnerable to generating summaries with outdated hallucinations and ignore the supporting evidence provided by source articles. it is crucial to develop techniques to prevent summarization systems from generating outdated information and design automatic evaluation metrics to detect outdated hallucinations. temposum provides a valuable data corpus which covers a wide range of temporal periods, along with human judgements from various summarization systems, to support future studies in this direction.