Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries (2023)

Abstract: Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. However, a single average performance score on the entire test set is inadequate in determining such model competencies. We propose a fine-grained evaluation protocol by partitioning a test set based on the lexical similarity of reference test summaries with training summaries. We observe up to a 5x (1.2x) difference in ROUGE-2 (entity recall) scores between the subsets with the lowest and highest similarity. Next, we show that such training repetitions also make a model vulnerable to rote learning, reproducing data artifacts such as factual errors, especially when reference test summaries are lexically close to training summaries. Consequently, we propose to limit lexical repetitions in training summaries during both supervised fine-tuning and likelihood calibration stages to improve the performance on novel test cases while retaining average performance. Our automatic and human evaluations on novel test subsets and recent news articles show that limiting lexical repetitions in training summaries can prevent rote learning and improve generalization.

## introduction modern neural summarization models generate summaries that obtain high evaluation scores on automated metrics such as rouge. while models obtain high scores on average, factors beyond their summarization-related capabilities, such as learning of data artifacts, may contribute to performance. for instance, we show summaries generated by the bart @xcite , brio @xcite and pegasus @xcite ) models trained on xsum data @xcite for a recent news article in article: the american people may soon hear directly from a key witness in the house january 6 select committee's investigation who can speak to former president donald trump's approving reaction to the us capitol riot -live testimony that could be the most important moment... bart: a key witness in the house intelligence committee's russia investigation is likely to testify in person, according to a source. brio: the special counsel's investigation into russia's meddling in the 2016 election and the trump campaign's links to it is expected to return to the public spotlight this week. pegasus: a key witness in the house intelligence committee's investigation into russian interference in the 2016 election has changed her lawyer, sources have told cnn. the document been taken from xsum, the fact investigation on russia's meddling would likely be correct due to the temporal adjacency among facts in reference testing and training summaries. however, this is an undesirable behavior from a summarization model given it ignores the true fact "investigation of january 6 attack" in the source document. since most summarization datasets are created by randomly partitioning data into train, validation, and test sets, training and test summaries also contain similar distributions of summary-worthy information. therefore, we argue that the existing practice of exclusively reporting average performance on the full test set may overestimate the true capability of a model to summarize the most salient information from a document describing novel information that is unseen in training documents. similar to the example in to validate this, we divide test data into subsets with different levels of lexical similarity relative to reference training summaries. we use the percentage of 4-grams in a reference test summary that overlaps with the 4-grams in training summaries. we assume low lexical similarity between test and training reference summaries likely represents novel summary-worthy information in the corresponding test document. we, then, evaluate the current sota summarization models, bart, pegasus, brio and reina @xcite , on different test subsets. indeed, we find that all models systematically underperform on the test subset with low lexical overlap with training summaries. our fine-grained analyses ( §4) further show that higher lexical similarity between training and test reference summaries may also result in higher remembered facts in generated summaries. interestingly, since all models generate the same factual error in naively, we can improve the performance on documents with novel summary-worthy content or reduce remembered factual errors by increasing the diversity of training summaries, as demonstrated in our empirical results in § 5.2. we find that models trained on a small yet diverse subset of the training samples have fewer entity remembering issues. however, this requires removing a large proportion of training samples and sacrificing the model's performance on those summaries that are similar to reference training summaries. we aim to use train data such that the resulting summarization model performs well on novel test summaries while retaining most or all of its average performance. accordingly, we propose to first finetune a base summarization model on all training data, and then calibrate @xcite that model on a subset of training data that maximizes lexical diversity. this first stage of fine-tuning enables the models to assign high probabilities to reference summaries, reaping the maximum benefit from all training samples. for the second-stage calibration, we use the briotraining paradigm, which teaches the base model to rank generated summaries by rouge score, on a subset of the most diverse training samples. thus, we guide the model to only promote the rank of its generated summaries with higher lexical diversity and minimize repetitions across different summaries. through empirical studies in this paper, we show ## related work several summarization works @xcite @xcite have studied repeated ngrams within a generated summary and proposed various means to minimize that. recently, @xcite shows the tendency of summarization models to self-repeat, i.e., repeat certain ngrams across different summaries. however, these works exclusively focus on minimizing or quantifying ngram repetitions within or across generated summaries. on the other hand, we use ngram overlaps to identify novel reference summaries in a dataset and focus on complementary analyses, such as, how the average performance of a model correlates with the lexical similarity between reference training and testing summaries. among the works that focus on artifacts in summarization data, @xcite excludes noisy summaries from training data, @xcite defines training curriculum based on the level of noise in a training sample, @xcite uses contrastive ensembling between models trained on clean and noisy summaries, and @xcite revises noisy training summaries through an auxiliary model. they all focus on factual errors in generated summaries and study the correlation between model hallucination and training data noise. to the best of our knowledge, we are the first to analyze the correlation between the lexical diversity of training reference summaries and information correctness and relevance of generated summaries and use that to improve the summarization model's performance. ## datasets and evaluation setup in our experiments, we consider four summarization datasets, @xcite , xsum, wikihow @xcite and samsum @xcite . cnn consists of highly extractive article highlight summaries on cnn and daily mail news articles. xsum includes highly abstractive, one-sentence summaries of bbc news articles. wikihow contains articles from the wik-ihow foot_0 knowledge base, with summaries that are concatenation of paragraph outline sentences. sam-sum includes chat dialogues with human-written reference summaries. the four datasets differ in content organization structure and domain. xsum and cnn summaries are lead-biased following the inverted pyramid writing style of news articles. contrarily, source articles in wikihow and samsum datasets contain summary-worthy content throughout the text. secondly, xsum, cnn and wikihow articles are structured text written in the third-person point of view while samsum involves multiple participants talking about themselves. these variations help evaluate the generic effects of lexical repetitions in summarization models and datasets. test data partitioning using lexical overlap with training summaries: we use 4-gram overlap to partition each test dataset into subgroups with different levels of lexical similarity (or novelty) with respect to training summaries. we choose the n-gram size of four to ensure sufficient samples in all partitions while estimating similarity. we first make a list of all unique 4-grams in the training summaries. then, for a test summary, we calculate the percentage of its 4-grams that also exist in the training 4-grams list. finally, we rank the test samples based on their percent 4-gram overlap and partition them into disjoint subsets such that each subset contains a sufficient and comparable number of samples. we fix the minimum percent overlap range width for each subset to 5 and increase the width in a multiple of 5 to adjust the as train-test reference summary lexical over-lap increases, e rem increases and e p rc decreases for all models. for instance, from t nov to t sim , the percentage of remembered entities for reina increases more than 3x on the xsum data. remembered entities are akin to factual hallucinations (such as information that might be correct due to relatedness of facts in training and test reference summaries, e.g., investigation on russia's meddling in 2016 election for a document from xsum test set, fig. 1 ) and as expected they are prevalent in test cases with reference summaries having a high percent train 4-gram overlap. it is worth noting that all models obtain higher entity precision (or, generally, reproduce fewer extrinsic entity errors) on the test subsets with the lowest 4gram overlap with training summaries, indicating that models would likely be factually more consistent if the expected summaries were lexically dissimilar to the training summaries. we also perform fine-grained evaluations for bart and brio models on wikihow and sam-sum datasets, as shown in fig. 6 in appendix, observing similar trends. higher lexical overlap between the train and test reference summaries leads to higher r2, e rec and e rem scores. on sam-sum, however, e rem decreases slightly with the increase in the lexical overlap. this may not be surprising given the smaller size of the samsum dataset (10-20x smaller than other datasets) and the very low lexical overlap between the training and test reference summaries. implications for modelling: we found that all models perform consistently worst on the t nov . analyzing brio and reina on xsum, neither model improves r2 over pegasus on the t nov . while brio uniformly improves in r2 over pegasus on the remaining xsum test subsets other than the t nov , reina obtains relatively higher r2 gain on test subsets with very high train-lexical overlap. similarly, on e rem , reina and brio models perform comparably on test subsets with lower train-lexical overlap, but as the train-lexical overlap increases, reina generates more remembered entities. this is not surprising since reina retrieves and uses training summary as context. such retrieval-based models are likely to bias the generated summaries toward the retrieved training summary. our fine-grained analysis, thus, provides deeper insight into a model's capabilities, e.g., a higher average performance for reina on xsum results from improvement on test subsets that are similar to training summaries, and it may obtain higher performance scores only if the reference summaries are expected to be lexically similar and containing the same factual information (or hallucination) as the training summaries. ## effect of lexical repetitions in reference training summaries in section §4, we show that high lexical overlap between the training and test reference summaries drives high average performance for all neural summarization models. the lower r2 and e rec on t nov or higher e rem and low e p rc on t sim suggest that all models are somewhat memorizing and generating information in reference training summaries and learning the dataset or its artifacts besides learning the task-related summarization capabilities. given these findings, what factors are driving the high-performance gap between t nov and t sim ? intuitively, the above behavior of models could be a result of lower information diversity in training data. to validate our hypothesis and study model behavior relative to training data diversity, we evaluate the effects of training 4-grams repetitions ( §5.1) in both model fine-tuning ( §5.2) and likelihood calibration ( §5.3). we also evaluate the effects of 4-grams repetitions in zero-shot settings, described in appendix §b.1. ## discussion: the performance trade-off and human evaluation from our experiments in § 5.2 and § 5.3, we observe that using all training samples is the most plausible choice for obtaining the best average performance. however, the improvement is not uniform across all test subsets. further, the intervention-based analysis on xsum shows that higher ngram repetitions make a model learn and reproduce data artifacts. to determine what makes the best strategy for utilizing the available training data, we perform a human study to compare the generic summarization capabilities of different models. we use pairwise evaluation between supervised models fine-tuned on θ 4g of 25 and all data for each of the xsum and cnn datasets. we also compare brio-25 and brio-all models trained on each of the xsum and cnn datasets. we choose θ 4g of 25 for comparison since the resulting models perform the best (or comparable to the model trained on all data) on t nov for both xsum and cnn for both fine-tuning and calibration. we conduct our study on 100 recent cnn articles from @xcite to maximize temporal exclusivity of information (events and entities) and minimize the influence of train-lexical overlap on relevance and correctness of generated summaries. this also mimics the practical use-case scenario where there is a temporal separation between the training and test data. since the 100 articles do not have reference summaries, we first ask our annotators (three authors) to highlight the most relevant information for each document, analogous to prior reference-free human evaluation @xcite . we compare the number of document highlights that are present in summaries generated by the two systems. if both systems include the same number of highlights, both systems are labeled equivalent on relevance. otherwise, the system that generated more highlighted information is rated better on relevance. for consistency, we compare the number of hallucinated facts in summaries generated by two systems and rank the system with fewer hallucinations as better. the inter-annotator agreements for relevance and consistency are 0.64 and 0.82 @xcite , respectively. we report the percentage of documents for which each model is rated better than the other in first, we observe that models fine-tuned on d θ 4g =25 generate more consistent summaries compared to the models that use all training data. this corroborates our results based on automated entity precision metric (e p rc ) as well as interventionbased analysis on xsum. on the relevance scale, we can observe that xsum-25 (ft) generates more relevant summaries compared to the xsum-all (ft), suggesting that controlling lexical repetitions in training data with artifacts is a superior technique to improve model performance. meanwhile, cnn models trained on d θ 4g =25 or all data achieve similar results. the main reason could be that the cnn dataset has fewer artifacts. in addition, comparing the results of brio-25 and brio-all on both cnn and xsum datasets, we see that controlling lexical repetitions during calibration is an effective strategy. this is also reflected in the results in § 5.3. another advantage to conducting the calibration step with fewer but lexically diverse training samples is that it is computationally more expensive than the supervised fine-tuning @xcite . thus, our proposed solution can achieve uniform improvement across all test subsets and also minimize the computational cost. ## conclusion in this work, we propose a fine-grained evaluation protocol by partitioning a test set based on the lexical similarity between reference test summaries and training summaries to evaluate the generalization capabilities of summarization models. our comparative evaluations of current sota summarization models show that they all systematically underperform in generating novel summary-worthy content. next, we show that higher lexical repetitions in training data contribute to this phenomenon. in addition, training repetitions make models vulnerable to learning data artifacts. finally, through a series of automatic and human evaluations, we show that controlling the lexical diversity of training data at different stages of training can help a model generate more relevant and consistent summaries on novel test subsets.