Length Does Matter: Summary Length can Bias Summarization Metrics (2023)

Abstract: Establishing the characteristics of an effective summary is a complicated and often subjective endeavor. Consequently, the development of metrics for the summarization task has become a dynamic area of research within natural language processing. In this paper, we reveal that existing summarization metrics exhibit a bias toward the length of generated summaries. Our thorough experiments, conducted on a variety of datasets, metrics, and models, substantiate these findings. The results indicate that most metrics tend to favor longer summaries, even after accounting for other factors. To address this issue, we introduce a Bayesian normalization technique that effectively diminishes this bias. We demonstrate that our approach significantly improves the concordance between human annotators and the majority of metrics in terms of summary coherence.

## introduction text summarization aims to condense lengthy documents into concise, coherent, and human-readable texts while preserving essential ideas. deep learning and large-scale datasets have substantially advanced this field, as demonstrated by models @xcite @xcite and datasets @xcite @xcite . progress in this field hinges on reliable metrics for evaluating generated summary quality. while human annotations are considered the gold standard, they can be costly, time-consuming, and subjective. automated evaluation metrics address these challenges, but assessing summary quality remains complex. efforts to improve text summarization metrics are ongoing (e.g., @xcite @xcite ). however, biases concerning generated summary length remain underexplored. if certain metrics favor shorter or longer summaries, evaluations 1 the code is available at sldbias may be flawed, as they should focus on quality, not extraneous factors. in this paper, we investigate the impact of generated summary length on 14 distinct metrics. our experiments reveal that: 1) the length of generated summaries affects most metrics to varying extents; 2) although some discrepancies result from the correlation between quality and summary length, they persist even after controlling for quality; 3) these effects are consistent across datasets, but the magnitude of the effects may differ; 4) metrics based on gram overlap (e.g., rouge) are more inclined to assign higher scores to longer summaries than metrics based on word/sentence embeddings (e.g., bertscore). in response to these findings, we propose a bayesian normalization strategy to diminish the influence of summary length on metrics. we demonstrate that our approach significantly improves alignment with human annotators in terms of summary coherence for the majority of metrics. ## related work numerous studies have explored the limitations of automatic metrics for text summarization, addressing issues like weak correlation with human judgment, poor adaptability to diverse corpora, and failure to capture linguistic nuances. for instance, @xcite found a modest correlation between human judgment and most metrics, while @xcite showed rouge metrics struggle with varied terminology and paraphrasing. cross-language studies @xcite demonstrated suboptimal performance for some metrics on non-english corpora. additionally, @xcite and @xcite critiqued automatic metrics for not detecting factual inconsistencies. a fundamental characteristic of a robust metric for this task is the capacity to consistently rate summaries of equal quality, even when they differ along other dimensions. prior research, such as @xcite ; @xcite , highlighted variable average scores for some metrics depending on reference/generation lengths. however, they neither proposed debiasing methods nor extended their focus beyond rouge and bartscore. @xcite addressed length bias by comparing generated text with a randomly chosen, length-controlled extractive summary. although this method can partially mitigate length bias, it is limited in applicability to various summarization techniques and has only been tested on the rouge family of metrics. nonetheless, we benchmark our method against theirs, demonstrating consistent superiority across multiple summarization approaches and metrics. ## the impact of length on metrics here, we investigate the effect of generated summary length on 14 summarization metrics, including both lexical-based and embedding-based metrics @xcite . we conduct experiments using three popular datasets: cnn/daily mail corpus (cnn/dm) @xcite , wikihow @xcite , and webis-tldr-17corpus (web-tldr) @xcite , with summaries generated by three models: bart @xcite , longformer @xcite , and t5 @xcite . more details can be found in appendix a. to isolate the impact of generated summary length, we control for the source article and reference summary lengths (appendix b shows that both factors also bias metrics). using equal frequency binning, we categorize samples based on source-article and reference lengths (10 buckets each, totaling 100 buckets). within each bucket, we compute the mean score for each five percentile of the generated summary length distribution, then average the scores across the 100 buckets to calculate the mean score for that percentile. as our study focuses on percentile rankings, the varying distribution of generated summary lengths is not a concern. for clarity, we shift scores by the minimum score for each metric in our figures. figure 1 shows the trend for bart (results for other models in appendix d). even after accounting for source article and reference lengths, most metrics are influenced by generated summary lengths to varying degrees. we also observe that the trends are generally consistent across datasets, with the exception of bertscore, which exhibits divergent behavior among different datasets. figure 1 : average metric values for varying generated summary lengths, controlling for the length of the source article and reference. note that scores have been shifted by the minimum value, resulting in the lowest score being 0. for clarity, scores have been rescaled from a 0-1 range to a 0-100 range. next, we control for summary quality to eliminate any correlations between generated summary length and quality by randomly shuffling generated summaries among samples, thus randomizing reference-summary pairings. we repeat our experiments with this modified dataset and display the results for bart in figure 2 (see appendix d for other models). while weaker, the trends persist for several metrics, indicating that metric length preference primarily drives these trends. interestingly, bertscore exhibits an opposite trend, potentially due to its reliance on text semantics, which may become diluted in longer text. ## modeling the effect of length we quantitatively examine the impact of generated summary length on various metrics using a bayesian network, considering four variables: source article length, reference summary length, generated summary length, and score. the bayesian network structure is shown in figure 3 . in our experiments, we manually design the structure and provide the bayesian network with the joint distribution of all variables based on text samples from our data. once trained, the network can be employed to calculate the expected score ŝ, given the source article length (l a ), reference length (l r ), and generated summary length (l g ), using the following equation: we use the trained networks to quantify the effect of generated summary length on each metric's score. for each sample, we keep l a and l r constant, increment l g by one (ten percentile) foot_0 , and input the updated values into the bayesian networks to predict new scores for each metric. we average the difference between these new scores and the original corresponding scores across all samples, yielding the mean score difference due to a tenpercentile increase in generated summary length. we experiment with all 14 metrics across three datasets and three models, using both shuffled and non-shuffled data. trends remain consistent across different models, with only one instance of disagreement (rouge-3 for the web-tldr dataset). comparing lexical-based and embedding-based metrics, we find embedding-based metrics are generally less prone to score increases with increasing generated summary length. non-shuffled experiments display higher sensitivity, on average, to generated summary length, suggesting other unknown factors may also influence scores. a limitation of our work is the presence of such unaccounted-for factors. our experiments in sections 3 and 4 conclude that: 1) controlling for source article length, reference summary length, generated summary quality, model, and dataset, generated summary length appears to bias most metrics, with longer summaries resulting in higher scores in a non-linear relationship; 2) trends are generally consistent across datasets and models, but effect sizes vary; 3) the relationship between generated summary length and score can be partially attributed to metrics' preference for longer text, but not all variance is accounted for, indicating other unknown variables contribute to the relationship. ## reducing length bias here, we explore strategies for mitigating the metrics' bias concerning the length of generated summaries and generating a length-adjusted score. intuitively, if all generated summaries had the same length as their corresponding reference summaries, the length would no longer be a confounding factor. since we do not want to control the length of the generations, we adjust the scores for a generated summary post hoc. each metric is adjusted based on the corresponding bayesian network's predicted score for a generated summary with the same length as the sample's reference length while keeping every other variable assignment the same as the sample. we employ a bayesian network to better capture the non-linear relationships between our variables. the bayesian network used in this section is identical to the previous network, except for ## conclusions & future work this paper investigates the relationship between generated summary length and 14 summarization metrics. our findings reveal a correlation between generated summary length and metric scores, even after accounting for generation quality, source article length, reference summary length, model, and dataset. we propose a bayesian-network-based approach to adjust metric scores based on generated summary length, resulting in an improved agreement between model-generated and coherencebased human rankings for most metrics. future work could examine new metrics for summarization tasks, such as conditioning on target length or measuring "information density" rather than absolute information.