ROUGE -K: Do Your Summaries Have Keywords? (2024)

Abstract: Keywords, that is, content-relevant words in summaries play an important role in efficient information conveyance, making it critical to assess if system-generated summaries contain such informative words during evaluation. However, existing evaluation metrics for extreme summarization models do not pay explicit attention to keywords in summaries, leaving developers ignorant of their presence. To address this issue, we present a keyword-oriented evaluation metric, dubbed ROUGE-K, which provides a quantitative answer to the question of – How well do summaries include keywords? Through the lens of this keyword-aware metric, we surprisingly find that a current strong baseline model often misses essential information in their summaries. Our analysis reveals that human annotators indeed find the summaries with more keywords to be more relevant to the source documents. This is an important yet previously overlooked aspect in evaluating summarization systems. Finally, to enhance keyword inclusion, we propose four approaches for incorporating word importance into a transformer-based model and experimentally show that it enables guiding models to include more keywords while keeping the overall quality.

## introduction summarization systems compress long documents into shorter ones to convey important information more effectively to readers @xcite . to convey all essential information correctly, it is crucial for summarization systems to include important, i.e., summaryrelevant keywords. in our analysis, human annotators find that summaries with more keywords, words that are relevant for the summary (see section 3), capture important information better than the ones with fewer keywords. however, existing 1 our code: https://github.com/sobamchan/rougek . reference a novel, hybrid deep learning approach provides the best solution to a limited-data problem (which is important to the conservation of the hawaiian language) r-1 bs hypothesis 1: in this paper, to shed light on this problem, we propose rouge-k, an extension of rouge which considers only those n-grams in the summaries that match a set of pre-defined keywords. we propose a simple heuristic that exploits the common structure of summarization datasets to extract keywords automatically, making it possible for our metric to scale in size and domain without additional annotation effort. correlation analysis reveals that there is only a weak strength of dependence between our new metric and existing ones as well as summary lengths, thus demonstrating that our approach can complement, rather than replace, previous metrics. through a manual evaluation, we find that human annotators show substantially higher agreement with rouge-k than with rouge and bertscore on relevance, in other words, how well summaries include important information, which is one of four commonly assessed aspects in manual evaluations of summaries @xcite . this shows that while one still would use traditional rouge to assess the overall qualities, our metric can provide a better index for evaluating the relevance of summaries. as a showcase of this new metric, we evaluate both extractive @xcite and abstractive @xcite @xcite state-of-the-art models on two extreme summarization datasets from different domains, namely xsum @xcite and scitldr @xcite , as well as a more traditional, non-extreme dataset, scisummnet @xcite . besides news text (xsum), we choose summarization of scientific publications (scisummnet and scitldr), since this is a domain where keyword inclusion within summaries plays a crucial role. surprisingly, the results reveal that these strong baseline models often fail to include essential words in their summaries, and that rouge-k enables us to better distinguish systems' performance than alternative metrics. we also apply our rouge-k to the evaluations of recent large language models (llms) and show how our metric better accounts for the powerful capabilities of llm-based summarizers when compared to traditional rouge metrics. finally, as a first attempt to address the limitations on summary keyword inclusion, we introduce four ways to incorporate a lightweight word importance feature into existing transformer-based models. experiments show that our methods can guide models to include more keywords without any additional annotations and negative effects on overall summarization quality. our contributions are the following ones: the need for another kind of rouge rouge @xcite is a long-running de facto standard evaluation metric for summarization systems. it is very popular due to its high correlation with human evaluation while keeping its simplicity and interpretability. however, several works report on its limitations @xcite , one of which is that it only takes the word surface into account and disregards semantics @xcite . because it considers all n-gram matches to be equally important, rouge fails to detect salient words that underpin a summary's quality. as an example, scitldr @xcite we show that autoregressive models can generate high fidelity images. we introduce a new inductive bias that integrates tree structures in recurrent neural networks. xsum @xcite opec, the oil producers' group is back in the driving seat. lenovo and acer have both unveiled smartphones with much larger than normal batteries. ## rouge-k we present rouge-k, an extension of rouge that exclusively focuses on essential words in summaries. its core idea is simple: rouge-k assesses the proportion of keywords from the reference summary that are included in the candidate summary. we compute rouge-k as: ## importance-guided summarization we next propose four ways to incorporate a soft guiding signal into bart to enforce the inclusion of keywords into the generated summaries. re-weighted encoding (rwenc). the first approach is to modify the representations within the model with tf-idf scores. concretely, we compute the attention matrix in transformer layers as: the first term within the softmax function is from the original transformer @xcite where q and k are query and key matrices respectively, and we introduce the second term t which is a matrix of tf-idf scores over the input text. this enhances the model to propagate higher values for the important words to the upper layers. we apply this modification to the 0-, 4-, 8-th encoder layers, empirically selected on the dev data. ## related work in the context of factual consistency evaluation @xcite @xcite , @xcite propose to use a ner model to detect hallucinated named entities in summaries. while their approach also focuses on specific words in summaries as our rouge-k, it is limited because (1) not all critical information consists of named entities, (2) strong ner models are not available in many domains, and (3) ner performance is unknown for summarization datasets. @xcite and @xcite propose to use vector representations to compute semantic similarity between reference and candidate summaries. @xcite instead propose to use a question-answering system to assess the summary quality. while these methods can exploit semantic knowledge stored in parameters in large models, as a side-effect, they introduce 'blackboxness' that hinders transparent model development. in contrast, we take a 'bottom-up' approach by proposing to focus on keyword availability. ## conclusion in this paper, we proposed rouge-k, an extension of rouge to quantify how summary-relevant keywords are included in summaries. using rouge-k, we showed human annotators prefer summaries with more keywords and how models often miss several essential keywords in their output. in a variety of experiments using the baseline provided by a large pre-trained language model (bart) we showed how rouge-k only moderately correlates with rouge and bertscore, thus indicating that it can complement them, and correlates with the length of the generated summaries on a par with rouge f1 and bertscore, despite being a recalloriented metric. finally, we proposed four ways to guide bart to include more keywords in its sum-maries. we plan in future work to further test our metric's applicability across different domains and languages, e.g., by relying on wikilingua (ladhak et al., 2020) and x-scitldr @xcite .