Supervising the Centroid Baseline for Extractive Multi-Document Summarization (2023)

Abstract: The centroid method is a simple approach for extractive multi-document summarization and many improvements to its pipeline have been proposed. We further refine it by adding a beam search process to the sentence selection and also a centroid estimation attention model that leads to improved results. We demonstrate this in several multi-document summarization datasets, including in a multilingual scenario.

## introduction multi-document summarization (mds) addresses the need to condense content from multiple source documents into concise and coherent summaries while preserving the essential context and meaning. abstractive techniques, which involve generating novel text to summarize source documents, have gained traction in recent years @xcite @xcite , following the advent of large pre-trained generative transformers. however, their effectiveness in summarizing multiple documents remains challenged. this is attributed not only to the long input context imposed by multiple documents but also to a notable susceptibility to factual inconsistencies. in abstractive methods, this is more pronounced when compared to their extractive counterparts due to the hallucination-proneness of large language models. extractive approaches, on the other hand, tackle this problem by identifying and selecting the most important sentences or passages from the given documents to construct a coherent summary. extractive mds usually involves a sentence importance estimation step @xcite @xcite , in which sentences from the source document are scored according to their relevance and redundancy with respect to the remaining sentences. then, the summary is built by selecting a set of sentences achieving high relevance and low redundancy. the centroid-based method @xcite is a cheap unsupervised solution in which each cluster of documents is represented by a centroid that consists of the sum of the tf-idf representations of all the sentences within the cluster and the sentences are ranked by their cosine similarity to the centroid vector. while the original method is a baseline that can be easily surpassed, subsequent enhancements have been introduced to make it a more competitive yet simple approach @xcite @xcite . in this work, we refine the centroid method even further: i) we utilize multilingual sentence embeddings to enable summarization of clusters of documents in various languages; ii) we employ beam search for sentence selection, leading to a more exhaustive exploration of the candidate space and ultimately enhancing summary quality; iii) we leverage recently proposed large datasets for multidocument summarization by adding supervision to the centroid estimation process. to achieve this, we train an attention-based model to approximate the oracle centroid obtained from the ground-truth target summary, leading to significant rougescore improvements in mono and multilingual settings. to the best of our knowledge, we are the first to tackle the problem within a truly multilingual framework, enabling the summarization of a cluster of documents in different languages. foot_0 2 related work typical supervised methods for extractive summarization involve training a model to predict sentence saliency, i.e. a model learns to score sentences in a document with respect to the target summary, either by direct match in case an extractive target is available or constructed @xcite @xcite or by maximizing a similarity score (e.g., rouge) with respect to the abstractive target summaries @xcite . attempts to reduce redundancy exploit the notion of maximum marginal relevance (mmr; @xcite or are coverage-based @xcite , seeking a set of sentences that cover as many concepts as possible while respecting a predefined budget. during inference, the model is then able to classify the sentences with respect to their salience, selecting the highest-scored sentences for the predicted summary. rather than training a model that predicts salience for each individual sentence, we employ a supervised model that directly predicts an overarching summary representation, specifically predicting the centroid vector of the desired summary. training this model can thus be more direct when training with abstractive summaries (as is the case in most summarization datasets), since computing the reference summary centroid is independent of whether the target is extractive or abstractive. regarding enhancements to the centroid method for extractive mds, @xcite refined it by substituting the tf-idf representations with word2vec embeddings @xcite , and further incorporated a redundancy filter into the algorithm. gholipour ghalandari (2017), on the other hand, retained the utilization of tf-idf sentence representations but improved the sentence selection process. recently, @xcite introduced modifications to the sentence scoring mechanism, incorporating novelty and position scores, and evaluated a diverse array of sentence embeddings with the proposed methodology, including contextual embeddings provided by elmo @xcite and bert @xcite . while there have been initiatives to foster research in multilingual extractive mds @xcite , the proposed approaches @xcite @xcite are only language-agnostic, requiring all the documents within each cluster to be in the same language. in contrast, we address extractive mds in a scenario where each cluster is multilingual. ## methodology the pipeline of our proposed model is divided into two stages. in the first stage, we use an attention model to obtain a cluster representation that replaces the naive centroid obtained by averaging sentence embeddings of the documents in a cluster. the rationale behind this approach is that the contribution of each sentence to the cluster centroid should depend on its relevance to the cluster summary. in order to capture the whole cluster context, a sentence-level attention model is employed, assigning variable weights to each sentence embedding so as to approximate the resulting average to the centroid that would be obtained by averaging the sentence embeddings of the target summary. in the second stage, an adapted version of the greedy sentence selection algorithm from gholipour ghalandari (2017) for extractive mds is used to select the sentences included in the predicted summary. this adapted version uses our proposed supervised centroid and also includes a beam search algorithm to better explore the space of candidate summaries. ## experimental setup herein, we outline the methods, datasets, and evaluation metrics employed in our experiments. ## results monolingual setting the rouge-2 recall (r2-r) of all the methods in the monolingual datasets are presented in ## conclusions we enhanced the centroid method for multidocument summarization by extending a previous approach with a beam search followed by a greedy search. additionally, we introduced a novel attention-based regression model for better centroid prediction. these improvements outperform existing methods across various datasets, including a multilingual setting, offering a robust solution for this challenging scenario. regarding future work, we believe an interesting research direction would be to further explore using the supervised centroids obtained by the cera and cerai models, by having them as a proxy objective to obtain improved abstractive summaries.