Entropy-based Sampling for Abstractive Multi-document Summarization in Low-resource Settings (2023)

Abstract: Research in Multi-document Summarization (MDS) mostly focuses on the English language and depends on large MDS datasets that are not available for other languages. Some of these approaches concatenate the source documents, resulting in overlong model inputs. Existing transformer architectures are unable to process such long inputs entirely, omitting documents in the summarization process. Other solutions address this issue by implementing multi-stage approaches that also require changes in the model architecture. In this paper, we introduce various sampling approaches based on information entropy that allow us to perform MDS in a single stage. These approaches also consider all source documents without using MDS training data nor changing the model’s architecture. Besides, we build a MDS test set of German news articles to assess the performance of our methods on abstractive multi-document summaries. Experimental results show that our entropy-based approaches outperform previous state-of-the-art on German MDS, while still remaining primarily abstractive. We release our code and MDS test set to encourage further research in German abstractive MDS.

## introduction in light of the ever-growing volume of available information, it becomes essential to be able to automatically summarize information from several sources. multi-document summarization (mds) aims at condensing the most important information from different documents. despite the advances in single-document summarization @xcite , summarizing multiple related documents remains a greater challenge due to its input length and the presence of redundant information @xcite . therefore, some research focuses on implementing multi-stage approaches 1 link to github repository. 2 link to multi-genews repository. multi-document summarization some prior work approaches mds as a multi-stage process @xcite that first extracts salient sentences from the source documents to then distill them into a summary using different methods such as graph-based modeling @xcite or modifying the attention mechanism (perez-beltrachini and lapata, 2021). in contrast, @xcite highlights the importance of adapting single-document summarization (sds) models to summarize multiple documents and propose an approach that adapts their attention weights. similarly, other works propose various changes in the model architecture @xcite . the main disadvantage of these approaches is that they are tailored to specific model architectures. more recently, @xcite introduce primera, a pre-trained model for mds that can be applied in zero-or few-shot settings. the source documents are concatenated and fed into the longformer transformer model, which can handle long inputs up to 4,096 or even 16k tokens with current gpus. nevertheless, primera is only available for english and there is no alternative for other languages. similarly, @xcite performs mds on german text using the pre-trained language model bart @xcite and concatenating the source documents as input. 3 however, bart input length is restricted to 1,024 tokens, which may end up excluding entire documents from the summarization process. overall, our entropy-based approaches present the following advantages over prior work: (a) they do not require a pre-step to extract salient information (b) nor changes in the sds model architecture with (c) no need for additional mds training data, and (d) still considering all source documents in the summarization process. this work is built upon the dynamic ensemble approach from @xcite , improving the decoding strategy by sampling on more informative predictions. ## entropy background in information theory, the entropy of a random variable denotes the amount of information, or lack thereof (i.e. uncertainty), associated with its possible outcomes. thus, given a probability distribution p over all possible outcomes x 1 , . . . , x n of a random variable x, we quantify the entropy of x using the standard shannon entropy equation: the entropy is then maximum for uniform distributions, where all outcomes are equally likely, indicating high uncertainty. in the context of automatic text generation, we can leverage entropy to quantify the confidence of probabilistic models in their predictions @xcite . more specifically, summarization models aim at generating a summary string y * of a given source document x that maximizes the scoring function: ## entropy-based mds given a set of documents @xmath0, . . . , x n }, the dynamic-ensemble approach (dyne) described in @xcite adapts single-to multidocument summarization as follows: at every decoding timestep t, it computes the output probabilites for each individual source document using a single-document summarization model; next, it averages these outputs to obtain a single logprobability distribution assigned to the token y: we leverage entropy information to adapt the dyne approach and implement various sampling strategies that select the most informative output at each decoding timestep t. ## datasets this section describes the datasets used to train and evaluate our mds approaches. specifically, we consider three pre-existing german datasets that are suitable for single-document-gewiki @xcite and 20m @xcite )-and multidocument summarization-auto-hmds @xcite . moreover, we build multi-genews, a mds test set in the news domain that is specifically tailored for abstractive mds. ## experiments we evaluate the performance of the entropy-based sampling approaches on our multi-genews and the auto-hmds datasets in terms of automatic rouge scores @xcite and extractive fragment density ρ @xcite . since we focus on abstractive summarization, the latter allows us to measure the degree of extractiveness of the summaries, and in turn, abstractiveness-higher ρ values indicate that the summary is more extractive and contains larger text chunks from the source article. furthermore, we collect human annotations on a subset of the multi-genews to assess the faithfulness of the generated summaries and get a deeper understanding on their quality (section 7). ## human evaluation we recruited three native german speakers to perform a manual evaluation on the multi-genews test set. 16 this evaluation task is twofold: (1) assess the relative quality of the summaries, ranking them accordingly @xcite and (2) the faithfulness of the generated summaries to the source articles @xcite , that is, whether the information presented in the summaries is supported by the articles. since the task requires to read a considerable amount of text, we ask the participants to annotate a sample of the mds test set. this sample comprises 20 randomly selected instances that meet the following criteria: (a) each instance consists of three source articles, (b) the generated summaries end with a punctuation mark to avoid incomplete sentences, and (c) the token-level edit distance among the summaries is above five to ensure lexical differences. for each participant, we randomly shuffle the evaluation instances and the required annotations to avoid any biases. additionally, we do not provide them with any information about which specific approach generated each summary. summary ranking task the objective of this task is to gain insights into human preferences of the generated summaries. for each instance (i.e. a set of related articles), we ask the participants to rank the generated summaries according to the 16 the participants received a voucher worth chf 75.-as compensation for their participation. this task is specially challenging when multiple summaries either contain similar information or suffer from hallucinations, or both. in fact, the annotators reported that it was hard to decide the rank of a summary between two consecutive positions. this negatively impacted on the inter-annotator agreement, resulting in a final kendall's tau coefficient of 0.22. in the analysis of this task, we concentrate on the relative performance of the approaches and only consider instances with a majority agreement among annotators. figure 2a illustrates the relative preference among the different approaches in this ranking task. the results demonstrate that the summaries from the approaches h min and h th are consistently rated higher than the others. in contrast, the h pmi summaries are the least preferred. these results are also supported in figure 2b , where we compare the frequency with which each approach was ranked within the top two and the bottom two positions. furthermore, figure 2b shows that while the base faithfulness annotation task van der poel et al. since h pmi is an enhanced version of h th , and h th is consistently preferred over h pmi (figure 2 ), we delve deeper into cases where h pmi shows an improvement in factuality, yet it receives a lower rating than h th . the results indicate that h pmi indeed redirects the prediction of the rest of the summary, specially when applied early on as stated in @xcite . however, it does not necessarily address the issue of hallucination. for example, the first text span of h th in ## conclusion in this work, we tackle multi-document summarization (mds) in low-resource settings where there is a lack of mds training data. we therefore present various sampling approaches built upon prior works that use single-document summarization models for the mds task. specifically, we leverage information entropy as a metric to measure the model certainty in each token prediction. the experimental results on german mds show that our h th approach, which specifically applies maximum probabilitic information entropy, achieves the state-of-art in german abstractive mds. in our experiments, we also assessed an extended version of the h th approach that applies pointwise mutual information (pmi) when all predictions exhibit uncertainty. although pmi has been used in prior work to address hallucination, we observe in the manual evaluation that pmi changes the prediction of the rest of summary, but it does not inherently tackle hallucination. future work should focus on addressing the issue of hallucination in automatic summarization, including further research on the efficacy of pmi to mitigate hallucinations. additionally, it would be interesting to explore alternative approaches to enhance the h th approach when there is uncertainty in the prediction. finally, we built a mds test set of german news articles that will help the research community to evaluate abstractive mds on german text.