Which Information Matters? Dissecting Human-written Multi-document Summaries with Partial Information Decomposition (2024)

Abstract: Understanding the nature of high-quality summaries is crucial to further improve the performance of multi-document summarization. We propose an approach to characterize human-written summaries using partial information decomposition, which decomposes the mutual information provided by all source documents into union, redundancy, synergy, and unique information. Our empirical analysis on different MDS datasets shows that there is a direct dependency between the number of sources and their contribution to the summary.

## introduction multi-document summarization (mds) consists of providing an abridged version of multiple documents. while some abstractive summarization approaches concatenate all documents into a single input @xcite , the large size of input text represents a major challenge in mds. therefore, most methods implement two-stage approaches that extract salient text spans based on different heuristics @xcite @xcite . the text is then fed into a summarization model under the assumption that a high-quality summary is based on such information. while earlier work quantifies the properties of human-written mds based on n-gram matching @xcite , we argue that there is a lack of indepth analyses. without an understanding of the nature of summaries, improving the quality of mds remains vague and without clear interpretability. this work sheds light on what information constitutes a high-quality multi-document summary. in particular, we propose to categorise the summary information into information provided by at least one source (union) or by a unique source, redundant information from all source documents, and even new information derived from considering them jointly (synergy). in information theory, partial information decomposition (pid) decomposes information in the same way to assess how information about a target is distributed among multiple source variables @xcite . we therefore implement pid in mds and present spider; a novel approach to quantify the degree to which the pid components-such as redundancy or unique information-contribute to a summary. 1 we then perform an empirical analysis on human-written summaries from different mds datasets using our approach. our results demonstrate that the number of sources has a direct dependence on how they contribute to the summary. we also show that, surprisingly, the order of the source documents matters, and the first three documents are frequently considered as the main source of unique information for any number of sources. to the best of our knowledge, we present the first fine-grained information analysis in human-written mds. we open-source spider foot_1 and hope that it helps to enhance the performance of future mds methods. we suggest that our pid approach could also be used to automatically build mds datasets that align with human quality in future work. ## information theory background mutual information (mi) @xcite has been widely used in nlp tasks to quantify the information that a source provides about an output @xcite @xcite @xcite . however, mutual information is insufficient in the context of mds, as it can only be applied to pairs of random variables. partial information decomposition (pid) tackles the multivariate problem and decomposes the information that a set of sources conveys about a target into union, redundancy, unique, and synergistic information @xcite . to date, pid has only been applied in nlp to measure morphological fusion in @xcite . figure 1 illustrates the relationships between the different pid components for two sources, x 1 and x 2 , and a target y . see a textual example in in this work, we consider the pid approach proposed in @xcite , which is based on the definitions of union and intersection from set theory. in contrast to previous work, kolchinsky (2022)'s approach can be applied to any number of sources, a crucial requirement in our mds setting. formally, given a set of source random variables @xmath0, ..., x n } and a target y , redundancy i ∩ (@xmath1. this redundancy is the maximum information q we can obtain about y that is less informative than any of the sources: finally, unique and synergistic information are derived from eq. ( 1 ) and eq. ( 2 ), respectively. ## decomposing information in mds we adopt the pid approach in @xcite to mds, considering sentences as units of infor-source x1 kimchi is fermented cabbage. source x2 fermented foods are rich in probiotics. target y fermented foods, such as kimchi, are rich in probiotics. unique x1 the nature and preparation of kimchi. unique x2 a general characteristic of fermented foods. redundancy information about fermented. synergy inferring that kimchi is a fermented food. we then define unique information and synergy using eq. ( 3 ) and (4) as in kolchinsky (2022) ## pid of human-written summaries we analyze the information components of humanwritten summaries from multiple mds datasets using our framework. specifically, we consider a random sample of 100 instances per dataset and number of sources, if available (see ## measuring synergistic information since synergy is negligible in the analyzed mds summaries (see section 4.2), we perform an additional experiment to assess whether our approach can measure synergistic information using the mul-tirc dataset @xcite . specifically, multirc is a reading comprehension dataset consisting of multi-sentence paragraphs and multiplechoice questions, which also specifies the sentences that are required to answer each question. most importantly, the dataset ensures that correct answers can only be derived by considering multiple sentences jointly. that is, synergistic information should be prominent in correct answers. to apply our pid approach, we transform mul-tirc into a mds dataset, where each instance comprises the set of sentences (sources) required to answer a question, and a question-answer pair concatenated into a single sentence (summary). although both correct and incorrect answers share the same question, the former should result in higher synergy than the latter. we also generate unrelated instances for each set of source sentences, where the question-answer pairs are randomly sampled from a different paragraph (see figure 6 ). ## conclusion we characterize the information present in humanwritten multi-document summaries to get insights into what information comprises a high-quality summary. in particular, we suggest to decompose the mutual information that the source documents provide about a summary and propose spider, a novel approach to quantify such information using partial information decomposition. we then analyze human-written summaries from widely used mds datasets. the results reveal that redundancy decreases, whereas unique information increases with the number of sources. furthermore, the order of the documents has an impact on the summarization process, as the first three documents contribute the most in terms of unique information.