Corpora Evaluation and System Bias Detection in Multi-document Summarization (2020)

Abstract: Multi-document summarization (MDS) is the task of reflecting key points from any set of documents into a concise text paragraph. In the past, it has been used to aggregate news, tweets, product reviews, etc. from various sources. Owing to no standard definition of the task, we encounter a plethora of datasets with varying levels of overlap and conflict between participating documents. There is also no standard regarding what constitutes summary information in MDS. Adding to the challenge is the fact that new systems report results on a set of chosen datasets, which might not correlate with their performance on the other datasets. In this paper, we study this heterogeneous task with the help of a few widely used MDS corpora and a suite of state-of-theart models. We make an attempt to quantify the quality of summarization corpus and prescribe a list of points to consider while proposing a new MDS corpus. Next, we analyze the reason behind the absence of an MDS system which achieves superior performance across all corpora. We then observe the extent to which system metrics are influenced, and bias is propagated due to corpus properties. The scripts to reproduce the experiments in this work are available at https://github.com/LCS2-IIITD/summarization_bias.git

## introduction multi-document summarization (mds) deals with compressing more than one document into a textual summary. it has a wide range of applications -gaining insights from tweets related to similar hashtags, understanding product features amongst e-commerce reviews, summarizing live blogs related to an ongoing match, etc. most studies on mds were performed during the duc 1 and tac foot_0 challenges starting in the early 2000s. each version of the challenges released a new dataset. most of the mds systems submitted to these challenges were unsupervised and extractive in nature. gradually, the data released in these challenges became the de facto for mds. these datasets were manually curated and had less than a hundred instances each. the recent development of deep neural architecture has led to a significant increase in the number of supervised document summarization systems. large labeled corpora which are mostly crowd-sourced have been introduced to meet the training requirements of the supervised systems. however, the crowd-sourced corpora widely differ in quality based on factors like genre, size of the community, presence of moderation in the community, etc. this is further aggravated by the complexity of the task, the hardness of accumulating labeled data, or more so in the definition of what constitutes a multi-document summary. recently, a few large datasets for mds have been introduced @xcite . however, there has been no study to measure the relative complexity of these datasets. we observe that existing mds systems behave differently on different corpora. for example, a system achieving state-of-the-art performance on one corpus fails to achieve reasonable performance on another. although the rouge points of mds systems are increasing day-by-day, manual inspection reveals an increased presence of bias in generated summaries. new systems are being introduced and evaluated on a few selected corpora, leading to difficulty in understanding whether the bias is introduced by the system or it is present in the corpus used for training. our research questions are as follows: q1. how should one model the quality of a mds corpus as a function of its intrinsic properties? q2. why do the rouge-based ranks of different mds systems differ across different corpora? how should an mds system which intends to achieve high rouge scores across all corpora, look like? q3. why do systems show bias on different metrics, and which other system and corpus attributes are the reason behind it? q4. is the task of mds almost solved, or is there still scope for improvement? we study five mds corpora -duc @xcite , tac @xcite , opinosis @xcite , multinews @xcite , and cqa-summ @xcite . we consider eight popular summarization systems -lexrank @xcite , textrank @xcite , mmr @xcite , icsisumm @xcite , pg @xcite , pg-mmr @xcite , @xcite , and copy-transformer @xcite . our major contributions are four-fold: • we propose a suite of metrics to model the quality of an mds corpus in terms of -abstractness, inter document similarity (ids), redundancy, pyramid score, layout bias and inverse-pyramid score. to the best of our knowledge, the current study is the first of its kind. ## background and proposed metrics throughout the paper, we use the term candidate documents for the documents participating in summarization, and the term reference to indicate the ground-truth summary. oracle summary is the extractive set of sentences selected from the candidate documents, exhibiting maximum rouge-n score w.r.t. the reference summary. it is an np-hard problem @xcite , and approximate solutions can be found greedily or using ilp solvers. here, we briefly introduce a suite of corpus and system metrics proposed by us to better understand the mds task. these metrics are further explained in detail in supplementary. ## discussion on research questions q1. how should one model the quality of an mds corpus as a function of its intrinsic metrics? what guidelines should be followed to propose mds corpora for enabling a fair comparison with existing datasets? the quality of an mds corpus is a function of two independent variables: the quality of the candidate documents and the quality of the reference summary. our findings suggest that a framework for future mds datasets should provide scores measuring their standing w.r.t. both the above factors. the former is usually crowd-source dependent, while the latter is usually annotator dependent. while inter document similarity, redundancy, layout bias and inverse-pyramid score are indicators of the properties of the candidate document, metrics such as abstractness of the reference summary and pyramid score are ground-truth properties. we divide the above metrics into two categories: objective and subjective. while all these metrics should be reported by imminent corpora proposers to enable comparisons with existing corpora and systems, we feel that the objective metrics average pyramid score and inverse-pyramid score must be reported as they are strong indicators of generic corpus quality. other subjective metrics such as ids, redundancy, abstractness etc. can be modeled to optimize task-based requirements. q2. why do the rouge-based ranks of different mds systems differ across corpora? how should an mds system which is to achieve reasonably good rouge score on all corpora look like? from q3. why do systems show bias on different metrics, and which other system and corpus attributes are the reason behind it? we begin by studying how abstractness of generated summaries is related to the abstractness of corpora the system is trained on. for this, we calculate the pearson correlation coefficient between the abstractness of generated summaries and references across different datasets. from on the other hand, extractive systems such as lexrank, textrank, mmr and icsisumm maintain a strongly positive correlation throughout the segments. on shuffling the source segments internally, we observe that extractive systems tend to retain their correlation with corpora while abstractive systems show no correlation at all (fig. 2 ), proving that in supervised systems, the layout bias in system summaries propagates from the layout bias present in corpora. q4. is the task of mds almost solved, or there is still plenty of scope remaining for improvement? in the previous sections, we computed the oracle extractive upper bound summary using greedy approaches to find the summary that obtains the highest rouge score given the candidate documents and references. we observe that the best summarization system on each corpus today obtains a score which is 39.6% of the extractive oracle upper bound on duc, 47.8% on tac, 75.02% on opinosis, 54.5% on multinews and 49.9% on cqasumm. this shows that there is enough scope for mds systems to achieve double the rouge scores obtained by the best system to date on each corpus except opinosis. therefore, we believe that the task of mds is only partially solved and considerable efforts need to be devoted to improving the systems. ## related work previous attempts to evaluate the quality of the benchmark summarization corpora are few in number and mostly from the time when corpora were manually accumulated. @xcite primarily used the intrinsic metrics of precision and recall to evaluate corpus quality. in addition, the authors proposed an extrinsic metric, called 'pseudo question answering'. this metric evaluates whether a summary has an answer to a question that is otherwise answerable by reading the documents or not. although effective, the cost of such an evaluation is enormous and is not scalable to modern day corpora sizes. for such corpora where multiple references are available, @xcite used an inter-annotator agreement to model the quality of the corpora. they also used non-redundancy, focus, structure, referential clarity, readability, coherence, length, grammaticality, spelling, layout, and overall quality as quantitative features for an mds corpus. recently, @xcite ) proposed an mds system that used the baseline pg model along with hierarchical structural attention to take into account long-term dependencies for superior results compared to baseline models. there have been a series of very recent studies that look into how to strengthen the definition and discover system biases in single-document summarization. very recently, @xcite studied how position, diversity and importance are significant metrics in analyzing the toughness of singledocument summarization corpora. another recent work @xcite extensively studied the layout bias in news datasets that most single-document summarization systems seem to exploit. two seminal works, namely @xcite and @xcite , exploited the theoretical complexity of summarization on the ground of importance, analyzing in-depth what makes for a good summary. @xcite ) mathematically modeled the previously intuitive concepts of redundancy, relevance and informativeness to define importance in single-document summarization. @xcite proposed a new singledocument summarization corpus and quantified how it compares to other datasets in terms of di-versity and difficulty of the data. they introduced metrics such as extractive fragment density and extractive fragment coverage to plot the quality of sds corpus. to the best of our knowledge, no comparative work exists for either corpora or systems in mds, and the current paper is the first in this direction. ## conclusion in this paper, we aimed to study the heterogeneous task of multi-document summarization. we analyzed interactions between widely used corpora and several state-of-the-art systems to arrive at a line of conclusions. we defined mds as a mapping from a set of non-independent candidate documents to a synopsis that covers important and redundant content present in the source. we proposed intrinsic metrics to model the quality of an mds corpus and introduced a framework for future researches to consider while proposing a new corpus. we analyzed how rouge ranks of different systems vary differently on different corpora and described what a system that achieves reasonable performance on all corpora would look like. we evaluated how different systems exhibit bias and how their behavior is influenced by corpus properties. we also commented on the future scope for the task of mds. future directions to take forward this work would include a causal analysis of how corpus bias is responsible for bias in model prediction across different corpora and systems. this might bring forward measures to de-bias nlp algorithms with/without de-biasing the corpora.