An Empirical Study on Topic Preservation in Multi-Document Summarization (2022)

Abstract: Multi-document summarization (MDS) is a process of generating an informative and concise summary from multiple topic-related documents. Many studies have analyzed the quality of MDS dataset or models, however no work has been done from the perspective of topic preservation. In this work, we fill the gap by performing an empirical analysis on two MDS datasets and study topic preservation on generated summaries from 8 MDS models. Our key findings include i) Multi-News dataset has better gold summaries compared to Multi-XScience in terms of its topic distribution consistency and ii) Extractive approaches perform better than abstractive approaches in preserving topic information from source documents. We hope our findings could help develop a summarization model that can generate topic-focused summary and also give inspiration to researchers in creating dataset for such challenging task.

## introduction multi-document summarization (mds) is a task to produce an informative and concise summary from multiple documents. in general, there are two different approaches to mds, which are extractive and abstractive summarization @xcite . extractive summarization refers to methods that select important sentences from input documents and produce a summary. these methods perform better at producing summary without grammatical errors. on the other hand, abstractive summarization refers to methods that have the ability to generate summaries with words that do not exist in input documents @xcite . the development of text summarization model has been supported by the growing amount and quality of available dataset. the available dataset types vary from news articles @xcite to scientific articles @xcite and wikipedia abstract @xcite . however, information is not scarce in this era, but "valuable" information is. many recent work @xcite @xcite have been focusing on generating topic-guided summaries using one-sizefits-all dataset which are not meant for this kind of work, making it difficult to evaluate whether the model is performing better than "generic" model in terms of the quality of generated topic-focused summary. there are also work @xcite @xcite focusing on the analysis of summarization models and datasets but none on topic-preservation. to the best of our knowledge, there is only one dataset @xcite created for topic-guided news summarization, but has been tailored for single document summarization. therefore, it is essential to deep dive into current available mds dataset, and investigate their suitability for developing topic-guided summarization models, and the pattern of high quality summaries in order to inspire future work in text summarization. in this paper, we conducted several experiments in analyzing the relevance of input and output documents in automated summarization and the pattern of model-generated summaries. to sum up, our contributions are two-folds: i) for mds dataset, we evaluated topic relation between source documents and gold summaries in widely used mds dataset, inspiring future work on creating high quality dataset for topic-aware summarization model; ii) for mds models, we investigated summaries generated from a wide range of state-of-the-art models in order to provide insights of how relevant it is to the source documents. our observations could inspire research directions towards better topic-preserving mds dataset and models. ## datasets and models in this work, we use two most commonly used multi-document summarization dataset multi-news and multi-xscience in our experiments. we run 8 mds models from non-deep learning based models to deep learning models including the recent transformer-based state-of-the-art models. for fair comparison, training/validation and testing for all models are performed on a high performance computing cluster powered by nvidia v100. ## methods and results we compare and analyze topic-related patterns of source document, gold summaries (provided in mds benchmark datasets), and the generated summaries (from 8 mds models). guided by topic modelling research, we adopt the topic related evaluation metrics in this work. we specifically study i) topic coherence, to identify the best settings; ii) number of documents in each topic, to study the overall topic distribution; iii) distances among topic distributions of summaries, to examine the document-level patterns; iv) topic words correlations in summaries, to analyze the word-level patterns. ## discussions our studies and observations raise the following questions that we believe need to be considered in the mds research: is extractive mds model better than abstractive mds model in preserving topics? from results in section 3.4, we find that in terms of topic preservation, extractive models work better than abstractive models. this could due to the "extract" nature of the former which shares the same vocabulary as the source document, resulting in higher word correlation between source documents and summaries. future work could focus on analyzing word semantic similarities instead of relying on topic distribution similarities only as abstractive models use words that are different from source documents to generate a summary. to improve the topic preservation of abstractive models, we could consider select semantically similar words to the words in the source document during generation. whether gold summary follows source documents' topic distribution? from section 3.4 we also find multi-news's gold summaries topic distribution are well aligned with the topic distributions in its source document, however multi-xscience does not perform well in this regard.this analysis could inspire future mds dataset contributors to take topic preservation into consideration when preparing gold summaries such that source documents and gold summaries have similar topic distribution. whether the number of documents are similar across all topics? dataset that is "heavy" on one topic can disadvantage summarization models in training as the vocabulary might be dominated by a specific topic causing topic information for other topics with less instances being discarded or normalised. this can be seen in figure 2 where the document count per topic for source documents and gold summaries are inconsistent. future dataset creation should focus on the topic distribution among all documents in data collected to make sure that the generation model captures equal information from all topics. ## conclusion and future work in conclusion, we have systemically and empirically analyzed two popular multi-document summarization datasets and summaries generated from a variety of state-of-the-art summarization models. our analysis over 100,000 documents reveals that source documents, gold summaries and model generated summaries are rarely topic coherent which cause the summary to be less informative for some usages. this analysis also lead to some inspiration and suggestions in creating better summarization models and dataset for real world application.