Error Analysis of using BART for Multi-Document Summarization: A Study for E nglish and G erman Language (2021)

Abstract: Recent research using pre-trained language models for multi-document summarization task lacks deep investigation of potential erroneous cases and their possible application on other languages. In this work, we apply a pre-trained language model (BART) for multi-document summarization (MDS) task using both fine-tuning and without fine-tuning. We use two English datasets and one German dataset for this study. First, we reproduce the multi-document summaries for English language by following one of the recent studies. Next, we show the applicability of the model to German language by achieving state-of-the-art performance on German MDS. We perform an in-depth error analysis of the followed approach for both languages, which leads us to identifying most notable errors, from made-up facts and topic delimitation, and quantifying the amount of extractiveness.

## introduction nowadays, we are confronted with an enormous amount of information through news, mails, social media, etc., which are difficult to absorb for a human being in one go. hence, there is a pressing need to compress and comprehend this information. capturing salient details from multiple sources to produce an abridged version is described as multi-document summarization (mds) @xcite and can be carried out in both an abstractive or extractive manner. mds has recently become one of the most interesting research topics in the field of natural language processing (nlp). as per the literature, whilst the state-of-theart models @xcite heavily rely on large datasets, recent advances with pre-trained language model systems @xcite @xcite have shown great potential for the summarization task. while there have been studies to gradually improve the performance of mds for the english language, mds for other languages has rarely been attempted. there has also been a lack of in-depth error analysis for the mds task. in this study, we attempt to analyze and address these issues. our main contributions are the following: firstly, we reproduce recent pre-trained and fine-tuned results for multi-document summarization with the bart model, introduced by @xcite , on two english datasets. further, we adapt the model for the german language and achieve state-of-theart performance for the german mds task, beating the most competitive baseline by a margin of 3.48-8.67%. secondly, we perform an analysis on the erroneous cases for both languages where we point out general errors and cross-lingual error similarities regarding factfulness and topic delimitation. additionally, we also investigate the extractiveness of the generated summaries. ## related work early approaches on extractive mds apply term frequency-inverse document frequency (tf-idf) @xcite @xcite . later, @xcite and @xcite , attempt the mds task with a topic and set-based methodology, respectively. initial attempts for abstractive multi-document summarization are made by @xcite and @xcite . @xcite use sentence-fusion for text generation to create summaries across different documents. @xcite build a model based on word frequency and latent dirichlet allocation (lda) for mds whereas phrase selection and merging approaches have also been tried @xcite for the same. in recent years, neural network architecture is being adapted for several nlp tasks, especially with the approach of using encoder-decoder architecture. here, relevant work includes @xcite , who propose an attention model for combining extractive and abstractive methods, which is supplemented with document-wide contextualization by @xcite and @xcite . in a different direction, several graph-based approaches are explored as well @xcite . @xcite show the feasibility of using wikipedia as an mds dataset whereas @xcite apply a pointer-generator network with a transformer model complemented with maximal marginal relevance (mmr). @xcite explores graph representation and proposes to leverage graphs for abstractive mds. most recently, fine-tuning pre-trained language models have gained a lot of attention for nlp tasks. for summarization, one such work by @xcite attempted to explore fine-tuning, whereas, in another work, @xcite fine-tune bert for summarization. later, @xcite adapt and fine-tune bart on mds. approaches regarding a systematic error analysis of those models were introduced by @xcite who compared bart to other abstractive and extractive methods. in another direction, attempts have also been made for single-document summarization for non-english text. for instance, single-document summarization of text in german language was done by @xcite who utilized transformer models for abstractive summarization on two datasets -swisstext 2019 foot_0 and common crawl foot_1 . evaluation of summarization models to non-english data was done by @xcite who applied an automatic evaluation paradigm on the german heterogeneous dataset dbs @xcite . since our main focus is on multi-document summarization, we do not explore the literature of single-document summarization extensively. ## datasets for our experiments we use three datasets that exhibit extractive characteristics: two english datasets -cnn/dm @xcite , @xcite and one german dataset -auto-hmds @xcite . cnn/dailymail this dataset is an english single-document summarization (sds) news dataset consisting of 311,971 news articles with an average length of ∼800 words from the cnn and dailymail websites including abstractive summaries. multi-news the multi-news dataset is an english mds news dataset consisting of 56,216 summaries and over 250,000 sources with an average of ∼2,100 words from 1,500 different sites. the summaries are linked to 2-10 human-written source documents retrieved from https://www.newser.com/ . ## methodology we consider the state-of-the-art bart model @xcite for the multi-document summarization (mds) task. first, we use only pre-trained bart, and next, we fine-tune the pre-trained bart model using each of the three datasets separately and analyze the performances. the details about the bart model are described below. description of bart model bart @xcite generalizes the concepts of bidirectional encoders from bert @xcite and autoregressive decoders from gpt-2 @xcite . the model is trained with text corrupted through an arbitrary noising function and a sequence-to-sequence model that learns to reconstruct the original text. the encoder reads the sequential input e.g. a document to summarize while the decoder generates the outputs autoregressively. both layers are connected by cross-attention where each decoder layer focuses on specific aspects over the final state of the encoder output creating sequences, closely connected to the initial input. the bidirectional encoder architecture takes all previous and subsequent tokens into account for predicting a masked token. in text generation, bert without any modification loses its strength of bi-directionalism and becomes directional to-wards past words, as following words have yet to be generated. here bart adopts the architecture of gpt-2 to predict future words only by utilizing previous words. the advantage of bart therefore is the combination of contextual embeddings from bert and text generation from gpt-2. transformation, as described in @xcite , can be implemented through token masking, token deletion, text infilling, sentence permutation, or document rotation. note that, in the work done by @xcite , authors apply the bart model only on single-document summarization (sds) task, not on the multi-document variant of the summarization task. therefore, to adapt the bart model for the mds task, we follow the approach prescribed by @xcite , where authors reuse the existing sds model for mds by merging multipleinput to single-input. on the other hand, the issue of redundant and overlapping information is one major point to be taken care of for any summarization task, especially for mds tasks. for that purpose, we rely on the n-gram blocking approach following the work done by @xcite . ## experimental results and error analysis for our experiments we make use of the pre-trained bart model 3 and fine-tune the model on the three datasets and compare the performance with competitive baselines. ## conclusion in this paper, we investigated the performance of one of the most recent pre-trained language models namely bart, for multi-document summarization tasks in english and german language. for the first time ever, we attempted fine-tuning bart for german language multi-document summarization and achieved state-of-the-art performance. we further analyzed the erroneous cases for both english and german language and attempted to find a set of patterns where bart went wrong. the insights obtained via this error analysis give rise to devise more sophisticated methods for the task of multidocument summarization addressing these errors, of which the most severe is the hallucination of facts. our code and data repository is available publicly foot_4 .