Summary Explorer: Visualizing the State of the Art in Text Summarization (2021)

Abstract: This paper introduces Summary Explorer, a new tool to support the manual inspection of text summarization systems by compiling the outputs of 55 state-of-the-art single document summarization approaches on three benchmark datasets, and visually exploring them during a qualitative assessment. The underlying design of the tool considers three well-known summary quality criteria (coverage, faithfulness, and position bias), encapsulated in a guided assessment based on tailored visualizations. The tool complements existing approaches for locally debugging summarization models and improves upon them. The tool is available at https://tldr.webis.de/

## introduction automatic text summarization is the task of generating a summary of a long text by condensing it to its most important parts. this longstanding task originated in automatically creating abstracts for scientific documents @xcite , and later extended to documents such as web pages @xcite and news articles @xcite . there are two paradigms of automatic summarization: extractive and abstractive. the former extracts important information from the tobe-summarized text, while the latter additionally involves paraphrasing, sentence-fusion, and natural language generation to create fluent summaries. neural summarization approaches trained on largescale datasets have significantly advanced both paradigms by improving the overall document understanding and text generation capabilities of the models to generate fluent summaries. currently, the progress in text summarization is tracked primarily using automatic evaluation with rouge @xcite as the de facto standard for quantitative evaluation. rouge has proven effective for evaluating extractive systems, measuring the overlap of word n-grams between a generated summary and a reference summary (ground truth). still, it only provides an approximation of a model's capability to generate summaries that are lexically similar to the ground truth. moreover, rouge is unsuitable for evaluating abstractive summarization systems, mainly due to its inadequacy in capturing all semantically equivalent variants of the reference @xcite @xcite . besides, a reliable automatic evaluation of a summary is challenging @xcite and strongly dependent on its purpose @xcite . a robust method to analyze the effectiveness of summarization models is to manually inspect their outputs from individual perspectives such as coverage of key concepts and linguistic quality. however, manual inspection requires obtaining the outputs of certain models, delineating a guideline that comprises particular assessment criteria, and ideally utilizing proper visualization techniques to examine the outputs efficiently. to this end, we present summary explorer (figure 1 ), an online interactive visualization tool that assists humans (researchers, experts, and crowds) to inspect the outputs of text summarization models in a guided fashion. specifically, we compile and host the outputs of several state-of-the-art models (currently 55) dedicated to english singledocument summarization. these outputs cover three benchmark summarization datasets comprising semi-extractive to highly abstractive ground truth summaries. the tool facilitates a guided visual analysis of three important summary quality criteria: coverage, faithfulness, and position bias, where tailored visualizations for each criterion streamline both absolute and relative manual evaluation of summaries. overall, our use cases (see section 5) demonstrate the ability of summary explorer to provide a comparative exploration of cnn dailymail xsum tl;dr a b c d e figure 1: overview of summary explorer. its guided assessment process works in four steps: (1) corpus selection, (2) quality aspect selection, (3) model selection, and (4) quality aspect assessment. exemplified is the assessment of the content coverage of the summaries of four models for a source document from the cnn/dm corpus. for each summary sentence, its two most related source document sentences are highlighted on demand. the state-of-the-art text summarization models, and to discover interesting cases that cannot likely be captured by automatic evaluation. ## related work leaderboards such as paperswithcode, 1 explaina-board 2 and nlpprogress 3 provide an overview of state of the art in text summarization mainly according to rouge. these leaderboards simply aggregate the scores as reported by the models' developers, where the reported scores can be obtained using different implementations. hence, a fair comparison become less feasible. for instance, the bottom-up model @xcite uses a different implementation of rouge, 4 compared to the banditsum model @xcite . 5 besides, for a qualitative comparison of the models, one needs to manually inspect the generated summaries, which are missing from such leaderboards. to address these shortcomings, visseq @xcite aids developers to locally compare their model's outputs with the ground truth, providing lexical and semantic comparisons along with statistics such as most frequent n-grams and sentence score distributions. lit @xcite provides similar functionality for a broader range 1 https://paperswithcode.com/task/text-summarization 2 http://explainaboard.nlpedia.ai/leaderboard/task-summ/ 3 https://nlpprogress.com/english/summarization.html 4 https://github.com/sebastiangehrmann/rouge-baselines 5 https://github.com/pltrdy/rouge of nlp tasks, implementing a work-bench-style debugging of model behavior, including visualization of model attention, confusion matrices, and probability distributions. closely related to our work is summvis @xcite , the recently published tool that provides a visual text comparison of summaries with a reference summary as well as a source document, facilitating local debugging of hallucinations in the summaries. summary explorer draws from these developments and adds three missing features: (1) qualitycriteria-driven design. based on a careful literature review of qualitative evaluation of summaries, we derive three key quality criteria and encode them explicitly in the interface of our tool. other existing tools render these criteria implicit in their underlying design. (2) a step-by-step process for guided analysis. from the chosen quality criteria, we formulate concise and specific questions needed for a qualitative evaluation, and provide a tailored visualization for each question. while previous tools utilize visualization and enable users to (de)activate certain features, they oblige the users to figure out the process themselves, which can be overwhelming to non-experts. (3) compilation of the state of the art. we collect the outputs of more than 50 models on three benchmark datasets providing a comprehensive overview of the progress in text summarization. summary explorer complements these tools and also provides direct access to the state of the art in text summarization, encouraging rigorous analysis to support the development of novel models. ## designing visual summary exploration the design of summary explorer derives from first principles, namely the three quality criteria coverage, faithfulness, and position bias of a summary in relation to its source document. these high-level criteria are frequently manually assessed throughout the literature. since their definitions vary, however, we derive from each criterion a total of six specific aspects that are more straightforwardly operationalized in a visual exploration (see figure 1 , step 2). to render the aspects more directly accessible to users, each is "clarified" by a guiding question that can be answered by a tailored visualization. below, the three quality criteria are discussed, followed by the visual design.