SUMMARY WORKBENCH : Unifying Application and Evaluation of Text Summarization Models (2022)

Abstract: This paper presents Summary Workbench, a new tool for developing and evaluating text summarization models. New models and evaluation measures can be easily integrated as Docker-based plugins, allowing to examine the quality of their summaries against any input and to evaluate them using various evaluation measures. Visual analyses combining multiple measures provide insights into the models’ strengths and weaknesses. The tool is hosted at https://tldr.demo.webis.de and also supports local deployment for private resources.

## introduction automatic text summarization reduces a long text to its most important parts and generates a summary. usually, a learning-based summarization model is developed in two basic steps: model development and model evaluation. given a collection of documents accompanied by one or more humanwritten (reference) summaries, first a set of features representing the documents is manually created or automatically extracted through supervised learning. the resulting model is then used to generate one or more (candidate) summaries, which are analyzed manually and/or with evaluation measures for their similarity to the reference summaries. these steps are iterated, optimizing the model and its parameters using a validation set. the models that perform best in the validation are selected for evaluation on the test set. with standardized test sets for each document collection, comparisons with models created earlier are reported. however, these steps are associated with comparatively tedious tasks: during model development, summaries of individual documents are often generated and immediately evaluated to identify deficiencies and improve the model, including comparisons to other models. the latter requires third-party models to be operational despite their heterogeneous software stacks. such "on-the-fly evaluation" during development entails that candidate and reference summaries as well as source documents are analyzed manually or by automatic measures. this multi-text comparison is often not supported by visualization, although this leads to a better understanding of the content coverage and possible selection biases of a model @xcite . the analysis of evaluation results for model selection also benefits from visual support @xcite . previous research in the field of automatic summarization has not yet resulted in a unified set of tools for these purposes which is the main goal of this paper. with summary workbench, we introduce the first unified combination of application and visual evaluation environments for text summarization models. currently, it integrates 15 well-known summarization models (26 variants in total) and 10 standard evaluation measures from the literature. with featuresum, it also includes a new feature-based extractive summarization model that implements features from the literature predating the deep learning era. underlying all of the above is a specification and interface that allows easy integration of new models and measures to facilitate large-scale experiments and their reproducibility. in what follows, section 2 reviews related work on tools to assist summarization research and development. section 3 overviews the key design principles of the summary workbench, and section 4 provides a complete overview of all the models and measures hosted to date. included are generalpurpose models, guided models that accept user prompts to guide summary generation, and models tailored to argumentative language and to news articles. a wide range of commonly employed evaluation measures are included, covering both lexical as well as semantic overlap measures. foot_0 ## related work the development of tools for summarization research has gained momentum recently, and several tools have been presented for (sub)tasks of the two steps above: tools such as huggingface @xcite , fairseq @xcite , summer-time @xcite , torchmetrics @xcite , sacrerouge @xcite , pytorch hub, foot_1 and tensorflow hub foot_2 focus on hosting several state-of-the-art text summarization models and automatic evaluation measures. these tools have significantly improved accessibility to working models. however, only some provide a very minimal interface for inference of summaries and their online/offline comparative analyses. many authors also choose to share their models independently, be they on github or elsewhere, as standalone repositories instead of integrating with any tools. to lower the bar of (latter) integration as much as possible, summary workbench simplifies model and measure integration as plugins (using docker). in this way, models under development or private ones can be locally compared to others and can be archived together with all their dependencies for reproducibility. similar efforts have been made in the information retrieval community via the docker-based toolkit such as anserini @xcite . tools such as lit @xcite , summ-vis @xcite , and summary explorer @xcite focus on qualitative model evaluation by providing static visual analyses of the relation between the summary and its source document. summary workbench adapts some of their visualizations next to new ones, and complements them with interactive visual analytics for quantitative evaluation according to multiple measures. users can explore the distribution of scores, select data points of interest and inspect them in relation to the source document to better understand the dataset. the success of past summarization research and development has relied a lot on in-depth manual error analyses. this being one of the most laborious tasks in every natural language generation evaluation, we believe that visually comparing summaries from multiple models for many different texts, and contextualizing manual review with multiple measures is crucial to both scale up error analysis, and to better understand the capabilities and limitations of the technology. as this still requires juggling many different, incompatible tools, the unified approach of summary workbench aims at lowering the bar for scaling up interactive experimentation. ## an interactive visual summarization model development & evaluation tool summary workbench implements two interactive views corresponding to the two basic summarization model development steps: a summarization view and an evaluation view. ## interaction use cases as a second use case, the interactive plotter allows comparing two variants of the same model architecture using any measure. here, we inspect the t5 model (its 3b and 11b variants) using bertscore to find that the larger variant generates a summary very similar to the reference while the smaller variant creates a summary that is topically related but not accurate in comparison to the reference. ## conclusion in this paper we present summary workbench, a tool that unifies the application and evaluation of text summarization models. the tool supports integrating summarization models and evaluation measures of all kinds via a docker-based plugin system that can also be locally deployed. this allows safe inspection and comparison of models on existing benchmarks and easy sharing with the research community in a software stack-agnostic manner. we have curated an initial set of 15 models (26 including all variants) and 10 evaluation measures and welcome contributions from the text summarization community. an extension of the tool's features to related text generation tasks such as paraphrasing and question answering is foreseen.