S ummer T ime: Text Summarization Toolkit for Non-experts (2021)

Abstract: Recent advances in summarization provide models that can generate summaries of higher quality. Such models now exist for a number of summarization tasks, including query-based summarization, dialogue summarization, and multi-document summarization. While such models and tasks are rapidly growing in the research field, it has also become challenging for non-experts to keep track of them. To make summarization methods more accessible to a wider audience, we develop SummerTime by rethinking the summarization task from the perspective of an NLP non-expert. SummerTime is a complete toolkit for text summarization, including various models, datasets, and evaluation metrics, for a full spectrum of summarization-related tasks. SummerTime integrates with libraries designed for NLP researchers, and enables users with easy-to-use APIs. With SummerTime, users can locate pipeline solutions and search for the best model with their own data, and visualize the differences, all with a few lines of code. We also provide explanations for models and evaluation metrics to help users understand the model behaviors and select models that best suit their needs. Our library, along with a notebook demo, is available at https://github.com/Yale-LILY/SummerTime .

## introduction the goal of text summarization is to generate short and fluent summaries from longer textual sources, while preserving the most salient information in them. benefiting from recent advances of deep neural networks, in particular sequence to sequence models, with or without attention @xcite @xcite , current state-of-the-art summarization models produce high quality summaries that can be useful in practice cases @xcite . moreover, neural summarization has broadened its scope with the introduction of more summarization tasks, such as query-based summarization @xcite , long-document summarization @xcite , multi-document summarization @xcite , dialogue summarization @xcite . such summarization tasks can also be from different domains @xcite @xcite . however, as the field rapidly grows, it is often hard for nlp non-experts to follow all relevant new models, datasets, and evaluation metrics. moreover, those models and datasets are often from different sources, making it a non-trivial effort for the users to directly compare the performance of such models side-by-side. this makes it hard for them to decide which models to use. the development of libraries such as transformers @xcite alleviate such problems to some extent, but they only cover a narrow range of summarization models and tasks and assume certain proficiency in nlp from the users, thus the target audience is still largely the research community. to address those challenges for non-expert users and make state-of-the-art summarizers more accessible as a tool, we introduce summertime, a text summarization toolkit intended for users with no nlp background. we build this library from this perspective, and provide an integration of different summarization models, datasets and evaluation metrics, all in one place. we allow the users to view a side-by-side comparison of all classic and state-of-the-art summarization models we support, on their own data and combined into pipelines that fit their own task. summertime also provides the functionality for automatic model selection, by constructing pipelines for specific tasks first and iteratively evaluation to find the best working solutions. assuming no background in nlp, we list "pros and cons" for each model, and provide simple explanations for all the evaluation metrics we support. moreover, we go beyond pure numbers and provide visualization of the performance and output of different models, to facilitate users in making decisions about which models or pipelines to finally adopt. the purpose of summertime is not to replace any previous work, on the contrary, we integrate existing libraries and place them in the same framework. we provide wrappers around such libraries intended for expert users, maintaining the userfriendly and easy-to-use apis. ## summertime the main purpose of summertime is to help nonexpert users navigate through various summarization models, datasets and evaluation metrics, and provide simple yet comprehensive information for them to select the models that best suit their needs. figure 1 shows how summertime is split into different modules to help users achieve such goal. we will describe in detail each component of summertime in the following sections. with section 3.1, we introduce the models we support in all subcategories of summarization; in section 3.2 we list all the existing datasets we support and how users can create their own evaluation set. finally in section 3.3, we explain the evaluation metrics included with summertime and how they can help users find the most suitable model for their task. ## future work an important piece of future work for summer-time is to include more summarization models (e.g., multilingual, query-aware, etc) to enlarge the number of choices for the users, and more datasets to increase the chance of users finding similar tasks or domain for evaluation when they do not have a dataset of their own. we also plan to add more visualization methods for the users to better understand the differences between the outputs of various models and the behavior of each individual model itself. moreover, we would like to enable fine-tuning for a subset of smaller models we support, to enable better performance on some domains or tasks for which no pretrained models are available. with all such potential improvements in the near future, we plan to supply summertime not only as a way for non-expert users to access state-of-art summarization models, but also as a go-to choice to quickly establishing baseline results for researchers as well. ## conclusion we introduce summertime, a text summarization toolkit designed for non-expert users. sum-mertime includes various summarization datasets, models and evaluation metrics and covers a wide range of summarization tasks. it can also automatically identify the best models or pipelines for a specific dataset and task, and visualize the differences between the model outputs and performances. summertime is open source under the apache-2.0 license and is available online.