S umm V is: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization (2021)

Abstract: Novel neural architectures, training strategies, and the availability of large-scale corpora haven been the driving force behind recent progress in abstractive text summarization. However, due to the black-box nature of neural models, uninformative evaluation metrics, and scarce tooling for model and data analysis the true performance and failure modes of summarization models remain largely unknown. To address this limitation, we introduce SummVis, an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of the models, data, and evaluation metrics associated with text summarization. Through its lexical and semantic visualizations, the tools offers an easy entry point for in-depth model prediction exploration across important dimensions such as factual consistency or abstractiveness. The tool together with several pre-computed model outputs is available at https://summvis.com .

## introduction the field of natural language processing has seen substantial progress in recent years driven by the availability of large-scale corpora @xcite , developments in neural architectures @xcite and training strategies @xcite . despite the promising results on benchmarks and recent findings in model analysis, the true performance, generalizability, and failure modes of modern neural models are not yet fully understood, due to the black-box nature of neural models and the unmanageable scale of recent datasets for manual analysis. software tooling for nlp research provides a plethora of mature and easy-to-use libraries for model development, such as pytorch @xcite or transformers @xcite , but offers disproportionately fewer tools for visual analysis and debugging, ## summvis in this section, we present summvis, an interactive visualization tool that provides rich text comparison in summarization systems, enabling finegrained analysis of models, data, and evaluation metrics. it comes pre-loaded with model outputs for state-of-the-art models over common benchmark datasets, as well as scripts for loading data for any dataset provided by the datasets api @xcite and any huggingface-compatible model. ## case study: debugging hallucination as discussed earlier, summvis supports joint analysis of the model, data, and evaluation metrics. we now demonstrate how we can draw from all three modes of analysis to study the problem of hallucination in summarization systems. through the unified view of summvis, we analyze the example shown in figure 4 and demonstrate the existence of hallucination, suggest a possible cause, and show how a common evaluation metric prefers hallucinated entities over faithful descriptors in this case. ## related work text summarization requires models to be adept at both natural language understanding (nlu) and natural language generation (nlg). a gap in either of these areas has consequences on the progress of summarization as a whole. an example of this is the lack of meaningful metrics in nlg for high entropy tasks @xcite . several recent works have realized the need for evolving benchmarks and evaluations @xcite @xcite . existing tools support some forms of text com-parison for summarization models. the newsroom dataset visualization tool @xcite highlights n-grams in the summary that overlap with the source article. the lit tool @xcite highlights words or characters that differ between reference and generated texts. however neither tool aligns @xcite the matched text. the csi framework @xcite and the seq2seqvis @xcite tool align the source document and summary, but use modelspecific attention mechanisms. summvis on the other hand supports a model-agnostic comparison between source document, reference summary, and generated summary, and aligns text along lexical and semantic dimensions. ## conclusion in this work we introduced summvis, an interactive visualization tool for analyzing text summarization models, datasets, and evaluation metrics. through a case study we showed that our tool can be used to efficiently identify the shortcomings and failure modes of state-of-the-art summarization models and datasets. together with the tool we released a set of pre-computed model outputs to enable easy, out-of-the-box use. we hope this work will positively contribute to the ongoing efforts in building tools for model evaluation and analysis and enable a deeper understanding of the performance of summarization models and the intricacies of datasets and metrics.