C onvo S umm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining (2021)

Abstract: While online conversations can cover a vast amount of information in many different formats, abstractive text summarization has primarily focused on modeling solely news articles. This research gap is due, in part, to the lack of standardized datasets for summarizing online discussions. To address this gap, we design annotation protocols motivated by an issues–viewpoints–assertions framework to crowdsource four new datasets on diverse online conversation forms of news comments, discussion forums, community question answering forums, and email threads. We benchmark state-of-the-art models on our datasets and analyze characteristics associated with the data. To create a comprehensive benchmark, we also evaluate these models on widely-used conversation summarization datasets to establish strong baselines in this domain. Furthermore, we incorporate argument mining through graph construction to directly model the issues, viewpoints, and assertions present in a conversation and filter noisy input, showing comparable or improved results according to automatic and human evaluations.

## introduction automatic text summarization is the process of outputting the most salient parts of an input in a concise and readable form. recent work in summarization has made significant progress due to introducing large-scale datasets such as the cnn-dailymail dataset @xcite and the new york times dataset @xcite . furthermore, the use of large self-supervised pretrained models such as bart @xcite and pegasus @xcite has achieved stateof-the-art performance across summarization tasks and strong performance in zero and few-shot settings @xcite . however, less work has focused on summarizing online conversations. headline: superbowl snippet: whether you're a football fan or not, what do you like about super bowl sunday? comment: ... in my opinion i think the falcons will stomp the patriots. i think tom brady will choke the super bowl. ... comment: i am big arizona cardinals fan so when they didn't even make the playoffs i was upset. ... comment: i'm not a very big football fan at all. so when it comes to superbowl sunday, i'm in it for the commercials and the half time show. ... comment: i am not exactly a football fan, but i enjoy watching the super bowl.... ... summary: several commenters list their favorite things about the super bowl, including half-time shows, the funny commercials, the puppy bowl, eating food, and spending time with family. a couple of commenters admit to not being football fans but still enjoying the super bowl. some commenters discuss whether they thought the falcons or the patriots were going to win, while others list teams they wish were in the game. unlike documents, articles, and scientific papers, which contain specific linguistic structures and conventions such as topic sentences and abstracts, conversational text scatters main points across multiple utterances and between numerous writers. as a result, the text summarization task in the conversational data domain offers a challenging research field to test newly-developed models @xcite . recently, @xcite introduced a dataset for chat-dialogue conversation summarization consisting of 16k examples, the first largescale dataset of its kind. previous work in conversation summarization was limited by the data available and focused primarily on meeting summarization, such as the ami @xcite and icsi @xcite datasets. the datasets used in recent conversation papers are often not uniform, ranging from visual dialogue data @xcite ) to customer-service dialogues @xcite , not initially intended for summarization. the availability of benchmark datasets for comparing methods has limited work in other conversation summarization domains and thus likely inhibited progress @xcite . we aim to address this research gap by crowdsourcing a suite of four datasets, which we call convosumm, that can evaluate a model's performance on a broad spectrum of conversation data. in determining the domains of data to collect, we use the general definition of conversation as "any discourse produced by more than one person" @xcite . we identify several key categories of data for which standard human-created development and testing datasets do not exist, namely (1) news article comments, (2) discussion forums and debate, (3) community question answering, and (4) email threads. we design annotation protocols motivated by work in quantifying viewpoints present in news comment data @xcite to crowdsource 250 development and 250 test examples for each of the above domains. we provide an example of comments to a new york times news article, and our crowdsourced summary in in addition to introducing manually-curated datasets for conversation summarization, we also aim to unify previous work in conversation summarization. namely, we benchmark a state-of-the-art abstractive model on several conversation datasets: dialogue summarization from samsum @xcite ), heuristic-generated community question answering from cqasumm @xcite , meeting summarization data from ami and icsi, and smaller test sets in the news comments, discussion forum, and email domains. we believe that such benchmarking will facilitate a more straightforward comparison of conversation summarization models across domains. to unify modeling across these conversational domains, we propose to use recent work in end-toend argument mining @xcite @xcite to instantiate the theoretical graph framework which motivated our annotation protocol, proposed by @xcite for conversation summarization. this protocol is employed to both identify and use the "issues-viewpoints-assertions" argu-ment structure (discussed in related work) for summarizing news comments. we construct this argument graph using entailment relations, linearize the graph, train a graph-to-text model @xcite , and experiment with argument mining as a way to reduce noise in long-text input. our contributions are the following: (1) we crowdsource datasets for four domains of conversational data and analyze the characteristics of our proposed datasets; (2) we benchmark state-of-theart models on these datasets as well as previous widely-used conversation summarization datasets to provide a clear baseline for future work; and several existing works have focused on conceptualizing conversation structure for summarization and how to present this structure to endusers. @xcite propose a conversation overview summary that aims to capture the key argumentative content of a reader comment conversation. @xcite use summarization as a means of probing online debates to discover central propositions, which they cluster to identify argument facets. @xcite identify three key components of conversational dialogue: issues (that individuals discuss), viewpoints (that they hold about these issues), and assertions (that they make to support their viewpoints). we build on this framework and advances in argument mining for end-to-end training for summarization. argument mining work in argument mining @xcite has aimed to identify these argumentative units and classify them into claims, premises, and major claims, or claims describing the key concept in a text. more recently, @xcite propose to finetune bert @xcite for identifying argumentative units and relationships between them within a text and across texts. @xcite are the first to propose an end-to-end approach for constructing an argument graph @xcite , a structured representation of claims and premises in an argumentative text; the graph is built by connecting claim and premise argumentative discourse units. we build on this framework for modeling discourse in conversational data. ## convosumm in this section, we introduce our dataset selection, our annotation protocol, and the characteristics of our crowdsourced dataset. ## argument graph summarization as our annotation protocol is motivated by the issues-viewpoints-assertions framework proposed in @xcite , we propose to instantiate a modified version of that work's theoretical, proposed graph model. ## experimental settings we use the fairseq codebase @xcite for our experiments. our base abstractive text summarization model is bart-large @xcite , a pretrained denoising autoencoder with 336m parameters that builds on the sequence-to-sequence transformer of @xcite . we finetune bart using a polynomial decay learning rate scheduler with adam optimizer @xcite . we used a learning rate of 3e-5 and warmup and total updates of 20 and 200, following previous few-shot transfer work @xcite . we could have equally fine-tuned other pretrained models such as pegasus @xcite or t5 @xcite , but @xcite find that bart largely performs equally well in few-shot settings when compared to pegasus. for the nyt and stack datasets, which contain sequences over the typical 1024 max encoder length with which bart is trained, we copied the encoder positional embeddings to allow sequences up to length 2048. to address the input-length of meeting summaries, which range from 6k to 12k tokens, we use the longformer @xcite , which allows for sequences up to length 16k to- @xcite and hmnet @xcite on the ami and icsi meeting summarization dataset along with our longformer and longformer-arg models. ## results we provide results for baseline, unsupervised extractive models in we train bart on 200 examples from our validation set for abstractive models, using the remaining 50 as validation and test on the final test set of 250 examples. we tested zero-shot transfer from cnndm and samsum in zero-shot settings, although these resulted in a much lower performance of about 28 rouge-1. few-shot model performance is shown in we also train on our argument mining-based approaches and show results in from modeling the argument structure or removing non-argumentative units. we provide full results for both variations in the appendix. ## conclusion we propose convosumm, a benchmark of four new, crowdsourced conversation datasets and stateof-the-art baselines on widely-used datasets that promote more unified progress in summarization beyond the news domain. our benchmark consists of high-quality, human-written summaries that call for abstractive summaries and a deeper understand-ing of the input texts' structure. we provide results for baseline models and propose to model the text's argument structure, showing that such structure helps better quantify viewpoints in non-linear input in both automatic and human evaluations. our analysis notes challenges in modeling relevance and consistency in abstractive conversation summarization when compared to news summarization.