TWEETSUMM - A Dialog Summarization Dataset for Customer Service (2021)

Abstract: In a typical customer service chat scenario, customers contact a support center to ask for help or raise complaints, and human agents try to solve the issues. In most cases, at the end of the conversation, agents are asked to write a short summary emphasizing the problem and the proposed solution, usually for the benefit of other agents that may have to deal with the same customer or issue. The goal of the present article is advancing the automation of this task. We introduce the first large scale, high quality, customer care dialog summarization dataset with close to 6500 human annotated summaries. The data is based on real-world customer support dialogs and includes both extractive and abstractive summaries. We also introduce a new unsupervised, extractive summarization method specific to dialogs.

## introduction text summarization is the task of creating a short version of a long text, retaining the most important or relevant information. in nlp, there are two types of summarization tasks-(1) extractive summarization, in which segments from the original text are selected to form a summary and (2) abstractive summarization, in which new natural language expressions are generated for summarizing the text. the past few years have witnessed a tremendous progress in creating both kinds of summaries using seq2seq models. however, these works have largely focused on documents such as news and scientific publications @xcite . in this paper, we focus on summarizing conversational data between customers and human support agents. in many enterprises, once an agent is done with handling a customer request, she is required to create a short summary of the conversation for record keeping purposes. at times, an ongoing conversation may also need to be transferred to another agent or escalated to a supervisor. this also requires creating a short summary of the conversation so far, as to provide the right context to the next handling agent. our main contribution is the release of tweet-summ, a dataset focused on summarization of dialogs, which represents the rich domain of twitter customer care conversations foot_0 . the dataset contains close to 6500 extractive and abstractive summaries generated by human annotators from 1100 dialogs. this is the first dataset released to the research community, which focuses on real dialogs, as opposed to previous works focusing on meeting conversations @xcite , general chitchat summarization @xcite , or topic descriptions of interviews @xcite . furthermore, the fact that each dialog was annotated by 3 different crowd-workers, resulting in an overall of 6 summaries for each dialog, provides diversity of summaries. we performed quality control and assessment to remove erroneous summaries, and to ensure that the collected tweet-summ summaries are of a high quality. we evaluate several summarization baselines and further provide a novel unsupervised extractive summarization algorithm, referred to as nrp summ which outperforms other unsupervised baselines for extractive summarization. figure 2 shows an example of a tweetsumm dialog along with a humangenerated abstractive summary and two machinegenerated summaries -abstractive and extractive summaries. we propose that the dataset quality and scale, is suitable for developing future models for the dialog summarization task. we hope that releasing tweetsumm for the community will foster further research. figure 1 : tweetsumm dialog and its summaries 2 tweetsumm dataset tweetsumm comprises of 1100 dialogs reconstructed from tweets that appear in the kaggle customer support on twitter dataset 2 , each accompanied by 3 extractive and 3 abstractive summaries generated by human annotators. the kaggle dataset, is a large scale dataset based on conversations between consumers and customer support agents on twitter.com @xcite . it covers a wide range of topics and services provided by various companies, from airlines to retail, gaming, music etc. thus, tweetsumm can serve as a dataset for training and evaluating summarization models for a wide range of dialog scenarios. for creating the 1100 dialogs of tweetsumm, we first reconstructed 49,155 unique dialogs from the kaggle customer support on twitter dataset (see section 2.1). second, we filtered short and long dialogs, containing less than 6 or more than 20 utterances, in order to focus on dialogs that are representative of most cases. this resulted in 45,547 dialogs with an average length of 22 sentences 3 . next, in order to represent the customer service 2 www.kaggle.com/thoughtvector/ customer-support-on-twitter 3 an utterance, sometimes termed turn, usually contains more than one sentence. scenario, in which a single customer interacts with a single agent, dialogs with more than two speakers were removed. from the remaining 32,081 dialogs, we randomly sampled 1100 dialogs. these dialogs were sent for generation of summaries using crowdsourcing on the appen.com platform, as described below. ## next response prediction summarizer we introduce a novel, unsupervised extractive summarization method (coined nrp summ) aimed at identifying the sentences that influence the entire dialog the most. the next response prediction model -to identify the influence of each sentence on the entire conversation, we utilize the next response prediction (nrp) task @xcite in dialog systems. the nrp task is defined as follows: given a dialog context, i.e., the list of sentences in the dialog up to a certain point (@xmath0, s 2 , ..., s k }), predict the next response sentence (c r ) from a given set of candidates {c 1 , ..., c r , ..., c n }. to train the nrp model, we used a binary classifier commonly used for glue tasks @xcite . we process the dialogs to construct triples @xmath1, candidate (ci), label (1/0)> from each dialog context. for each c, we create a set of k + 1 (@xmath2: one triple containing the correct response (c r ) (@xmath3, and k triples containing incorrect responses randomly sampled from the dataset (@xmath4. the dialog context c and a candidate response c i are fed together to bert as a sequence ([cls] c [sep] ci [sep]). the hidden state of the [cls] token was used as the representation of the pair. training is done using positive and negative examples with cross-entropy loss. a model trained on the nrp task associates a probability (p r ) for the response (c r ), given the context c. we trained two nrp models, (1) a model predicting the next response given the prior sentences (nrp-fw), and (2) a model predicting the prior utterance given subsequent utterances (nrp-bw). salient sentence identification-the intuition behind this approach is that the removal of the critical sentences from a dialog context will entail a larger drop in probability in predicting a subsequent and prior responses. we follow the hypothesis that the critical sentences for the nrp task will also be salient sentences for the summary. the sentence removal occurs in two steps. in the initial step, we feed the entire context to the nrp model and identify the probability of predicting the next (or prior) sentence. in the next step, we remove one sentence at a time from the context, and input the new context to the nrp model and identify the probability of predicting the same next (or prior) utterance. then, we assign the drop in probability as a score to the removed sentence. to identify the salient sentences in predicting the next response, we remove one sentence at a time from the dialog context (c\s i ) and use that as the input to a trained nrp-fw model and identify the probability (p f w r ) for the corresponding response (c r ). then, we assign the drop in probability (p r -p f w r ) as a score to the removed sentence s i in the context. we follow the same process to identify the drop in probability in predicting the prior sentence, given the same dialog context and masked sentence (using nrp-bw model), and assign that as another score for the masked sentence. the averaged score for each sentence is used during salient sentence identification. for the evaluation, we use the top two customer sentences and the two top agent sentences as the extractive summary of the dialog. ## experiments and results we aim to confirm that tweetsumm is suitable as a ground-truth dataset for the dialog summarization task. to this end, we apply and analyze several baseline summarization models as well as nrp summ, to the dataset, as detailed below. we randomly split the dialogs and their associated summaries into three sets: 80% for the training set, 10% for the validation and the rest 10%, for the test set. ## conclusion in this paper, we release tweetsumm, the first open large-scale dataset focused on summarization of customer-support dialogs. we conducted automatic and human evaluation studies to ensure the high-quality of the human-generated extractive and abstractive summaries. to test the applicability of the dataset, we evaluated various baselines, as well as a new extractive summarization method, nrp summ, and showed that while automatically generated abstractive summaries achieve high quality, there is still much room for improvement. we believe tweetsumm will help foster research in this real-world scenario, which was previously little studied due to lack of suitable datasets.