CSDS : A Fine-Grained C hinese Dataset for Customer Service Dialogue Summarization (2021)

Abstract: Dialogue summarization has drawn much attention recently. Especially in the customer service domain, agents could use dialogue summaries to help boost their works by quickly knowing customer’s issues and service progress. These applications require summaries to contain the perspective of a single speaker and have a clear topic flow structure, while neither are available in existing datasets. Therefore, in this paper, we introduce a novel Chinese dataset for Customer Service Dialogue Summarization (CSDS). CSDS improves the abstractive summaries in two aspects: (1) In addition to the overall summary for the whole dialogue, role-oriented summaries are also provided to acquire different speakers’ viewpoints. (2) All the summaries sum up each topic separately, thus containing the topic-level structure of the dialogue. We define tasks in CSDS as generating the overall summary and different role-oriented summaries for a given dialogue. Next, we compare various summarization methods on CSDS, and experiment results show that existing methods are prone to generate redundant and incoherent summaries. Besides, the performance becomes much worse when analyzing the performance on role-oriented summaries and topic structures. We hope that this study could benchmark Chinese dialogue summarization and benefit further studies.

## introduction text summarization aims to compress a long input text and generate a condensed summary @xcite . it can help people capture the gist of a long document quickly. traditional summarization tasks mainly focus on news reports @xcite @xcite . however, as the communication tools become convenient, enormous information is presented in a conversational format, such as meeting records, daily chatting, and customer service logs. these dialogues usually cost more time to read since they are longer and have more complicated structures. thus, summarizing information from dialogue becomes essential in practical use. compared with news and documents, dialogues have two main unique features. first, dialogues have multiple speakers, and each of them has different viewpoints. in some cases, we only focus on the main viewpoint of one participant. a roleoriented summary will help achieve this goal. second, a dialogue often has multiple topics, and each topic concerns a different issue. since the summary expresses the primary information of the dialogue briefly and clearly, it should contain the topic flow of the dialogue by summing up each topic separately and be organized in a more structural format @xcite . specifically, both features are rather crucial for practical applications in the customer service domain: (1) the user-oriented summary could reflect the frequency of users' issues, and the agent-oriented summary could help check the quality of agents' services. (2) topic-based structural summary could help an agent clearly acquire the user's problems and previous service progress, figuring out the solved and unsolved problems. besides, role-oriented and structural summaries are also valuable for other dialogue domains such as debating and court trials. although several dialogue summarization datasets have been proposed recently @xcite @xcite , none of them adds dialogue features (i.e., different speakers' roles or topic structure) in summaries, limiting the application of these datasets. therefore, we aim to construct a fine-grained chinese dataset for customer service domain dialogue summarization (csds). to achieve this goal, we employ question-answer (qa) pairs as the annotation format since it is the basic granularity in a customer service di-0 q: ## related work most of the widely-used summarization datasets belong to document summarization. early datasets are provided by some evaluation tasks, such as duc @xcite and tac @xcite . recently, there exist various types of document summarization datasets such as news reports @xcite , wiki passages @xcite , and scientific papers @xcite . different from document summarization, dialogue summarization aims to summarize a conversation into a narrative text. @xcite ; @xcite provide two datasets for dialogue summarization at the earliest, and the task of their data is to summarize a meeting transcript into a few short sentences. @xcite incorporate them and propose a query-based summarization task. @xcite propose an english daily conversation summarization dataset on fictitious dialogues, providing a new daily chatting scenario. other datasets provided by @xcite and @xcite also show the potential of dialogue summarization in other scenarios. as for chinese dialogue summarization datasets, song et al. (2020) construct a medical dialogue summarization dataset, where most of the summaries are extractive and relatively easy to generate. almost all the above datasets only provide an overall summary for each dialogue without further annotations. in the customer service domain, @xcite provide a related dialogue summarization dataset, which is the most similar to our work. however, their dataset only contains an overall summary from the agent's perspective for each dialogue. besides, their publicly available data are difficult to analyze since all the sentences are given by word indexes. on the contrary, our dataset is readable and has more fine-grained annotations such as role-oriented summaries and topic-based structural summaries, both valuable and convenient for further research in this area. ## dataset difficulties in this section, we want to analyze the difficulties of csds further. according to the fine-grained features in csds and the challenges mentioned in section 3.6, we raise the following two questions. integrating messages from other roles for the role-oriented summary is difficult. compared with the overall summary, the role-oriented summary focuses on a single role's utterances. it needs to integrate messages from other roles to make the summary understandable, especially for agent summary. to analyze whether existing methods could learn to integrate different roles' information for agent summary, we compare the summary quality of samples which need to be integrated and those that do not need separately. note that in section 3.3, we complete some agent summaries to make them well-understood without contexts. thus, these completed summaries are considered as requiring information integration and others as not requiring integration. as shown in we also analyze how the qa pair matching ratio changes with the number of qa pairs in the reference summary and present it in figure 2 . the similar trends for three different methods indicate that it is harder to separate qa pairs for different topics when the number of qa pairs increases in reference summaries. therefore, how to summarize dialogues with the correct topic structure needs further study. besides, our evaluation algorithm can also be used as a standard metric to compare the performance of different methods on csds. at last, we present some examples of mistakes made by existing methods on csds in appendix h. from these cases, we conclude that more elaborate methods are needed for dialogue summarization. ## conclusion in this paper, we introduce a novel customer service dialogue summarization dataset named csds. this dataset is fine-grained in two folds. summaries for different roles are provided, and summaries contain topic structure information, which could help promote research and applications in this area. we also do elaborate experiments on csds and draw some instructive conclusions on method performance and dataset difficulties. in the future, we hope that we can propose new models for csds to summarize different roles or summarize the content for a specific topic. more suitable automatic evaluation metrics, especially for comparing the topic structure in the summary, are also worth studying.