Are We Summarizing the Right Way? A Survey of Dialogue Summarization Data Sets (2021)

Abstract: Dialogue summarization is a long-standing task in the field of NLP, and several data sets with dialogues and associated human-written summaries of different styles exist. However, it is unclear for which type of dialogue which type of summary is most appropriate. For this reason, we apply a linguistic model of dialogue types to derive matching summary items and NLP tasks. This allows us to map existing dialogue summarization data sets into this model and identify gaps and potential directions for future work. As part of this process, we also provide an extensive overview of existing dialogue summarization data sets.

## introduction dialogue summarization is a long-standing task in the nlp field and has recently gained traction through the emergence of novel data sets @xcite and community efforts like the automin foot_0 shared task or the summ-dial@sigdial 2021 special session foot_1 . dialogues can take a wide variety of forms ranging from formal interviews on a specific topic, political debates to informal conversations over the telephone. 3 therefore, the question of what is a suitable, appropriate summary of this type of data emerges. while abstractive and extractive summaries have emerged as the de-facto standard for summarization of continuous text (either single or multiple documents), the situation is less clear for dialogue summarization: what is a "proper" summary for the different types of dialogues? several dialogue corpora with associated humanwritten summaries have been created. these sum-maries differ significantly in type, style, and focus, depending on the instructions that were given to the human annotators. it is usually not clear why the particular summary type was chosen for the dialogue corpus at hand. in fact, we are not aware of any well-founded theory that answers this questions. to close this gap, we leverage the wellestablished linguistic model of dialogue types by @xcite to identify suitable summary items for the different types of dialogues. this results in a combination of linguistically defined dialogue types, their features, and the suitable summary items. we then place all existing dialogue data sets with summaries that we are aware of into this matrix. this allows us to map the available resources and to identify gaps, which opens up directions for future work. more precisely, this work presents four contributions: we also present the overview of existing dialogue summarization data sets in a comprehensive tabular overview in table a in the appendix. ## dialogue types in linguistics the analysis of dialogues within linguistics is mainly investigated in the fields of conversation analysis and pragmatics. a large body of work investigates speech acts @xcite , i.e. dialogues are decomposed into individual turns and their communicative intents are analysed. a smaller body of work focuses on establishing a typology of dialogues. among these works, walton and krabbe (1995) is a wellestablished model that is often cited and discussed. ## data sets -an overview we next provide an overview of existing dialogue summarization data sets. the overview is complemented by table a in the appendix which offers a compact and comprehensive outline of the data sets including descriptions, sizes, covered languages, and available summary types. we divide the data sets into the domains that they cover (meetings, broadcast interviews, customer and patient support, spontaneous conversation) and discuss applicable dialogue types. dialogues can be either spoken or written. while several corpora of written or more formal dialogues and their summarization have emerged recently @xcite , we here focus on corpora for summarization of (transcripts of) spoken dialogues, which is considerably different than summarization of text, as described for example in @xcite . work on summarizing spoken dialogues (i.e. involving more than one speaker) started in the late 1990s and early 2000s (see for example @xcite ). these already covered a great variety of different types of dialogues, such as tv discussions (newshour, cnn crossfire), phone calls (callhome, callfriend) and meetings. an overview of these early approaches into summarizing dialogues can be found in @xcite . at the same time, the verbmobil project, which focused on negotiations dialogues, also worked on summarising these @xcite . 6 ## mapping data sets to summary items given the overview of dialogue summarization resources and their mapping to dialogue types under the linguistic model in the previous section, and the summary items assigned to the dialogue types in section 2.2, we are now able to tabulate the corpora and the summary items 7 to see what areas in this space are covered and where there are opportunities for future work. specifically, we tabulate corpora and the nlp targets that are mapped to the summary items and insert the paper references that cover the summary item for a given corpus. we perform this mapping under the requirement that a resource explicitly annotates a given nlp target in a structured manner. that is, while a general, abstractive, manual summary of a meeting might include e.g. action items or decisions, they might not be marked explicitly as such in the summaries or the underlying transcripts. in such a setting, the resource would not enable the creation of summarization systems that explicitly extract e.g. action items. 8