On the Need for Thoughtful Data Collection for Multi-Party Dialogue: A Survey of Available Corpora and Collection Methods (2021)

Abstract: We present a comprehensive survey of available corpora for multi-party dialogue. We survey over 300 publications related to multi-party dialogue and catalogue all available corpora in a novel taxonomy. We analyze methods of data collection for multi-party dialogue corpora and identify several lacunae in existing data collection approaches used to collect such dialogue. We present this survey, the first survey to focus exclusively on multi-party dialogue corpora, to motivate research in this area. Through our discussion of existing data collection methods, we identify desiderata and guiding principles for multi-party data collection to contribute further towards advancing this area of dialogue research.

## introduction to say research in conversational agents and natural language generation has seen an explosive growth in recent years would be an understatement, as evidenced by the increasing number of papers published on this topic. however, most current research in this area has focused on two-party or dyadic conversations. this focus is important, since many open questions remain with dialogue systems in dyadic settings, such as modeling longterm dialogue context modeling and infusion of knowledge, persona and empathy @xcite @xcite nevertheless, there is still a pressing need to focus on more naturally occurring conversations which consist of more than two speakers @xcite , also known as multiparty dialogue. humans naturally tend to work in groups and teams. conversational agents capable of working in multi-party dialogue situations stand to advance the future of work, since they can be integrated into teams, e.g., in surgery, search and rescue, or manufacturing and design. the settings for such agents could be informal (e.g. chatroom assistants) or formal (e.g. meeting assistants) settings. particularly, with conversational assistants such as amazon alexa, there is a push to develop ai to understand multiple users and act as teammates @xcite . at the same time, methods and models built for two-party cannot simply be generalized for multi-party conversations. some challenges that are unique to multi-party dialogue include speaker identification (figuring out who is speaking), turntaking (understanding whether to respond or not) and tailoring the content of the response to each agent or person @xcite . several of these challenges can be approached through data-driven methods @xcite . given that corpora are the currency for data-driven methods, and facilitate further research on building data-driven multi-party dialogue systems, we present this systematic survey of existing corpora for multi-party dialogue. we describe how these corpora (section 3) were collected (section 4) along with the tasks that are undertaken on these corpora. our key goal is to identify desiderata that could help guide data collection efforts towards making research in multi-party dialogue more mature (section 5). our survey follows prior efforts in systematic reviews of dialogue corpora @xcite , evaluation of chatbots @xcite , and nlg evaluation @xcite . @xcite provide a meticulous survey of the state-of-the-art in natural language generation, however they do not include a separate discussion on corpora. the systematic review of dialogue corpora conducted by @xcite does not primarily focus on multi-party corpora. @xcite provide a systematic survey on the evaluation of dialogue systems, which includes a section of datasets and benchmarks, but again the focus is not primarily towards multi-party dialogue systems. consequently, the goal of this article is to make the following contributions: as part of our screening process, we limit the discussion to corpora that (a) have already been used in existing research in conversational systems; (b) which have a text component, and focus on the english language; and (c) which include multiple speakers in the majority of conversations, finally resulting in 343 papers. we release our annotated references to the 343 papers on github foot_0 . unsurprisingly, we found that majority of corpora papers were published in lrec and sigdial venues, in addition to *acl venues. organizing corpora by genre: next, we organized all included corpora into a new taxonomy (figure 1 ). corpora are first categorized by whether they include spoken or written dialogue. spoken corpora are further divided as unscripted vs. scripted. within these type-based divisions, the corpora are then arranged by their main sources. the unscripted spoken corpora are thus arranged into 4 main categories -informal discourse mainly consisting of informal interactions such as radio talk shows, formal discourse mainly consisting of formal interactions such as debates, spontaneous speech mainly consisting of spontaneous interactions such as teenage talk, and meetings and interviews mainly focused on data from sources such as tv interviews. similarly, the scripted spoken corpora are arranged into scripts and dialogues from plays, movies and tv series. lastly, the written corpora are arranged into four categories-synchronous mainly consisting of chatroom talk, and online game-playing forums with users mainly conversing about game progression; and asynchronous mainly consisting of posts made on online forums and short text messages on microblog websites with character limits for posts. tables 1 and 2 present additional details about each corpus, including the name and source citation, topics presented, quantitative details such as number of dialogues, words, total length, and speakers, as well as whether they are multi-modal. all the available corpora have been used for datadriven research on multi-party dialogue. we thus include the task descriptions each corpus has been used for in the past. these tasks range from machine reading comprehension and turn-taking to speaker-identification. ## existing corpora for multi-party dialogue in the subsections below, we outline the descriptions of each corpus. ## data collection methods several methods of data collection have been used to collect the aforementioned corpora. we organize these into three main categories and discuss in detail below. aggregated from various sources: bnc, cancode, and micase employ the aggregation method to build the corpora. they pull information from various sources, including text from sources such as newspapers, journals, publicly available government meetings, radio phone-ins, academic writings, seminars, advising sessions etc. these corpora incorporate multiple types of speech, and often include speech surrounding multiple topics (especially bnc and cancode, micase mainly focuses on academic settings to collect data). they are thus great candidates for studying language semantics and have been employed to study largescale vocabularies @xcite and word sense disambiguation @xcite in the past. transcribed from pre-recorded media: single (or double) source origins, such as colt, crd3, and iq2, maintain focus on certain themes, such as formal meeting data. these are not collected within specialized environments, but consist of either transcribed speech recorded in the wild, transcribed interviews & meetings, and online forum or social media data. this category also includes scripted corpora, which are usually collections of various scripts & dialogues from plays, movies and tv series, such as tvd and subtle. having a set theme allows these corpora to be used for generating themed text such as meld being used for character identification as a part of the 2018 semeval challenge @xcite . collected in specialized environments: most multi-modal corpora employ specialized environments or equipment to collect data that can be synchronized across multiple modalities. most focus on data collection using audio, which can then be transcribed. specialized room environments with studio-quality recording (icsi, ami), closetalking mics (icsi, idiap wolf, teams), and a combination of far-and close-field mics (cosine, ## desiderata for data collection given the multitude of corpora available and the modeling tasks that need to be undertaken to develop conversational agents for multi-party dialogue, we outline here three key criteria for future efforts in data collection: ## discussion the three desiderata listed above provide us with a set of guidelines for thinking about the challenges for thoughtful data collection. this (potentially non-exhaustive) list of questions is inspired by the current movement in several research fields to preregister studies in advance @xcite and the needs for datasheets for datasets @xcite .