Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure (2020)

Abstract: Research into the area of multiparty dialog has grown considerably over recent years. We present the Molweni dataset, a machine reading comprehension (MRC) dataset with discourse structure built over multiparty dialog. Molweni’s source samples from the Ubuntu Chat Corpus, including 10,000 dialogs comprising 88,303 utterances. We annotate 30,066 questions on this corpus, including both answerable and unanswerable questions. Molweni also uniquely contributes discourse dependency annotations in a modified Segmented Discourse Representation Theory (SDRT; Asher et al., 2016) style for all of its multiparty dialogs, contributing large-scale (78,245 annotated discourse relations) data to bear on the task of multiparty dialog discourse parsing. Our experiments show that Molweni is a challenging dataset for current MRC models: BERT-wwm, a current, strong SQuAD 2.0 performer, achieves only 67.7% F1 on Molweni’s questions, a 20+% significant drop as compared against its SQuAD 2.0 performance.

## introduction research into multiparty dialog has recently grown considerably, partially due to the growing ubiquity of dialog agents. multiparty dialog applications such as discourse parsing and meeting summarization are now mainstream research @xcite @xcite @xcite @xcite . such applications must consider the more complex, graphical nature of discourse structure: coherence between adjacent utterances is not a given, unlike standard prose where sequential guarantees hold. in a separate vein, the area of machine reading comprehension (mrc) research has also made unbridled progress recently. most existing datasets for machine reading comprehension (mrc) adopt well-written prose passages and historical questions as inputs @xcite @xcite @xcite . reading comprehension for dialog -as the intersection of these two areas -has naturally begun to attract interest. @xcite constructed a small dataset for passage completion on multiparty dialog, but which has been easily dispatched by cnn+lstm models using attention. the dream corpus @xcite is an mrc dataset for dialog, but only features a minute fraction (1%) of multiparty dialog. friendsqa is a small-scale span-based mrc dataset for multiparty dialog, which derives from tv show friends, including 1,222 dialogs and 10,610 questions @xcite . the limited number of dialogs in friendsqa makes it infeasible to train more complex model to represent multiparty dialogs due to overfitting, and the lack of annotated discourse structure prevents models from making full use of the characteristics of multiparty dialog. dialog-based mrc thus varies from other mrc variants in two key aspects: c1. utterances of multiparty dialog are much less locally coherent than in prose passages. a passage is a continuous text where there is a discourse relation between every two adjacent sentences. therefore, dialogue 1 nbx909: how do i find the address of a usb device ? 𝑈 1 likwidoxigen: try taking it out to dinner and do a little wine and dine and it shoudl tell ya 𝑈 2 likwidoxigen : what sort of device ? 𝑈 3 babo: ca n't i just copy over the os and leave the data files untouched ? 𝑈 4 nbx909 : only if you do an upgrade 𝑈 5 nuked: should i just restart x after installing 𝑈 6 likwidoxigen: i 'd do a full restart so that it re-loads the modules 𝑈 7 q1: why does likwidoxigen do a full restart? a1: it re-loads the modules q2: what does nbx909 want to do? a2: find the address of a usb device q3: how to restart network? a3: na. figure 1 : (dialog 1) a corpus example from molweni. there are four speakers in the dialog: nbx909, likwidoxigen, babo, and nuked. in total, the speakers make seven utterances: u 1 to u 7 . our annotators proposed three questions against the provided dialog: q1-3, where q1 and q2 are answerable questions, and q3 is unanswerable. due to the properties of informal dialog, the instances in our corpus often have grammatical errors. c2. multiparty dialog subsumes the special case of two-party dialog. in most cases, the discourse structure of a two-party dialog is tree-like, where discourse relations mostly occur between adjacent utterances. however, in multiparty dialog, such assumptions hold less often as two utterances may participate in discourse relations, though they are very distant. prior mrc works do not consider the properties of multiparty dialog. to address this gap in understanding of multiparty dialog, we created molweni. in dialog 1 (cf. fig 1 ), four speakers converse over seven utterances. we additionally employ annotators to read the passage and contribute questions: in the example, the annotators propose three questions: two answerable and one unanswerable. we observe that adjacent utterance pairs can be incoherent, illustrating the key challenge. it is non-trivial to detect discourse relations between non-adjacent utterances; and crucially, difficult to correctly interpret a multiparty dialog without a proper understanding of the input's complex structure. we derived molweni from the large-scale multiparty dialog ubuntu chat corpus @xcite . we chose the name molweni, as it is the plural form of "hello" in the xhosa language, representing multiparty dialog in the same language as ubuntu. our dataset contains 10,000 dialogs with 88,303 utterances and 30,066 questions including answerable and unanswerable questions. all answerable questions are extractive questions whose answer is a span in the source dialog. for unanswerable questions, we annotate their plausible answers from dialog. most questions in molweni are 5w1h questions -why, what, who, where, when, and how. for each dialog in the corpus, annotators propose three questions and find the answer span (if answerable) in the input dialog. to assess the difficulty of molweni as an mrc corpus, we train bert's whole word masking model on molweni, achieving a 54.7% exact match (em) and 67.7% f 1 scores. both scores show larger than 10% gap with human performance, validating its difficulty. due to the complex structure of multiparty dialog, human performance just achieves 80.2% f 1 on molweni. in particular, annotators agreed that knowledge of the correct discourse structure would be helpful for systems to achieve better mrc performance. this comes to the second key contribution of molweni. we further annotated all 78,245 discourse relations in all of molweni's dialogs, in light of the potential help that annotated discourse structure might serve. prior to molweni, the stac corpus is the only dataset for multiparty dialog discourse parsing @xcite . however, its limited scale (only 1k dialogs) disallow data-driven approaches to discourse parsing for multiparty dialog. we saw the additional opportunity to empower and drive this direction of research for multiparty dialog processing. ## the molweni corpus our dataset derives from the large scale multiparty dialogs dataset -the ubuntu chat corpus @xcite . we list our three reasons in choosing the ubuntu chat corpus as the base corpus for annotation. the discourse dependency structure of each multiparty dialog can be regarded as a discourse dependency graph where each node is an utterance. to learn better graph representation of multiparty dialogs, we filter the ubuntu chat corpus for complex dialogs -those dialogs with 8-15 utterances and 2-9 speakers. as multiparty dialog is already intensely complex for the current state of the art, we chose to further simplify in our selection criteria, additionally filtering out dialogs with long utterances (more than 20 words). finally, we randomly chose a subset of ∼10,000 dialogs with 88,303 utterances from the ubuntu dataset. we give an overview of molweni's key demographics in in total, our subjects annotated 9,754 dialogs, slightly fewer than 10,000 dialogs, consisting of 88,303 utterances, and contributed 30,066 questions for machine reading comprehension and 78,245 discourse relation annotations. there are 8,771 dialogs in the demarcated train set for both machine reading comprehension and discourse parsing tasks. 883 dialogs are used for dev set. each annotator is asked to propose three questions per dialog. there are 100 dialogs in common for all ten annotators, and these 100 dialogs comprise our test set. each dialog in the training set and develop set has three questions. our annotation team proposed a total of 2,871 questions for the 100 dialogs in test sets. detailed statistics are shown in ## experiments we now introduce our baseline experiments on our dataset. we consider the two tasks of discourse parsing and machine comprehension for multiparty dialog. ## conclusion in this paper, we introduce molweni, a multiparty dialog dataset for machine reading comprehension (mrc). compared with traditional textual structure, the dialog is concatenated by the utterances from multiple participants. we believe that discourse structure can provide potential help for understanding the dialog. therefore, we ask annotators to label the discourse dependency structure of the multiparty dialog and propose questions for the dialog. annotation on a large number of dialog shows that tagging discourse structure can significantly help taggers understand dialog and raise higher quality questions. in the future, we will try to propose novel discourse parsing models for multiparty dialog and apply discourse structure in the reading comprehension task of multiparty dialog.