MIDAS : A Dialog Act Annotation Scheme for Open Domain H uman M achine Spoken Conversations (2021)

Abstract: Dialog act prediction in open-domain conversations is an essential language comprehension task for both dialog system building and discourse analysis. Previous dialog act schemes, such as SWBD-DAMSL, are designed mainly for discourse analysis in human-human conversations. In this paper, we present a dialog act annotation scheme, MIDAS (Machine Interaction Dialog Act Scheme), targeted at open-domain human-machine conversations. MIDAS is designed to assist machines to improve their ability to understand human partners. MIDAS has a hierarchical structure and supports multi-label annotations. We collected and annotated a large open-domain human-machine spoken conversation dataset (consisting of 24K utterances). To validate our scheme, we leveraged transfer learning methods to train a multi-label dialog act prediction model and reached an F1 score of 0.79.

## introduction human-machine conversations have different dynamics compared to human-human conversations due to the power imbalance between humans and machines in conversational settings. such differences include content and voice quality @xcite . for instance, humans tend to use a more authoritative voice when they talk to a machine. we found that "commands" account for about 9% of utterances in a human-machine social conversation corpus we collected, while a similar dialog act, "action-directive", only accounts for 0.4% in a human-human conversation corpus (switchboard dialog act corpus (swda) @xcite ). moreover, a human-machine dialog act scheme is not simply used to understand the dialog flow but also to help dialog systems plan their 1 code, data, and trained models are available at https: //github.com/diandyu/midas_dialog_act in this paper, we propose a hierarchical multilabel dialog act annotation scheme, midas, specifically designed for real-time open-domain humanmachine spoken conversations. we show a comparison to major dialog act schemes in ## related work previous dialog act annotation schemes are mostly designed for task-oriented dialogs with a specific task, such as maptask @xcite and date @xcite , or in a specific setting @xcite . there are a few dialog act schemes designed for human-human task-independent conversations, such as the discourse annotation and markup system of labeling (damsl, core and allen, 1997) and swbd-damsl @xcite . swbd-damsl is used to annotate the switchboard @xcite corpus, a task-independent telephone conversation corpus, with inter-annotator agreement of κ = 0.80. swbd-damsl is also applied to annotate human-human meeting conversations @xcite . in this paper, we design a dialog annotation scheme specifically for humanmachine social chitchat conversations without any topic constraints. @xcite introduces a human-machine dialog act annotation scheme with 14 tags. however, the scheme is designed for modeling conversation topics instead of training dialog act predictors. the scheme has tags such as information request, general chat, and multiple goals, and the annotation is performed on unsegmented user utterances. even though the small number of tag categories makes annotation more reliable, it may not provide enough information for understanding user semantics. for example, tags such as multiple goals do not provide explicit information about user intent. in contrast, we propose a dialog act annotation scheme that focuses on improving open-domain dialog system understanding. we also build a dialog act predictor based on the annotated corpus. previously, most popular annotation schemes, such as damsl, use mutually-exclusive tags @xcite (0.01% of the labeled utterances have multiple labels so we do not consider damsl as multi-label) and each utterance is labeled with a single tag in swbd-damsl @xcite . however, @xcite argues that conversation utterances are complex. each functional segment can have four to five functions on average, so dialog act tags should serve multiple functions. dynamic interpretation theory (dit) @xcite and its extension, dit++ @xcite try to solve this problem by supporting multi-dimension and multi-function tags. the 88 tags are organized in a hierarchical structure and separated into dimensionspecific and general-purpose functions. the fifth version of dit++, iso 24617-2 (iso standard, @xcite , is introduced to incorporate not only linguistic theory but also empirical discourse analysis on real domain-independent conversations. although much effort has been expended in designing iso, no large dataset was annotated with the scheme except dialogbank @xcite . this is probably due to the complexity of the scheme and the lack of clear guidelines on how to use contextual information @xcite . due to the huge complexity of open domain social conversations, we propose a hierarchical structure in the annotation scheme and allow one utterance to have multiple dialog acts. we limit the number of dialog acts to 23 to capture the fundamental acts necessary to the system while keeping the total number relatively small (two annotators reached 0.94 in kappa). we ## midas annotation scheme we present midas, a contextual hierarchical multilabel dialog act annotation scheme for humanmachine conversations. midas follows dit++ and iso @xcite to ensure that the scheme facilitates both annotation and the training of automatic dialog act predictors. midas focuses on helping dialog systems understand their human users, while previous schemes mainly focus on analyzing human-human dialog with not fully open-domain data (swbd-damsl), mutually-exclusive tags (damsl), or lack of contextual information (dit++ and iso). therefore, midas provides a unique hierarchical structure and proposes a set of new labels for the humanmachine setting while inheriting labels from previous schemes. a complete description of midas is in appendix a.2. similar to chowdhury et al. (2016); mezza et al. 2018 , we also show a potential mapping from swbd, swbd-damsl, and iso to midas in appendix a.4. we discuss the three main features of midas: hierarchical structure, multi-label format, and context consideration respectively. ## dataset and annotation process we collected 380k human-machine conversations asr outputs using gunrock, the 2018 alexa prize winning social bot @xcite . the average number of turns in a conversation and the average tokens per user utterance is 21.76 and 2.85, respectively. detailed annotation results, including dialog act tag distribution, are shown in appendix a.2 and appendix a.3. user1: what do you want to talk about user2: what can you tell me what the top books are right now user1: i am so excited to talk to you about books. i'm actually a pretty big bookworm, and i love to read when i'm not chatting user2: oh [seg] what are some titles of the books you've read user1: recently, i'm reading the great gastby. it's really thought provoking, and i can see why some people call it the great american novel. how about you? what book do you like? user2: i have {not} read a book in a while [seg] do you have recommendations in the sci fi we note that there may be some limitations in the corpus we annotated in terms of both asr results, and the current interactions between humans and machines. our motivation is that compared to carefully corrected transcriptions such as swda, our scheme and annotated corpus can be more ro-bust against asr errors and noisy inputs. since the proportion of utterances with asr errors are relatively small in our corpus, models trained with the corpus should still perform well on utterances without such errors from more advanced asr systems. in addition, we believe that our hierarchical multilabel schemes with a manageable size and structure can be easily modified to customized models, as well as more advanced human-machine conversational models. moreover, our scheme is designed to bridge the gap in human and machine interactions, and are not restricted by the annotation corpus. more importantly, this scheme and annotated corpus can be extended outside of predictions and understanding tasks to language generation tasks. we leave detailed comparison and application to additional tasks such as language generation to future work. for dialog act prediction, user utterances in human-machine dialogs are asr outputs and have no punctuation. therefore, we train a model to segment utterances into semantic units for preprocessing. note that utterance segmentation is a different line of research and is out of the scope of this paper. we focus on dialog act prediction on each segmented unit following @xcite . previous research detects sentence boundaries by predicting the exact punctuation in the training dataset @xcite . however, correct punctuation also relies on deep semantic understanding beyond the sentence surface forms. for instance, a misused question mark can lead the dialog act model to predict a sentence as a question. so following @xcite , we only predict the boundary of a sentence instead of predicting punctuation to avoid introducing errors. it is expensive to annotate sentence boundaries, so we use the cornell movie-quotes corpus @xcite to train a sentence segmentation model. the cornell dataset contains 300k utterances from movie transcripts. we reformat the transcripts by replacing punctuation to sentence breaker tokens (denoted as @xcite ). we then train a sequence-to-sequence (seq2seq) model @xcite to predict sentence breaker similarly to @xcite and @xcite . the input to the model is a reformatted sentence, and the output is the same sentence with added sentence breaker tokens. an example can be seen in the last user2 utterance in ## dialog act prediction we formulate the dialog act prediction problem as a multi-label classification problem. following @xcite , we evaluate the proposed methods using f1 score. for evaluation simplicity, we predict one or two labels in the following experiments. we leverage both unlabeled data and annotated data to improve classification performance. ## results and analysis we found that incorporating context improves the model performance. adding text information as context improves the bert model from 71.30% to 79.11% in f1. we also compare the impact of different context embedding methods on dialog act classification performance. the results show that replacing text with dialog act achieves a high precision, but suffers from a low recall. this is because an utterance can have multiple intents while dialog act itself does not provide enough context information to achieve accurate prediction. for example, when "i don't think so" is a response to a simple yes/no question such as "have you read the book", it is a negative_answer. but if it is a response to a more complex yes/no question, such as "do you want to talk about books", then it has two tags, command and negative_answer. the latter conveys user's implicit request on changing the topic. therefore, only using dialog act as context could lead to high recall but low f1. we found combining both the previous segment's dialog act label and its surface text achieves the best performance in f1 (79.44%). however the performance improvement over including text only is not statistically significant. this suggests that dialog act and text may contain more overlapping information than complementary information. we also found that fine-tuning the model using the supervised dialog act prediction task on the swda data did not improve performance in f1 but improved recall slightly. the reduced performance may be due to difference in the data. even though both datasets are open domain conversational data, the swda task uses pre-processed switchboard data that does not have asr errors. moreover, swda is human-human conversations, and they are more coherent and consistent compared to human-machine conversations. another reason is that swda dataset has exactly one label for each utterance. when fine-tuning on our multilabel task, the pre-trained single-label model may tend to predict more labels to quickly reduce loss but fail to learn better representations. we further looked into the errors made by our best model (bert_f-da+text) and found that the model confuses statement non_opinion and general opinion. this is most likely caused by only including one turn context. sometimes, users have questions that break the conversation flow, such as "can you say it again clearly". the model needs to consider not only this utterance but also the prior turns to perform dialog act prediction. we plan to incorporate longer context in future work. in addition, some of the nonsense sentences are misclassified as statement non_opinion such as "it doesn't outside break a car". it is also worth noting that some incorrectly segmented units resulted in inaccurate dialog act prediction. for instance, the utterance "we are they love each other" is incorrectly segmented into "we" and "are they love each other" given the context "you must be great friends with your dogs". the second segment is incorrectly predicted as a yes/no question. ## conclusion we propose midas, a dialog act scheme designed for open-domain human-machine conversational systems. midas is a hierarchical annotation scheme that supports multiple labels. we annotated 24k sentences from a human-machine social conversation dataset using midas. we also trained dialog act classification models based on the annotated dataset. we tested different transfer learning techniques to improve model performance. we found that fine-tuning a pre-trained bert model on unannotated target human-machine conversations improved model performance. however, fineturning the model on a supervised dialog act task with human-human conversations did not improve model performance. midas has been deployed in real world dialog systems, demonstrating its efficacy in the open-domain dialog setting. beyond prediction in open-domain dialogs for human request understanding, our proposed scheme can also be applied to task-oriented dialogs together with chit-chats. furthermore, midas can be used in end-to-end dialog systems for cases such as probing and model interpretation. in addition, our scheme and method can be extended to controlled text generation. we will explore broader applications in future work.