Towards Emotion-aided Multi-modal Dialogue Act Classification (2020)

Abstract: The task of Dialogue Act Classification (DAC) that purports to capture communicative intent has been studied extensively. But these studies limit themselves to text. Non-verbal features (change of tone, facial expressions etc.) can provide cues to identify DAs, thus stressing the benefit of incorporating multi-modal inputs in the task. Also, the emotional state of the speaker has a substantial effect on the choice of the dialogue act, since conversations are often influenced by emotions. Hence, the effect of emotion too on automatic identification of DAs needs to be studied. In this work, we address the role of both multi-modality and emotion recognition (ER) in DAC. DAC and ER help each other by way of multi-task learning. One of the major contributions of this work is a new dataset- multimodal Emotion aware Dialogue Act dataset called EMOTyDA, collected from open-sourced dialogue datasets. To demonstrate the utility of EMOTyDA, we build an attention based (self, inter-modal, inter-task) multi-modal, multi-task Deep Neural Network (DNN) for joint learning of DAs and emotions. We show empirically that multi-modality and multi-tasking achieve better performance of DAC compared to uni-modal and single task DAC variants.

## introduction dialogue act classification (dac) is concerned with deciding the type i.e., communicative intention @xcite of the speaker's utterance. dac is very important in the context of discourse structure, which in turn supports intelligent dialogue systems, conversational speech transcription and so on. considerable works have been done on classical machine learning (ml) based dac @xcite , @xcite , @xcite , etc. and deep ## related works the tasks of er and dac are extensively explored. dialogue act frameworks: dac has been investigated since late 90s @xcite , @xcite and early 2000's @xcite , @xcite . much of this research, however, uses chat transcripts with only the text mode, due partly due to unavailability of multi-modal open-source dataset. in @xcite , authors apply stacked lstm to classify speech acts. in @xcite , the author developed a hierarchical network based approach using bi-lstms and the crf. a contextual self-attention system fused with hierarchical recurrent units was proposed by the authors of (raheja and tetreault, 2019) to develop a sequence label classifier. the authors of @xcite proposed a method for the capture of long-range interactions that span a series of words using a convolutional network based approach. in @xcite , authors proposed several ml and dl based approaches such as conditional random fields, clustering and word embeddings to identify das. however, all these works identify das by utilizing solely the textual modality without the use of emotional cues. emotion aware das. within a multi-modal setting, little work is available in the literature to study the impact of emotional state in the evaluation of das. the effect of integrating facial features as a way of identifying emotion to classify das was examined by authors in @xcite . they exhibited their work for tutorial dialogue session typically task-oriented and applied logistic regression to identify das. but they studied only the cognitive-affecting states such as confusion and flow as the emotional categories to learn das. in @xcite , authors examined the impact of affect analysis in da evaluation for an unsupervised dac model. the authors made use of lexicon based features from wordnet affect and sentiwordnet to map them with emotion labels to model the das in a lsa based approach. authors of @xcite , also inspected the impact of emotions mediated with intention or das for an in-game japanese dialogue. their goal was to construct da-emotion combinations from the pre-annotated corpus. however, such stringent associations or dis-associations amongst da-emotion pairs may not truly hold for real life conversations. ## dataset to facilitate and enhance the research in multimodal dac assisted with user emotion, we introduce a new dataset (emotyda) consisting of short videos of dialogue conversations manually annotated with its da along with its pre-annotated emotions. ## proposed methodology this section describes the proposed multi-task, multi-modal approach followed by the implementation details. ## results and analysis emotyda contains dialogues pertaining to dyadic and multi-party speakers, so, we performed experi as is also evident from the results, the multi-task variant performs consistently well throughout all the experiments compared to its single task dac variant. as a baseline, we also show that using emotion as a feature in the single task dac counterpart doesn't outperform the proposed multi-task variant. this shows that the joint optimization of both these tasks boosts the performance of dac. figure 6 shows the visualization of the learned weights of different words for a sample utterance for the single task dac as well as the multi-task model to highlight the importance of incorporating er as an auxiliary task. the true da label of the utterance in figure 6 is disagreement with emotion as anger. with the multi-task approach, the attention is laid on appropriate disagreement bearing words whereas with single task, attention is learnt on agreement words such as yes which here has just been used in a sarcastic way to disagree. it is also observed that the experiments with dyadic conversations attain better results as compared to multi-party conversations. this is supposedly due to the constant change of speakers in multi-party conversations that misleads the classifier to learn suboptimal features, thus, stressing on the role of using speaker information as valuable cues for dac. ## conclusion and future work in this paper, we investigate the role of emotion and multi-modality in determining das of an utterance. to enable research with these aspects, we create a novel dataset, emotyda, that contains emotion-rich videos of dialogues collected from various open-source datasets manually annotated with das. consequently, we also propose an attention based (self, inter-modal, inter-task) multi-modal, multi-task framework for joint optimization of das and emotions. results show that multi-modality and multi-tasking boosted the performance of da identification compared to its unimodal and single task dac variants. in future, conversation history, speaker information, fine-grained modality encodings can be incorporated to predict da with more accuracy and precision.