Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification (2022)

Abstract: Dialogue Act Classification (DAC) that determines the communicative intention of an utterance has been investigated widely over the years as a standalone task. But the emotional state of the speaker has a considerable effect on its pragmatic content. Sentiment as a human behavior is also closely related to emotion and one aids in the better understanding of the other. Thus, their role in identification of DAs needs to be explored. As a first step, we extend the newly released multi-modal EMOTyDA dataset to enclose sentiment tags for each utterance. In order to incorporate these multiple aspects, we propose a Dual Attention Mechanism (DAM) based multi-modal, multi-tasking conversational framework. The DAM module encompasses intra-modal and interactive inter-modal attentions with multiple loss optimization at various hierarchies to fuse multiple modalities efficiently and learn generalized features across all the tasks. Additionally, to counter the class-imbalance issue in dialogues, we introduce a 2-step Deferred Optimisation Schedule (DOS) that involves Meta-Net (MN) learning and deferred re-weighting where the former helps to learn an explicit weighting function from data automatically and the latter deploys a re-weighted multi-task loss with a smaller learning rate. Empirically, we establish that the joint optimisation of multi-modal DAC, SA and ER tasks along with the incorporation of 2-step DOS and MN learning produces better results compared to its different counterparts and outperforms state-of-the-art model.

## introduction dialogue act classification (dac) constitutes an important means for understanding a speaker's communicative intention (for example, question, command, apology etc.) in any dialogue system @xcite , @xcite . thus, da seeks to analyze the pragmatics of a conversation instead of just its literal meaning. authors of @xcite went a step ahead and established in a multi-modal setting (including text, audio and video) that a speaker's true communicative content is greatly influenced by its emotional state of mind @xcite . utterances such as "oh sure" or "ya why not" can be understood as "agreement" or "disagreement" (if implied sarcastically). however, the emotional state of the speaker might enclose cues giving it another definition altogether. sentiment and emotion are frequently viewed as two different entities @xcite @xcite etc., but are often interpreted in a similar way and are therefore used interchangeably due to their subjective character. but sentiment and emotion are not literally the same, but are strongly linked. for example, emotions such as happy and joy are inherently related to a positive sentiment. thus, the speaker's emotion and sentiment are intertwined and one aids in better understanding of the other. as a result, information pertaining to emotion, as well as sentiment, provides a better comprehension of the speaker's state of mind. this strong relationship between emotion and sentiment drives us to incorporate the speaker's sentiment as well as its emotion while modeling das. additionally, we seek to address the classimbalance issue for the task of dac, as not all das are equally represented or are equally occurring in a conversation. when the training dataset has a high degree of class-imbalance, the testing criterion necessitates strong generalisation on less frequent classes @xcite . to address this issue, a sample reweighting approach is typically utilised @xcite @xcite , which involves creating a weighting function that maps training loss to sample weight. currently, employing this strategy requires manually pre-specifying the weighting function. however, this approach is not scalable in practice ow-ing to the variations of an ideal weighing scheme based on the investigating task and training data at hand. in this paper, we leverage from the concept of meta-learning @xcite to develop a method capable of learning an explicit weighting function from the data itself in an adaptable manner, named, meta-net (mn) learning. simultaneously, we apply an effective training schedule (inspired by @xcite ) on top of mn learning, namely, two-step deferred optimization schedule (2-step dos). the 2-step dos postpones or defers the re-weighting so that the classifier learns an initial representation while avoiding some of the complexities involved with re-weighting or re-sampling (incase of classimbalance). the contributions of this work are as follows : (i) we propose a dual attention mechanism (dam) based multi-task framework for multi-modal dac, sa and er in conversations. we leverage the information pertaining to emotional state and sentiment of the speaker to identify das; (ii) additionally, we introduce a 2-step dos that involves mn learning and deferred re-weighting to counter the class-imbalance issue for the task of dac; (iii) in order to integrate these various facets, we extend the newly created dataset, emotyda, to encompass annotations of the sentiment tags. we surmise that this extended characteristic of emotyda will introduce novel sub-task for future investigation: sentiment and emotion aided dac; (iv) we illustrate the gain in different measures that jointly optimizing these three tasks (dac, sa and er) using our proposed framework with the incorporation of 2-step dos and mn learning produces better results compared to its different counterparts and state-of-the-art model. in @xcite @xcite , authors presented several dl based approaches to study the role of sentiment in identifying speech acts for a social media platform called mastodon. in @xcite , authors made an attempt to determine correlation between das and basic emotion tags for an in-game japanese conversation. in @xcite , authors introduced a large-scale, multi-modal conversational data annotated with das and emotions in order to establish that emotion indeed aided the task of dac. however, they did not make use of sentiment of the speaker which is yet another crucial user behavior that can aid in understanding the das better. in @xcite , authors introduced the concept of studying speech acts in correlation with sentiment and emotion but it was meant for the social media communication in twitter with no dialogic structure. authors of @xcite @xcite proposed several correlated tasks in a dialogue system that leverages with the addition of sentiment and/or emotion in its learning process. ## dataset the newly created multi-modal (i.e., text, audio and video), emotion-da dataset: emotyda @xcite , consists of 1341 dyadic and multiparty conversations resulting in a total of 19,365 utterances and approximately 22 hours of recordings. in this dataset, utterances are annotated with 12 da tags with the corresponding 10 emotion tags. the details of the da and emotion tags are mentioned in the appendix below. so, this dataset is manually re-annotated for its related sentiment labels. emotyda dataset is curated using conversations from meld @xcite and iemocap @xcite datasets. in case of meld, pre-annotated sentiment labels of the utterances were already existing. we chose to use the same sentiment labelling as released in the source dataset. however, the iemocap dataset contains solely pre-annotated emotion tags without any sentiment labels foot_0 . three annotators were hired for the task of sentiment annotation. they were asked to manually annotate the utterance by viewing the corresponding video and context to assign its sentiment label, namely positive, negative and neutral. we observed an inter-annotator score of 78% which can be considered reliable. statistics of the dataset related to sentiment and relation between the different tasks are shown in figure 1 . other statistics as well as the process of resolving disagreement amongst annotators are reported in the appendix below. ## proposed methodology the proposed approach and implementation details will be outlined in this section. problem statement. for the multi-task set-up, let us consider a training set, {x i , y i , w i , z i } n @xmath0, where x i is the i-th sample, y i , w i and z i are the label vectors for dac, sa and er tasks, respectively and n is the number of training instances. f (x, w) denotes the multi-task, multi-modal classifier, called the primary network (say) and w is its parameters. the task is to find the optimal parameter, w * , by minimizing the multi-task training loss (combined loss from each of the three tasks), 1/n n @xmath1, where l train i (w) = l(y i , f (x i , w)). ## results and analysis we carried out a number of experiments to assess the efficacy of the proposed method. experiments were carried out for various combinations of multitasking with dac as the crucial task, as well as for varying modalities, in addition to the single task dac variation along with mn and dos based learning. this was followed by experiments in a conversational framework and compared against single utterance classification. in in figure 4 , we present a visualization of the learned weights of an utterance from the ia u layer (as this layer contains word-wise attention scores). the true da tag of this particular utterance is disagreement. the importance of disagreement bearing words are learnt well for the multi-task approach as opposed to the single-task dac model where attention is on compliance bearing word such as fine. with dac+sa, dac+er and dac+sa+er respectively, the degrees of importance of correct/incorrect words have increased/decreased gradually as enhanced information is learnt due to the effect of different tasks and its combinations. during a detailed analysis, it was observed that expressive das such as 'greeting", "acknowledge", "apology", "command", "agreement", "disagreement" are sensitive to the presence of sentiment and emotion etc. for e.g., utterance such as "that's very amusing indeed" was identified as "agreement" in the single task dac model, but was correctly classified as "disagreement" in the proposed multi-task, dac+sa+er model as the sentiment and emotion of the utterance were "negative" and "angry", respectively, given the context that the speaker was disagreeing with the hearer in a sarcastic manner. it was also observed that for longer utterances comprising of composite sentences, sentiment and emotion of the speaker did play significant role in correctly identifying the da tag. for eg., an utterance such as "hey, i'm, uh. i'm really sorry about what happened. i don't um-i mean what you can you do?"