Towards Sentiment and Emotion aided Multi-modal Speech Act Classification in T witter (2021)

Abstract: Speech Act Classification determining the communicative intent of an utterance has been investigated widely over the years as a standalone task. This holds true for discussion in any fora including social media platform such as Twitter. But the emotional state of the tweeter which has a considerable effect on the communication has not received the attention it deserves. Closely related to emotion is sentiment, and understanding of one helps understand the other. In this work, we firstly create a new multi-modal, emotion-TA (‘TA’ means tweet act, i.e., speech act in Twitter) dataset called EmoTA collected from open-source Twitter dataset. We propose a Dyadic Attention Mechanism (DAM) based multi-modal, adversarial multi-tasking framework. DAM incorporates intra-modal and inter-modal attention to fuse multiple modalities and learns generalized features across all the tasks. Experimental results indicate that the proposed framework boosts the performance of the primary task, i.e., TA classification (TAC) by benefitting from the two secondary tasks, i.e., Sentiment and Emotion Analysis compared to its uni-modal and single task TAC (tweet act classification) variants.

## introduction identification of speech acts is one of the preliminary means of determining the communicative intent or pragmatics of a speaker (for example, statement, request, question etc.). this is true for dialogue system, speech transcription, social media such as twitter, myspace etc. twitter is one of the leading micro-blogging services. by 2019, 330 million users were active monthly and 500 million tweets were sent per day 1 . identification of tweet acts (tas-speech acts in twitter) is highly beneficial for twitter as well as tweeters. for twitter, it helps decipher a particular subject in terms 1 https://www.omnicoreagency.com/twitter-statistics/ of speech acts and discrepancy identification. it also helps in social media monitoring by analysing topic alteration or spamming. it assists the followers in monitoring and scanning the subject with the most advantageous speech acts based on their needs. this helps reduce their search space and encourages them to obtain useful information from out of millions of tweets. it gives the tweeter a greater sense of the content, mood and trend. a person's emotional state and sentiment greatly impacts its intended content @xcite . often sentiment and emotion are treated as two different problems @xcite , @xcite , @xcite , (hossain and muhammad, 2019), @xcite . however, sentiment and emotion are are closely related. for example, emotions such as happy and joy are inherently related to a positive sentiment. but emotion is much more nuanced and fine-grained compared to sentiment @xcite . emotion along with sentiment provides better understanding of the state of mind of the tweeter. for example, a question or statement is associated with anticipation. an opinion is many times associated with anger or disgust. the close association between emotion and sentiment motivates considering tweeter's sentiment along with emotion while deciphering the tweet acts. for expressive tas such as "expression", "request", "threat" etc., the tweeter's sentiment and emotion can aid in classifying true communicative intent and vice-versa. additionally, multi-modal inputs, i.e., the combination of text and other nonverbal cues (emojis in tweets) @xcite help create reliable classification models aiding the identification of emotional state and sentiment of the tweeter which in turn help in determining correct tas. in this paper, we leverage the relationships as delineated above to predict tas of tweets in a multimodal framework. in this multi-task framework, tac is treated as the primary task and sentiment analysis (sa) and emotion recognition (er) as auxiliary (i.e., secondary) tasks. contributions of this paper are as follows : i. we create a new dataset called emota consisting of tweets with high-quality annotations of tas, including emotionally aided and multi-modal cues; ii. we establish the need for considering the sentiment and emotional state of the tweeter while identifying tas. iii. we propose a dyadic attention mechanism (dam) based multi-task adversarial learning framework for multi-modal tac, sa and er. in dam, we incorporate intra-modal and inter-modal attention to integrate information across multiple modalities and learn generalized features across multiple tasks; iv. we illustrate performance gains by jointly optimizing tac, sa and er. multi-modal and multi-task tac performs significantly better than its uni-modal and single task tac variants. ## related works there exist plenty of works which address the task of tac as a standalone problem. in @xcite , @xcite , authors proposed machine learning based approaches for tac namely support vector machines (svm), logistic regression etc. in @xcite , authors proposed a first ever public dataset for the identification of speech acts in twitter followed by a capsule based network built on top of bert for tac. in @xcite , authors highlighted the importance of identification of tweet acts and established it to be one of the elementary steps for detection of rumours in twitter. in @xcite , authors proposed an attention based model built on top of the transformer for predicting tas. in @xcite , authors proposed a capsule based network built on top of bert for tac. all these works utilized only the textual modality to identify tas without any sentiment or emotional correlation of the tweeter. in @xcite , authors proposed a lstm based study for jointly optimizing sa and tac in a decentralized social media platform called mastodon. however, they modelled their task as a multi-party conversation pretty different in essence to that of twitter analysis. in @xcite , authors presented a semi-supervised approach to identify speech acts in emails and different forums. these works, however, use datasets that comprise of face-to-face or telephone data that can not directly aid in advancing work on endless data in electronic mode such as micro-blogging networks, instant-messaging, etc. apart from these, identification of speech acts has been studied extensively for dialogue conversations starting from early 2000's with @xcite being one of the benchmark works where the authors presented varieties of approaches such as hidden markov models, neural networks and decision trees to identify dialogue acts on a benchmark dialogue data known as the switchboard (swbd) @xcite dataset. in @xcite , authors studied the role of emotion in identifying dialogue acts for a dyadic conversation by considering thee textual and the audio modality of the utterances in the conversation. in @xcite , authors proposed studying the role of emotion in determining dialogue acts on a dyadic and multi-party conversational dataset in a multi-modal framework (incorporating text, audio and video). however, tweets are unstructured and noisy communications with spelling mistakes, random coinages with limitations in expression because of character constraint per tweet. this makes it very different from face-to-face or other conversations. ## proposed methodology the proposed multi-tasking, multi-modal approach and implementation details are outlined in this section. ## results and analysis a series of experiments were conducted for evaluating the proposed approach. experiments were conducted for single task and several combinations of multi-task framework with tac being the pivotal task along with varying modalities. a thorough ablation study is performed to analyze the importance of each of the attention mechanisms of the proposed architectural framework along with several variations of multi-task learning (e.g., fs, sp etc.). note that we aim to enhance the performance of tac with the help of other two auxiliary tasks. following this, we report results and analysis with tac strictly being the pivotal task in all the task combinations. since, the dataset is unbalanced for all the task categories, we report results for different dimensions of tac in the following set-up: as stated above, we treat sa and er as auxiliary tasks aiding the primary task, i.e., tac. however, we report the performance of sa and er tasks on the proposed model for single as well as multi-task frameworks in comparison amongst different multi-task architecture. in terms of varying ways of multitasking such as fs, sp along with adversarial loss (adv), it was observed that sp model gave better results compared to fs model. additionally, incorporating adversarial loss further boosted the performance of different multi-task models. intuitively, as tac shares lesser amount of correlation with sa and er compared to sa and er themselves, fs model was not sufficient enough to learn diverse features across different tasks. this observation is in conformity with the existing literature. we also demonstrate the importance of different attentions used for the best performing multi-task model, i.e., sp+adv. furthermore, we also report results by replacing bert model to extract textual representation with glove embeddings @xcite . results indicate that each of these aspects contributed significantly to aid the performance of the proposed multi-tasking framework. all the reported results here are statistically significant @xcite . ## conclusion and future work in this paper, we studied the role of sentiment and emotion in speech act classification in twitter. we curate a novel dataset emota, that contains pre-annotated tweets with emotions collected from open-source dataset and annotated with tas and sentiment categories. we propose a dyadic attention mechanism based multi-modal (emojis and text), adversarial multi-task framework for joint optimization of tas, sentiment and emotions. the dam (dyadic attention mechanism) module employs intra-modal and inter-modal attention to fuse multiple modalities and learn generalized features across all the tasks. results show that multi-modality and multi-tasking boosted the performance of ta identification compared to its uni-modal and single task tac variants. in future, attempts will be made to predict tas with more precision by incorporating fine-grained modality encodings and also identifying which other nlp tasks (e.g., named entity recognition) might assist tac as a task.