Multilogue-Net: A Context-Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation (2020)

Abstract: Sentiment Analysis and Emotion Detection in conversation is key in several real-world applications, with an increase in modalities available aiding a better understanding of the underlying emotions. Multi-modal Emotion Detection and Sentiment Analysis can be particularly useful, as applications will be able to use specific subsets of available modalities, as per the available data. Current systems dealing with Multi-modal functionality fail to leverage and capture - the context of the conversation through all modalities, the dependency between the listener(s) and speaker emotional states, and the relevance and relationship between the available modalities. In this paper, we propose an end to end RNN architecture that attempts to take into account all the mentioned drawbacks. Our proposed model, at the time of writing, out-performs the state of the art on a benchmark dataset on a variety of accuracy and regression metrics.

## introduction multi-modal emotion detection and sentiment analysis in conversation is gathering a lot of attention recently considering its potential use cases owing to the rapid growth of online social media platforms such as youtube, facebook, instagram, twitter etc. @xcite @xcite @xcite , especially knowing that information obtained from any combination of more than one of the available modalities (e.g. text, audio, video) can be used to produce meaningful results. the current state of the art systems on multimodal emotion detection and sentiment analysis do not treat the modalities in accordance to the information they are capable of holding (e.g. textual information is significantly more likely to hold contextual information then audio or video features are), lack an adequate fusion mechanism, and fail to effectively capture the context of a conversation in a multi-modal setting. in addition to the lack of proper usage of the available modalities, models also fail to effectively capture the flow of a conversation, the separation between speaker and listener states, and the emotional effect a speaker's utterance has on the listener (s) in dyadic conversations. our proposed model multilogue-net, attempts to embed basic domain knowledge and takes insight from @xcite , assuming that the sentiment or emotion governing a particular utterance predominantly depends on 4 factors -interlocutor state, interlocutor intent, the preceding and future emotions, and the context of the conversation. interlocutor intent amongst the mentioned is particularly difficult to model due to its dependency of prior knowledge about the speaker, but modelling the other 3 separately, yet in an interrelated manner was theorized to produce meaningful results if managed to be captured effectively. the key intention was to attempt to simulate the setting in which an utterance is said, and use the actual utterance at that point to be able to gain better insights regarding emotion and sentiment of that utterance. the model uses information from all modalities learning multiple state vectors (representing interlocutor state) for a given utterance, followed by a pairwise attention mechanism inspired by @xcite , attempting to better capture the relationship between all pairs of the available modalities. the model uses two gated recurrent units (gru) @xcite for each modality for modelling interlocutor state and emotion. along with these gru's, the model also uses an interconnected context network, consisting of the same number of gru's as the number of available modalities, to model a different learned context representation for each modality. the incoming utterance representations and the historical gru outputs are used at every timestamp to be able to arrive at a prediction for that timestamp. the model produces m different representations at every timestamp (where m is the number of modalities), where each representation is the emotional state at that timestamp as conveyed by each of the modalities. these m representations are used by the fusion mechanism to incorporate information from each of the m representations to be able to arrive at the final prediction for that timestamp. we understand that the usage of the pairwise attention mechanism, along with the emotion gru are what make the model flexible across tasks. the usage of only the text representation as input to the context gru's has been observed to be key to the results, as the context of the conversation would be better captured by textual information then it would have with audio or video information. we believe that multilogue-net performs better than the current state of the art @xcite on multi-modal datasets because of better context representation leveraging all available modalities. 1 the remaining sections of the paper are arranged as follows: section 2 -discusses related work; section 3 -discusses the model in detail; section 4 -provides experimental results, dataset details, and analysis; section 5 contains our ablation studies and its implications; and finally section 6 -speaks on potential future work, and concludes our paper. ## ablation studies and analysis until now, some architectural considerations, such as the use of egru and the fusion mechanism, have been briefly explained but not empirically justified. this section aims to get empirical evidence regarding the effectiveness of these modules. since our model completely hinges around the usage of the context and state gru's, our ablation studies and analysis have focused on the fusion mechanism and emotion gru (egru ) only. ## conclusion in this paper, we have presented an rnn architecture for multi-modal sentiment analysis and emotion detection in conversation. in contrast to the current state-of-the-art models, our model focuses on effectively capturing the context of a conversation and treats each modality independently, taking into account the information a particular modality is capable of holding. our model consistently performs well on benchmark datasets such as cmu-mosi and cmu-mosei in any multi-modal setting. the model can be further extended to have better feature extractors, and increase both the number of modalities and the number of participants in the conversation. due to the lack of availability of datasets consisting of these extensions with emotion or sentiment labels, we have left this to our future work.