PFA - ERC : Psuedo-Future Augmented Dynamic Emotion Recognition in Conversations (2024)

Abstract: AI systems’ ability to interpret human emotions and adapt to variations is becoming more crucial as AI gets embedded into everyone’s daily lives. Emotion Recognition in Conversations (ERC) is based on this fundamental challenge. Current state-of-the-art technologies in ERC are limited due to the need for future information. We introduce High-Dimensional Temporal Fusion Transformer (HiTFT), a time-series forecasting transformer that predicts pseudo-future information to overcome this constraint. This retains the models’ dynamic nature and provides future information more efficiently than other methods. Our proposed method combines pseudo future embeddings with an encoder that models the speaker’s emotional state using past and pseudo-future information as well as inter and intra speaker interactions; these speaker states are then passed through a decoder block that predicts the inferred emotion of that utterance. We further evaluate our method and show that it achieves state of the art performance on three ERC datasets - MELD, EmoryNLP, and IEMOCap.

## introduction directly or indirectly, chatbots and language models have become a crucial component of cyber infrastructure. with the high penetration of large language models (llms) in various domains, such as healthcare, law, and many other safetycritical systems, any unexpected output generated by these models could lead to catastrophic consequences. the goal of affective computing systems is to correctly identify users' emotions based on their responses and direct future conversations toward a desired emotion such as happiness or neutrality. emotion recognition in conversations (erc) is a field of active research in affective computing. the erc task goes beyond the conventional recognition of emotions at the sentence level; * * equal contribution. it considers the complex interplay of conversational context and speaker's emotional states across multiple utterances. an example of our proposed erc system in action is described in figure 1 . researchers have developed several methods to effectively exploit conversational features, from prompt engineering @xcite to sophisticated speaker interaction tracking systems @xcite @xcite . these systems depend on future dialogue discourse to accurately predict the emotion of the current utterance. this reliance limits their applicability in real-time scenarios where such information is unavailable. @xcite have explored the use of pseudo-future contexts but with significant computational overhead, as they rely on pre-trained llms with billions of parameters for extracting these features. while pre-trained language models offer extensive linguistic knowledge, their computational overhead limits system usability in real-time edge systems equipped with low-power hardware. this highlights the need for models that can operate efficiently without sacrificing the accuracy of erc systems in real-time use cases. this paper introduces a novel dynamic forecasting method that significantly reduces computational overhead while enhancing real-time responsiveness for erc tasks. unlike existing methods that rely heavily on pre-trained llms for future context prediction, our method leverages a temporal fusion transformer (tft)-based architecture to generate pseudo-future embeddings. this timeseries approach to erc, which forecasts emotional contexts rather than relying on computationally expensive llms, not only achieves comparable or superior accuracy but also improves inference speed and memory efficiency. our contributions are particularly focused on the following key areas: ## related methods traditional approaches in emotion recognition in conversations (erc) have primarily focused on analyzing discrete emotions from static texts, often overlooking the dynamic nature of conversations and speaker interactions. the evolution towards text-based erc has seen the adoption of advanced methodologies, particularly in modeling conversation context. use of rnns and transformers: early erc studies relied on transformer models and recurrent neural networks (rnns). these methods captured the continuous and inter-related nature of dialogues. dialogxl @xcite ) uses a memory-augmented transformer xl to exploit the hierarchical structure of conversations. dia-loguecrn @xcite incorporates a reasoning module using rnns for contextual features. advanced designs like sged @xcite include a speaker-guided decoder network with an attention-based speaker state encoding system. however, rnns suffer from vanishing gradient issues, limiting modelling long-term context dependencies, while transformers can be computationally expensive, posing challenges for real-time applications. ## methodology the primary goal of erc systems is to identify the underlying emotion for each utterance in a dialogue. formally, a dialogue in our context is defined as a sequence of utterances, denoted as @xmath0, u 2 , ..., u n } and corresponding emotions @xmath1, e 2 , ..., e n }, where n represents the total number of utterances in the conversation. e @xmath2, c 2 , ...c m } for m emotion classes. each utterance u i consists of multiple tokens(words), and each utterance is linked to a unique speaker s i . all speakers involved in dialogue d are represented by s @xmath3,s 2 ,s 3 ,..s k } where @xmath4. for each utterance, we store a list of speaker and utterance pairs in order of occurrence, which helps us to map the speakers with their utterances. this mapping is used later to decode inter and intra-speaker dependency vectors. ## decoder our decoder network employs a gated recurrent unit (gru) architecture to maintain simplicity and efficiency. when processing an utterance u i , we align the speaker state vector with the utterance's representational vector: the match vector m i is then concatenated with the emotional embedding of the previously predicted emotion and passed to the gru as suggested by @xcite enabling the decoder to track the dynamic interplay of emotions: finally, we combine the utterance representation with the gru output and pass this through a feedforward neural network to predict the emotion: for model training, we minimize the crossentropy loss: in the pre-training step, our high-dimensional temporal fusion transformer (hitft) was trained for 100 epochs with initial learning rate of 1e-4 and weight decay of 1e-3. once the pre-training is complete, the hitft model parameters are frozen during further fine-tuning of the speaker modeling encoder-decoder network. additionally, the speaker modeling network was fine-tuned on past, current and pseudo-future predicted tft embeddings, with an initial learning rate of 5e-5, followed by a linear decay after a 20% epoch warm-up period. we use the pytorch framework for implementing our method and use the adamw optimizer. the trained model processes dialogue data in realtime with an average inference time of 0.5 milliseconds per conversation. ## results to determine the optimal number of future time steps for pseudo-future embeddings, we increment the number of predicted future time steps τ from one to six as shown in figure 3 and evaluate model performance on meld dataset. initially, wf1 scores rose exponentially due to valuable additional information. however, further increases led to a decline in wf1 scores as excessive data points introduced noise, making it hard for the model to decode anything from current utterance. we fix τ = 4 for all further analysis. ## conclusion in this study, we present a novel approach to erc using a cutting-edge hi-dimensional temporal fusion transformer (hitft) model built to forecast pseudo-future utterance embeddings. this methodology improves the erc task by combining dynamic temporal dependencies and speaker context, making our proposed method better for modelling long range conversational interactions. our method efficiently manages the complexities and varying lengths of dialogues. it also adeptly addresses the challenges of real-time processing and class imbalance inherent in traditional erc systems. we perform rigorous evaluations of the proposed method using iemocap, meld, and emorynlp datasets and our model achieves state-of-the-art performance on iemocap and emorynlp datasets. this research uses advanced approaches to bring a fresh perspective to the erc domain, creating a new standard for future research and applications. it will shape real-time, efficient erc systems, enabling robust context-aware computing systems in everyday interactions.