A Training-Free Debiasing Framework with Counterfactual Reasoning for Conversational Emotion Detection (2023)

Abstract: Unintended dataset biases typically exist in existing Emotion Recognition in Conversations (ERC) datasets, including label bias, where models favor the majority class due to imbalanced training data, as well as the speaker and neutral word bias, where models make unfair predictions because of excessive correlations between specific neutral words or speakers and classes. However, previous studies in ERC generally focus on capturing context-sensitive and speaker-sensitive dependencies, ignoring the unintended dataset biases of data, which hampers the generalization and fairness in ERC. To address this issue, we propose a Training-Free Debiasing framework (TFD) that operates during prediction without additional training. To ensure compatibility with various ERC models, it does not balance data or modify the model structure. Instead, TFD extracts biases from the model by generating counterfactual utterances and contexts and mitigates them using simple yet empirically robust element-wise subtraction operations. Extensive experiments on three public datasets demonstrate that TFD effectively improves generalization ability and fairness across different ERC models.

## introduction emotion recognition in conversations (erc) has garnered substantial research interest in recent years @xcite @xcite . this attention stems from its promising applications in various domains, including recommendation systems and dialogue generation @xcite . erc aims to identify the emotion of each utterance in conversations. current efforts primarily focus on modeling context-and speaker-sensitive dependencies @xcite . this includes recurrent-based network @xcite et @xcite @xcite @xcite , transformerbased network @xcite @xcite , and graph-based network @xcite @xcite . although these endeavors have achieved satisfactory results, they all overlook unintended biases present in the datasets, thereby impeding their generalization capabilities and fairness @xcite . different from other tasks, in erc, these biases may manifest at both the utterance level, such as label and speaker biases, as well as the word level, like the neutral word bias. label bias, as depicted in fig. 1 , is evident in the emorynlp dataset's training set, where 32.9% (2485 utterances) are labeled as 'neutral' and 6.3% (474 utterances) are labeled as 'sad'. numerous prior studies @xcite have indicated that models trained on such imbalanced data are susceptible to the tendency of predominantly predicting the majority class. speaker and neutral word biases arise from trained models exhibiting strong associations between specific words and particular emotion categories. for instance, in emorynlp, it is highly probable for richard burke to be associated with emotions such as 'joyful', 'neutral', and 'peaceful', rather than 'mad' as depicted in figure 1 . additionally, neutral words also display such biases in emotion distribution. consequently, models tend to unfairly assign utterances containing these emotionless keywords to specific categories based on biased statistical information, rather than relying on intrinsic textual semantics @xcite @xcite . existing debiasing methods mainly involve two approaches: data-level manipulations, like resampling @xcite and generating counterfactual samples @xcite , aim to balance the training set but increase training time. model-level balancing mechanisms, such as counterfactual reasoning @xcite and reweighting @xcite , adjust category influence during training but require careful strategy selection and retraining from scratch. in contrast, our proposed training-free debiasing framework (tfd) focuses on counterfactual reasoning during the prediction process without extra data augmentation and any training costs. because generating counterfactual examples while maintaining conversation structure integrity is challenging, inspired by @xcite , we employ a masking mechanism to obtain counterfactual examples from the original input data to extract biases suffered from trained models. we then effectively mitigate these extracted biases using empirically robust element-wise subtraction operations. our main contributions are as follows: to tackle this, researchers have integrated external knowledge sources such as comet @xcite , senticnet @xcite and conceptnet @xcite into their models @xcite @xcite . despite these advancements, unintended dataset biases have been neglected, which hinders the generalization of erc models. dataset debiasing: to address dataset bias, one approach is to manipulate the data during model training to prevent unintended biases from being captured. these manipulations include techniques like data balance or resampling methods mentioned in studies such as @xcite @xcite , as well as data augmentation methods described by @xcite . another common strategy is to incorporate model-level balancing mechanisms. these mechanisms aim to mitigate bias by using unbiased embeddings @xcite , adjusting thresholds @xcite , and applying instance weighting techniques @xcite @xcite . however, the data-level strategy results in additional manual costs for data manipulations and longer training time. on the other hand, the model-level strategy necessitates a meticulous selection of balancing techniques and retraining whenever there are changes in the balancing mechanisms. ## experiments 4.1 datasets we conduct experiments on three datasets: iemo-cap @xcite , emorynlp (zahiri and choi, 2018), and meld @xcite . the statistics of datasets are shown in iemocap consists of dyadic sessions where actors perform improvisations or scripted scenarios. each utterance is labeled with one of the emotions: angry, happy, sad, neutral, excited, or frustrated. since there is no dedicated validation set in this dataset, we follow the approach from @xcite by using the last 20 dialogues from the training set for validation. meld is a multi-party conversation dataset collected from the tv show friends. each utterance is annotated with one of the emotions: fear, surprise, anger, disgust, sadness, neutral, or joy, and one of the sentiments: neutral, negative, or positive. emorynlp consists of multi-party sessions from the tv show friends and each utterance is labeled with one emotion from the set: joyful, scared, peaceful, sad, powerful, mad, or neutral, along with one sentiment, as suggested in @xcite , from the set: neutral, negative, or positive. ## conclusion in this paper, we introduce tfd with counterfactual reasoning as a solution to address unintended dataset biases in erc, which performs the debiasing operation only during the prediction phase without additional data augmentation or training expenses. through extensive experiments on three public datasets, we demonstrate that tfd effec-tively promotes the generalization capability and fairness of various erc methods. {"prompt":"1. speaker0: guess what? ->","completion":" excited"} {"prompt":"2. speaker1: what? ->","completion":" neutral"} {"prompt":"3. speaker0: i did it, i asked her to marry me. ->","completion":" excited"} {"prompt":"4. speaker0: yes, i did it. ->","completion":" excited"} {"prompt":"5. speaker1: when? ->","completion":" excited"} {"prompt":"6. speaker0: oh my god, it was just last weekend. ->","completion":" excited"} {"prompt":"7. speaker1: oh, what, how-where how did you do it? ->","completion":" excited"} {"prompt":"8. speaker0: she--,well, she said yes, first of all, let me say that right off the bat. well, i would like to assume, too. but you never know these things, right? ->","completion":" happy"} {"prompt":"9. speaker1: okay good. i i assumed. ->","completion":" excited"} {"prompt":"10. speaker1: oh-->","completion":" happy"} {"prompt":"11. speaker0: i did it up at yosemite. we went camping right like we usually do. you know we go camping up there all the time. but, um-the waterfalls had died down a little bit, you know. and so uh-we-i had her climb up. ->","completion":" happy"} {"prompt":"12. speaker0: you know-we did some-you know some rock climbing up the waterfalls and went up to this little pool that was up there. and then, i-it's great. um-you have to climb this little rock-->","completion":" happy"} {"prompt":"13. speaker1: uh-huh ->","completion":" neutral"} {"prompt":"14. speaker0: and you can climb it and it's covered in algae you know. so, i wanted her to climb it to go take a picture, like, back in this little waterfall area, and she got back there and she took her picture. and then, i had her come back. and i was going to do the same thing.