IUCL at WASSA 2022 Shared Task: A Text-only Approach to Empathy and Emotion Detection (2022)

Abstract: Our system, IUCL, participated in the WASSA 2022 Shared Task on Empathy Detection and Emotion Classification. Our main goal in building this system is to investigate how the use of demographic attributes influences performance. Our (official) results show that our text-only systems perform very competitively, ranking first in the empathy detection task, reaching an average Pearson correlation of 0.54, and second in the emotion classification task, reaching a Macro-F of 0.572. Our systems that use both text and demographic data are less competitive.

## introduction emotion classification has become increasingly important due to the large-scale deployment of artificial emotional intelligence. in various aspects of our lives, these systems now play a crucial role. for example, customer care solutions are now gradually shifting to a hybrid mode where an ai will try to solve the problem first, and only when it fails, will a human intervene. the wassa 2022 shared task covers four different tasks on empathy detection, emotion classification, personality prediction, and interpersonal reactivity index prediction. we participated in task 1 on empathy detection and task 2 on emotion classification. most of the existing emotion classification tasks are restricted to only using signals such as video, audio, or text, but seldom using demographic data, partly because such information is often not available. however, using demographic information also raises ethical concerns. in the current shared task, additional demographic information was made available, thus implicitly inviting participants to investigate the interaction between empathy, emotion, and demographic information. in this work, we will compare two different systems, one using demographic data and one that does not. our text-only system performs very competitively. in the evaluation, we ranked first in the empathy detection task and second in the emotion classification task foot_0 . adding demographic information to the systems makes them less competitive. the remainder of the paper is structured as follows: in section 2, we will discuss the related work on emotion classification. in section 3, we will present our two systems and discuss their differences. we will also discuss the challenges we encountered and how we addressed them. in section 4, we will present the evaluation results of our systems and the performance of our other systems. we will also discuss the implications of these results. in section 5 we will conclude and discuss future research efforts. ## related work though empathy detection is relatively new, a considerable amount of work has been carried out in the related areas of emotion detection (e.g. @xcite , sentiment analysis (e.g. @xcite , and stance detection (e.g. @xcite @xcite . after initial success using svms (e.g. @xcite , bert and other transformerbased models @xcite have become the mainstream architecture for handling these related tasks (e.g. @xcite . while most data sets use twitter feed, the current task uses essays as data points, which are considerably longer than tweets, and thus necessitates procedures to mitigate problems arising from the length of the input sequence. in such settings, transformerbased models have evolved to handle longer input sequences by strategic truncating @xcite , either taking the front, the end, or the middle part of the text or using a sliding window method. additionally, packages such as the one by @xcite provide us with methods and implementations to incorporate categorical and numerical features. categorical and numerical features can be treated as additional tokens, or they can be treated as a different modality and handled by co-attention @xcite . ## methodology in this section we will describe our systems and how we approach the empathy prediction and emotion classification tasks with two different systems. ## results and analysis in this section, we discuss our results for the two tasks, empathy detection and emotion classification. ## conclusion our system, iucl, participated in the empathy detection and the emotion classification tasks of the wassa 2022 shared task. our text-only systems rank first in the empathy task and second in the emotion task. we come to the following conclusions: 1. there is a complex interaction between the size of the training data and the complexity of the task, classification for emotion detection and regression for empathy. given a small training data set and a small set of labels, only minimal finetuning is required. 2. using demographic attributes as features decreases performance given the small training set, and it may raise ethical concerns. we plan to further investigate the biases in this data set and their implications to both the machine learning systems and society in the future.