Empathy and Distress Prediction using Transformer Multi-output Regression and Emotion Analysis with an Ensemble of Supervised and Zero-Shot Learning Models (2022)

Abstract: This paper describes the participation of the SINAI research group at WASSA 2022 (Empathy and Personality Detection and Emotion Classification). Specifically, we participate in Track 1 (Empathy and Distress predictions) and Track 2 (Emotion classification). We conducted extensive experiments developing different machine learning solutions in line with the state of the art in Natural Language Processing. For Track 1, a Transformer multi-output regression model is proposed. For Track 2, we aim to explore recent techniques based on Zero-Shot Learning models including a Natural Language Inference model and GPT-3, using them in an ensemble manner with a fine-tune RoBERTa model. Our team ranked 2nd in the first track and 3rd in the second track.

## introduction emotion analysis is a popular and established task in natural language processing (nlp) with a large number of studies conducted during the last few years @xcite . emotion detection can be considered as the main task in this area which consists of mapping textual units to different emotion categories within a text following different psychological models such as ekman's theory @xcite , with six basic emotions, or plutchik's @xcite with the addition of anticipation and trust. two inextricably related concepts to emotions that have received less attention are empathy and distress. the former is defined as the ability to sense other people's emotions, coupled with the ability to imagine what someone else might be thinking or feeling, while the latter is a self-focused, negative affective state that arises when one feels upset due to witnessing an entity's suffering or need @xcite . a linked task that plays an important role in the study of these concepts is personality trait detec-tion, which is related to author profiling and is commonly defined as the task of detecting the five basic personality traits (extraversion, agreeableness, openness, conscientiousness, and neuroticism) in the text @xcite . we refer the reader to a recent survey in the task @xcite . all these concepts together have potential applications and play an important role in helping victims of abuse @xcite @xcite , mental and physical health support @xcite , and in the study of reaction to news stories @xcite . in this paper, we present our participation as sinai team in the shared task on empathy and personality detection and emotion classification (wassa 2022). within this shared task, four main tracks are proposed that aim to develop models that can predict empathy, distress, emotion, and personality traits in reaction to english news articles. track 1: empathy prediction (emp) consists in predicting both the empathy concern and the personal distress at the essay level. track 2: emotion classification (emo) refers to detecting the emotion at the essay level. track 3: personality prediction (per) aims to predict the big five personality traits, and track 4: interpersonal reactivity index prediction (iri) consists of predicting each dimension of assessment of empathy: perspective taking, fantasy, empathic concern. our team sinai has participated in the first and second tracks. ## data the dataset provided by the organizers of wassa 2022 shared task is an extension of the one presented in @xcite which is composed of posts in reactions to news articles where there is harm to a person, group, or other. person-level demographic information (age, gender, ethnicity, income, education level) is included for each post. a set of 2,130 training documents annotated with empathy, distress, and emotions is provided (see dataset #instances training 1,860 development 270 test 525 ## system description in this section, we describe the systems our team sinai developed for track 1 (emp) and track 2 (emo) at wassa 2022. ## experimental setup all the transformer based models have been finetuned on a single nvidia ampere a100 gpu by making use of the hugging face's transformer library @xcite . regarding the hyperparameters used, we computed a grid search in order to find out the combination that maximized each task's metric on the development set. the batch size values tested during the optimization were 8, 16 and 32. concerning the learning rate, the range of values we tested during the grid search was 1e-5, 2e-5, 3e-5, 4e-5 and 5e-5. we also set the maximum length of the tokenizer (the length from which the tokenizer will truncate a tokenized sequence) equal to the longest essay in the training set as tokenized by the roberta's byte-pair encoding tokenizer, that is, 221. regarding the epochs, we trained every model until an early stopping mechanism determined the model was starting to overfit on the training data, which usually happened between epochs 2 and 3, depending on the model. ## results in this section, we present the results obtained by the systems we developed as part of our participation in wassa 2022 track 1 and track 2. to evaluate our systems, we used the official competition metrics given by the organizers. specifically, the average of the two pearson correlations is computed for emp and the macro f 1 -score for emo. further, for the latter we report macro precision and recall scores. the experiments are conducted in two phases: the model selection phase and the evaluation phase, which are explained in the following two sections. ## conclusion this paper presents the participation of the sinai research group in the shared task on empathy and personality detection and emotion classification (wassa 2022). for the first task, we explore how different raw language models and models finetuned on emotions work for the empathy and distress prediction. for this task, we observe that the raw language model roberta in a multi-output regression fashion together with the features of gender, age and income perform better than the models which contain emotion knowledge. therefore, this shows that not all models previously fine-tuned on emotions help in the prediction of empathy and distress. regarding the track 2, emotion detection, we have experimented with recent zsl models including nli and gpt-3. results on the development set suggest that they are promising options for emotion detection when no labeled data is available. therefore, our proposal for this task is an ensemble model that takes advantage of both supervised and zsl models. our final results in both track 1 (emp) and track 2 (emo) demonstrate the success of our proposal's approaches since we ranked 2 nd and 3 rd among all the participants, respectively. as future work, we plan to further explore zsl models as they have shown promising results in the emotion classification task.