Curtin OCAI at WASSA 2023 Empathy, Emotion and Personality Shared Task: Demographic-Aware Prediction Using Multiple Transformers (2023)

Abstract: The WASSA 2023 shared task on predicting empathy, emotion and other personality traits consists of essays, conversations and articles in textual form and participants’ demographic information in numerical form. To address the tasks, our contributions include (1) converting numerical information into meaningful text information using appropriate templates, (2) summarising lengthy articles, and (3) augmenting training data by paraphrasing. To achieve these contributions, we leveraged two separate T5-based pre-trained transformers. We then fine-tuned pre-trained BERT, DistilBERT and ALBERT for predicting empathy and personality traits. We used the Optuna hyperparameter optimisation framework to fine-tune learning rates, batch sizes and weight initialisation. Our proposed system achieved its highest performance – a Pearson correlation coefficient of 0.750 – on the onversation-level empathy prediction task1 . The system implementation is publicly available at https: //github.com/hasan-rakibul/WASSA23-empathy-emotion.

## introduction empathy refers to an individual's capacity to comprehend and express appropriate emotions in response to others' emotions, perspectives and beliefs @xcite . this ability can foster relationships and reduce stress and unhappiness among individuals through interaction. the importance of empathy is evident across a broad range of real-life human interactions, such as patient-doctor @xcite , teacher-student @xcite and human-robot @xcite interactions. the workshop on computational approaches to subjectivity, sentiment & social media analysis 1 at the time of writing this paper, official rankings on any tasks and evaluations of several tasks in which we participated have not been published yet. wassa 2023 challenge extends from the 2022 challenge @xcite , employed transformer-based architectures, such as bert @xcite . transformer-based models were also found to be the best-performing model in the wassa 2021 shared task on empathy prediction @xcite . apart from wassa competition, transformer models are also used in predicting empathy in essays written by medical students about simulated patient-doctor interactions @xcite . transformer models are deemed highly suitable for undertaking text-based empathy prediction owing to their inherent ability to effectively capture long-range dependencies through attention mechanism @xcite . fine-tuning pretrained transformers harnesses prior knowledge, leading to enhanced performance while minimising training time. @xcite reported the best performance by just fine-tuning a bert-based model in their system for the wassa 2022 shared task. we, therefore, choose to fine-tune pre-trained transformers to predict empathy and personality traits in this challenge. in our prediction pipeline, we utilise numerical information from the datasets, such as participants' demographic information and income, because previous research by @xcite showed demographic information is an important cue in text-based empathy prediction. overall, this paper has made the following contributions: (1) we use novel strategies to incorporate numerical demographic and other data in the text-based prediction pipeline, (2) we summarise longer text sequences to fit into the pipeline, and (3) we augment training samples by paraphrasing the textual data. ## system description the general prediction system for essay-level tasks is illustrated in figure 1 . in the case of conversation-level tasks, demographic and other personal information are not available in the conversation-level dataset. in that case, our prediction models involve only conversations and summarised articles, followed by paraphrasing to augment the training dataset. ## result & analysis to determine which feature sets are most effective for predicting empathy, we conducted an experiment in which we combined different features (essay, demographic, demographic_essay and article) and trained a distilbert @xcite model using 5-fold cross-validation for 10 epochs. huggingface's tokeniser allowed us to tokenise pairs of sequences together by automatically concatenating them with a special [sep] token. we then used these pairs of features and evaluated their performance, as presented in conversion from the longer version of the article text to its summarised shorter version improved the performance ( it is worthwhile to note that the use of data augmentation techniques such as paraphrasing can introduce very similar samples in the dataset. it may bias the evaluation metrics, especially when similar samples are present in both the training and validation sets. the cross-validated pearson correlation coefficient reported in among the pre-trained transformer models we experimented with (bert, distilbert and al-bert), the bert base model was the bestperforming model. accordingly, we used bert and tuned the other hyperparameters. we conducted 200 and 100 optuna trials for essay-based empathy and distress prediction models, respectively. as the best set of hyperparameters is always found within the first 50 trials in the essay-level empathy and distress prediction models, 50 trials were run for other essay-level prediction models. in the case of conversation-level tasks, 100 optuna trails were run. we investigated the relative importance of learning rate, seed and batch size (see appendix a). our findings are consistent with prior research by @xcite , which highlighted the impact of seed value on the fine-tuning performance of bert-based models. however, the relative importance of hyperparameters varied across the prediction tasks, indicating the task-specific nature of finetuning pre-trained transformer models. it guided us to train separate models for separate tasks. we observed that text summarisation and data augmentation (paraphrasing) improved model performance on the development set. on the test dataset, the final model achieved pearson correlation coefficients of 0.750, 0.683 and 0.573 for conversation-level empathy, emotional polarity and emotional intensity prediction, respectively. for essay-level tasks, we achieved pearson correlation coefficients of 0.187 and 0.344 for empathy and distress predictions, respectively. the average pearson correlation coefficient for conversation-level tasks was 0.669, while it was 0.266 for essay-level empathy and distress prediction. the test performance of essay-level personality and interpersonal reactivity index predictions, as well as the official rankings, have not been published at the time of writing this paper. nevertheless, our system achieved its best performance of a pearson correlation coefficient of 0.750 in predicting conversation-level empathy. ## conclusion empathy is a vital human attribute to support and care for others. this paper outlines a comprehensive system for predicting empathy, emotion and other personality traits as part of the wassa 2023 shared task. to this end, we first map the numerical demographic information into meaningful text since individuals' demographic information, such as age, sex and race, may affect their empathic capacity. our system utilises pre-trained transformers to map numerical information into meaningful text, summarise longer text sequences, paraphrase text sequences to augment smaller training datasets and finally predict the degree of empathy and other personality traits.