LLM - GE m: Large Language Model-Guided Prediction of People’s Empathy Levels towards Newspaper Article (2024)

Abstract: Empathy – encompassing the understanding and supporting others’ emotions and perspectives – strengthens various social interactions, including written communication in healthcare, education and journalism. Detecting empathy using AI models by relying on self-assessed ground truth through crowdsourcing is challenging due to the inherent noise in such annotations. To this end, we propose a novel system, named Large Language Model-Guided Empathy _(LLM-GEm)_ prediction system. It rectifies annotation errors based on our defined annotation selection threshold and makes the annotations reliable for conventional empathy prediction models, e.g., BERT-based pre-trained language models (PLMs). Previously, demographic information was often integrated numerically into empathy detection models. In contrast, our _LLM-GEm_ leverages GPT-3.5 LLM to convert numerical data into semantically meaningful textual sequences, enabling seamless integration into PLMs. We experiment with three _NewsEmpathy_ datasets involving people’s empathy levels towards newspaper articles and achieve state-of-the-art test performance using a RoBERTa-based PLM. Code and evaluations are publicly available at [https://github.com/hasan-rakibul/LLM-GEm](https://github.com/hasan-rakibul/LLM-GEm).

## introduction empathy refers to an inherent ability to understand and convey suitable emotional responses in reaction to the emotions and viewpoints of others @xcite . seminal work by @xcite proposed the widely-recognised empathy measurement scale by defining empathy as having six aspects: sympathetic, moved, compassionate, tender, warm and softhearted. empathic capability is key in cultivating interpersonal relationships and mitigating stress and discontent among individuals in our society in various human-to-human interactions. empathic doctors are better equipped to understand their patients' concerns, leading to improved communication and patient outcomes @xcite . this empathic connection is not confined to face-to-face interactions but extends to written communication, such as medical reports and informative articles that convey a compassionate understanding of patients' experiences. in education, especially with the shift towards online learning due to the covid-19 pandemic, empathy is critical in helping teachers understand their students' emotional states and create a positive learning environment @xcite . in addition to verbal communication, empathy in the education sector also surfaces in written communications, such as emails and feedback on assignments, where the tone and language reflect a genuine concern for students' well-being. in examining the role of empathy in written journalism, consider the poignant example of a newspaper article detailing a local family's struggle after a devastating house fire. the journalist's empathic narrative goes beyond factual reporting, weaving a story that not only informs but also connects readers emotionally to the human experiences within the news. assessment of empathy levels is crucial in determining interaction quality @xcite . empathy deficits often lead to conflicts and miscommunications, which can be resolved by measuring empathy levels as the first step, but such measurement is challenging, even for humans @xcite . research endeavours in computational empathy remain limited @xcite compared to other domains of affective computing, such as emotion @xcite , primarily due to the lack of high-quality data. the aphorism, 'garbage in, garbage out', signifies how inaccurate data results in inaccurate outputs @xcite . while crowdsourcing platforms (e.g., amazon mechanical turk, crowd-flower and prolific) offer a simpler and faster way to get a sizeable participant pool, they suffer from false information @xcite . such erroneous data result from carelessness and multitasking and threaten the validity of findings relying on such data @xcite . however, such crowdsourcing with self-assessment annotation is a major source of data collection in computational social science and human behaviour studies, such as empathy @xcite and emotion @xcite . computational empathy using crowdsourced annotation, therefore, often provides suboptimal performance (figure 1 ). in addition, the subjective nature of empathy necessitates consideration of people's demographic information, which is normally represented as numbers in the datasets. we, therefore, leverage demographic information into our prediction pipeline and introduce llm-gem, a large language model (llm)-guided empathy prediction system. while earlier studies, such as @xcite , employed gpt-3 llm for direct data annotations, to the best of our knowledge, no work has focused on using llm to refine human annotations. to this end, we leveraged the enhanced capabilities of gpt-3.5 to reduce labelling errors in pre-existing crowdsourced annotations. it will be particularly useful when there is already some noisy crowdsourced annotation. we experiment with three publicly available datasets to predict people's empathy levels toward newspaper articles, where our system results in competitive performance by outperforming prior work. our major contributions include (1) application of gpt-3.5 llm to convert numerical demographic information to semantically meaningful text in order to seamlessly integrate them with a pre-trained language model (plm), (2) employing gpt-3.5 llm to reduce annotation errors caused by crowdsourcing, and (3) defining annotation selection threshold to systematically select between crowdsourced annotation and llm annotation. ## related work a workshop on computational approaches to subjectivity, sentiment & social media analysis (wassa) has organised a series of competitions on predicting people's empathy towards newspaper articles. in these challenges from 2021 to 2023, several works @xcite @xcite predicted empathy by fine-tuning roberta plm followed by some multi-layer perceptron (mlp) layers. apart from these, some studies @xcite @xcite finetuned bert plm, and some other studies @xcite @xcite leveraged an ensemble approach with fine-tuning multiple plms. @xcite experimented with multi-task learning and reported that a simple fine-tuning of the roberta base model resulted in better performance (0.480 vs 0.508 pearson correlation coefficient (r)). fine-tuning plms, therefore, has become the conventional approach to predict people's empathy towards newspaper articles. among different plms, roberta has become the most frequently used prediction model in empathy detection studies, as reported in a recent survey covering computation empathy studies from 2013 to @xcite . several authors experimented with various approaches to ensure data quality in empathy predictions. as a data augmentation technique, vasava et al. (2022) translated texts to a random language using google translate and then back to english. they combined five demographic features before the final layer of their empathy prediction pipeline. @xcite also harnessed demographic and personality data, which yielded a validation pearson correlation coefficient (r) of 0.53. notably, this performance surpassed that achieved without incorporating demographic and personality information. data augmentation and demographic information, therefore, help to predict empathy levels. as for data annotation employing llm, wang et al. (2021) experimented with gpt-3 in annotating data for several natural language processing tasks, including sentiment analysis, question generation and topic classification. they concluded that gpt-3 is a cost-effective way of annotating data but is not as reliable as human annotations. in this paper, we systematically select annotations between gpt-3.5 llm and crowdsourcing to minimise existing noise in crowdsourced annotations. ## conclusion and future work empathy plays a crucial role in social dynamics, such as education, health and business. evaluating people's empathy levels using computational tools such as ai requires good-quality data. computational social science often involves collecting data and annotation from crowdsourcing, which often has noise. to this end, our system, llm-gem, aims to minimise annotation noise and ensure data quality. we experiment with three datasets predicting people's empathy levels towards newspaper articles. we define an annotation selection threshold to systematically select between llm and crowdsourced annotations, which achieves sota performance. our annotation error mitigation method can be applicable to other self-annotation datasets with necessary adaptations in the prompts (to include/change the details of the problem, range of annotation labels, etc.). for example, @xcite collected self-annotation to detect empathy in social media, where similar error analysis and possible inclusion of llm may help mitigate annotation noise, if any. similarly, hossain and rahman (2022) used crowdsourcing selfannotated data to detect customers' empathy behaviour, where our llm-based annotation noise removal can be helpful. apart from these, it could be applicable to other similar self-annotated datasets across different computational social science and human behaviour studies. future work can further investigate better loss functions that closely estimate the pearson r evaluation metric. finally, experimenting with plms that are pre-trained on emotion and empathy-related datasets would be another avenue we leave for future work.