Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models (2023)

Abstract: The emotions we experience involve complex processes; besides physiological aspects, research in psychology has studied cognitive appraisals where people assess their situations subjectively, according to their own values (Scherer, 2005). Thus, the same situation can often result in different emotional experiences. While the detection of emotion is a well-established task, there is very limited work so far on the automatic prediction of cognitive appraisals. This work fills the gap by presenting CovidET-Appraisals, the most comprehensive dataset to-date that assesses 24 appraisal dimensions, each with a natural language rationale, across 241 Reddit posts. CovidET-Appraisals presents an ideal testbed to evaluate the ability of large language models — excelling at a wide range of NLP tasks — to automatically assess and explain cognitive appraisals. We found that while the best models are performant, open-sourced LLMs fall short at this task, presenting a new challenge in the future development of emotionally intelligent models. We release our dataset at https://github.com/honglizhan/CovidET-Appraisals-Public.

## introduction emotions constitute a crucial aspect of people's lives, and understanding them has a profound impact on improving public mental health problems as well as policy-making (choudhury and @xcite @xcite . the emotions we experience involve complex processes: the same situation can often result in different emotional experiences, based on an individual's subjective evaluations. these are called cognitive appraisals, and have been extensively studied in psychology through theoretical, behavioral, and hand-coded studies @xcite @xcite @xcite smith and i've recently heard of stories that people who are getting mildly sick a er being vaccinated are still coming down with long covid. people on the covid long hauler subreddit are saying that we're going to have a huge problem on our hands because no vaccine protects against long covid… this isn't going to end, isn't it? right as i think i've tasted freedom, i nd out information that makes me want to stay inside forever like a hermit... ## background and related work cognitive appraisal theories. the cognitive appraisal theories of emotion state that emotions arise from an individual's subjective understanding and interpretation of situations that hold personal importance for their overall well-being @xcite @xcite @xcite @xcite @xcite @xcite . in practical terms, people interpret and appraise situations along a range of different dimensions, and it is the specific manner in which they appraise their situations that give rise to the distinct emotions they experience. the primary focus of cognitive appraisal theories of emotions revolves around the identification of these appraisal dimensions that are associated with specific emotional experiences and how these dimensions contribute to distinguishing between different emotional states @xcite @xcite @xcite . while appraisal theorists agree on the impor-tance of motivationally-relevant appraisals in triggering emotions, they have not reached a consensus on the specific appraisal dimensions that play a significant role in this process @xcite . various theories have put forth distinct sets of appraisal dimensions that are considered crucial in triggering and distinguishing emotions. from prior literature, @xcite identified and assembled a taxonomy of all appraisal dimensions that have been studied, and produced a condensed list of 24 cognitive appraisal dimensions which we focus on in this paper. cognitive appraisals in nlp. appraisals provide the necessary computational structure allowing for the distillation of real-life situations that depend on a multitude of factors into a (large but) finite set of appraisal dimensions @xcite . despite its importance, however, few works have explored the implications of cognitive appraisals on emotions in nlp. @xcite experimented with a small set of cognitive appraisal dimensions (including attention, certainty, effort, pleasantness, responsibility, control, and circumstance) to assist the automatic detection of emotions in text, and found that accurate predictions of appraisal dimensions boost emotion classification performance. they introduced a dataset of 1, 001 sentences following the template "i feel [emotion], when ..." (average sentence length: 27 tokens). in comparison, our work covers a much wider range of 24 appraisal dimensions found in prior literature, over lengthy (176 tokens on average) reddit posts that were natural and emotionally charged. we also collect natural language rationales as a key contribution to reveal human's in-depth understanding of such cognitive appraisals in context. recent studies @xcite acknowledged both what happened and how one appraised the situation as inherent components of emotion triggers, although the appraisal of events was not explicit in their work. instead we provide datasets and perform evaluation on appraisals explicitly, such that language models can build on this work to achieve a comprehensive and explicit understanding of cognitive appraisals from written text. llms on emotion-related tasks. autoregressive llms have been explored extensively in emotion-related tasks such as sentiment analysis @xcite @xcite , emotion recognition @xcite , disclosing the representation of human emotions encapsulated in llms @xcite , and interpreting mental health analysis @xcite . however, few have tapped into the understanding of cognitive appraisals of emotions innate in llms. in this work, we dive into the extent to which llms comprehend the profound cognitive appraisals underlying emotions in situations, and further elicit natural language rationales from the language models to disclose the reason behind such predictions from the otherwise baffling black-box llms @xcite . aligning with @xcite who performed human evaluation on rationales generated by gpt, we additionally perform an in-depth human evaluation of the rationales from human annotators and llms alike on the novel task of providing natural language explanations for cognitive appraisals of situations that underlie narrators' emotional experiences. ## dataset analysis how do the scales distribute across dimensions and emotions? the distribution of the likertscale ratings is shown in figure 3 . the rat- we analyze the connections between our likertscale annotations and covidet's emotion annotations. figure 4 shows the mean likert-scale rating for each dimension within each post with respect to the perceived emotion. while it is evident that most dimensions show consistency (the posts are all related to covid-19), some emotions stand out distinctly in particular dimensions. for example, trust and joy have higher likert-scale ratings on dimensions pfc (problem-focused coping) and gcnd (goal conduciveness) compared to other emotions, suggesting the inter-correlation between these appraisal dimensions with positive emotions. we further explore whether appraisal dimensions alone are indicative of perceived emotions already annotated in covidet in appendix §d.1. what are the characteristics of the natural language rationales? on average, each rationale is 1.2 sentences (std.@xmath0.4) and 28.9 tokens (std.@xmath1.0) long. following @xcite , we also measure the abstractiveness of the rationales from our human annotators by calculating the percentage of novel bigrams in the rationales with respect to the reddit posts and in- are rationales repetitive? we also look into automatic measures of similarity to assess how much rationales from different annotators, or from different dimensions/posts, differ from one another. specifically, we calculate bleu-4 @xcite , rouge-l @xcite , and re-scaled bertscore @xcite between our two annotators' rationales. we establish 2 random baselines for comparison: (1) rationales of the same dimension from different posts; models. we evaluate the following instructiontuned llms 1 : 1) chatgpt, i.e., gpt-3.5-turbo; 2) flan-t5-xxl (11b) @xcite , which is the instruction fine-tuned version of t5 @xcite ; 3) alpaca (7b, 13b) @xcite is fine-tuned from llama (7b and 13b) @xcite ) on 52k instructionfollowing examples created with gpt text-davinci-003 in the manner of self-instruct @xcite ; 4) dolly-v2 (7b, 12b) @xcite is an instruction-tuned llm trained on ~15k demonstrations consisting of both instructions and responses. prompts and setup. the templates for prompting the llms are shown in appendix figure 17 . after extensive experimentation, we found that only chatgpt is able to generate both a rating and a rationale with a single prompt; this type of "1-step" prompting leads to ill-formed responses for other models. thus, for models other than chat-gpt, we instead use a pipeline or "2-step" prompting similar to the strategy used in @xcite : we first elicit the rating for the appraisal dimension, then conditioned on the response for the rating we further elicit the rationale for the selection. we carry out all our experiments on 4 nvidia a40 gpus. we use the huggingface transformers @xcite library for model inference. we set the temperature value of all models to 0.1. 2 to enable a fair comparison of models, we sample from the llms five times with different model initializations and report average values for both scales and rationales. ## evaluation: likert-scale ratings we report model performance for likert-scale ratings on the 21 applicable dimensions using two 1 while we have also experimented with non-instructiontuned llms (including gpt-3 davinci and llama (7b and 13b), they largely fail to generate sensible outputs for this task. we showcase examples of responses from non-instructiontuned models in appendix §a. for these reasons, we do not include their results in this paper. 2 we experimented with higher temperatures on a validation set consisting of 10 reddit posts annotated by our group which are not included in covidet-appraisals, and the models yielded worse and more unstable performance. standard regression metrics: mean absolute error (mae) and spearman's correlation. we treat the selection of the na labels as a binary classification task and report f1 measures across all 24 dimensions. for the 40 gold examples that were doubly annotated by human annotators, we consider a dimension as na when both annotators select the label. results. to evaluate the performance, we clean the responses elicited from the llms. specifically, we use regular expressions to extract the first numeric value ranging from 1-9 from the scale responses foot_1 . the results of the models' performance are shown in for the na labels ( for the likert-rating predictions, results show that chatgpt-3.5 consistently yields the highest performance compared to the other language models, with a significant spearman's correlation of 0.388 and an mae of 1.694. we note that flan-t5-xxl is the second best-performing model. alpaca and dolly perform poorly on our task, with negative correlations with the gold labels foot_2 . interestingly, we notice a drop in performance when the size of the model parameters increases for alpaca. the results highlight the challenging nature of our task, and the gap between open-sourced llms vs. chatgpt @xcite . additionally, we also measure the systems' performance on all 24 appraisal dimensions, including the 3 appraisal dimensions where the na rates are around 80%. results revealed marginal change in performance across all llms. for most llms the performance dropped as expected: measured with spearman's ρ, chatgpt-3.5 (↓ 0.018), alpaca-7b (↓ 0.008), and dolly-12b (↓ 0.007). on the other hand, the performance of flan-t5 (↑ 0.005), alpaca-13b (↑ 0.027), and dolly-7b (↑ 0.020) increased. ## evaluation: rationales as rationalizing emotional appraisals with natural language is a novel task, we perform both automatic ( §7.1) and human evaluation ( §7.2). ## conclusion to achieve a more accurate and holistic understanding of emotions from written text, nlp models need to work towards understanding the subjective cognitive appraisals of emotions underlying our thorough evaluation of llms' performance on assessing emotion appraisal dimensions emphasizes that covidet-appraisals is a challenging benchmark, and our in-depth human evaluation of the natural language rationales indicates potential areas of improvement (e.g., improving the factuality and usefulness of the rationales) for open-source llms.