Modeling Subjectivity in Cognitive Appraisal with Language Models (2025)

Abstract: As the utilization of language models in interdisciplinary, human-centered studies grow, expectations of their capabilities continue to evolve. Beyond excelling at conventional tasks, models are now expected to perform well on user-centric measurements involving confidence and human (dis)agreement- factors that reflect subjective preferences. While modeling subjectivity plays an essential role in cognitive science and has been extensively studied, its investigation at the intersection with NLP remains under-explored. In light of this gap, we explore how language models can quantify subjectivity in cognitive appraisal by conducting comprehensive experiments and analyses with both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative results demonstrate that personality traits and demographic information are critical for measuring subjectivity, yet existing post-hoc calibration methods often fail to achieve satisfactory performance. Furthermore, our in-depth analysis provides valuable insights to guide future research at the intersection of NLP and cognitive science.

## introduction large language models (llms) are increasingly deployed in high-stakes scenarios such as healthcare @xcite and law @xcite where the integration of human oversight is essential to mitigate potential risks. however, designing such systems poses significant challenges, as real-world human decision-making is prone to occasional errors and subjectivity @xcite @xcite . moreover, inherent disagreements among human annotators, stemming from the subjective nature a1 ). while models exhibit adequate capability in modeling the mean (b), they are lacking in modeling the variance (a). one important example of subjectivity is in how people experience and regulate emotions. divergent emotional reactions may emerge from the same situation. in psychology, such variation is attributed to subjectivity in cognitive appraisal, the process by which individuals evaluate and interpret events they experienced in relation to their beliefs, goals, and prior experiences. @xcite @xcite @xcite . for example, when experiencing a romantic breakup, some individuals may blame themselves and appraise the situation to be "self-responsible" (causing emotions like guilt or regret) while others may appraise it to be "others-responsible" (causing emotions like anger). recent work has examined the ability of llms to identify subjective appraisals in how individuals evaluate their situations @xcite @xcite , which in turn shapes their emotional experiences. despite the progress achieved by these studies, existing methods often fail to capture the nuanced variability among individuals. as illustrated in figure 1 , while models perform well in capturing average tendencies (mean), they struggle to model the inter-individual variance that is central to subjective interpretation. as a result, the models may end up representing only a narrow subset of individuals and lack the ability to properly model cognitive appraisal across varying population. modeling subjectivity enables user-centric evaluations that reflect individual preferences, which is crucial for the development of personalized and socially responsible systems. for instance, llms can be guided to generate tailored re-appraisals for emotion, grounded in estimated appraisal variance @xcite . while this concept has been extensively studied in cognitive science @xcite @xcite , its investigation at the intersection with nlp remains underexplored @xcite . to bridge this gap, we investigate how language models can be used to model subjectivity in cognitive appraisal. motivated by the increasing demand for human-centred evaluation and the growing interdisciplinary applications of llms, we frame our study around three key research questions: 1) to what extent can language models quantify subjectivity in cognitive appraisal? 2) can their ability to measure subjectivity be improved, and if so, how? 3) what insight can be gained from modeling subjectivity for practical applications? to address these questions, we conducted a series of experiments and analyses across various scenarios using both finetuned models and prompt-based llms. in summary, our contributions are as follows: ## modeling subjectivity in cognitive appraisal evaluation in psychology, cognitive appraisal evaluation (cae), the task of predicting human appraisal judgments on a given situation, is typically conducted by asking individuals to provide likertscale ratings foot_1 on a set of predefined appraisal dimensions related to an event or situation @xcite . within the nlp community, researchers commonly frame cae as a classification task, where human-provided appraisal ratings serve as ground truth labels @xcite . sd fl pe pl upl gr gs ug sd sn at nc ef europe americas asia africa oceania mixed/multiple other/unspecified 1.0 1.5 2.0 2.5 3.0 figure 2: appraisal variance across different geographic locations in the envent dataset (hofmann et al., 2020). the x-axis represents the appraisal dimensions as defined by hofmann et al. (2020), while the y-axis represents the origins of the participants. see § a1 for a detailed description of the appraisal dimensions. however, numerous studies have shown that human appraisal ratings are not entirely consistent, even when assessing the same situation @xcite . as illustrated in figure 2 , individuals from different geographical regions exhibit varying appraisal ratings for the same scenarios. from a micro perspective, variability in certain appraisal dimensions differs across geographical groups. for example, the variance of attention (at) among individuals from oceania is significantly lower than that of individuals from africa. from a macro perspective, the overall appraisal patterns also differ across geographical locations. for example, individuals from mixed/multiple locations tend to have higher consensus across different appraisal dimensions compared to those from the americas. to address the challenge of capturing the subjectivity inherent in human appraisal judgments, we adopt the boltzmann policy for rating cognitive appraisals and propose a novel distributionestimation task for cae. formally, given a situation, s, and an appraisal dimension, δ @xmath0, the goal is to model the underlying distribution of the rating, r δ i , p(r δ @xmath1. assuming that the rating at δ i follows some latent distribution @xmath2, parameterized by θ ∈ θ (i.e. r δ i ∼ d(θ)), the distribution estimation task can be formulated as: point estimate metrics to evaluate the quality of the modeled distribution, a convenient measure is to assess the estimated mean and variance. we use sample mean and sample variance obtained from repeated measurements as the ground truth and compute the mean absolute error (mae) for both the estimated mean, µ-mae, and the estimated variance, σ 2 -mae. distribution metric while the point estimates provide assessments of the quality of the estimated mean or variance, they do not reflect the holistic quality of the estimated distribution. to comprehensively evaluate the discrepancy between the sample distribution and the modeled distribution, a common measure is the kl divergence. however, kl divergence suffers from asymmetry and vulnerability to regions with zero probability density. as such, we opt to use the wasserstein distance, which is a proper metric that measures the distance between two probability distributions using optimal transport plan @xcite . formally, given two discrete probability distributions p and q, the wasserstein distance foot_2 is defined as ## preliminary explorations to investigate language models' capability in modeling the subjectivity in cae, we examine two categories of methods: fine-tuning pre-trained autoencoding language models (plms) and zero-shot prompting of autoregressive llms. ## experiments to address our research questions, we systematically evaluate two categories of methods across three datasets: envent dataset ( setup details for fine-tuned models, we use deberta-v3-large as the backbone @xcite . each model is trained for 30 epochs with a linear warmup for the first 10% of the training steps. we employ adamw @xcite as the optimizer. we set the maximum learning rate at 5e-5 with a batch size of 32 and select the optimal model weights based on mse loss on the development set. for in-domain experiments, we train on 4,680 examples from the envent dataset and select the best checkpoint using 540 validation examples. for out-of-domain experiments, we fine-tune on envent training set and evaluate on the fge and covidet datasets. since the appraisal dimensions used in out-of-domain datasets slightly differ from those in envent, we manually aligned their dimensions and ratings for fair comparision. further details are provided in §a. baselines the baseline models selected for comparison can be broadly categorized into two groups: fine-tuned models: cade-lsm estimates subjectivity using label-smoothing @xcite ; cade-vae, a latent variable model infers appraisal distribution using vae @xcite . prompt-based models include: llama3.1-8b and llama3.3-70b (ai@meta, 2024); qwen2.5-7b and qwen2.5-72b (qwenteam, 2024). ## conclusion in this paper, we take an important first step toward modeling subjectivity in cognitive appraisal with language models. we conduct detailed experiments and analysis across various scenarios with both fine-tuned plms and prompt-based llms. our findings reveal notable inconsistencies in modeling subjectivity, with no single model reliably identifying it across all appraisal dimensions under varying conditions. the thorough quantitative and qualitative examination indicates that personality traits and demographical information play a vital role in measuring subjectivity, whereas existing post-hoc calibration approaches fail to achieve improved results. furthermore, our qualitative analysis provides valuable insights for future research and development in the intersection of nlp, cognitive modeling, and psycholinguistics.