Evaluating Large Language Models on Social Signal Sensitivity: An Appraisal Theory Approach (2024)

Abstract: We present a framework to assess the sensitivity of Large Language Models (LLMs) to textually embedded social signals using an Appraisal Theory perspective. We report on an experiment that uses prompts encoding three dimensions of social signals: Affect, Judgment, and Appreciation. In response to the prompt, an LLM generates both an analysis (Insight) and a conversational Response, which are analyzed in terms of sensitivity to the signals. We quantitatively evaluate the output text through topical analysis of the Insight and predicted social intelligence scores of the Response in terms of empathy and emotional polarity. Key findings show that LLMs are more sensitive to positive signals. The personas impact Responses but not the Insight. We discuss how our framework can be extended to a broader set of social signals, personas, and scenarios to evaluate LLM behaviors under various conditions.

## introduction the research paradigm is displayed in figure 1 . the framework is meant to assess specific capabilities of llms, identify limitations, and address challenges in utilizing sociolinguistic theories in such evaluations. our contributions are as follows: we make our code and data publicly available below. 1 2 related work from a technical perspective, this paper investigates the specific capabilities of llms to operate in contextually appropriate ways in different social settings. from a linguistic perspective, we are specifically interested in appraisal theory @xcite to define a space of social signals because of its prevalence in the field of language technologies. thus, we review past work from both a technical and linguistic perspective. ## conclusion in this study, we design a systematic framework to evaluate the sensitivity of gpt-3.5 and gpt-4 to key social signals based on appraisal theory, i.e. affect, judgment, and appreciation. the results confirm that these models demonstrate statistically significant sensitivity to the three social signals. however, our findings also uncover their limited sensitivity to negative aspects of social signals. future research could extend these findings by including a wider range of llms and exploring additional output measures to enhance our understanding of llms' capabilities in social contexts. through this work, we provide a generalizable framework that can be extended to a broader set of social signals and language framing beyond appraisal theory, as well as various social scenarios and personas, thus systematically evaluating the elicited behaviors of llms under diverse and complex conditions.