S ocial G aze: Improving the Integration of Human Social Norms in Large Language Models (2024)

Abstract: While much research has explored enhancing the reasoning capabilities of large language models (LLMs) in the last few years, there is a gap in understanding the alignment of these models with social values and norms. We introduce the task of judging social acceptance. Social acceptance requires models to judge and rationalize the acceptability of people’s actions in social situations. For example, is it socially acceptable for a neighbor to ask others in the community to keep their pets indoors at night? We find that LLMs’ understanding of social acceptance is often misaligned with human consensus. To alleviate this, we introduce SocialGaze, a multi-step prompting framework, in which a language model verbalizes a social situation from multiple perspectives before forming a judgment. Our experiments demonstrate that the SocialGaze approach improves the alignment with human judgments by up to 11 F1 points with the GPT-3.5 model. We also identify biases and correlations in LLMs in assigning blame that is related to features such as the gender (males are significantly more likely to be judged unfairly) and age (LLMs are more aligned with humans for older narrators).

## introduction large language models (llms) have demonstrated remarkable capabilities in understanding and generating human-like text @xcite . as these models become increasingly pervasive in our daily lives, aligning them with human social norms becomes increasingly germane. misaligned llms can have serious consequences, such as the propagation of harmful stereotypes and inappropriate decision-making in delicate social situations @xcite . the exploration of 4 . socialgaze aligns the model's understanding of social acceptance with human consensus. llms' abilities in reasoning about socially acceptable behaviors thus becomes imperative. social acceptability evaluates the appropriateness of an individual's actions in a social context. its understanding is a crucial prerequisite for conversational assistants to play a role in mediation and counseling by evaluating and navigating multifaceted social interactions. however, social acceptability can be highly complex and nuanced such as choosing between sharing a harsh truth or a kind lie, and it is unclear how well llms grasp this concept. in this work, we focus on the ability of a model to understand socially acceptable behavior. to investigate social acceptability in current large language models, we use social anecdotes -narrative accounts that depict scenarios of social conflict, where a narrator engages with one or more opposing parties. these situations span a variety of human experiences, ranging from friendly disagreements to familial disputes and workplace confrontations. the opposing parties in these anecdotes, individuals or groups, are defined as those who are in disagreement with the narrator. given the social anecdote detailing a conflict, we introduce a two-fold task of assigning a so-cial judgment -a binary label indicating social acceptability/unacceptability of the narrator's actions (whether the narrator is "the asshole" or not), and generating a rationale -a natural language justification of the judgment providing transparency and interpretability to the assessments. figure 1 illustrates an example of a social anecdote (top) and corresponding social judgment and rationale generated by . broadly, our findings reveal a significant misalignment between llm judgments and human consensus. in llms with 13b parameters or fewer, generated rationales often omit crucial events that inform these judgments. in contrast, more capable llms such as gpt-3.5 tends to produce harsher and more judgmental rationales compared to human responses on social forums. to address this discrepancy, we draw inspiration from (1) judicial processes, where deliberation and consideration of multiple perspectives precede rendering of verdicts @xcite , and from (2) planning and agentic workflows, where structured reasoning and self-refinement improve decision making @xcite @xcite . based on these insights, we introduce the socialgaze framework, which guides llms first to distill the anecdote into a summary, then spotlight key narrative events from multiple perspectives, and utilizes this enriched context to render a final judgment and rationale. figure 1 (bottom right) illustrates the change in rationale and judgment resulting from socialgaze. our experiments on social anecdotes from the r/aita subreddit demonstrate the effectiveness of the deliberation process in social-gaze at making models understand social acceptability across language models. additionally, given the inherently subjective and context-dependent nature of social judgments, we also examine several narrative features and their influence on llms' understanding of social acceptability. our analysis reveals the following: understanding and addressing these biases in social contexts is crucial. if left unchecked, such behaviors can lead to systemic discrimination and unfair treatment in automated decision-making processes, and erode trust in llm-based systems. the implications of such biases can be deep and very tangible, potentially impacting areas from hiring practices and law enforcement to social services and interpersonal communications. empirically, while models equipped with so-cialgaze are more robust to narrative lengths, they still exhibit similar biases concerning gender and age as vanilla prompting models. these results highlight the need for future strategies to mitigate biases inherent in prompt-based models, particularly if they are to serve as conversational interfaces for social interactions. ## evaluating social acceptance in this section, we provide a formal definition of the social acceptance task ( §2.1), followed by a detailed description of the socialgaze framework ( §2.2). ## results in this section, we evaluate socialgaze on its ability to judge the narrator ( §4.1) and generate good rationales for its judgment ( §4.2). ## analysis in this section, we dissect performance across varying narrative lengths, and examining potential age and gender biases, uncovering its strengths and limitations with and without socialgaze. additionally, we also analyze model behaviour for various narrator roles and influence of addressing the narrator in second person or third person on social acceptance, in appendix c. 8 gender bias in social judgements we conducted a gender bias study to explore potential disparities in how llms equipped with social-gaze judge anecdotes involving conflicts between narrators and their romantic partners of a different gender. this study centered on scenarios where the narrator's gender was explicitly stated as male or female in the social anecdote (approximately 300 in number). to assess the bias, we manipulated the anecdotes by swapping the genders of the narrators and their partners, and evaluated any changes in the models' judgments. detailed methods for extracting and manipulating gender information using prompts are provided in appendix b. while f1 scores for social judgments involving male and female narrators are similar across models (see in the long term, such biases can perpetuate harmful stereotypes and reinforce unfair treatment of men in social contexts. for instance, in scenarios involving conflict resolution or social mediation, a bias towards blaming male narrators can lead to unjust outcomes and exacerbate gender disparities. age bias in social judgements in this analysis, we investigate if llms augmented with so-cialgaze exhibit bias towards certain age groups. from our evaluation set, we analyze 742 narratives which included explicit mentions of the narrator's age. we categorize the posts based on the age of the narrator into three bins: '<20' years, '20-30' years, and '>30' years. in figure 3 , we plot the macro f1-scores of social judgments based on the age groups for all models. the results indicate a trend of improved performance with increasing narrator age, i.e., models predict social judgments better for anecdotes from mature groups. based on a narrative length analysis, we identify that older ('>30') narrators typically provide more detailed accounts (averaging 409 words) than their younger ('<20') counterparts (383 words), likely contributing to better performance (more on the effect of narrative length in rq4). notably, the performance gap between '20-30' and '>30' groups widens with socialgaze, suggesting its effectiveness in utilizing the richer contexts provided by older narrators. additionally, the nature of conflicts exhibited by various age groups might also play a role. interestingly, socialgaze shows significant improvements in the '<20' age group category for models like vi-cuna 13b and llama 2 7b, with gains over 9.5%, contrasting their near-random performance yta (you're the asshole) rationale: while it's understandable that you were in a difficult situation with your kids and you were unwell, refusing to help an elderly man in need when he asked for assistance does not excuse your behavior. it would have been a compassionate gesture to offer help or find someone else who could assist him, especially given the gravity of his situation. your actions can be viewed as selfish and lacking empathy in this scenario. ## conclusion we introduce socialgaze, a deliberative framework that enhances the social reasoning capabilities of large language models (llms). by employing multi-perspective deliberation, socialgaze significantly improves the alignment of llms with human judgments.