Do LLM s Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLM s (2025)

Abstract: Large language models (LLMs) can lead to undesired consequences when misaligned with human values, especially in scenarios involving complex and sensitive social biases. Previous studies have revealed the misalignment of LLMs with human values using expert-designed or agent-based emulated bias scenarios. However, it remains unclear whether the alignment of LLMs with human values differs across different types of scenarios (e.g., scenarios containing negative vs. non-negative questions). In this study, we investigate the alignment of LLMs with human values regarding social biases (HVSB) in different types of bias scenarios. Through extensive analysis of 12 LLMs from four model families and four datasets, we demonstrate that LLMs with large model parameter scales do not necessarily have lower misalignment rate and attack success rate. Moreover, LLMs show a certain degree of alignment preference for specific types of scenarios and the LLMs from the same model family tend to have higher judgment consistency. In addition, we study the understanding capacity of LLMs with their explanations of HVSB. We find no significant differences in the understanding of HVSB across LLMs. We also find LLMs prefer their own generated explanations. Additionally, we endow smaller language models (LMs) with the ability to explain HVSB.The generation results show that the explanations generated by the fine-tuned smaller LMs are more readable, but have a relatively lower agreeability.

## introduction large language models (llms) have demonstrated remarkable capabilities in understanding and generating texts, and their various applications produce 1 code and data are available at https://github.com/ ku-nlp/evaluate-alignment-hvsb widespread impact @xcite . the training of llms relies on corpora that reflect human values, and thus llms are expected to learn and reproduce these values @xcite @xcite . human values regarding social biases (hvsb) reflect the human insight about social biases in a contextual scenario. if llms fail to align with hvsb, they may unconsciously reinforce stereotype biases @xcite , harming users and wider disadvantaged groups @xcite . therefore, it is crucial to study the alignment of llms with hvsb. many datasets have been proposed to evaluate @xcite @xcite or mitigate @xcite social biases in llms. these datasets contain different scenarios that reflect stereotype biases in human values. for example, question-and-answer (q&a) scenarios @xcite , conversation scenarios @xcite , and realistic scenarios of misconduct emulated by an llm (e.g., gpt-4o; ope-nai (2024a)) @xcite . the annotated examples in these datasets reflect the value judgments of their authors or crowd-sourced annotators and thus can be considered as proxies for hvsb. however, no previous studies have utilized these datasets to investigate the alignment between llms and hvsb across different scenario types. in this study, we investigate the alignment preferences of llms with hvsb on different bias scenarios. we study the following aspects: 1) the degree of alignment of llms with hvsb. we investigate the alignment of 12 llms from four model families on four human-annotated datasets. our study makes the following three contributions. first, we introduce our pipeline for evaluating the alignment of llms with hvsb (section 2). second, we find that scenario type is a key factor influencing alignment: llms align better with hvsb in scenarios involving negative questions or harmful stereotype answers. our experiments also reveal several additional findings. for example, increasing the scale of model parameters does not necessarily reduce llms' misalignment rate and attack success rate @xcite . llms within the same model family exhibit a certain level of alignment correlation. moreover, few-shot learning fails to enhance the alignment of llms with hvsb (section 4). third, we study the understanding capacity of llms with their explanations of hvsb (section 5). our results indicate no significant differences in the understanding of hvsb across llms, while llms prefer their own generated explanations. in addition, by endowing smaller language models (lms) with the ability to explain hvsb, we find that the explanations generated by the smaller lms can improve human readability while decreasing model agreeability. ## evaluation methods to evaluate the alignment of llms with hvsb, we consider two tasks: 1) judgment: investigating the degree of misalignment of llms with hvsb. ## experiment settings our study investigates the alignment of llms with hvsb. in this section, we describe the experimental settings to test llms' alignment of hvsb. ## conclusion to study the alignment of llms with hvsb, in this work, we propose a new pipeline that considers two aspects to evaluate the alignment of llms with hvsb. on the one hand, we evaluate whether llms can accurately judge hvsb. on the other hand, we evaluate whether llms can understand hvsb. our experiments on 12 llms from four model families show a certain degree of misalignment between llms and hvsb. moreover, the degree of misalignment differs significantly across different types of scenarios (e.g., scenarios containing negative and non-negative questions). in addition, we find that llms prefer their own generated explanations, which reflects a judgmental bias. our study will help provide evaluation methods and references for aligning hvsb with llms.