Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLM s (2025)

Abstract: Human judgments are inherently subjective and are actively affected by personal traits such as gender and ethnicity. While Large LanguageModels (LLMs) are widely used to simulate human responses across diverse contexts, their ability to account for demographic differencesin subjective tasks remains uncertain. In this study, leveraging the POPQUORN dataset, we evaluate nine popular LLMs on their abilityto understand demographic differences in two subjective judgment tasks: politeness and offensiveness. We find that in zero-shot settings, most models’ predictions for both tasks align more closely with labels from White participants than those from Asian or Black participants, while only a minor gender bias favoring women appears in the politeness task. Furthermore, sociodemographic prompting does not consistently improve and, in some cases, worsens LLMs’ ability to perceive language from specific sub-populations. These findings highlight potential demographic biases in LLMs when performing subjective judgment tasks and underscore the limitations of sociodemographic prompting as a strategy to achieve pluralistic alignment. Code and data are available at: https://github.com/Jiaxin-Pei/LLM-as-Subjective-Judge.

## introduction from sentiment analysis to dialogue generation, large language models (llms) have demonstrated impressive capabilities in various natural language processing (nlp) tasks @xcite . recent research has begun exploring whether these models possess social knowledge analogous to that of humans @xcite . for example, @xcite replicate eight classic psychological experiments on llms to test their ability to reason about moral and legal issues. @xcite examines how llms' "instrumental knowledge" relates to the more ordinary "worldly" knowledge of human agents. building on these insights, llms have been applied to large-scale labeling tasks requiring social understanding, and often with promising results @xcite . in terms of subjective tasks, researchers have explored llms' zero-shot potential in areas such as character simulation @xcite and hate speech detection @xcite . however, llms face significant challenges in handling subjective tasks. it is well acknowledged that social biases and stereotypes embedded in their training data can lead to inadequate representation of diverse human experiences @xcite . as a result, using llms for subjective tasks risks producing outcomes that disproportionately favor certain demographic groups, leading to biased or unfair results @xcite . @xcite found that when responding to value-based questions, llms tend to align more closely with the perspectives of lower-income, moderate, and protestant or roman catholic individuals. despite these early findings, limited research has explored whether llms exhibit similar systemic biases with certain social groups across other subjective nlp tasks, highlighting the need for further investigation into their broader implications. subjective tasks present an additional challenge because language perception is shaped by social context and identity @xcite . for instance, a text perceived as polite or inoffensive by one group may be interpreted differently by another. ideally, llms should capture the full spectrum of subjective judgments. steerable pluralism, as described by @xcite , refers to an llm's ability to be faithfully adjusted to represent specific perspectives. yet, @xcite found that many current llms have limited steerability to take on various persona, due to both inherent biases in their baseline behavior and asymmetries in how they adapt across different persona dimensions. these limitations suggest that while steerability is a promising direction, it requires more refinement to effectively capture diverse perspectives. sociodemographic prompting, which involves enriching prompts with demographic or individualspecific information, has gained increasing attention in recent research. this approach has shown potential for improving data augmentation and simulating human behavior for social science applications @xcite . despite its promise, the effectiveness of sociodemographic prompting remains debated, as model performance can be sensitive to the phrasing, structure, or order of prompts @xcite . for example, @xcite finds that the impact of adding demographic information varies significantly depending on the model, task, and prompt design. moreover, some studies suggest that sociodemographic prompting can exacerbate stereotypes and biases @xcite or reduce model performance on certain tasks @xcite . given these mixed findings and the focus of previous studies on specific nlp tasks, our work extends the literature by examining (1) whether llms' predictions systematically align more with certain social groups on two more subjective tasks and (2) how llms can effectively account for identity-based differences in perception when handling subjective language tasks with sociodemographic prompting. leveraging the popquorn dataset @xcite , we evaluate nine popular llms on their ability to understand demographic differences in subjective tasks, offensiveness and politeness. the two tasks are occasionally related but distinct. politeness pertains to notions of status differences and interpersonal distance, while offensiveness involves violations of expected social norms. offensiveness is not as broad as impoliteness, as varying levels of politeness can be perceived as non-offensive. exploring these subtly different tasks offers a more comprehensive evaluation of llms' potential biases in subjective nlp tasks. overall, our results reveal that intrinsic biases persist in llms when applied to these tasks. the study highlights the limitations of llms in understanding and aligning gender and racial differences in subjective judgment. while some research aims to directly use llms to simulate group-specific social behaviors, our findings underscore the risks of unintentionally reinforcing racial and gender biases when applying sociodemographic prompting to subjective tasks. ## methods data we use the popquorn dataset @xcite to evaluate llms' capacity to tackle subjective nlp tasks. popquorn includes 45,000 annotations from a demographically representative u.s. sample. we focus our analysis on two identity types: gender and ethnicity. to ensure statistical robustness, we focus on the gender categories man, woman and ethnic groups asian, black and white as they have sufficient annotations. for this study, we analyze annotators' offensiveness and politeness ratings on a 5-point likert scale. we compute average scores for each identity group to capture perceptions from specific demographics. the mean overall offensiveness score is 1.88 (@xmath0.76), and politeness scores average 3.31 (@xmath1.91). scores from men, women, and white annotators closely mirror the overall distribution, while black and asian annotators show diverging means and higher variance. figure 3 in appendix a shows the distributions of both overall and identity-specific scores for offensiveness and politeness tasks. models to enhance the generalizability of our findings, we conduct experiments with a range of open-source and close-source llms: flan-t5-xxl @xcite , flan-ul2 @xcite , tulu2-dpo-7b, tulu2-dpo-13b @xcite , gpt-3.5, gpt-4 (openai, 2023), llama-3.1-8b-instruct @xcite , mistral-7b-instruct-v0.3 @xcite , and qwen2.5-7b-instruct @xcite . prompts we design prompts to instruct the models to predict offensiveness and politeness scores for each instance. to ensure the prompts elicit valid responses, we conduct preliminary experiments on a small subset of data. an example prompt ( ## are model predictions closer to certain demographic groups? while individual judgments may vary, llms can generate only a single prediction unless explicitly instructed to output a distribution. therefore, when llms are applied to judgment tasks, it is crucial to examine whether their predictions align more closely with certain demographic groups. analysis to measure the alignment between llm and certain demographic groups, we define baseline prediction error (e base ) as the absolute difference between llms' predictions using identityfree prompts and human ratings from a specific demographic group: for each task and demographic identity type, we apply separate linear mixed effect models to examine changes in baseline prediction error of a specific demographic group (target group) compared to the reference group, controlling for instance-level variations with instance id as a random effect. for example: a regression coefficient β = 0 indicates that there is no difference in baseline prediction errors between the target and reference groups. a positive β means that baseline prediction errors are larger for the target group, suggesting that the llm predictions are closer to the reference group than to the target group. the aggregated results are visualized in figure 1 , while ## does sociodemographic prompting improve alignment with demographic groups? given the intrinsic bias of llms in subjective judgment tasks, a natural question arises: does adding demographic information in prompts steer llms to generate more diverse outputs that better align with specific groups? in this section, we conduct a series of analyses to answer this question. ## discussion with the large-scale deployment of llms in our society, it becomes increasingly important to study whether llms are able to understand the preferences of different groups of people. our results suggest that llms are more aligned toward certain demographic groups than others on subjective perception tasks. for both of our tasks, we find that all of our tested llms provide answers which are closer to the annotations of white annotators compared to other demographic groups. our findings contribute to the newly growing knowledge of types of demographic biases inherent in llms when asked to solve subjective tasks @xcite , signaling caution for potential applications such as deploying llms for generating annotations at large scale @xcite . our results also suggest that directly inserting demographic features into prompts, unfortunately, does not reliably help models adopt the perspectives of target groups. the ability of llms to consider various opinions, at least from the perspective of demographic groups, seems limited at its current stage. furthermore, we observe that newer models, such as mistral-0.3 and qwen-2.5, exhibit reduced alignment on different task types and identity-based prompts. this may be due in part to increasingly strict guardrails designed to mitigate harmful outputs, which can also affect model performance by increasing refusal rates and limiting functionality @xcite . given that our tasks include sensitive keywords (e.g., vulnerable identity, offensive, not polite), these safety mechanisms may further contribute to the diminished effectiveness of identity-based prompting in newer models. ## conclusion in this study, we study llms capability to account for demographic differences in subjective judgment tasks. we find that llms' predictions are closer to white people's perceptions for both tasks and across 9 models compared with asian and black people. we further explore whether incorporating demographic information into the prompt helps mitigate this bias. surprisingly, we find that adding identity tokens (e.g. black and man) does not consistently help to improve the models' performance at predicting demographic-specific ratings. our results suggest that llms may hold implicit biases on subjective nlp tasks and sociodemographic prompting is not an effective approach to address this bias yet. researchers and practitioners should be careful when using llm as judges on subjective tasks.