Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting (2024)

Abstract: Annotators’ sociodemographic backgrounds (i.e., the individual compositions of their gender, age, educational background, etc.) have a strong impact on their decisions when working on subjective NLP tasks, such as toxic language detection. Often, heterogeneous backgrounds result in high disagreements. To model this variation, recent work has explored sociodemographic prompting, a technique, which steers the output of prompt-based models towards answers that humans with specific sociodemographic profiles would give. However, the available NLP literature disagrees on the efficacy of this technique — it remains unclear for which tasks and scenarios it can help, and the role of the individual factors in sociodemographic prompting is still unexplored. We address this research gap by presenting the largest and most comprehensive study of sociodemographic prompting today. We use it to analyze its influence on model sensitivity, performance and robustness across seven datasets and six instruction-tuned model families. We show that sociodemographic information affects model predictions and can be beneficial for improving zero-shot learning in subjective NLP tasks.However, its outcomes largely vary for different model types, sizes, and datasets, and are subject to large variance with regards to prompt formulations. Most importantly, our results show that sociodemographic prompting should be used with care when used for data annotation or studying LLM alignment.

## introduction how messages are perceived, is often not only dependent on their factual content, but also on the receiver's subjective interpretation: for instance, two dataset annotators might have different equally valid opinions about what the "correct" offensiveness label for a particular tweet should be (e.g., gender: female race: white age: 25-34 edu: m.sc. politics: liberal instruction-tuned llm gender: male race: asian age: 18-24 edu: b.sc. politics: liberal "what the f*** did you guys do to it?" + + very toxic slightly toxic figure 1 : we instruct llms to make predictions for subjective nlp tasks from different perspectives using sociodemographic profiles. we show that, besides sociodemographics, outcomes are largely influenced by model choice or prompt formulation. @xcite . as previously shown, this variation is, at least to some extent, tied to sociodemographic characteristics of the receivers, like their gender identity, age, and educational background (e.g., @xcite @xcite . thus, nlp models need to account for a comprehensive set of sociodemographic factors to make more socially-aware predictions. accordingly, modeling the effect of those factors on subjective tasks has emerged as an important research direction for nlp. as such, researchers have proposed new data collection paradigms -cf. perspectivism @xcite -and trained models for reflecting the decisions of particular sociodemographic groups @xcite . most recently, researchers @xcite @xcite have explored sociodemographic prompting of large language models (llms): the idea is to enrich a particular input prompt with additional sociodemographic informa-tion (cf. figure 1 ). the models' output should then be aligned with the population described. this technique has led to promising applications in dataset augmentation @xcite and simulation of social computing platforms @xcite . still, our knowledge on the effect of including sociodemographic profiles is scarce and the existing literature disagrees on its usefulness: for instance, @xcite showed that sociodemographic prompting can be used to simulate human populations -a promise for more efficient sociological surveys. other work, in turn, points to the danger of stereotypical bias reflected when prompting models with sociodemographic profiles @xcite . our work makes an important step towards a better understanding of the influence of sociodemographic prompting on models' decisions. we use subjective nlp tasks for evaluation as they have been shown to induce disagreement among annotators related to sociodemographic factors. our study analyzes the effects of sociodemographic prompting on model output (sensitivity), zero-shot performance, and robustness. concretely, we make the following contributions: overall, our results provide important insights for future research on sociodemographic prompt-ing, and, in particular, when applying sociodemographic prompting in sensitive scenarios, for instance, in the context of sensitive data annotation or when studying llm alignment. ## related work the sociodemographic background of annotators has been identified as an influential factor in text annotation for subjective nlp tasks @xcite @xcite @xcite . consequently, researchers started to integrate such information into nlp models to enable more socially aware predictions @xcite @xcite @xcite . with the increasing performance of llms, researchers investigated to what extent they are influenced when prompted with sociodemographic information. @xcite examine whether instruction-tuned llms accurately reflect or conform to human disagreements but limit their experiments to a single nli dataset. they conclude that models deviate from human annotators in terms of accuracy and disagreement level. by analyzing disagreement for q&a, @xcite find that users' opinions and their sociodemographic background are not mutual predictors. for predicting individual opinions, they show that combining sociodemographic information and relevant past opinions performs best. several works @xcite @xcite ) analyze llm's alignment with specific sociodemographic groups and show that model responses are biased towards responses by participants from western countries. notably, @xcite observe that misalignment persists even after explicitly steering the lms towards particular demographic groups. @xcite suggest using gpt-3 as testbed before conducting large-scale population surveys. they propose algorithmic fidelity to evaluate alignment with different human subpopulations and present it as a costefficient proxy for specific human sub-populations in social science research. finally, it has been shown @xcite @xcite that prompting large models with sociodemographic information is prone to amplify existing stereotypical biases @xcite . in contrast to studying llm alignment, our work extends previous work showing that integrating sociodemographic information in a supervised manner is helpful to improve annotator-specific predictions @xcite . we use sociodemographic prompting because it allows to diversify model predictions without the need for privacy-concerning data collection of annotators' sociodemographic information. beyond our objective, our results offer important insights into influential factors for sociodemographic prompting in general. ## sociodemographic prompting throughout this work, we prompt a language model with (or without) sociodemographic information for obtaining predictions for the classification tasks we study. in the following, we discuss the main concepts our methodology relies on. prompting. prompting refers to the act of providing an initial input or cue to a language model, guiding its subsequent output generation. llms rely on these prompts to produce contextually relevant and coherent responses. ## sensitivity we investigate how sensitive the predictions are, i.e., to what extent llms' predictions change when instructed to answer from viewpoints characterised by particular sociodemographic backgrounds. ## performance we analyze the impact of sociodemographic prompting on models' zero-shot performances. ## robustness previous work demonstrated that prompting for text classification is influenced by the prompt format @xcite . thus, we investigate whether sociodemographic prompting is robust as indicated by the extent to which predictions change when reformulating the instruction. ## discussion and recommendations while all llms are sensitive to sociodemographic prompting, we identified model scale and the number of instruction-tuning tasks as relevant factors for improving model performance. toxicity detection and sentiment classification are the tasks which benefit the most from this technique. further, the model family and prompt formulation have a strong influence on model predictions. thus, we emphasize that sociodemographic prompting should be used with care, especially in human response simulation @xcite and data annotation @xcite . from these findings, we can extract actionable suggestions. first, estimating the degree of sociodemographic "alignment" of any llm should not be merely based on the outcome of prompting with varying sociodemographic profiles. our work points out the need of a general evaluation framework for studying the sociodemographic alignment of llms. second, if any sociodemographic prompting experiment is conducted, a robustness analysis should accompany the work to evaluate the validity of the findings. sociodemographic prompting is effective at modeling disagreement. we also acknowledge potential applications of sociodemographic prompting in the future. @xcite trained a model for disagreement prediction in subjective nlp tasks.their approach relies on the existence of annotated data alongside the sociodemographic information of the annotators. thus, we investigate whether we can use sociodemographic prompting as an efficient method to identify instances which will likely result in disagreement during annotation. we compare the original annotations with the result of sociodemographic prompting and calculate a binary f1 score. true positives are instances which received disagreement in both setups. conversely, true negatives are instances which received no disagreeing votes in both setups. we present the results in figure 5 . surprisingly, the best-performing zero-shot models ( §6) are not the best at modeling disagreement. with a mean performance of 0.62, flan-t5 (11b) produces the best and most consistent results across all datasets. this is confirmed by our statistical analysis ( §a.9 for details). for the two datasets (dp, diaz) with original sociodemographic information and lowest iaa overall, we observe the best performances across different model sizes. as both datasets induce increased prediction changes ( §5), we hypothesize that sociodemographic prompting is more sensitive if there is larger disagreement in the original annotation. we interpret this as a promising result for using zero-shot sociodemographic prompting to estimate whether a text is likely to induce disagreement among annotators. we leave the exploration of integrating additional training signals (e.g., few-shot) as future work. ## conclusion we study sociodemographic prompting for subjective nlp tasks and employ a comprehensive study across seven datasets and seven instruction-tuned llms from different model families. our results show that these models are sensitive to sociodemographic prompting and using this technique can improve zero-shot performance. however, we also observe a strong influence of the prompt formulation and model family. thus, we argue that sociodemographic prompting should be used with care in sensitive applications and requires comprehensive evaluation when used for data annotation or studying sociodemographic alignment of language models.