Personas with Attitudes: Controlling LLM s for Diverse Data Annotation (2025)

Abstract: We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results indicate that persona-prompted LLMs generate more diverse annotations than LLMs prompted without personas, and that the effects of personas on LLM annotations align with subjective differences in human annotations. These effects are both controllable and repeatable, making our approach a valuable tool for enhancing data annotation in subjective NLP tasks such as toxicity detection.

## introduction many nlp tasks depend on human-annotated data, often gathered via crowdsourcing. @xcite distinguish between two paradigms for handling label disagreement among annotators with diverse backgrounds. the prescriptive paradigm minimizes subjectivity and aims for a single groundtruth label per instance, typically through majority vote, which facilitates model training. this approach assumes that the aggregated judgment of diverse annotators approximates the true label. in contrast, the descriptive paradigm embraces subjectivity, using disagreement to explore diverse perspectives and improve model robustness. neither paradigm is inherently superior because each serves different goals, with the prescriptive paradigm supporting efficient training and the descriptive paradigm enabling the study of annotation diversity and the understanding of differences in perceptions across annotators with diverse beliefs and backgrounds. in this work, we propose to combine the idea of using llms as annotators with the large pool of personas offered by the persona hub @xcite to both increase and control the diversity of the generated annotations. to explore the feasibility of injecting personas into llm prompts to diversify and steer the models' zero-shot annotations in all its facets, we organize this work into two studies. study 1 covers the prescriptive paradigm towards diverse annotations, assuming the existence of a single label per instance and evaluating our approach's annotation diversity by comparing the persona-prompted llm annotations to it. study 2, in contrast, is in line with the descriptive paradigm, exploring the approach's ability to reconstruct the diversity found in human annotations and to controllably replicate the observed effects of human subjectivity. while each study tests the suitability of our llm personaprompt approach for a specific paradigm, together they show that the approach increases annotation diversity (study 1) and that persona effects are not random but follow predictable and controllable patterns, similar to human subjectivity (study 2). in this paper, therefore, we set out to answer the following two research questions: researchers have explored the abilities and performance of llms in annotating datasets for different types of constructs. @xcite , @xcite , @xcite and @xcite provide overviews of scenarios for which the idea of using llms as annotators has already been tested, including tasks as diverse as determining the relevance of texts for specific issues, the detection of humor, or the extraction of medical information. other researchers argued that llms are particularly well suited for the annotation of subjective constructs like hate speech, offensive language and toxicity @xcite . they argue that using llms can counter the instability in annotations that often arises from the varying social backgrounds of human annotators. even more recently, researchers have started to explore the performance impacts of aggregating annotations generated with different llms @xcite , acting upon the assumption that an increased diversity of the crowd of annotators -be they human or llm -would lead to gains in performance. @xcite explore yet another angle of llm annotations by comparing the annotation performance of gpt-4 with the annotations resulting from a carefully designed and conducted crowdworker annotation pipeline. yet another line of research is moving beyond the use of llms to predict the groundtruth label only, proposing to personalize llms via the inclusion of socio-demographic information in order to steer the llm annotations towards those provided by human annotators. recent work has explored the use of persona prompting to influence nlp annotation tasks (e.g., @xcite ; @xcite ). while both studies make valuable contributions, they differ from our approach in a key aspect: they define personas using one or two demographic attributes, thereby operating at a subgroup level. in contrast, our method is the first to employ rich, individualized persona descriptions, allowing for variation among individuals within the same subgroup (e.g., two "white men" may receive different prompts). this enables a more nuanced and fine-grained investigation of subjectivity in annotation tasks. among other prominent approaches are argyle et al. ( 2023 while the generation of dataset labels via llms might be interpreted as a form of synthetic training data generation, this description is usually reserved for efforts that synthetically create the instances to annotate, not (only) the corresponding labels. @xcite offer an extensive review of the state of the literature together with a detailed discussion of associated challenges and opportunities. fundamental for this work, @xcite introduce the persona hub, a collection of 1,000,000,000 diverse persona descriptions, as a way to diversify the synthetic instances that llms generate, and show that their persona descriptionswhen included in the prompts used to synthesize, e.g., novel math or logical reasoning problems -are successful in increasing the diversity of the resulting datasets and thereby also the generalizability of the models trained on the datasets' tasks. ## data to systematically test the impact of including persona descriptions in the prompts used to collect toxicity annotations from different llms, we rely on two external sources of data described next. ## methodology experiments in all studies were conducted with two different llms from two popular families of open-source models -the mistral-7b-instruct-v0.1 model and the qwen2-7b-instruct model. further justification for our choice of models and details on model deployment can be found in appendix a.3. figure 1 provides an overview of the experimental setup used in our two studies. ## results the following sections present the results of the experiments described above. all results can be reproduced using the code made available on github 3 together with the publicly available datasets shared by @xcite and @xcite . ## discussion and conclusion in this work we explored the potential of personalizing llms through persona-based prompts to enhance diversity and control in data annotation tasks. by injecting persona descriptions into llm prompts, we observed an increase in the variability of model annotations compared to annotation runs that did not include persona descriptions, demonstrating through various experiments that personas can influence model outputs in a consistent and controllable manner. we show that our persona-based approach to llm data annotations offers a novel way to simulate human subjectivity in annotations, which can be particularly useful in tasks that require diverse and subjective perspectives, such as the detection of toxicity. our findings suggest that personas not only introduce desirable diversity in annotations, but that they also enable researchers to guide llms toward specific annotation behaviors, making them -under certain conditions -more aligned with groups of human annotators and being successful in replicating effects of annotation subjectivity also found in human annotations.