Assessing the Reliability of LLM s Annotations in the Context of Demographic Bias and Model Explanation (2025)

Abstract: Understanding the sources of variability in annotations is crucial for developing fair NLP systems, especially for tasks like sexism detection where demographic bias is a concern. This study investigates the extent to which annotator demographic features influence labeling decisions compared to text content. Using a Generalized Linear Mixed Model, we quantify this influence, finding that while statistically present, demographic factors account for a minor fraction (~8%) of the observed variance, with tweet content being the dominant factor. We then assess the reliability of Generative AI (GenAI) models as annotators, specifically evaluating if guiding them with demographic personas improves alignment with human judgments. Our results indicate that simplistic persona prompting often fails to enhance, and sometimes degrades, performance compared to baseline models. Furthermore, explainable AI (XAI) techniques reveal that model predictions rely heavily on content-specific tokens related to sexism, rather than correlates of demographic characteristics. We argue that focusing on content-driven explanations and robust annotation protocols offers a more reliable path towards fairness than potentially persona simulation.

## introduction reliable annotations are foundational to machine learning in nlp, guiding models toward accurate predictions. according to @xcite , annotation involves humans labeling or transforming data inputs into "gold data", which guides machine learning practitioners in building their models. for instance, to create a gold dataset for a model that corrects grammatical errors, annotators might be asked to identify mistakes in a range of sample sentences. however, creating high-quality annotations is not a straightforward task since it requires thoughtful consideration of the criteria that make annotations effective, consistent, and unbiased. however, creating high-quality annotations is not a straightforward task since it requires thoughtful consideration of the criteria that make annotations effective, consistent, and unbiased. this raises the following question, what defines a robust annotation process? when it comes to evaluating annotation quality, several studies highlight inter-annotator agreement (iaa), as defined by @xcite , as a standard metric for labeled datasets @xcite . however, achieving high iaa is often challenging, particularly for subjective language tasks that rely on human judgments. for tasks like sexism detection, where subjectivity is inherent, addressing annotator agreement challenges is essential, as disagreements can significantly influence the performance of nlp models trained on this data. in some cases, disagreement often arises from ambiguous sentences or vague label definitions, which can make it difficult for annotators to reach an agreement @xcite . traditionally, aggregating judgments from multiple annotators to create a single "ground truth" for each data instance is widely used to address the inherent ambiguity and subjectivity in language interpretation. this approach is similar to initial methods for handling annotator disagreement, which focuses on estimating a "true" label. however, @xcite shows that even when annotators are provided with additional context, there is not always a single correct answer, and disagreements still persist. recent studies indicate a significant shift in how annotator disagreements are handled, particularly in subjective tasks involving human judgments @xcite @xcite . current research primarily focuses on developing models that can learn from these disagreements. while nlp researchers aim for consistency among annotators, some level of disagreement is both inherent and unavoidable in human annotation processes @xcite . as @xcite showed, annotators' demographic factors, personal perspectives, and differing value systems can lead to discrepancies in annotations. building on this foundation, researchers have systematically analyzed how the characteristics of annotators and the way tasks are framed can skew evaluation outcomes. for instance, hosking et al. demonstrate that annotator assertiveness and the linguistic complexity of model outputs significantly bias judgments of factuality and consistency in crowdsourced error annotations. their study finds that responses that sound more confident are judged as more accurate, even if they contain the same number of errors. similarly, @xcite reveal that factors such as cultural background, age, gender, and personal values lead to substantial variation in how responses are rated for helpfulness, creativity, and alignment with individual beliefs. these findings underscore the challenge of distinguishing true model performance from annotatorinduced biases and motivate the need for more scalable and consistent annotation methods under controlled conditions. this sparked researchers to explore the potential of genai models as substitutes for human annotators. several studies have shown that large language models (llms), when provided with demographic information, can imitate specific annotator groups by tailoring their outputs to reflect attributes such as gender, race, age, or education @xcite . however, llms often align more closely with certain demographics (e.g., younger, white, male) unless explicitly directed otherwise @xcite . to the best of our knowledge, the incorporation of xai techniques to guide these models is still rare @xcite @xcite . this creates critical gaps in evaluating how demographic biases impact annotation reliability and whether genai models, with xai guidance, can effectively substitute human annotators, especially in subjective tasks such as sexism detection. for instance, @xcite presents an explainability-enhanced sexism detection pipeline that bridges model predictions with token-level explanations, illustrating efforts to improve transparency in sexism detection. in this study, we use data from the exist 2024 challenge @xcite , a shared task on sexism detection in social networks-a highly subjective task. foot_0 our primary goal is to assess annotation reliability and examine how demographic biases influence annotator decisions. using a generalized linear mixed model (glmm), we analyze both fixed and random effects, revealing that demographic variables account for nearly 8% of the variance in labeling behavior, suggesting the presence of demographic biases in human judgments.we also evaluate llm performance by simulating annotation and classification under various prompting scenarios, model configurations, and temperature settings. our methodology compares state-of-the-art models across open-source frameworks and proprietary apis, exploring how prompt modifications affect outcomes. to improve explainability, we employ shap values to reveal the influence of specific tokens on predictions across demographic groups. by integrating shap analyses into persona prompting, we examine how demographic attributes shape predictions. results show that combining shap with persona prompting enhances both interpretability and reliability of llm-generated annotations. bias statement this paper examines how demographic factors, such as gender, ethnicity, education, and region, may influence both human and llm annotations in detecting sexist content on social media. we focus on potential representational harms, wherein certain demographic groups' viewpoints or sensitivities to biased language might be underrepresented or misjudged. by highlighting differences in labeling behaviors across diverse annotator backgrounds, we aim to reduce the risk that an nlp system trained on these annotations will inadvertently perpetuate stereotypes or unfairly discount certain cultural experiences. we take the normative stance that all groups deserve unbiased and respectful treatment in both data collection and model predictions. our ultimate goal is to ensure that technology, especially in sensitive tasks like sexism detection, does not exacerbate inequalities or reinforce harmful narratives. ## related work recent studies have explored how annotators' personal backgrounds, experiences, and identities influence labeling outcomes, particularly in subjective tasks @xcite . however, findings in this area are mixed. some studies report significant correlations between demographic features and annotation results @xcite , while others observe minimal statistically significant differences, especially regarding gender @xcite . these conflicting results highlight the complexity of the relationship between annotator characteristics and labeling decisions. contrasts are particularly evident in tasks such as identifying sexist content, offensive language, and political ideologies, where an individual's personal experiences and group affiliations can significantly influence their perception and categorization of content @xcite . the diversity of findings underscores the need for ongoing research to better understand the intricate interplay between annotator attributes and labeling outcomes. this understanding is crucial for developing more robust and inclusive nlp models that can effectively incorporate diverse perspectives in the annotation process. while some studies attempt to enhance data quality by analyzing disagreements among annotators, systematic investigations into how annotators' demographic biases affect annotation results remain limited @xcite . ## results and discussion do demographic biases mainly drive labeling differences, or does tweet content play a larger role? to investigate this question, we first fit a flat logistic regression model with annotator demographic features as fixed effects. this provides a baseline assessment of demographic influence without accounting for tweet-specific or annotatorlevel variability. we then extend the analysis using a mixed-effects logistic regression model, incorporating crossed random intercepts for annotators and nested random effects for tweets within languages. this approach captures both annotator variability and tweet-specific differences while retaining demographic features as fixed effects. our findings show that incorporating tweet-level and annotator-level variability in the mixed-effects model substantially improves performance over the flat model. the mixed model achieves higher accuracy (73.73% vs 48.76%) and f1 score (75.77% vs 45.09%), along with better fit indicated by lower aic and bic values and a higher auc. a kappa value of (47.06%) and an intraclass correlation coefficient (icc) of 92.3% highlight the importance of accounting for tweet-specific differences, which the flat model ignores. notably, the random effect for tweets shows high variance (33.72), indicating that tweet content is the main source of labeling variability. the annotator random effect (5.54) also contributes meaningfully, while the language effect (0.30) has minimal influence. these findings confirm the mixed-effects model as a more accurate and nuanced approach for understanding the labeling process. ## conclusion this study evaluated the reliability of llm annotations for sexism detection, focusing on the roles of annotator demographics and model explainability. mixed-effects modeling showed that demographic factors, while sometimes statistically significant, accounted for only 8% of the variance in human labels, tweet content and individual differences were the main drivers. we tested the use of demographic personas to guide llms but found this strategy had limited, inconsistent, and sometimes negative effects on performance. shap analysis confirmed that content drove model decisions. these findings suggest that bias mitigation should focus less on broad demographic corrections and more on content and individual-level understanding. simulated personas may oversimplify complexity and risk reinforcing stereotypes. this limitation is underscored by evidence that llm often exhibits uniform stylistic patterns @xcite , showing that current models cannot fully emulate the diverse differences of human annotators. instead, explainability tools that highlight contentrelevant features offer a more promising path toward fairness and reliability in nlp.future research should explore richer ways to capture diverse perspectives and improve content-based guidance in llm annotations.