Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models (2025)

Abstract: Modern language models are trained on large amounts of data. These data inevitably include controversial and stereotypical content, which contains all sorts of biases related to gender, origin, age, etc. As a result, the models express biased points of view or produce different results based on the assigned personality or the personality of the user. In this paper, we investigate various proxy measures of bias in large language models (LLMs). We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores. However, if we reformulate the task and ask a model to grade the user’s answer, this shows more significant signs of bias. Finally, if we ask the model for salary negotiation advice, we see pronounced bias in the answers. With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle: modern LLM users do not need to pre-prompt the description of their persona since the model already knows their socio-demographics.

## introduction as large language models (llms) are being increasingly adapted for personalization, accounting for a diverse and ever-growing user base has become even more critical @xcite @xcite . since using llms to solve everyday tasks is becoming omnipresent, this growing dependence also raises a number of concerns related to hidden biases in models' behavior @xcite . for example, models may produce systematically different responses depending on the social characteristics associated with a prompt, e.g., gender or race @xcite . at the same time, in april 2025, openai officially announced a feature of personalized responses in @xcite , which allows it to generate answers based on prior information from the user, including, for example, the user's gender. in light of this, an important question arises: how does the personalization of user expertise influence the responses generated by llms? in this paper, we examine a set of scenarios in which llm responses could be affected by the additional user information provided. though a complete removal of the undesirable bias using an automated procedure is shown to be impossible, as it is only distinguishable from the rules and structure of language itself by negative consequences in downstream applications @xcite , the research in the direction of debiasing language models is being rapidly developed @xcite . in socio-economic studies, one attempt to measure bias is through the analysis of the gender pay gap across different countries @xcite . this implies quantifying the impact of such biases in financial terms, also taking into account factors such as seniority and professional field @xcite . to address this bias, various efforts have been made, one of which is the implementation of diversity training programs @xcite . however, the results indicate that notable changes were primarily observed among participants already predisposed to inclusivity; among others, the impact was limited @xcite . this suggests that one-off diversity trainings, which are commonplace in organizations, are unlikely to serve as stand-alone solutions for promoting workplace equality, especially given their limited effectiveness among the very groups policymakers aim to influence most. in this context, llms seem to be similar in the sense that one-off attempts to debias the outputs on the set of predefined keywords also have mixed results. in our study, we gradually increase the complexity of tasks given to llms to examine how this affects gender bias. in this paper, we discuss the methodology for llm bias detection: ## methods we conduct a series of experiments with a range of language models for different personae. in this section, we outline the experimental setup with prompts, choice of models, personae, and data. ## experimental results in this section, we report the results of the experiments we conducted and interpret them. ## discussion the significant differences in experiment 1 are in absolute minority and are mostly scattered among models, subjects, and persona groups. the small proportion of significant numbers and the lack of dependency do not allow us to claim that there is some "directional" bias towards some personae. our results also add up to the research on evaluation method comparison, showing that evaluation by generation is noisier than the one based on probability. in experiment 2, the picture is similar, though the proportion of significant results among all is larger, and the bias is directed. we hypothesise that the models might be more agreeable to the statements of personae, towards whom the stereotypical bias is usually directed, regardless of whether the person is right or wrong, as a result of an improper alignment during training. the results of experiment 3, however, show that when we ground the experiments in the socioeconomic context, in particular, the financial one, the biases become more pronounced. when we combine the personae into compound ones based on the largest and lowest average salary advice, the bias tends to compound. this presents a major concern with the current development of language models. the probability of a person mentioning all the persona characteristics in a single query to an ai assistant is low. however, if the assistant has a memory feature and uses all the previous communication results for personalized responses, this bias becomes inherent in the communication. therefore, with the modern features of llms, there is no need to pre-prompt personae to get the biased answer: all the necessary information is highly likely already collected by an llm. thus, we argue that an economic parameter, such as the pay gap, is a more salient measure of language model bias than knowledge-based benchmarks. as a possible form of measuring the bias, we propose the results we present in ## conclusion in this paper, we have studied various proxy measures of bias on a range of models. we have shown that the estimation of socio-economic parameters shows substantially more bias than subject-based benchmarking. furthermore, such a setup is closer to a real conversation with an ai assistant. in the era of memory-based ai assistants, the risk of persona-based llm bias becomes fundamental. therefore, we highlight the need for proper debiasing method development and suggest pay gap as one of reliable measures of bias in llms.