M c BE : A Multi-task C hinese Bias Evaluation Benchmark for Large Language Models (2025)

Abstract: As large language models (LLMs) are increasingly applied to various NLP tasks, their inherent biases are gradually disclosed. Therefore, measuring biases in LLMs is crucial to mitigate its ethical risks. However, most existing bias evaluation datasets are focus on English andNorth American culture, and their bias categories are not fully applicable to other cultures. The datasets grounded in the Chinese language and culture are scarce. More importantly, these datasets usually only support single evaluation task and cannot evaluate the bias from multiple aspects in LLMs. To address these issues, we present a Multi-task Chinese Bias Evaluation Benchmark (McBE) that includes 4,077 bias evaluation instances, covering 12 single bias categories, 82 subcategories and introducing 5 evaluation tasks, providing extensive category coverage, content diversity, and measuring comprehensiveness. Additionally, we evaluate several popular LLMs from different series and with parameter sizes. In general, all these LLMs demonstrated varying degrees of bias. We conduct an in-depth analysis of results, offering novel insights into bias in LLMs.

## introduction due to their excellent performance in understanding and generating human language, large language models (llms) are widely used in daily interactions with humans and various downstream tasks. however, it has been observed that llms can inadvertently express stereotypes and biases towards certain demographic groups @xcite @xcite @xcite . a significant reason is that the training corpora have yet to be strictly filtered, and llms inherit many unfair or stereotypical expressions during the training process @xcite . figure 1 illustrates this phenomenon that some language models tend to associate men with programmers and doctors, while women are linked to homemakers and nurses @xcite . applying such language models to nlp tasks may further reinforce these stereotypes, thus damaging social fairness and causing harm to certain demographic groups. although numerous studies @xcite @xcite have been dedicated to evaluating biases in llms, most of them face three limitations, as illustrated in figure 2 . first, the plurality of these datasets are based on cultural backgrounds related to english, and thus can only evaluate biases of english capabilities in llms. they cannot measure the biases present in other cultural backgrounds. second, existing evaluation benchmarks pay less attention to categories with regional and cultural characteristics. additionally, other noteworthy categories also receive relatively scant consideration. third, most previous works using question-answering @xcite @xcite @xcite or counterfactual-inputting @xcite to evaluate llms, which cannot fully and comprehensively measure bias. to address the issues mentioned above, we introduced mcbe, a multi-task chinese bias evaluation benchmark. this is a comprehensive chinese bias evaluation benchmark for llms. mcbe consists of 4,077 bias evaluation instances and covers 12 single bias categories, including gender, religion, nationality, socioeconomic status, age, appearance, health, region, lgbtq+, worldview, subculture, and race. each bias category contains numerous bias evaluation instances for detailed evaluation. furthermore, we have introduced 5 evaluation tasks, including preference computation, bias classification, scenario selection, bias analysis, and bias scoring, to more thoroughly quantify the potential chinese biases in llms. figure 3 illustrates the overall structure of the mcbe. in summary, our key contributions are as follows: biases have been identified in different nlp tasks. in machine translation, as @xcite found, there is a "male default" phenomenon, such as specific roles being translated with gender assumptions. in coreference resolution, @xcite and @xcite disclosed biases where models may wrongly link gender pronouns to occupations based on gender stereotypes. in text generation, venkit et al. (2023) discussed nationality bias in gpt-2, like using negative descriptions for people from countries with lower gdps. ## tasks for bias evaluation accurately evaluating biases in llms is challenging, as simply observing whether a model provides a fair answer is insufficient to determine the severity of its biases. therefore, we design 5 evaluation tasks in mcbe to measure model bias systematically: preference computation, subcategory classification, scenario selection, bias analysis, and bias scoring. each task is scored from 0 to 100, with higher scores indicating lower bias. the prompts we used in the evaluation experiments can be found in appendix d.3. our research is under the assumption that an unbiased model should not demonstrate a particular preference for any demographic group @xcite . however, a score of 100 does not guarantee that the model is entirely unbiased. it merely suggests a lower likelihood of displaying apparent biases. ## results and discussion in this section, we discuss the bias performance of the models across bias categories and evaluation tasks. to maintain consistency with previous multi-task evaluation benchmarks @xcite , we derive a relatively reasonable comprehensive ranking by calculating the average score, similar to the overall grade in school examinations, aiming to provide participants with an intuitive reference. the experimental settings can be found in appendix f, and the all models' scores maps in all bias categories and tasks can be found in appendix g. ## conclusion this paper expands efforts to evaluate chinese bias in llms by introducing multi-task chinese bias evaluation benchmark (mcbe), which encompasses 4,077 bias evaluation instances categorized into 12 single bias categories and 82 subcategories. mcbe introduces the concept of bias evaluation instance and goes beyond single-task evaluation by providing diverse tasks to quantify bias in llms. extensive experiments demonstrate the effectiveness of mcbe in evaluating chinese biases in chinese and multilingual llms. these experiments examine the differences in bias manifestation across llms with different parameter sizes and structures, and offer novel insights into the possible reasons behind these varying bias mani-festations in llms.