JBBQ : J apanese Bias Benchmark for Analyzing Social Biases in Large Language Models (2025)

Abstract: With the development of large language models (LLMs), social biases in these LLMs have become a pressing issue.Although there are various benchmarks for social biases across languages, the extent to which Japanese LLMs exhibit social biases has not been fully investigated.In this study, we construct the Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the English bias benchmark BBQ, with analysis of social biases in Japanese LLMs.The results show that while current open Japanese LLMs with more parameters show improved accuracies on JBBQ, their bias scores increase.In addition, prompts with a warning about social biases and chain-of-thought prompting reduce the effect of biases in model outputs, but there is room for improvement in extracting the correct evidence from contexts in Japanese. Our dataset is available at https://github.com/ynklab/JBBQ_data.

## introduction biases in large language models (llms) may lead to the reproduction of bias in downstream tasks such as language generation. as discussed by @xcite , nlp models contain various types of bias, among which we focus on social bias, namely, stereotyping behavior toward groups or individuals based on their social identity. for instance, stereotyping behavior observed in text generation can influence readers' perceptions of minority groups, thereby reinforcing societal stereotypes against these groups, and using such biased texts as training data introduces additional biases into the subsequent llms @xcite . various social bias benchmarks have been provided @xcite @xcite @xcite @xcite @xcite @xcite , but most are constructed in english, and benchmarks in other languages are not yet fully developed. in addition, although some llms have recently been developed specifically for japanese @xcite , it remains unclear the extent to which japanese llms exhibit biases against a range of social categories. to evaluate social biases and stereotypes in llms, question-answering (qa) tasks have been widely used. the bias benchmark for qa (bbq) was originally provided for english @xcite but has recently been made multilingual @xcite @xcite . these qa benchmarks provide contexts that target attested social biases against several different socially relevant categories. the categories of bias measurement are culturally relative (e.g., english bbq is rooted in us culture), but there are cultural differences in the ways that socioeconomic status and religion are perceived. this makes it difficult to apply all the categories used in bbq to other languages as they are. to transfer a bias benchmark from one language to another, it is necessary to adjust the context and add examples, in addition to translating the template. considering these points, we have created a japanese social bias dataset to evaluate social biases in japanese llms. to ensure both the efficiency and quality of the data creation, we used a semi-automatic method to create the japanese bias benchmark for qa (jbbq) based on english bbq. while bbq has nine categories in total, we selected the five involving stereotypes for adjustment to japanese contexts: age, disability status, 1 gender identity, physical appearance, and sexual orientation. in addition, we added examples particular to the japanese background for each category. for example, we added templates of stereotypes about x-gender, which is unique to japan, to the gender identity category (see section 3.2). another example is templates of stereotypes about the physical characteristics of people living in japan (e.g., low height) in the physical appearance category. using jbbq, we analyze the extent of social bias in japanese llms from a comprehensive perspective, namely, (i) the effects of the number of parameters and instruction tuning, (ii) the effects of prompts augmented with a warning about social bias, (iii) the effects of outputting the evidence contained in contexts leading to label predictions, and (iv) different qa task settings. our main contributions are as follows: ## related work various social bias benchmarks have been constructed in english. bbq @xcite is a qa dataset for assessing whether models can correctly understand the context of various social categories, and is widely used to evaluate social biases in llms. we describe the details of bbq in section 3. crows-pairs @xcite is a dataset for analyzing the social biases of masked language models with fill-in-the-blank questions about social categories. seegull @xcite is an english dataset consisting of tuples of identities (nationality and region) and attributes associated with those identities, and reflects regional differences in stereotypes by annotating stereotype scores for various regions. recently, these datasets have been provided for languages other than english, including chinese bbq (cbbq, huang and xiong 2024), korean bbq (kobbq, @xcite , basque bbq (basqbbq, zulaika and saralegi 2025), french crows-pairs @xcite , and multilingual bbq @xcite and seegull @xcite . our jbbq dataset will contribute to extending multilingual bbq. there is growing awareness of the safety and reliability of japanese llms, and there are several relevant datasets for japanese, such as those for harmful expressions @xcite , expressions of human rights violations @xcite , common sense morality @xcite , and hate speech dataset @xcite . however, these studies did not focus directly on analyzing social biases in japanese llms. most closely related to our study, anantaprayoon et al. (2024) used a natural language inference (nli) task to construct a dataset for gender biases in japanese, and they analyzed those in pre-trained models in japanese. instead, we selected qa tasks as appropriate downstream tasks for evaluating current generative language models. we created a japanese social bias benchmark for qa tasks based on the english bbq dataset in order to analyze biases for various social categories, such as age and physical appearance, not just gender. ## dataset creation jbbq was constructed semi-automatically in two steps: (i) machine translation of bbq templates and manual modification for japanese templates, and (ii) manual filtering and adding japanese templates. we begin by briefly introducing the original bbq dataset, then we describe our data creation method. ## conclusion in this study, we constructed the japanese social bias qa dataset jbbq and used it to analyze social biases in japanese llms from various perspectives. the experimental results showed that while instruction tuning helped the models to answer unknown labels for ambiguous questions, the model improvement on disambiguated questions was small. in addition, more parameters led to improved accuracy on qa tasks but also increased bias scores. regarding the results for different prompt settings, warnings about social biases and chain-of-thought prompting decreased the effect of social biases in the model outputs. however, the current japanese llms failed to extract correct evidence from contexts for some questions. comparing the bias detection and qa tasks showed that the models that performed well on the bias detection tasks also performed well on the qa tasks, but the bias detection tasks were more challenging than the qa tasks. in future, we will expand jbbq to realize a more detailed analysis of social biases in japanese llms. we believe that jbbq will be a useful benchmark testbed for assessing biases in japanese llms.