You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022)

Abstract: Evaluating bias, fairness, and social impact in monolingual language models is a difficult task. This challenge is further compounded when language modeling occurs in a multilingual context. Considering the implication of evaluation biases for large multilingual language models, we situate the discussion of bias evaluation within a wider context of social scientific research with computational work. We highlight three dimensions of developing multilingual bias evaluation frameworks: (1) increasing transparency through documentation, (2) expanding targets of bias beyond gender, and (3) addressing cultural differences that exist between languages. We further discuss the power dynamics and consequences of training large language models and recommend that researchers remain cognizant of the ramifications of developing such technologies.

## introduction machine learning (ml) systems, especially large language models (llms), are prone to (re)produce harmful outcomes and social biases @xcite @xcite . despite recent advances in llms @xcite , they have shown to disproportionately produce harmful content when addressing certain topics @xcite and demographics @xcite @xcite -in part due to the training data used @xcite @xcite , and the design of modeling processes @xcite . in response, previous work has explored ways in which such social biases can be measured and counteracted @xcite @xcite . typically, these issues have been addressed either by conceptualizing the underlying systemic discrimination as "bias" or by developing evaluation datasets that shed light on how llms produce harmful social outcomes. however, in the former case, as @xcite points out, these conceptualizations often lack clear descriptions, e.g., type of systemic discrimination and affected demographics. this results in a highly under-specified "bias", which could lead to a downstream issue in the validity of the technical approaches that are developed @xcite . similarly, the ill-defined "bias" is further compounded by the specifics of many benchmarks. often, benchmarks exhibit discrepancies between understandings of the unobservable theoretical constructs against which "bias" is being measured and their operationalization @xcite . furthermore, many prior benchmark datasets were developed with specific modeling architectures in mind @xcite . they are limited to english and are culturally anglo-centric. 1 in this position paper, we present an overview of the current state-of-the-art concerning challenges and measures taken to address bias in language models. specifically, we document the challenges of evaluating language models, with a focus on the generation of harmful text. by engaging our challenges with the relevant social scientific literature, we propose (1) a more transparent evaluation of bias via scoping and documentation, (2) focusing on the diversity of stereotypes for increased inclusivity, (3) careful curation of culturally aware datasets, and (4) creation of general bias measures that are independent of model architecture but capture the context of the task. we recognize that many of the challenges that we have encountered and described here are large open problems that will require joint work to address. our goal is to analyze these challenges and provide scaffolding for future work. ## challenges of bias evaluating the social impacts and harmful biases llms exhibit is an important development step. however, despite the increased interest in developing bias benchmarks, the field still faces various challenges in evaluating llms with off-the-shelf benchmarks. in this section, we provide examples of existing bias measures currently used in nlp. we then discuss the challenges that originate from these: (1) they rely on vague definitions of bias, (2) are restricted to particular model architectures, (3) have limited relevance for different cultural contexts, and (4) are difficult to validate and interpret. ## the elephant in the room: power, privilege, and point of view throughout the paper, we have primarily discussed bias in language models as a mechanical phenomenon. however, it is important to situate these discussions within the context and power dynamics of the way that nlp is practiced -both in research and in application @xcite . in this section, we discuss sociopolitical influences on ai ethics and bias research in nlp. we argue that contemporary developments of llms have been an exercise in financial, institutional, ecological, linguistic, and cultural privilege. they are the consequence of the political will to create totalizing technologies and evaluation of bias, fairness and social impact should be viewed as a countervailing power mechanism, although in some cases serve to obscure these. ## addressing bias researchers have developed various strategies to address bias in large language models. as discussed in earlier sections, however, these strategies are insufficient to tackle multiple dimensions of bias. below, we enumerate a few ways in which bias can be addressed by the research community to effectively engage with our aforementioned concerns: (1) moving towards a more transparent way of evaluating bias, (2) focusing on the diversity of stereotypes and increasing inclusivity, and (3) considering the impact of linguistic and cultural differences on the identification and mitigation of bias in designing culturally comparable datasets. we would like to highlight that these suggestions are not exhaustive. they will, however, guide the work in this area. ## conclusion recent improvements in llms to mimic human text have led to a surge in research that seeks to identify and address the harms arising from their training and deployment. however, the considerations on social harms that arise has been limited to narrow, anglo-centric, contradictory, and often underspecified definitions of fairness and bias. furthermore, the development of contemporary methods has conflated task-specific and architecture-specific designations. compounded with the structural inequalities around resources, language, and identity, this has yielded an overreliance on prestige forms of english for developing llms and interrogating and addressing the social biases that they harbor. situating these methods within such englishes has had the consequence of over-emphasizing western-centric social categories. moreover, datasets for evaluating social biases in llms have traditionally failed to denote and specify the context within which biases are situated. such concerns have been the cause for questions around the validity of the developed measures, and in particular for multilingual llms. to address such challenges, we propose that developing methods for multilingual llms requires researchers to provide thorough documentation of their approaches, including documenting the scope, demographics of speakers, and potential annotators. additionally, we also recommend that researchers situate their bias evaluation methods within the specific context of the languages that the model operates on. in doing so, bias evaluation methods can be made to specifically address biases under the conditions and contexts that they occur in each of the model's languages. furthermore, we recommend that researchers examine diversity issues beyond gender bias, with a particular focus on intersectional issues @xcite . finally, we recommend that researchers are cognizant of the social and environmental harms that developing llms have. for instance, developing ever-larger language models that achieve marginal improvements for english may bring a smaller benefit than developing a llm for other languages. thus, in a consideration of developing a new language model, we implore researchers to consider ways in which harms can be limited, or the benefits can come to compensate for their costs.