Toward Cultural Bias Evaluation Datasets: The Case of B engali Gender, Religious, and National Identity (2023)

Abstract: Critical studies found NLP systems to bias based on gender and racial identities. However, few studies focused on identities defined by cultural factors like religion and nationality. Compared to English, such research efforts are even further limited in major languages like Bengali due to the unavailability of labeled datasets. This paper describes a process for developing a bias evaluation dataset highlighting cultural influences on identity. We also provide a Bengali dataset as an artifact outcome that can contribute to future critical research.

## introduction bias, in the context of computing systems, is where sociotechnical systems systematically and unfairly discriminate against certain individuals or social groups in favor of others @xcite . people often identify through their perceived memberships in certain groups @xcite . while computational linguists have studied gender and racial biases @xcite , systematic discrimination of language technologies based on various cultural factors like religion and nationality has received little attention. moreover, critical studies examining these biases mostly focused on nlp systems in a handful of languages, whereas many languages with sizeable numbers of speakers do not have enough resources like datasets to pursue similar studies. according to @xcite , whereas 0.28% of global languages (e.g., english, spanish, japanese) reap benefits from nlp breakthroughs, 88.38% of languages have virtually no data to use. they also found that while english and bengali are the third and sixth largest languages by the number of native speakers @xcite , the former has hundreds of times more resources than the latter in linguistic data consortium, language resources and evaluation, and wikipedia, and thou-sands more resources in the web overall. the difference in available resources like labeled datasets impedes the progress of critical studies aimed at fairness, transparency, and identifying biases in such under-represented languages. in the absence of native resources, many of these tools first translate non-english text to english for downstream nlp tasks, creating the potential for colonial imposition on indigenous languages @xcite . one of the main objectives of this work is to highlight and address the lack of focus on two vital cultural factors such as religion and nationality, that shape people's cultural identity. in addition to its large number of native speakers and a thriving cultural community online, the religious diversity of this ethnolinguistic group, with 71% muslims and 28% hindus, and their postcolonial division into two nationalities, bangladeshi (59%) and indian (38%) makes the bengali language an interesting case for developing a cultural bias evaluation dataset @xcite . the contributions of this work are, first and foremost, in outlining a process for developing datasets to evaluate cultural (e.g., religious, national) biases in nlp systems. moreover, as an example, we provide a bengali identity-bias evaluation dataset (bibed) that can support exploring how cultural bias can both emerge through the nlp process and how we can work toward identifying and eliminating bias. in the next section, we will review the relevant literature. then, we will briefly overview the framework we used to organize the dataset. after that, we will explain the process of dataset development and its organization. ## related work in this section, we will discuss how culture shapes people's identities across various dimensions and mediates their interaction through and with technologies and prior works studying bias in language technologies toward or against different identities. in this work, we draw on the definition of identity, which views it as a social construct shaped by people's perceived membership in different groups @xcite . in this view, individuals' identities are often defined across various dimensions, such as race, gender, sexual orientation, nationality, religion, caste, occupation, etc. @xcite . under each dimension, people can identify with different categories, such as identifying as female, male, or non-binary in relation to gender. people express these identities based on broader social, and cultural logics @xcite institutionalized within religious and national communities @xcite . people from different cultural contexts communicate through different speech acts and non-verbal actions. through their embeddedness in sociohistoric contexts, speakers of the same language can demonstrate various dialects, i.e., geo-cultural variations (e.g., german language in austria and germany) @xcite or sociolects, i.e., dialects of particular social classes @xcite . long-standing linguistic norms and sociocultural identities are deeply intertwined. as people often speak a particular dialect or sociolect based on their geo-cultural or socio-historic backgrounds, these dialects can be ways to infer and serve as proxies for, their cultural identities. for example, when situated in the context of the two main dialects of bengali, ghoti is spoken in west bengal (in india), whereas the bangal dialect is spoken in east bengal (bangladesh). these regions were partitioned by the british colonizers based on their socioeconomic structure and religion-based demography (see @xcite for reviewing how colonial history shaped the societies in bengal). hence, bangal and ghoti dialects are often used as proxies for indian and bangladeshi identities and associated with muslim and dalit hindu agrarian identities and uppercaste hindu elite identities, respectively @xcite . when different identities come together, such as race, gender, nationality, and religion-what is known as intersectionality @xcite -this can create differential power and bias in how people might experience sociotechnical systems. norms around different intersectional identities guide how algorithms on these systems perceive individuals' digital identities and influence the creation of datasets that are often used to make decisions @xcite @xcite . many state-of-the-art computing platforms (e.g., recommendation systems) heavily rely on creating digital identities to model their users and their preferences (cheney-lippold, 2017) that often fail to account for cultural contexts @xcite . postcolonial computing scholars who study cultural imposition and the role of cultural contexts in designing and deploying technology @xcite have critiqued the commitment to reductionist representations for complex human identities and relationships @xcite . with over-simplification, using non-inclusive datasets and stereotypical categories as the ontological basis to construct computational identities without considering cultural differences, technology can exhibit algorithmic coloniality @xcite , exclusion (simpson and semaan, 2021), impose hegemonic classification, and cause cultural erasure @xcite . for example, @xcite found content moderation on quora to estimate bengali users' national and religious identities based on their linguistic performances and prioritize indian hindu dialects while marginalizing bangladeshi muslim dialects. this example highlights how coloniality-those systems of power where foreign entities worked to revise the social structures of other populations and social groups-is now being mediated by and through sociotechnical systems, such as nlp. decolonial scholars who study ways to resist technology-mediated cultural imposition (ali, 2016; bird, 2020) emphasized the necessity of diverse representations and including local and indigenous voices in developing technology. in the context of computational linguistics, "diverse perspectives" can mean both studies focusing on different languages and those about variations of the same language @xcite . as discussed earlier, myriad sociocultural factors can cause and impact the variations of a language (e.g., dialects), which is less explored in the current body of literature @xcite . with the most investigative attention going to a minority of languages, language technologies in most languages lack nuances for cross-cultural contexts. for example, the body of bengali nlp research is quite small compared to its large number of speakers, especially little of which addresses the language's sub-cultural variations in different religious and national communities, creating a risk of reinforcing societal biases based on identities through those research. given the numerous ways biases can get embedded in computing systems, critical researchers across various fields have examined computing systems resulting in increased interest in social justice, fairness, accountability, transparency, algorithmic audits, and critical data studies @xcite @xcite . along that line, computational linguists have studied bias in language technologies from various perspectives @xcite . in these works, while gender bias received substantial attention @xcite , they have also examined biases based on different identity dimensions such as race @xcite , age @xcite , disability @xcite , occupation @xcite , caste @xcite , and political affiliations @xcite for various computational linguistic tasks like sentiment analysis @xcite , machine translation @xcite , and language generation @xcite . however, two major cultural identity dimensions such as religion and nationality, have not received much attention @xcite @xcite . the prevalence of religion and nationality as two intersecting dimensions in how people both see themselves and engage in the everyday performance of self through speech and other actions is more visible and complex in diverse contexts of the indic languages @xcite . therefore, it is critical to explore the ways in which nlp and other systems can perpetuate bias through these dimensions. while doing so, it is important to culturally contextualize nlp metrics and models. instead of plainly translating english models into bengali, hindi, etc., we need to carefully consider the dimensions of fairness and types and sources of bias specific to that cultural context @xcite . to address this gap, this paper proposes a methodology for developing culturally centered bias-evaluation datasets in nlp. within the complex ecosystem of language technologies, to identify the sources of bias and un-derstand how societal prejudices get translated into technology to affect downstream tasks, researchers have focused on word embedding @xcite , pre-trained language models @xcite , and training datasets @xcite . methodologically, researchers have used both qualitative and quantitative approaches to study the biases of similar systems @xcite @xcite . for quantitative critical algorithmic studies, nlp researchers have compiled datasets for detecting and evaluating various kinds of bias @xcite . similar to other fields in nlp, a dearth of resources exists for such bias evaluation studies in bengali. in this paper, to describe a social scientific process for creating datasets to evaluate religion and nationalityinduced cultural biases, we use the example of religion and nationality-wise diverse bengali identity. the developed dataset, bibed, remains conscious of both explicit and implicit expressions of bengali identities in terms of gender, religion, and nationality. ## resource description framework to improve support for reusing scholarly data, @xcite motivated good data management through fair (findable, accessible, interoperable, and reusable) principles. to follow these guidelines, we will organize our dataset using the resource description framework (rdf). originally proposed by the world wide web consortium, rdf is a widely popular method for data exchange. in this section, we will briefly overview this framework. rdf is a flexible, simple yet structured, and decentralized standard for representing relationships between data @xcite . using this framework, we can make statements about resources (e.g., documents, data objects). an rdf statement, often called a triple, consists of three components. these are (a) subject-the resource or entity being described, (b) predicatethe relationship or attribute, and (c) object-the value related to the subject @xcite . for example, an rdf triple about a person named karim's ability to speak in bengali can be written as: @xmath0. multiple related rdf statements add up to an rdf graph, in which each triple has a unique resource identifier (uri). the use of uris and uniform triple formats sup-port easier aggregation of datasets from different sources compared to tabular data formats. rdf data can be stored in various formats, popular ones being json, xml, and turtle 1 . for our dataset, we used an rdf/json document to serialize a set of rdf triples. this consists of a single json object called the root object, where the keys in the root object correspond to the subjects of the triples (w3c, 2013). a triple is structured as follows: ## dataset creation to describe the process of developing a culturally centered bias evaluation dataset, we focus on three dimensions of identity: gender, religion, and nationality. for each dimension, we included binary categories in the context of bengali identity, as shown in ## organizing dataset with rdf for a dataset like ours compiled from templates, lists reflecting pre-defined identity dimensions and categories, and linked data sources, describing the organization of the dataset is more useful. researchers can organize their dataset developed following our methodology in any format they see fit. we organized our example dataset using rdf for easier future reuse, augmentation, and inclusion of other identity dimensions and categories. in bibed foot_0 , there are more than 121 thousand sentences that explicitly or implicitly represent bengali identity based on gender (female-male), religion (hindu-muslim), or nationality (bangladeshi-indian). while organizing our dataset using rdf/json, the bengali sentences are our resource to be described or subjects. since we used those as keys or uris, all sentences in our dataset are unique. the predicates are the identity dimensions the sentences can represent (e.g., gender). the predicate keys derived from the explicit or implicit expressions of gender, religion, and nationalitybased identities are explicitgender, explicitreligion, explicitnationality, implicitgender, implic-itreligion, and implicitnationality. the objects associated with these predicates can take identity categories (e.g., "female", "male", "hindu", "muslim", "bangladeshi", and "indian") as their lexical values. again, for ebe and colloquial vocabulary-based ibe phases where we generated synthetic sentences in pairs or translated using pairs of colloquial vocabularies for an existing sentence from @xcite ) dataset, we included a predicate key pairresource that will contain a uri, that means a unique sentence as its corresponding object. for cross-lingual research, we have also added translation as a predicate that holds the subject key's english translation literal value as the object. the translations were done through a combination of manual effort (in the case of noun phrases-based ibe) and identifying corresponding english translations from @xcite (in the cases of ebe and colloquial vocabulary-based ibe). figure 1 shows an entry from bibed. here, the bengali sentence "৩৬ শতাংেশর েব-িশ নারী এই ভাবনার সােথ একমত।" (from the first row in ## dataset content dataset papers in nlp traditionally describe their corpus using approaches like topic modeling, word frequency, and some kind of baseline classification @xcite . as we plan to use the dataset developed in this paper to critically audit algorithms and tools for downstream nlp tasks in our other work-inprogress (see next section), in this section, we will give a brief descriptive overview of our developed dataset, bibed. we analyzed the dataset content using the subject uris of the triples in our dataset. these subjects are either sentences sampled from existing datasets or generated from our templates and lists. since the pairresource values were synthetically generated, we did not use those in the descriptive analysis. first, we removed stopwords from the sentences using the list by stopwords iso foot_1 . after removing punctuation and numeric literals from the sentences, we tokenized the sentences and stemmed the tokens using the bltk foot_2 and banglastemmer foot_3 packages. on average, the sentences have 18.78 words (median 15 words) and are 147.13 characters (median 114 characters) long. there are 108608 unique words (excluding stopwords and after stemming). most frequent (top 15) words in our dataset are: "ভারতীয়" (indian), "সাল" (year), "হেয়" (being), "একজন" (a person), "নারী" (woman), "মিহলা" (woman), "মু স লম" (muslim), "সােথ" (with), "িহেসব" (consider/calculation), "পািন" (water), "িহ ু " (hindu), "পু রুষ" (man), "বাংলােদ-শী" (bangladeshi), "সময়" (time), and "জাতীয়" (national). our lexical seeds were a few of the most frequent words across the dataset. other frequent words may come from sources used in building the datasets @xcite , from which we sampled sentences. ## downstream applications and future work we intend the methodology to inspire the development of bias evaluation datasets in other cultural contexts. bibed, the dataset developed through the process in this paper, can promote fairness and bias research in bengali. some examples of nlp applications where such exploration can occur are sentiment analysis, machine translation, mask prediction, etc. this paper is an early outcome of a large project investigating the continuation of colonial marginalization of under-represented bengali identities through technology. our prior research highlighted how human content moderators could marginalize users based on religion and nationality @xcite . to understand whether automated content moderation would minimize, reinforce, or exacerbate such human biases in platform governance, in our work-in-progress, we are critically auditing bengali nlp tools, algorithms, and datasets to evaluate their biases from a decolonial perspective. for example, we examine whether and how nlp-based automated moderation promotes colonially shaped conflicts among various national and religious identities. currently, we focus on downstream nlp tasks like sentiment analysis, hate speech detection, and machine translation, which have traditionally been vital components of automated content moderation @xcite @xcite . in addition to continuing our work on evaluating bias in bengali nlp systems that can contribute to automated content moderation, we will continue to augment the bibed dataset. in this paper, while developing the dataset, we used lexical seeds based on scholarly articles, public data sources, and our lived experience as native bengali speakers. prior research has highlighted that selecting these lexical seeds or keywords can implicitly introduce researchers' biases in an artifact @xcite . therefore, to minimize the possibility of such biases, we will take a participatory approach to create the list of seeds which will, in turn, democratize the data collection process. ## conclusion this paper describes a process for developing bias evaluation datasets highlighting cultural factors like religion and nationality. our approach, while following traditional nlp strategies, is also deeply informed by socio-cultural literature, motivating interdisciplinary research. in doing so, we also created a sample artifact, i.e., a bengali biasevaluation dataset. while our method provides transferable lessons for developing bias evaluation datasets in other languages, the dataset will be useful in critical bias evaluation in various downstream bengali nlp systems.