An Empirical Study on the Characteristics of Bias upon Context Length Variation for B angla (2024)

Abstract: Pretrained language models inherently exhibit various social biases, prompting a crucial examination of their social impact across various linguistic contexts due to their widespread usage. Previous studies have provided numerous methods for intrinsic bias measurements, predominantly focused on high-resource languages. In this work, we aim to extend these investigations to Bangla, a low-resource language. Specifically, in this study, we (1) create a dataset for intrinsic gender bias measurement in Bangla, (2) discuss necessary adaptations to apply existing bias measurement methods for Bangla, and (3) examine the impact of context length variation on bias measurement, a factor that has been overlooked in previous studies. Through our experiments, we demonstrate a clear dependency of bias metrics on context length, highlighting the need for nuanced considerations in Bangla bias analysis. We consider our work as a stepping stone for bias measurement in the Bangla Language and make all of our resources publicly available to support future research.

## introduction language models, encompassing both contextfree and contextualized variants, have increasingly demonstrated human-like biases (e.g., @xcite . with the introduction of newer concepts, such as more sophisticated language models, correspondingly nuanced strategies for bias detection have become necessary @xcite @xcite . as a consequence, sentence-level bias detection strategies have emerged. however, bias detection strategies primarily concentrate on english, with limited research in other languages. recent efforts target bias detection in dutch, arabic, and chinese languages @xcite @xcite . in the case of indic languages, @xcite conducted a comprehensive analysis of bias linked to binary gender associations in the hindi language. moreover, @xcite underscore the vital role of cultural awareness in examining bias measurement by conducting socially aware experiments on the hindi language. despite these valuable contributions, bangla, the sixth most spoken language in the world with over 230 million native speakers comprising 3% of the world's total population foot_0 , has received scant attention in bias analysis and remains an underrepresented language in the nlp literature due to a lack of quality datasets @xcite . this gap in research significantly limits our understanding of the bias characteristics present in existing language models under various linguistic contexts for this widely spoken language. addressing this limitation, this work endeavours to introduce bangla, a low-resource language into the realm of bias analysis, through a study focused on gender bias. we also posit the question: does the amount of contextual information provided to a model influence the application of bias measurement methods in contextual settings? to answer this query, in this study, we present (1) an empirical investigation comprising the creation of a dataset tailored for intrinsic gender bias measurement in bangla, (2) discussions on necessary adaptations to apply existing bias measurement methods for bangla, and (3) an examination of the impact of varying context lengths on bias measurement methodologies within a bangla-based framework. our findings reveal notable dependencies of bias metrics on context length, shedding light on nuanced considerations for bias analysis in language models. ## methodology our research focuses on bias measurement in contextual settings. we provide two intrinsic bias measurement methodologies for comparison. we choose these methods because they represent two distinct approaches to bias measurement: embedding extraction and mask prediction. ## data preparation we adopt the data preparation procedures based on the specific requirements of each experiment foot_2 . ## conclusion in this research, we aim to examine bias in bangla language models through creating a curated dataset and assert that the bias result outcome is influenced by the amount of context used in templates. further exploration can be conducted on other lowresource languages. in future, we plan to investigate the effects of bias in downstream applications of bangla language models, with the goal of developing language-specific debiasing methods to mitigate harmful bias in bangla embeddings and extend these efforts to generative models as well.