Assessing Political Inclination of B angla Language Models (2023)

Abstract: Natural language processing has advanced with AI-driven language models (LMs), that are applied widely from text generation to question answering. These models are pre-trained on a wide spectrum of data sources, enhancing accuracy and responsiveness. However, this process inadvertently entails the absorption of a diverse spectrum of viewpoints inherent within the training data. Exploring political leaning within LMs due to such viewpoints remains a less-explored domain. In the context of a low-resource language like Bangla, this area of research is nearly non-existent. To bridge this gap, we comprehensively analyze biases present in Bangla language models, specifically focusing on social and economic dimensions. Our findings reveal the inclinations of various LMs, which will provide insights into ethical considerations and limitations associated with deploying Bangla LMs.

## introduction the field of natural language processing (nlp) has experienced a transformative paradigm shift driven by the advent of pre-trained large-scale language models (lms) @xcite . these models have unleashed novel opportunities in specific areas such as text generation @xcite , question answering @xcite , sentiment analysis @xcite , machine translation @xcite , document summarization @xcite , and a myriad of other linguistic tasks. language models gain these capabilities from training on a vast corpus, enabling them to understand syntactic, language conventions, and nuances with remarkable accuracy @xcite . however, this capability does not come without its complexities. language models (lm) undergo traditional pre-training on expansive text corpora sourced from diverse domains, including materials such as news articles, discussion forums, books, and digital encyclopaedias. these sources often encompass a range of political inclinations, social biases, stereotypical beliefs, and ideas that tend toward extremes @xcite . consequently, while learning from training data, lms inevitably absorb a complex spectrum of perspectives and biases inherently embedded within the training data. the implications of these biases are extensive, profound, and have far-reaching implications @xcite . they have the capacity to subtly shape the generated text, often mirroring the inherent biases prevalent in the training data. in today's interconnected world, ai-generated content is integral to human communication, spanning domains such as news article composition and virtual assistant responses. the need to rigorously examine and mitigate these embedded biases extends beyond scientific exploration; it represents a vital ethical responsibility. one specific dimension of bias that requires a thorough examination is political bias @xcite . politics is a fundamental aspect of human society, exerting significant influence in various domains @xcite . the potential for language models to impact political discourse, whether by their use in the summarization of news articles, engagement in political dialogues, or the generation of political content, underscores the importance of examining political biases within these models. in this paper, we explore political inclination and bias in a low-resource language like bangla (mainly spoken in bangladesh), which is almost non-existent. despite the growing importance of bangla as the sixth most spoken language in the world @xcite and its significance in contemporary digital communication, bias analysis within this domain remains relatively unexplored. within this context of limited linguistic resources, our research aims to explore and analyze political leaning and biases present in bangla language models, contributing to the understanding of this underexplored area. we assess the political inclination of bangla language models, particularly focussing on social and economic dimensions. we also discuss the implications of using biased models and the need for mitigation strategies. ## related works bias identification and mitigation have been subjects of significant research interest @xcite . various forms of bias in language models have been extensively studied, from stereotypical to social and political biases @xcite . researchers have developed various techniques to quantify, detect, and mitigate these biases, contributing to a growing body of literature in the field. @xcite examined societal biases within pre-trained language models, investigating six sensitive attributes, including race, gender, religion, appearance, age, and socioeconomic status. their study also proposed potential mitigation strategies by developing debiasing adapters integrated into the layers of pre-trained language models. similarly, gender bias within lms has garnered significant research attention. recent studies have convincingly demonstrated the inherent gender bias present in these models @xcite . researchers have proposed various metrics to quantify and measure this bias @xcite . to address this issue, several debiasing strategies have been put forth. @xcite suggested a debiasing approach that modifies the loss function by incorporating terms aimed at equalizing probabilities associated with male and female words in the model's output. @xcite applied the theory of causal mediation analysis to develop a method for interpreting the components of a model that contribute to its bias. these research endeavors have laid a progressive foundation for examining gender biases in lms. furthermore, researchers have investigated various aspects of bias within lms @xcite . @xcite conducted research on generative models, particularly gpt-2 @xcite , and uncovered occupational biases. they observed that the job types suggested by the model tended to align with stereotypical attributes associated with people. similarly, venkit et al. (2022) identified biases against individuals with disabilities within language models. these explorations span a wide range of areas, encompassing the study of stereotypical bias @xcite , demographic bias @xcite , bias against lgbtq+ communities @xcite , and more. collectively, these research efforts provide valuable insights and directions to examine various aspects of bias within language models. while these studies illuminate diverse dimensions of bias, the field of political orientation and inclination within lms, especially in languages like bangla, remains relatively uncharted. @xcite conducted extensive experiments on english-language models to study their political inclination and identify potential sources of bias. however, further investigation of political biases within language models is imperative. this need is particularly pronounced in languages such as bangla, where such analyses are virtually nonexistent. recognizing this important and aligning with the united nations' leave no one behind (lnob) principle, our study focuses on uncovering biases in bangla language models. ## methodology we employed a two-step methodology to gauge the political inclinations present in language models. while recent studies have predominantly centered around assessing inclinations based on how language models treat specific individuals (aher et al., @xcite , our approach aligns with the methodology proposed by @xcite , guided by principles from political spectrum theories. we evaluate political positions along two axes: social values, which range from liberal to conservative, and economic values, which range from left to right. this approach provides a more comprehensive perspective, going beyond a simple left versus right distinction. hence, we examine the orientations of language models using the widely accepted political com-pass test foot_0 , rooted in these theories. this test assesses an individual's political stance in a twodimensional space by analyzing their responses to 62 political statements. participants express their level of agreement or disagreement with each statement, and their answers are then used to compute their social and economic scores via a weighted summation. to be precise, the political compass test translates a series of responses denoting agreement levels {strong disagree, disagree, agree, strong agree} into a two-dimensional point (s soc , s eco ). here, the social score (s soc ) and economic score (s eco ) fall within the range of @xcite . we employ this test as a tool by translating statements in the political compass to bangla (appendix a) for evaluating the political leaning of pre-trained lms in bangla. ## results based on our observations in figure 1 , it is evident that bangla language models exhibit political inclinations along various political and social axes. notably, the pre-trained fill-mask language model, multilingual bert, showed a more authoritarian leaning with a social score (s soc ) of 4.15. this inclination can be plausibly attributed to the nature of the training data used by multilingual bert. existing literature suggests that models trained on older text data tend to demonstrate right-wing or conservative tendencies. conversely, models trained on contemporary web content tend to exhibit fewer right-leaning tendencies, primarily because modern web pages often contain more liberal content. in contrast, our findings reveal that banglabert adopted a relatively neutral stance on social issues. this neutrality can be attributed to banglabert's training data, which includes the wikipedia dump dataset and datasets from webpages. wikipedia articles typically maintain a neutral stance, and the corpus sourced from webpages tends to contain fewer right-wing discussions. this aligns with our presumption banglabert multilingual bert gpt-2 (bangla) gpt-3 (ada) gpt-3 (text-davinci-003) economic left/right: 0.88 social libertarian/authoritarian: -0.05 economic left/right: 0.38 social libertarian/authoritarian: 4.15 economic left/right: 0.75 social libertarian/authoritarian: 1.64 economic left/right: 0.5 social libertarian/authoritarian: -1.9 economic left/right: 1.63 social libertarian/authoritarian: 2.72 that web content, in general, features less rightleaning discourse compared to the data used by multilingual bert. these findings underscore the significant influence of training data on language model political leaning, emphasizing the importance of understanding and mitigating biases within language models. similarly, the text generation models developed by openai exhibit significantly less authoritarian leaning compared to the multilingual berts. specifically, gpt-3 (ada) and gpt-3 (text-davinci-003) display considerably lower levels of authoritarianism when compared to multilingual bert. this contrast can be largely attributed to ope-nai's approach, which involves human-in-theloop and reinforcement learning feedback mechanisms. these mechanisms are designed to reduce right-leaning tendencies and prevent extreme biases in the generated content. similarly, gpt-2 (bangla) displays more libertarian leaning, likely stemming from its training on mostly web crawl corpus data. it's worth highlighting that the average magnitude of opinions on social issues (s soc ) is 2.07, whereas for economic issues (s eco ), it's 0.83. this observation underscores that language models tend to express stronger opinions on so-cial issues compared to economic ones. this discrepancy can probably be attributed to the training data's emphasis on social topics, as the data primarily originates from social media sources where economic discussions are relatively less prevalent. for a more comprehensive analysis, further research is imperative. future investigations could involve subjecting these models to various data types to discern whether the observed biases are inherent to the model's architecture or primarily influenced by the training data. such inquiries would provide valuable insights into the root causes of bias in language models and contribute to ongoing efforts to address and mitigate these biases effectively. moreover, it is essential to acknowledge that deploying politically inclined language models carries potential harm, especially in contexts like news article summarization, political discussions, or content generation. ## conclusion in this paper, we investigated political biases within bangla lms, uncovering diverse inclinations across social and economic dimensions influenced by their training data sources and methods. multilingual bert exhibited authoritarian tenden-cies attributed to older data, while banglabert maintained a relatively neutral stance owing to its predominantly neutral training data. additionally, gpt-3 models displayed reduced authoritarianism, reflecting openai's mitigation efforts. gpt-2 (bangla) showcased more libertarian inclinations, likely due to its training on web crawl corpus data. our research highlights the significance of comprehending and mitigating biases in bangla lms and contributes to the ongoing discourse on fairness and ethical ai deployment.