From Measurement to Mitigation: Exploring the Transferability of Debiasing Approaches to Gender Bias in M altese Language Models (2025)

Abstract: The advancement of Large Language Models (LLMs) has transformed Natural Language Processing (NLP), enabling performance across diverse tasks with little task-specific training. However, LLMs remain susceptible to social biases, particularly reflecting harmful stereotypes from training data, which can disproportionately affect marginalised communities.We measure gender bias in Maltese LMs, arguing that such bias is harmful as it reinforces societal stereotypes and fails to account for gender diversity, which is especially problematic in gendered, low-resource languages.While bias evaluation and mitigation efforts have progressed for English-centric models, research on low-resourced and morphologically rich languages remains limited. This research investigates the transferability of debiasing methods to Maltese language models, focusing on BERTu and mBERTu, BERT-based monolingual and multilingual models respectively. Bias measurement and mitigation techniques from English are adapted to Maltese, using benchmarks such as CrowS-Pairs and SEAT, alongside debiasing methods Counterfactual Data Augmentation, Dropout Regularization, Auto-Debias, and GuiDebias. We also contribute to future work in the study of gender bias in Maltese by creating evaluation datasets.Our findings highlight the challenges of applying existing bias mitigation methods to linguistically complex languages, underscoring the need for more inclusive approaches in the development of multilingual NLP.

## introduction large language models (llms) have revolutionised natural language processing (nlp), demonstrating remarkable capabilities across diverse tasks through few-shot and zero-shot learning, often without task-specific training @xcite @xcite . this shift from task-specific models to versatile foundational models has accelerated progress in nlp applications. however, these advances come with concerns, particularly regarding the propagation of social biases. llms are trained on massive, unfiltered internet datasets, which often encode societal stereotypes and inequities @xcite . these biases disproportionately affect marginalised communities, resulting in issues such as harmful sentiment, stereotyping, and underrepresentation @xcite . for instance, @xcite found that llms are 3-6 times more likely to associate occupations with stereotypical genders, amplifying biases beyond societal perceptions and factual data. most bias research has focused on english, benefiting from its high resources and relatively simple grammar. however, methods developed for english may not generalise to other languages, especially those with low resources and morphologically complex structures. maltese, an official eu language, exemplifies these challenges. it is a low-resource language with a semitic core and romance influences, written in latin script, and exhibits complex gendered grammar @xcite . current maltese-specific bert-based models, such as bertu (monolingual) and mbertu (multilingual mbert further pretrained on maltese) @xcite , fill a critical gap in language model availability for the language. however, bias evaluation and mitigation remain relatively unexplored. this research aims to address this gap by examining gender bias in maltese lms and experimenting to determine the extent to which english-centric bias techniques can be applied to this linguistically unique context. we focus on the following specific objectives: bias statement: this paper addresses binary gender bias in maltese language models, which creates representational harm by reinforcing limiting societal stereotypes and excludes gender diversity. this problem is especially acute for gendered and low-resource languages. left unaddressed, this bias risks creating unequal performance in downstream applications. our work is motivated by the conviction that this is a systemic flaw and that adapting debiasing methods is a critical step toward building equitable nlp technologies that counteract, rather than amplify, societal imbalances in under-resourced languages. ## related work the growing adoption of llms across nlp applications has heightened concerns about social biases embedded in these models. this section reviews key approaches to bias evaluation and mitigation, emphasising their applicability to morphologically rich and low-resource languages. the work by @xcite significantly influenced the discourse on mitigating bias and catalysed innovative research in the field, highlighting how gender bias in word embeddings can reflect and magnify societal prejudices. the approaches towards bias measurement and mitigation within language models have mostly focused on two principal approaches: pre-processing and in-training techniques @xcite . pre-processing techniques are designed to modify model inputs -whether through data adjustments, prompt engineering, or the application of bias-reducing algorithms -without changing the model's trainable parameters. these techniques aim to create a fairer input landscape for the models to operate within. conversely, in-training techniques target bias mitigation during the training phase, optimising the learning process itself to foster a more equitable representation of language from the outset. turning our attention to non-english models, languages with grammatical gender present challenges for evaluation metrics designed for english, as these metrics assume no inherent link between gender and professions. however, in gendered languages, such associations are often expected due to gender-specific noun forms. we highlight some works that have looked into bias in other languages. @xcite addressed this issue in dutch, a germanic language with grammatical gender, by analysing roberta, a dutch language model @xcite . they examined gender bias using template-based sentence probes and fairness metrics such as demographic parity ratio and equal opportunity. rather than treating gendered noun associations as bias, their study focused on whether the model exhibited a preference for male pronouns, which they considered a more relevant indicator of bias in a gendered language. chávez mulsa and spanakis (2020) analysed gender bias in dutch word embeddings using weat and seat. their findings confirmed the presence of gender bias in dutch word embeddings and showed that english-based bias measurement and mitigation techniques could be adapted for dutch with appropriate translations and careful language-specific adjustments. @xcite extended this research to english and german, analysing gender bias in profession-related words. they fine-tuned bert on the gap corpus using counterfactual data substitution to reduce bias. while their method was effective in english, it was less successful in german due to the language's complex morphology and gender distinctions. this emphasises the need for cross-linguistic studies on bias and mitigation strategies. in the same paper, they also introduce the bias evaluation corpus with professions (bec-pro), a template-based corpus designed to measure gender bias in both english and german. their findings highlight that bias detection methods effective in english may not directly transfer to other languages. in german, a gender-marking language, grammatical gender influences associations, with feminine forms being more marked than the default masculine forms. additionally, despite both english and german belonging to the same language family, linguistic similarities do not guarantee that bias detection methods will work equally well across languages. despite these advancements, it remains a reality that most existing research has predominantly focused on bias measurement and mitigation within english language models. this focus has exposed a significant gap in understanding how these methodologies can be effectively transferred and adapted to other languages. the linguistic diversity and unique grammatical structures of non-english languages may present distinct challenges and opportunities for bias mitigation, necessitating further research. it is essential to recognise, as noted by @xcite , that relying on a single metric fails to provide a comprehensive understanding of the biases present in a language model and their manifestations. moreover, this multiplicity of metrics introduces uncertainty regarding the most appropriate methods for measuring bias, complicating the evaluation process. ## methodology concentrating on bias measurement and mitigation for the maltese language, the publicly available pretrained maltese lms, bertu and mbertu @xcite , were leveraged to deepen our understanding of the possibility of transferability of these methods within the unique linguistic context of maltese. all code and datasets used in this work are publicly available. 1 ## results we systematically examine the results from the performance metrics, compare them across different models and datasets, and explore the implications of these findings. ## visual evaluation inspired by @xcite , we use t-sne plots to visualise the latent semantic space of gender-triggering adjectives in maltese. this projection of high-dimensional embeddings helps identify gender bias by analysing how gendered terms cluster. given that counterfactual data augmentation (cda) yielded the best debiasing results, we present visualisations for bertu and mbertu before and after applying cda. this was done using three gender word-pairs: "tabib-tabiba" (doctor), "avukat-avukata" (lawyer) and "ghalliemghalliema" (teacher). the t-sne plots for bertu and mbertu using tabib-tabiba can be found in figures 1 and 2 . the remaining figures are included in appendix a. the t-sne visualisations for gendered word pairs in bertu and mbertu reveal persistent gender bias in the monolingual model, while the multilingual model exhibits more balanced representations. for tabib-tabiba (doctor) and avukat-avukata (lawyer), baseline bertu shows clear gendered associations, with tabiba and avukata (female forms) closely linked to inkompetenti (incompetent), while tabib and avukat (male forms) are associated with kompetenti (competent). additionally, positive and professional adjectives tend to cluster around male terms, reinforcing societal stereotypes. in contrast, baseline mbertu displays a more diverse distribution, suggesting that multilingual exposure mitigates some of these biases. after applying cda, bertu still exhibits incomplete debiasing, as tabib and tabiba remain significantly distant in embedding space, and professional adjectives continue to favour male forms. similarly, avukat retains closer ties to positive adjectives than avukata, indicating that bias is reduced but not eliminated. meanwhile, mbertu achieves a more neutral distribution post-debiasing, with key adjectives like kompetenti and professjonali positioned equidistantly between male and female forms, indicating more effective bias mitigation. for ghalliem-ghalliema (teacher), baseline bertu reflects a different stereotype: positive adjectives such as professjonali (professional) and intelliġenti (intelligent) are more closely linked to ghalliema (female teacher), while negative terms like ikrah (ugly) and kattiv (cruel) are associated with ghalliem (male teacher). this mirrors societal norms that favour women in educational roles while casting men in a harsher light. after cda, bertu shows improved gender balance, with ghalliem and ghalliema appearing closer together and adjectives more evenly distributed. baseline mbertu already presents a more neu-tral representation of ghalliem and ghalliema, with positive and negative adjectives distributed more equitably. post-debiasing, the visualisation remains essentially unchanged, suggesting that mbertu was less biased to begin with. ## final observations and conclusions our analysis revealed that both bertu and mbertu exhibit measurable gender bias, with bertu showing a higher degree of bias. this aligns with findings in english models, where monolingual bert displayed more bias than multilingual mbert, likely due to the latter's exposure to diverse linguistic contexts. the bias primarily favoured male-associated terms, particularly in occupational stereotypes, though negative connotations for male terms were also observed, highlighting the complexity of bias patterns. among the debiasing techniques tested, cda was the most effective, significantly reducing bias in both crows and seat scores. however, it occasionally introduced grammatical errors in maltese, and the full impact of this technique on the model was difficult to determine without access to appropriate resources. dropout regularization had a limited impact, slightly reducing bias in crows but increasing implicit bias in bertu, while showing improvement for mbertu. guidebias did not generalise well to maltese, increasing bias in both models. auto-debias was effective for monolingual models but increased bias in multilingual ones, suggesting its effectiveness depends on the model architecture. these results highlight the need for multiple evaluation metrics, as different techniques produced conflicting results across crows and seat. a more nuanced approach is required to fully understand and mitigate bias in language models. in summary; this research highlights the need for further investigation into bias in multilingual language models, particularly in low-resource languages with complex gender systems, such as maltese. to aid future work in the area, we publicly share all our experimental and evaluation data, including the maltese debiasing dataset. future work could significantly expand upon these findings by offering more targeted recommendations, such as identifying which debiasing techniques are better suited for specific tasks (classification versus generation). it would also be beneficial to further analyse what common language features would suit specific debiasing approaches, as well as how debiasing affects llm performance on nlp downstream tasks such as named-entity recognition and sentiment analysis. while existing debiasing techniques have demonstrated varying levels of effectiveness, our findings underscore the need to refine these methods to better address linguistic and cultural nuances. future work should focus on developing more robust, language-agnostic debiasing strategies and comprehensive evaluation metrics that can accurately capture different forms of bias across diverse languages. additionally, bias research must extend beyond gender and racial biases to include other critical aspects such as age, socioeconomic status, regional dialects, and disability, which remain largely underexplored. understanding and mitigating these biases is essential for ensuring fairness in ai systems that serve diverse communities. our findings contribute to the growing body of research on bias in low-resource languages, emphasising the necessity of adapting mitigation strategies beyond english-centric approaches. as language models continue to shape digital interactions and decision-making systems, it is crucial to prioritise equitable and inclusive ai development. through continued research and refinement, we can move closer to creating language technologies that are fair, representative, and culturally aware.