M ascu L ead : le premier tableau de bord de biais de genre (2025)

Abstract: Nous présentons MascuLead, le premier tableau de bord centré sur une tâche de détection de biais de genre pour les langues fleuries. Nous appliquons un cadre existant sur plusieurs Grands Modèles de Langue et comparons leurs performances. Nous soulignons également l’importance d’inclure des références de biais dans les tableaux de bord, et nous questionnons la notion même de tableaux de bord.

## on the urge of including bias benchmarks in leaderboards the field of natural language processing (nlp) is experiencing exponential growth, leading to exponential development of large language models (llms). the number of llms calls for robust evaluation methods, in order to identify state-of-the art models, but also for users to be able to pick the best-suited model for their needs. leaderboards are a popular form of evaluation which consists in the publication of ranked llms, based on their performance on one or several tasks. for example, the open llm leaderboard @xcite compares over 4,500 llms on 6 benchmarks, and has been liked by over 13,000 huggingface users 1 . a french version of this leaderboard is also available, covering about 30 llms on the same 6 translated benchmarks @xcite . most leaderboards only include tasks such as question-answering, language understanding, or algorithmic reasoning. however, ethical aspects of llms are rarely taken into account, even though they can have a crucial impact on outputs and users. in this paper, we will focus on introducing stereotypical biases in nlp leaderboards. numerous studies propose evaluation metrics for different types of stereotypical biases regarding gender @xcite , race @xcite , socio-economic status (cercas curry et al., 2024), etc. even if this subfield of research focuses on english and us-centric stereotypes, efforts are carried out to address more diverse languages and socio-cultural contexts @xcite . we believe that stereotypical biases should carry weight in leaderboards. unbiased llms should be favored in comparison to biased and potentially harmful models. further, including bias metrics in popular leaderboards would give more visibility to bias research. it could also "encourage researchers and engineers to pay more attention and direct more resources towards developing more exhaustive bias mitigation techniques and tackling more sources of biases" @xcite . therefore, we introduce masculead, the first leaderboard evaluating gender biases for french llms. 1 as of april 2025. we use and extend the framework presented in @xcite to build our leaderboard. we also develop an online demonstrator to perform gender detection and bias evaluation, as well as update leaderboards foot_0 . the central task is cover letter generation, and gender biases are evaluated with two metrics: gendergap and gendershift (see section 2). this framework presents advantages that make it a fit candidate to constitute a leaderboard task: it can be easily adapted to inflected languages and it does not rely on a corpus, which prevents overfitting foot_1 . we focus on french only and apply this framework on more llms than the original publication, generating 74,490 new cover letters with mistral-7b-v0.3, mistral-7b-instruct-v0.3, croissantllmbase, croissantllmchat, gemma-2-2b, gemma-2-2b-it, llama-3.2-3b, and llama-3.2-3b-instruct. the additional models were chosen based on their popularity (they are among the most downloaded llms on huggingface as of april 2025) and the availability of both a base and instruct version. the same two combinations of hyperparameters as the original article were selected foot_2 . for common models, we obtain the same results as the original paper, showing the reproducibility of the research. ## comparing llms with masculead the resulting leaderboards are presented in tables 1 to 3 . the highest ranked llms are the ones that performed the best with the framework, and are supposedly the least biased according to the gendergap (gg) and gendershift (gs) metrics. according to @xcite , gg is the difference in proportions of masculine vs. feminine generated texts. hence, the ideal gg is 0, i.e. there are as many masculine as feminine generated texts. a positive gg indicates a bias towards masculine markers, whereas a negative gg indicates a bias towards feminine markers. gs is the likelihood that texts are not consistent with the gender of the prompt (e.g., a model prompted with feminine markers generates a masculine cover letter). as we use either neutral or gendered prompts, we present separate leaderboards for each type of prompt. we still display the separate leaderboards ( all leaderboards highlight the important biases generated by the llama models (including vigogne, a llama model fine-tuned for french). mistral-7b-v0.3, xglm-2, and croissantbase remain in the top 5 across all leaderboards. we can thus establish that all instruct models appear more biased than their base counterparts. further, the most biased models seem to be the most popular, but also best performing models according to traditional benchmarks (see ## discussion -leaderboards flaws masculead allows for a first evaluation and comparison of llms in terms of gender biases in french. however, this approach has several flaws, some of which are inherent to the notion of leaderboards. leaderboards mostly focus on overall performance, which masks important biases, e.g. in relation to gender stereotypes. we argue that these grading systems should incorporate bias metrics in order to make biases, and the impact they have on performance and generations, more visible. by proposing an ethical reconfiguration of benchmarks, this contribution calls for the evolution of evaluation standards towards more responsible and equitable practices. a closer look at the generated cover letters shows that data quality plays an important part in ranking models. some models, such as mistral-7b-v0.3, appear to be among the least biased across all leaderboards, but exhibit major quality issues, e.g. irrelevant texts, gibberish foot_3 , or the production of only a few words in french before switching to english foot_4 . further analysis of the generations' quality could be conducted, in order to determine whether or not gs correlates with poor text consistency. if so, gs could be a relevant metrics to assess both gender bias and the general quality of generations. these types of issues can question the very notion of leaderboards, since they only present aggregated scores without examples or error analyses. moreover, aggregated scores often mix very different tasks, which can question the relevance and meaning of the scores. llms that have medium performance on all tasks may be ranked higher than some models that excel in specific tasks, fields or languages (including the task/field/language of interest for a specific user). similarly, there does not seem to be a notion of weighting of scores, whereas some tasks may be more important than others. as shown by @xcite and @xcite , the matter of general evaluation of llms and other nlp systems is at stake and should be further investigated. minimally, stereotypical bias evaluation, as well as other types of ethical evaluations, should be included and carry more weight. we hope that masculead constitutes a first step towards bias inclusion in leaderboards, and that it will be expanded with other bias tasks, bias types, and more linguistic and cultural contexts.