Assessing Gender Bias in W ikipedia: Inequalities in Article Titles (2021)

Abstract: Potential gender biases existing in Wikipedia’s content can contribute to biased behaviors in a variety of downstream NLP systems. Yet, efforts in understanding what inequalities in portraying women and men occur in Wikipedia focused so far only on *biographies*, leaving open the question of how often such harmful patterns occur in other topics. In this paper, we investigate gender-related asymmetries in Wikipedia titles from *all domains*. We assess that for only half of gender-related articles, i.e., articles with words such as *women* or *male* in their titles, symmetrical counterparts describing the same concept for the other gender (and clearly stating it in their titles) exist. Among the remaining imbalanced cases, the vast majority of articles concern sports- and social-related issues. We provide insights on how such asymmetries can influence other Wikipedia components and propose steps towards reducing the frequency of observed patterns.

## introduction gender bias can be defined as a systematic preference or discrimination against people of a particular gender @xcite . nlp systems that exhibit such biased behavior can perform better for the favored gender, e.g., speech recognizers that achieve higher accuracy for male voices @xcite , or reinforce harmful stereotypes, e.g., social media platforms that misgender lgbt+ community members @xcite . 1 biased behaviors have (in-)direct origins in statistical patterns occurring in data that the nlp models are trained on @xcite . such patterns contribute to different types of biases, such as selection bias that comes with the gold-standard annotated resources, or semantic bias related to the information encoded in pre-trained word embeddings. as noted by @xcite , understanding the origins of observed biases is crucial in developing countermeasures for them. as a consequence, diagnosing what type of biases occur in the primary training data is an important initial step in this process. wikipedia is one of the largest and most commonly used sources of training data for nlp models: it serves as unannotated data for pre-training word representations @xcite , a textual source for annotated corpora @xcite , or treebanks (de @xcite . wikipedia is also a community-based effort created by a predominantly male group of editors @xcite . this homogene-ity of wikipedia authors constantly raises the question of possible inequalities in ways that people of different genders are represented in wikipedia articles @xcite @xcite @xcite . for example, @xcite analyzed articles about notable men and women and observed systemic lexical and structural asymmetries, i.e., articles about women contain more family-and gender-related words, and more hyperlinks to articles about men than the other way around. in their following analysis, @xcite showed that gender inequalities can be observed also on other dimensions, e.g., women have to be more notable to have their biographies than men. these patterns occurring in wikipedia have been shown to directly influence nlp models, such as word embeddings @xcite or relation extraction @xcite . all the above-mentioned studies focus on diagnosing gender bias in biographies. biographies are also the main focus of the wikipedia community when it comes to mitigating the content gender gap. for example, the project women in red 2 aims at increasing the number of female biographies. however, as pointed out @xcite , inequalities in the way that women and men are portrayed in wikipedia are present also in other domains. for example, articles about national sports teams such as england women's national football team and england national football team commonly omit the word men's in titles of men's teams, presenting them as if they are the default concept. yet, a computational method to find such gender-related asymmetries on a bigger scale and understanding of how often they occur in wikipedia articles are missing. in this paper, we make an initial step in assessing gender-related inequalities in wikipedia titles. we design a simple three-step heuristic for filtering articles that describe specific concepts and topics, i.e., the above-mentioned english football team, in an unbalanced way (section 2). we apply the proposed method to four wikipedia editions: turkish, english, german, and polish, and find coherent patterns across all four languages (section 3). only half of the articles that use gender-indicating words such as men's or female in their titles have 2 en.wikipedia.org/wiki/wikipedia: wikiproject_women_in_red their symmetrical counterparts describing the same concept for the other gender. among the remaining imbalanced cases, the vast majority either describe male-related concepts as generic or represent women as an exception within a more general topic. we discuss the possible harmful effect of the diagnosed title inequalities on other wikipedia components, such as cross-lingual hyperlinks and nlp models that use them (section 4). finally, we propose steps towards reducing the frequency of discovered patterns (section 5). ## methodology our main goal is to recognize topics and concepts that are described in wikipedia asymmetrically, i.e., with respect to only one gender. since finding all possible wikipedia inequalities is an immense challenge, in this paper, we focus on article titles. in section 4, we provide additional insights on other wikipedia components that have a direct connection to titles thus should be investigated further in the future. we design a simple three-step methodology for spotting wikipedia inequalities in titles, which is presented in figure 1 . below we describe each of the steps in more detail, but first, explain the selection of languages used for evaluation. languages we process four languages: turkish, english, german, polish (in the order of gender marking in the language) @xcite @xcite . german and polish are germanic and slavic languages respectively and both have grammatical gender, though gender marking is more prominent in polish. in these languages, nouns and (thirdperson) pronouns have grammatical gender. determiners and attributive adjectives agree with nouns. reflexive pronouns, however, are not gendermarked. polish, in addition, has gender agreement in predicative adjectives, and verbs in past and future tenses agree with the gender of the subject. english is a germanic language that falls under natural gender languages. almost all nouns are gender-neutral, but third-person personal pronouns are gendered. turkish is a genderless language from the turkic language family. there are no gendered nouns or pronouns in principle, exceptions are mostly loanwords (e.g. aktör -aktris). having no explicit gender markers makes it harder to observe gender bias linguistically, e.g., approaches using pronouns @xcite human female sexuality women in islam 80's ladies . . . women amazing x-men list of male jazz singers human male sexuality . . . men women: human female sexuality men: human male sexuality generic: human sexuality social w|m|g social: human female sexuality social: women in islam name: 80's ladies . . . women name: amazing x-men list: list of male jazz singers social: human male sexuality . . . men women: women in islam men: -generic: islam social w|g women: -men: amazing x-men generic: -name m . . . step 1: title filtering step 2: meta-categories step 3: grouping et al. @xcite are not applicable to turkish. @xcite utilizes sociolinguistic tests to demonstrate that turkish indeed exhibits inherent gender bias that she refers to as covert gender. step 1: title filtering the first step in figure 1 consists of filtering all wikipedia titles that describe a particular topic or concept from a genderrelated perspective and dividing them into men and women groups. for this, we manually select a list of word indicators, i.e., words such as men or female, and search for titles that contain at least one of them (underlined words in figure 1 ). it is important to note that by filtering the articles only through men and women indicators we leave out non-binary genders @xcite and do not address the non-binary gender biases in this women men english women, ladies, men, gentlemen female, feminine male, masculine german frauen-, damen-, männer-, herrenweiblich, -innen männlich polish kobiety, kobiecy, żeński mężczyźni, męski turkish kadın, kadınlar, erkek, erkekler bayan, bayanlar ## inequalities in titles in this section, we apply the described methodology to four wikipedia editions: english, german, polish, and turkish. 7 first, we look at statistics of filtered articles and tuples to provide insights on the coverage of the proposed method (section 3.1). then, we detect types of inequalities occurring in titles (section 3.2) and finally assess which concepts and topics they relate to the most (section 3.3). ## beyond captured inequalities our methodology clearly illustrates that gender bias exists in wikipedia titles in various patterns. nevertheless, these patterns do not constitute an exhaustive list; more fine-grained mismatches could be captured if we extend our observations towards asymmetrically named titles and other wikipedia components. before we move on with our observations, we define the structure of a wikipedia article, exemplified in figure 3 . aside from the self-explanatory title and content parts, there is a hatnote in some wikipedia pages that is always placed at the very top of a page and that helps readers distinguish the page they are at, especially after a redirection or visiting a disambiguation page. on the left column 4 . lexically asymmetric tuples sometimes it is hard to find concept-related articles because the naming of symmetric concepts hence their wikipedia titles are not aligned. articles about the spanish football league in the english wikipedia represent such a case. the article about the male league is titled la liga, which is the more commonly known version of the longer name campeonato nacional de liga de primera división (division national league championship). the women's article, on the other hand, is titled primera división (women), which is a shorter version of primera división de la liga de fútbol femenino. our method recognizes the second title as an instance of women, but it is not able to pair it with la liga and the tuple falls into the symmetrical w (sports) group instead of the asymmetrical w|g (see semantically asymmetric tuples in american collegiate athletics, a common word choice for representing women's teams is adding lady to the name of men's teams, e.g. statesmen vs. lady statesmen of delta state university @xcite . this phenomenon extends to professional sports such as professional golfers' in turkish, on the other hand, lady is used in the sports category in a different way. the corresponding word bayan is used as an euphemism for kadın (woman), as the latter is considered impolite or has a sexual connotation from a sexist perspective @xcite . this general use in language is carried over to national federations of olympic sports. @xcite report that 27 out of 34 federations use bayan to refer to female athletes as of 2016. it is not only in result sections or in regulations, but also in league and team names where male counterparts employ erkek (man). in terms of automatic processing, the problem is two-way. if we consider, for instance, erkekler ligi vs bayanlar ligi (men's league vs. ladies' league) as a concept-related pair, then seemingly there is symmetry; such tuples fall into the symmetric w|m group. if we do not pair these two titles, then they belong in m and w groups respectively, and as sports articles are not treated as inequalities. in both cases, we do not capture the implicit semantic bias behind these two titles, i.e., bias related to historical and societal factors. title-content mismatch to remove inequalities in wikipedia, editors commonly edit generic titles so that they become men titles (e.g. national @xmath0. an additional approach is to redirect generic titles to men titles to facilitate search. yet, if this procedure is done uncarefully and the content is not edited accordingly, it can create pages with title-content mismatch. for instance, searching for türkiye millî voleybol takımı (turkey national volleyball team) redirects to türkiye erkek millî voleybol takımı (turkey men's national volleyball team). since the women article also exists, the tuple belongs to the w|m group thanks to editors; the titles are symmetrical. however, the content still uses the generic reference: "türkiye millî voleybol takımı . . . türkiye'yi uluslararası erkek voleybol karşılaşmalarında temsil eden takımdır." ("turkey national volleyball team . . . is the team representing turkey in the international men's volleyball matches."). therefore there is a mismatch between the title and content of the page. cross-lingual mismatch interwiki links are beneficial in nlp tasks such as machine translation @xcite , bilingual dictionary extraction @xcite , or multilingual named entity recognition @xcite . when a title in one language is edited after interwiki links are established between languages, it might cause a cross-lingual mismatch. at the time of writing, the pages for the turkish national basketball teams are tr: türkiye millî basketbol takımı g+h de: türkische basketballnationalmannschaft en: turkey men's national basketball team fr: équipe de turquie de basket-ball az: türkiyә milli basketbol komandası g du: turks basketbalteam es: selección de baloncesto de turquía pl: reprezentacja turcji w koszykówce mężczyzn m ru: мужская сборная турции по баскетболу a w|g tuple: türkiye kadın millî basketbol takımı -türkiye millî basketbol takımı. the generic title also has the hatnote explaining that although the title is generic, the page is about men, and there is a separate women article. however, interwiki links connect the page to its english counterpart which is titled turkey men's national basketball team, that is, men indicator is explicit only on the english side. the generic to men mapping in titles depicts a cross-lingual mismatch. nlp tasks that would use this pair would fail to extract an exact translation. following multiple interwiki links shows that approaches vary across languages, as summarized in finally, cross-lingual mismatches come also with semantic asymmetries, such as the use of bayan (lady) instead of kadın (woman). interwiki still links them to non-turkish titles with the word woman or its equivalent, causing a translation mistake. ## discussion: towards debiasing titles the types of inequalities that we presented in this paper call for different debiasing measures. lexically asymmetric tuples can only be paired via mutual hatnotes. in the specific case of la liga, adding (men) to the title in parallel to its female counterpart would at least prevent the male generic bias. title-content mismatches could be avoided if the content could be updated together with the title. cross-lingual mismatches are the easiest to catch as there are wiki bots designed for that purpose. 9 among meta-categories, debiasing the sports category is seemingly the simplest one. for instance, the turkish basketball federation foot_8 symmetrically names national basketball teams by explicitly using kadın and erkek in their titles. the w|g inequality comes from wikipedia editors. inserting erkek to the generic türkiye millî basketbol takımı (and updating the content consequently) would bring balance to this article tuple. however, sometimes the imbalance is actually in the real world. the international cycling union (uci) foot_9 defines both cups and teams as w|g tuples, e.g., uci road world cup vs. uci women's road world cup, and wikipedia titles follow suit. after all, it is not wrong to take the actual name of a concrete entity. the next step is subject to a debate: should wikipedia editors continue to reflect the real world or should they already convert the title to men and keep the existing generic naming, perhaps with a redirection for search purposes? the lists category with a w|g or m|g inequality can be balanced by introducing the missing gender-specific articles or going into the complete opposite direction and keeping only the generic page. if the answer to this question is to create gender-specific pages, the non-binary gender values come into the picture to be represented in separate lists pages. such lists exist, e.g. list of transgender political office-holders, yet, they are few. wikipedia's metadata system facilitates such a listing by providing non-binary values for annotating biographies; currently, it is a set of seven: male, female, non-binary, intersex, transgender female, transgender male, agender. foot_10 however, page specification raises more questions: how about the people who do not want to be identified with one of the values, where should they be placed? a substantial amount of the w only tuples in the social category consists of women's rights movements. 10% of the social titles in the english wikipedia contains the words suffrage or rights. while w seems to be an imbalanced category by definition, the reason behind it is not that women are dominating the scene. on the contrary, it shows the reaction to suppression and that women had to fight over rights that should be the default. simi-larly, w|g tuples such as women in engineering vs. engineering seem to be biased towards women by definition, yet the urge to discuss such topics separately comes from the complete opposite reason. men have been considered the 'norm' such that women are a deviating subset. it is hard to develop an umbrella strategy for debiasing the social category. for some of these titles, the question is whether they should be debiased rather than how they could be debiased. in our opinion, for instance suffrage pages, e.g., women's suffrage in alabama should remain as is, but the w|g tuple women's suffrage vs. suffrage can be extended to a w|m|g tuple. at the time of writing, the title men's suffrage redirects to suffrage. instead, a separate men page that very briefly explains the historical developments (i.e., why there has been no need to coin an explicit term for men) would be more informative and more in line with wikipedia's encyclopedic nature. ## conclusion inequality in male and female wikipedia biographies has driven a lot of attention in recent years, both from the research community as well as the press. 13 the editors' community is consequently tackling this problem and fighting the gender gap. as recently shown by @xcite , these efforts pay off and not only more biographies about women are being added, but also nlp models such as word embeddings trained on wikipedia articles are exhibiting less stereotypical biases. in this paper, we aimed at raising awareness that gender inequalities in wikipedia extend beyond biographies. women and men are systematically represented in article titles in a different way, especially in such domains as sports and social issues. we showed that such inequalities can be computationally assessed by investigating asymmetrical tuples, i.e., titles that describe a particular topic or concept for only one gender. the topics we argue here by no means cover all possible inequalities in wikipedia titles, let alone inequalities in its overall components. yet, even a simple but competent, systematic approach is strong enough to demonstrate the existing gender bias. with this paper, we hope to draw attention to such bias and open possible solutions to discussion. 13 see en.wikipedia.org/wiki/wikipedia: wikiproject_women_in_red/about_us for a list of recent news articles regarding this subject.