Developing a Multilingual Corpus of W ikipedia Biographies (2023)

Abstract: For many languages, Wikipedia is the most accessible source of biographical information. Studying how Wikipedia describes the lives of people can provide insights into societal biases, as well as cultural differences more generally. We present a method for extracting datasets of Wikipedia biographies. The accompanying codebase is adapted to English, Swedish, Russian, Chinese, and Farsi, and is extendable to other languages. We present an exploratory analysis of biographical topics and gendered patterns in four languages using topic modelling and embedding clustering. We find similarities across languages in the types of categories present, with the distribution of biographies concentrated in the language’s core regions. Masculine terms are over-represented and spread out over a wide variety of topics. Feminine terms are less frequent and linked to more constrained topics. Non-binary terms are nearly non-represented.

## introduction wikipedia's decentralised organisation and independent communities in different languages have led it to be considered a 'global repository of knowledge' @xcite . easily and openly accessible, in many language environments it has displaced more traditional and specialised resources as a 'default' encyclopedic source, shaping the language landscape. this effect is noticeable in nlp research and development, where wikipedia is a staple training data source for language models that imitate wikipedia when generating texts or making writing suggestions. when it comes to biographical information, wikipedia has become a primary source of reference, especially outside of education systems. because of the near-monopoly that wikipedia has on public knowledge in these environments, it can define which persons are perceived as notable, which aspects of their lives deserve a mention, and how the persons are presented. the community guidelines of wikipedia are built around the concept of 'neutrality', but the content is still inevitably shaped by societal biases, such as gender gaps @xcite . besides the content of the biographical articles, wikipedia also shapes the expectations the reader has in terms of format, language, style and inclusion. a 'biography' can come to invoke a wikipedia-like structure, becoming a commonly accepted way of summarising a person's life. most automatically extracted datasets consists of wikipedia backup dumps or rely heavily on the connection to wikidata. using wikidata, however, makes it harder to modify or update the dataset, or replicate it for another domain. manually collected datasets, on the other hand, are limited in size. they are almost inevitably biased towards longer, popular or better-categorised articles because poor categorisation prevents other articles to be discovered in the first place. these problems become even more apparent when creating a multilingual dataset. while different editions share a general article structure and templates, they are far from identical. as we discuss further in the paper, straightforward parsing approaches can fail if adapted directly because of the subtle markdown changes. for our biographical dataset, this often results in the omission of less well-documented (often marginalised) people. this paper contributes an adaptable method for curating a multilingual corpus of wikipedia biographies. we analyse general statistics and structures of the biographies for corpora in several languages and compare them with existing literature. finally, we release the code foot_0 and instructions on how to create a biography dataset from an up-to-date wikipedia dump adaptable to any language. ## the corpus the code for creating the wikipedia biographies corpora released with this paper is created with the purpose of exploring cultural and narrative trends, including social bias analysis. the wikipedia articles that are collected should meet the criteria of being a biography (section 2.1). in this section, we describe the process of identifying biographies and extracting clean text; and present a data card with basic corpora statistics. ## demo cases two demo cases were designed to demonstrate some general usage of the corpora and to acquire more latent information about their contents. the corpora are well-situated to study societal structures and how information is relayed, and make comparisons across languages. the first demo case (section 4.1) is a study on topical differences between languages with a focus on gendered themes. the second demo case (section 4.2) is a cluster analysis of the corpora to visualise how the biographies are broadly divided into clusters depending on their running text. ## limitations the aim of this work was to collect as many wikipedia articles as possible that fit the criteria for being a biography. while the corpora presented in this study are largely biographies, there are articles that evade the filter, e.g. the wikipedia category 'mountaineering deaths' includes both biographies and articles about accidents. although these articles are easy to manually identify, we do not remove them as they must be considered individually and we found them to be extremely uncommon. false negatives are harder to identify. we generally assume that all biographies have at least one of the patterns: 'born', 'death', or 'living'. in cases where a person does not have these, it may be the case that they have a sufficiently mythological status (for example, the buddha is not captured in our english corpus). more likely, however, it could be due to human error when editing the page. we recommend making manual checks with samples of biographies expected to be in the data. this is especially important when considering biographies of people belonging to marginalised groups, who may be less likely to be seen as 'significant' and thus not be properly curated. we attempted to strip the text from all links, tables and other clutter to only have the running text of the biography easily accessible in the dataset. we can not guarantee that the texts do not contain any errors from the cleaning, and some nlp applications may require different information (such as extracting links) which we do not provide. while analysing underlying gendered patterns can be done through topic modelling, this technique is not well-suited to languages where gram-matical and social gender overlap. it also fails in data-sparse contexts, such as for gender-diverse populations. our demo is provided as a proof-ofconcept of the utility of the corpora for social bias analysis, and as a warning that they should not be used uncritically: more detailed analysis and mitigation, tailored to specific use cases, will be necessary for all social biases. finally, these corpora should not be used as a benchmark for pre-trained models that where constructed using wikipedia in their training data. ## conclusions the multilingual wikipedia biographies is mainly a method for extracting an up-to-date high-quality dataset from the wikipedia dump. the method is easily adaptable to other languages including those with low resources. some general structures were common between languages such as masculine terms being generally more prevalent in topics compared to feminine, which were constrained to more specific topics. the distribution of biographies is naturally highest in the language's core region and gradually declines as it extends outward. the corpora allow for comparing the structures and composition of biographies in multiple languages which is important for understanding how wikipedia biographies shape how information about individuals, and society, as a whole is shared.