Measuring Intersectional Biases in Historical Documents (2023)

Abstract: Data-driven analyses of biases in historical texts can help illuminate the origin and development of biases prevailing in modern society. However, digitised historical documents pose a challenge for NLP practitioners as these corpora suffer from errors introduced by optical character recognition (OCR) and are written in an archaic language. In this paper, we investigate the continuities and transformations of bias in historical newspapers published in the Caribbean during the colonial era (18th to 19th centuries). Our analyses are performed along the axes of gender, race, and their intersection. We examine these biases by conducting a temporal study in which we measure the development of lexical associations using distributional semantics models and word embeddings. Further, we evaluate the effectiveness of techniques designed to process OCR-generated data and assess their stability when trained on and applied to the noisy historical newspapers. We find that there is a trade-off between the stability of the word embeddings and their compatibility with the historical dataset. We provide evidence that gender and racial biases are interdependent, and their intersection triggers distinct effects. These findings align with the theory of intersectionality, which stresses that biases affecting people with multiple marginalised identities compound to more than the sum of their constituents.

## introduction the availability of large-scale digitised archives and modern nlp tools has enabled a number of sociological studies of historical trends and cultures @xcite @xcite . analyses of historical biases and stereotypes, in particular, can shed light on past analysing historical documents poses particular challenges for modern nlp tools @xcite . misspelt words due to wrongly recognised characters in the digitisation process, and archaic language unknown to modern nlp models, i.e. historical variant spellings and words that became obsolete in the current language, increase the task's complexity @xcite @xcite . however, while most previous work on historical nlp acknowledges the unique nature of the task, only a few address them within their experimental setup. in this paper, we address the shortcomings of previous work and make the following contributions: (1) to the best of our knowledge, this paper presents the first study of historical language associated with entities at the intersections of two axes of oppression: race and gender. we study biases associated with identified entities on a word level, and to this end, employ distributional models and analyse semantics extracted from word embeddings trained on our historical corpora. (2) we conduct a temporal case study on historical newspapers from the caribbean in the colonial period between 1770-1870. during this time, the region suffered both the consequences of european wars and political turmoil, as well as several uprisings of the local enslaved populations, which had a significant impact on the caribbean social relationships and cultures @xcite . (3) to address the challenges of analysing historical documents, we probe the applied methods for their stability and ability to comprehend the noisy, archaic corpora. we find that there is a trade-off between the stability of word embeddings and their compatibility with the historical dataset. further, our temporal analysis connects changes in biased word associations to historical shifts taking place in the period. for instance, we couple the high association between caribbean countries and "manual labour" prevalent mostly in the earlier time periods to waves of white labour migrants coming to the caribbean from 1750 onward. finally, we provide evidence supporting the intersectionality theory by observing conventional manifestations of gender bias solely for white people. while unsurprising, this finding necessitates intersectional bias analysis for historical documents. bias in historical documents. historical corpora have been employed to study societal phenomena such as language change @xcite and societal biases. gender bias has been analysed in biomedical research over a span of 60 years @xcite , in englishlanguage books published between 1520 and 2008 @xcite , and in dutch newspapers from the second half of the 20th century @xcite . levis sullam et al. (2022) investigate the evolution of the discourse on jews in france during the 19th century. @xcite study the temporal change in stereotypes and attitudes toward women and ethnic minorities in the 20th and 21st centuries in the us. however, they neglect the emergent intersectionality bias. when analysing the transformations of biases in historical texts, researchers rely on conventional tools developed for modern language. however, historical texts can be viewed as a separate domain due to their unique challenges of small and idiosyncratic corpora and noisy, archaic text @xcite . prior work has attempted to overcome the challenges such documents pose for mod we study the dynamics of intersectional biases and their manifestations in language while addressing the challenges of historical data. ## datasets newspapers are considered an excellent source for the study of societal phenomena since they function as transceivers -both producing and demonstrating public discourse @xcite . as part of this study, we collect newspapers written in english from the "caribbean newspapers, 1718-1876" database, 1 the largest collection of caribbean newspapers from the 18th-19th century available online. we extend this dataset with english-danish newspapers published between 1770-1850 in the danish colony of santa cruz (saint croix) downloaded from danish royal library's website. ## bias and its measures biases can manifest themselves in natural language in many ways (see the surveys by @xcite ; @xcite ; @xcite ). in the following, we state the definition of bias we follow and describe the measures we use to quantify it. ## experimental setup we perform two sets of experiments on our historical newspaper corpus. first, before we employ word embeddings to measure bias, we investigate the stability of the word embeddings trained on our dataset and evaluate their understanding of the noisy nature of the corpora. second, we assess gender and racial biases using tools defined in §4.2. ## results first, we investigate which training strategies of word embeddings optimise their stability and compatibility on historical corpora ( §6.1). next, we analyse how bias is manifested along the gender and racial axes and whether there are any notice-able differences in bias across different periods of the caribbean history ( §6.2). ## conclusions in this paper, we examine biases present in historical newspapers published in the caribbean during the colonial era by conducting a temporal analysis of biases along the axes of gender, race, and their intersection. we evaluate the effectiveness of different embedding strategies and find a tradeoff between the stability and compatibility of word representations on historical data. we link changes in biased word usage to historical shifts, coupling the development of the association between manual labour and caribbean countries to waves of white labour migrants coming to the caribbean from 1750 onward. finally, we provide evidence to corroborate the intersectionality theory by observing conventional manifestations of gender bias solely for white people.