A ra WEAT : Multidimensional Analysis of Biases in A rabic Word Embeddings (2020)

Abstract: Recent work has shown that distributional word vector spaces often encode human biases like sexism or racism. In this work, we conduct an extensive analysis of biases in Arabic word embeddings by applying a range of recently introduced bias tests on a variety of embedding spaces induced from corpora in Arabic. We measure the presence of biases across several dimensions, namely: embedding models (Skip-Gram, CBOW, and FastText) and vector sizes, types of text (encyclopedic text, and news vs. user-generated content), dialects (Egyptian Arabic vs. Modern Standard Arabic), and time (diachronic analyses over corpora from different time periods). Our analysis yields several interesting findings, e.g., that implicit gender bias in embeddings trained on Arabic news corpora steadily increases over time (between 2007 and 2017). We make the Arabic bias specifications (AraWEAT) publicly available.

## introduction recent research offered evidence that distributional word representations (i.e., word embeddings) induced from human-created text corpora exhibit a range of human biases, such as racism and sexism @xcite . with word embeddings ubiquitously used as input for (neural) natural language processing (nlp) models, this brings about the jeopardy of introducing stereotypical unfairness into nlp models, which can reinforce existing social hierarchies, and therefore be harmful in practical applications. for instance, consider the seminal gender bias example "man is to computer programmer as woman is to homemaker", which is algebraically encoded in the embedding space with the analogical relation man-computer programmer ≈ woman-homemaker @xcite . the existence of such biases in word embeddings stems from the combination of (1) human biases manifesting themselves in terms of word co-occurrences (e.g., the word woman appearing in a training corpus much more often in the context of homemaker than together with computer programmer) and (2) the distributional nature of the word embedding models @xcite @xcite , which induce word vectors precisely by exploiting word co-occurrences, i.e., thus also encoding the human biases as a (negative) side-effect, which represents, expressed according to the taxnomy of harms proposed by @xcite , a representational harm, more specifically, stereotyping. in order to quantify the amount of bias in word embeddings, @xcite proposed the word embedding association test (weat), which is based on the associative difference in terms of semantic similarity between two sets of target terms, e.g., male and female terms, towards two sets of attribute terms, e.g., career and family terms. most recently, the weat test, measuring the degree of explicit bias in the distributional space, has been coupled with other tests, aiming to measure other aspects of bias, such as the amount of implicit bias @xcite or the presence of the analogical bias @xcite . while there is evidence that distributional vectors often encode human biases, the amount of biases does not seem to be universal across different languages and corpora, as recently shown by @xcite in the analysis of distributional biases across seven different languages. in this work, we focus on the multi-dimensional analysis of biases in arabic word embeddings. the motivation for this work is twofold: (1) arabic is one of the most widely spoken languages in the world: foot_0 this means that the biases encoded in language technology for arabic have the potential for affecting more people than for most other languages; (2) language resources for arabic -large corpora @xcite , pretrained word embeddings @xcite , and datasets for measuring semantic quality of arabic embeddings @xcite -are publicly available, allowing for the analyses of biases that these resources potentially hide. as a first step in the analysis of language technology biases for arabic, we present araweat, an arabic extension to the multilingual xweat framework @xcite . because the weat test @xcite , though it has the notable advantage of drawing inspiration from psychology literature, has recently been shown to systematically overestimate the bias present in an embedding space @xcite , in this work, we couple it with several other bias tests, designed to capture and quantify other aspects of human biases: embedding coherence test @xcite , bias analogy test @xcite and implicit bias tests @xcite . our work, which is to the best of our knowledge the first study on quantifying biases in arabic distributional word vector spaces, yields some interesting findings: biases seem more prominent in vectors trained on texts written in egyptian arabic than those written in modern standard arabic (msa). also, the implicit gender bias in arabic news corpora seems to be steadily on the rise over the ten year period between 2007 and 2017. finally, we find evidence that the explicit bias effects, as measured by the weat test, in embeddings trained on the entire arabic news corpus roughly correspond to averaging the biases measured across embeddings trained on temporally disjunct subsets of the corpus. ## araweat we present araweat, our framework allowing for multi-dimensional analysis of bias in arabic distributional word vector spaces. ## findings we present and discuss the findings of our analysis employing araweat. ## conclusion language technologies aim to avoid reflecting negative human biases such as racism and sexism. yet, the ubiquitous word embeddings, used as input for many nlp models, seem to encode many such biases. in this work, we extensively quantify and analyze the biases in different vector spaces built from text in arabic, a major world language with close to 300m native speakers. to this effect, we translate existing bias specifications from english to arabic and investigate biases in embedding spaces that differ over several dimensions of analysis: embedding models, corpora sizes, type of text, dialectal vs. standard arabic, and time periods. our analysis yields interesting results. first, we confirm some of the previous findings for other languages, e.g., that biases are generally not more pronounced in user-generated text and that embeddings trained on larger corpora lead to more prominent biases. secondly, our results suggest more bias is present in dialectal (egyptian) arabic corpora than in modern standard arabic corpora. next, our diachronic analysis suggests that the implicit gender bias of arabic news text steadily increases over time. finally, we show that the bias effects of the whole corpus can be predicted from bias effects of its subcorpora. we hope that araweat, our framework for multidimensional analysis of stereotypical bias in arabic text representations, fuels more research on bias in arabic language technology.