Dictionary-based Debiasing of Pre-trained Word Embeddings (2021)

Abstract: Word embeddings trained on large corpora have shown to encode high levels of unfair discriminatory gender, racial, religious and ethnic biases. In contrast, human-written dictionaries describe the meanings of words in a concise, objective and an unbiased manner. We propose a method for debiasing pre-trained word embeddings using dictionaries, without requiring access to the original training resources or any knowledge regarding the word embedding algorithms used. Unlike prior work, our proposed method does not require the types of biases to be pre-defined in the form of word lists, and learns the constraints that must be satisfied by unbiased word embeddings automatically from dictionary definitions of the words. Specifically, we learn an encoder to generate a debiased version of an input word embedding such that it (a) retains the semantics of the pre-trained word embedding, (b) agrees with the unbiased definition of the word according to the dictionary, and (c) remains orthogonal to the vector space spanned by any biased basis vectors in the pre-trained word embedding space. Experimental results on standard benchmark datasets show that the proposed method can accurately remove unfair biases encoded in pre-trained word embeddings, while preserving useful semantics.

## introduction despite pre-trained word embeddings are useful due to their low dimensionality, memory and compute efficiency, they have shown to encode not only the semantics of words but also unfair discriminatory biases such as gender, racial or religious biases @xcite @xcite @xcite . on the other hand, human-written dictionaries act as an impartial, objective and unbiased source of word meaning. although methods that learn word embeddings by purely using dictionaries have been proposed @xcite , they have coverage and data sparseness related issues because precompiled dictionaries do not capture the meanings of neologisms or provide numerous contexts as in a corpus. consequently, prior work has shown that word embeddings learnt from large text corpora to outperform those created from dictionaries in downstream nlp tasks @xcite . we must overcome several challenges when using dictionaries to debias pre-trained word embeddings. first, not all words in the embeddings will appear in the given dictionary. dictionaries often have limited coverage and will not cover neologisms, orthographic variants of words etc. that are likely to appear in large corpora. a lexicalised debiasing method would generalise poorly to the words not in the dictionary. second, it is not known apriori what biases are hidden inside a set of pretrained word embedding vectors. depending on the source of documents used for training the embeddings, different types of biases will be learnt and amplified by different word embedding learning algorithms to different degrees @xcite . prior work on debiasing required that the biases to be pre-defined @xcite . for example, hard-debias (hd; @xcite and gender neutral glove (gn-glove; @xcite require lists of male and female pronouns for defining the gender direction. however, gender bias is only one of the many biases that exist in pre-trained word embeddings. it is inconvenient to prepare lists of words covering all different types of biases we must remove from pretrained word embeddings. moreover, such pre-compiled word lists are likely to be incomplete and inadequately cover some biases. indeed, @xcite showed empirical evidence that such debiasing methods do not remove all discriminative biases from word embeddings. unfair biases have adversely affected several nlp tasks such as machine translation @xcite and language generation @xcite . racial biases have also been shown to affect criminal prosecutions @xcite and career adverts @xcite . these findings show the difficulty of defining different biases using pre-compiled lists of words, which is a requirement in previously proposed debiasing methods for static word embeddings. we propose a method that uses a dictionary as a source of bias-free definitions of words for debiasing pre-trained word embeddings foot_1 . specifically, we learn an encoder that filters-out biases from the input embeddings. the debiased embeddings are required to simultaneously satisfy three criteria: (a) must preserve all non-discriminatory information in the pre-trained embeddings (semantic preservation), (b) must be similar to the dictionary definition of the words (dictionary agreement), and (c) must be orthogonal to the subspace spanned by the basis vectors in the pre-trained word embedding space that corresponds to discriminatory biases (bias orthogonality). we implement the semantic preservation and dictionary agreement using two decoders, whereas the bias orthogonality is enforced by a parameter-free projection. the debiasing encoder and the decoders are learnt endto-end by a joint optimisation method. our proposed method is agnostic to the details of the algorithms used to learn the input word embeddings. moreover, unlike counterfactual data augmentation methods for debiasing @xcite , we do not require access to the original training resources used for learning the input word embeddings. our proposed method overcomes the abovedescribed challenges as follows. first, instead of learning a lexicalised debiasing model, we operate on the word embedding space when learning the encoder. therefore, we can use the words that are in the intersection of the vocabularies of the pre-trained word embeddings and the dictionary to learn the encoder, enabling us to generalise to the words not in the dictionary. second, we do not require pre-compiled word lists specifying the biases. the dictionary acts as a clean, unbiased source of word meaning that can be considered as positive examples of debiased meanings. in contrast to the existing debiasing methods that require us to predefine what to remove, the proposed method can be seen as using the dictionary as a guideline for what to retain during debiasing. we evaluate the proposed method using four standard benchmark datasets for evaluating the biases in word embeddings: word embedding association test (weat; @xcite , word association test (wat; @xcite , sem-bias @xcite and winobias @xcite . our experimental results show that the proposed debiasing method accurately removes unfair biases from three widely used pre-trained embeddings: word2vec @xcite , glove @xcite and fasttext @xcite . moreover, our evaluations on semantic similarity and word analogy benchmarks show that the proposed debiasing method preserves useful semantic information in word embeddings, while removing unfair biases. ## related work dictionaries have been popularly used for learning word embeddings @xcite @xcite . methods that use both dictionaries (or lexicons) and corpora to jointly learn word embeddings @xcite @xcite or post-process @xcite have also been proposed. however, learning embeddings from dictionaries alone results in coverage and data sparseness issues @xcite and does not guarantee bias-free embeddings @xcite . to the best of our knowledge, we are the first to use dictionaries for debiasing pre-trained word embeddings. @xcite proposed a postprocessing approach that projects gender-neutral words into a subspace, which is orthogonal to the gender dimension defined by a list of genderdefinitional words. they refer to words associated with gender (e.g., she, actor) as gender-definitional words, and the remainder gender-neutral. they proposed a hard-debiasing method where the gender direction is computed as the vector difference between the embeddings of the correspond-ing gender-definitional words, and a soft-debiasing method, which balances the objective of preserving the inner-products between the original word embeddings, while projecting the word embeddings into a subspace orthogonal to the gender definitional words. both hard and soft debiasing methods ignore gender-definitional words during the subsequent debiasing process, and focus only on words that are not predicted as gender-definitional by the classifier. therefore, if the classifier erroneously predicts a stereotypical word as a genderdefinitional word, it would not get debiased. zhao et al. (2018b) modified the glove (penning @xcite objective to learn gender-neutral word embeddings (gn-glove) from a given corpus. they maximise the squared ℓ 2 distance between gender-related sub-vectors, while simultaneously minimising the glove objective. unlike, the above-mentioned methods, @xcite proposed a post-processing method to preserve gender-related information with autoencoder @xcite , while removing discriminatory biases from stereotypical cases (gp-glove). however, all prior debiasing methods require us to pre-define the biases in the form of explicit word lists containing gender and stereotypical word associations. in contrast we use dictionaries as a source of bias-free semantic definitions of words and do not require pre-defining the biases to be removed. although we focus on static word embeddings in this paper, unfair biases have been found in contextualised word embeddings as well @xcite @xcite . adversarial learning methods @xcite @xcite for debiasing first encode the inputs and then two classifiers are jointly trained -one predicting the target task (for which we must ensure high prediction accuracy) and the other protected attributes (that must not be easily predictable). however, @xcite showed that although it is possible to obtain chance-level development-set accuracy for the protected attributes during training, a post-hoc classifier trained on the encoded inputs can still manage to reach substantially high accuracies for the protected attributes. they conclude that adversarial learning alone does not guarantee invariant representations for the protected attributes. @xcite found that iteratively projecting word embeddings to the null space of the gender direction to further improve the debiasing performance. to evaluate biases, @xcite proposed the word embedding association test (weat) inspired by the implicit association test (iat; @xcite . @xcite showed that weat to be systematically overestimating biases and proposed a correction. the ability to correctly answer gender-related word analogies @xcite and resolve genderrelated coreferences @xcite have been used as extrinsic tasks for evaluating the bias in word embeddings. we describe these evaluation benchmarks later in § 4.3. ## dictionary-based debiasing let us denote the n-dimensional pre-trained word embedding of a word w by @xmath0. moreover, let us assume that we are given a dictionary d containing the definition, s(w) of w. if the pre-trained embeddings distinguish among the different senses of w, then we can use the gloss for the corresponding sense of w in the dictionary as s(w). however, the majority of word embedding learning methods do not produce sense-specific word embeddings. in this case, we can either use all glosses for w in d by concatenating or select the gloss for the dominant (most frequent) sense of w foot_2 . without any loss of generality, in the remainder of this paper, we will use s(w) to collectively denote a gloss selected by any one of the above-mentioned criteria with or without considering the word senses (in § 5.3, we evaluate the effect of using all vs. dominant gloss). next, we define the objective functions optimised by the proposed method for the purpose of learning unbiased word embeddings. given, w, we model the debiasing process as the task of learning an encoder, e(w; θ e ) that returns an m(≤ n)dimensional debiased version of w. in the case where we would like to preserve the dimensionality of the input embeddings, we can set @xmath1, or @xmath2. because the pre-trained embeddings encode rich semantic information from a large text corpora, often far exceeding the meanings covered in the dictionary, we must preserve this semantic information as much as possible during the debiasing process. we refer to this constraint as semantic preservation. semantic preservation is likely to lead to good performance in downstream nlp applications that use pre-trained word embeddings. for this purpose, we decode the encoded version of w using a decoder, d c , parametrised by θ c and define j c to be the reconstruction loss given by (1). following our assumption that the dictionary definition, s(w), of w is a concise and unbiased description of the meaning of w, we would like to ensure that the encoded version of w is similar to s(w). we refer to this constraint as dictionary agreement. to formalise dictionary agreement empirically, we first represent s(w) by a sentence embedding vector s(w) ∈ r n . different sentence embedding methods can be used for this purpose such as convolutional neural networks @xcite , recurrent neural networks @xcite or transformers @xcite . for the simplicity, we use the smoothed inverse frequency (sif; @xcite for creating s(w) in this paper. sif computes the embedding of a sentence as the weighted average of the pre-trained word embeddings of the words in the sentence, where the weights are computed as the inverse unigram probability. next, the first principal component vector of the sentence embeddings are removed. the dimensionality of the sentence embeddings created using sif is equal to that of the pre-trained word embeddings used. therefore, in our case we have both w, s(w) ∈ r n . we decode the debiased embedding e(w; θ e ) of w using a decoder d d , parametrised by θ d and compute the squared ℓ 2 distance between it and s(w) to define an objective j d given by (2). recalling that our goal is to remove unfair biases from pre-trained word embeddings and we assume dictionary definitions to be free of such biases, we define an objective function that explicitly models this requirement. we refer to this requirement as the bias orthogonality of the debiased embeddings. for this purpose, we first project the pre-trained word embedding w of a word w into a subspace that is orthogonal to the dictionary definition vector s(w). let us denote this projection by φ(w, s(w)) ∈ r n . we require that the debiased word embedding, e(w; θ e ), must be orthogonal to φ(w, s(w)), and formalise this as the minimisation of the squared inner-product given in (3). note that because φ(w, s(w)) lives in the space spanned by the original (prior to encoding) vector space, we must first encode it using e before considering the orthogonality requirement. to derive φ(w, s(w)), let us assume the ndimensional basis vectors in the r n vector space spanned by the pre-trained word embeddings to be b 1 , b 2 , . . . , b n . moreover, without loss of generality, let the subspace spanned by the subset of the first k(< n) basis vectors b 1 , b 2 , . . . , b k to be b ⊆ r n . the projection v b of a vector @xmath3. to show that vv b is orthogonal to v b for any @xmath4, let us express v -v b using the basis vectors as given in (5). we see that there are no basis vectors in common between the summations in (4) and ( 5 ). therefore, considering that s(w) defines a direction that does not contain any unfair biases, we can compute the vector rejection of w on s(w) following this result. specifically, we subtract the projection of w along the unit vector defining the direction of s(w) to compute φ as in (6). we consider the linearly-weighted sum of the above-defined three objective functions as the total objective function as given in (7). here, α, β, γ ≥ 0 are scalar coefficients satisfying α + β + γ = 1. later, in § 4 we experimentally determine the values of α, β and γ using a development dataset. ## conclusion we proposed a method to remove biases from pretrained word embeddings using dictionaries, without requiring pre-defined word lists. the experimental results on a series of benchmark datasets show that the proposed method can remove unfair biases, while retaining useful semantic information encoded in pre-trained word embeddings.