Neutralizing Gender Bias in Word Embeddings with Latent Disentanglement and Counterfactual Generation (2020)

Abstract: Recent research demonstrates that word embeddings, trained on the human-generated corpus, have strong gender biases in embedding spaces, and these biases can result in the discriminative results from the various downstream tasks. Whereas the previous methods project word embeddings into a linear subspace for debiasing, we introduce a Latent Disentanglement method with a siamese auto-encoder structure with an adapted gradient reversal layer. Our structure enables the separation of the semantic latent information and gender latent information of given word into the disjoint latent dimensions. Afterwards, we introduce a Counterfactual Generation to convert the gender information of words, so the original and the modified embeddings can produce a gender-neutralized word embedding after geometric alignment regularization, without loss of semantic information. From the various quantitative and qualitative debiasing experiments, our method shows to be better than existing debiasing methods in debiasing word embeddings. In addition, Our method shows the ability to preserve semantic information during debiasing by minimizing the semantic information losses for extrinsic NLP downstream tasks.

## introduction recent researches have disclosed that word embeddings contain unexpected bias in their geometry on the embedding space @xcite . the bias reflects unwanted stereotypes such as the correlation between gender foot_0 and occupation words. @xcite enumerated that the automatically generated analogies of (she, he) in the word2vec @xcite show the gender biases in significant level. an figure 1 : the process view of our method. we can improve the embedding space from (a) to (b) with a better-aligned structure between gender word pairs by the proposed latent disentanglement. afterwards, (c) we generate the gender-counterfactual embedding of the gender-biased word while keeping a geometrically aligned relationship with the gender word pairs to guarantee that the pair of word embeddings only differs from gender information, not hurting semantic information. (d) we obtain the gender-neutralized word embedding by interpolating the embedding from the pair of original-counterfactual word embeddings. from the various gender debiasing methods for pre-trained word embeddings, the widely recognized method is a post-processing method, which projects word embeddings to the space that is or-thogonal to the gender direction vector defined by a set of gender word pairs. however, if the gender direction vector includes a component of semantic information foot_1 , the semantic information will be lost through the post-processing projections. to balance between the gender debiasing and the semantic information preserving, we propose an encoder-decoder framework that disentangles a latent space of a given word embedding into two encoded latent spaces: the first part is the gender latent space, and the second part is the semantic latent space that is independent to the gender information. to disentangle the latent space into two sub-spaces, we use a gradient reversal layer by prohibiting the inference on the gender latent information from the semantic information. then, we generate a counterfactual word embedding by converting the encoded gender latent into the opposite gender. afterwards, the original and the counterfactual word embeddings are geometrically interpreted to neutralize the gender information of given word embeddings, see figure 1 for the illustration on our debiasing method. our contributions are summarized as follows: we evaluated the proposed method and other baseline methods with several quantitative and qualitative debiasing experiments, and we found that the proposed method shows significant improvements from the existing methods. additionally, the results from several nlp downstream tasks show that our proposed method minimizes performance degradation than the existing methods. we can divide existing gender debiasing mechanisms for word embeddings into two categories. the first mechanism is neutralizing the gender aspect of word embeddings in the training procedure. @xcite proposed the learning scheme to generate a gender-neutral version of glove, called gn-glove, which forces preserving the gender information in pre-specified embedding dimensions while other embedding dimensions are inferred to be gender-neutral. however, learning new word embeddings for large-scale corpus can be difficult and expensive. the second mechanism post-processes trained word embeddings to debias them after the training. an example of such post-processings is a linear projection of gender-neutral words toward a subspace, which is orthogonal to the gender direction vector defined by a set of gender-definition words @xcite . another way of constructing the gender direction vector is using common names, e.g. john, mary, etc @xcite , while the previous approach used gender pronouns, such as he and she. in addition to the linear projections, @xcite utilizes other alternatives, such as flipping and subtraction, to reduce the gender bias more effectively. beyond simple projection methods, @xcite proposed a neural network based encoder-decoder framework to add a regularization on preserving the gender-related information in feminine and masculine words. ## methodology our model introduces 1) the siamese network structure @xcite with an adapted gradient reversal layer for latent disentanglement and 2) the counterfactual data augmentation with geometric regularization for gender debiasing. we process the gender word pairs through the siamese network with auxiliary classifiers to reflect the inference of gender latent dimensions. afterwards, we debias the gender-neutral words by locating it to be at the middle between a reconstructed pair of original gender latent variable and counterfactually generated gender latent variable. same as previous researches @xcite , we divide a whole set of vocabulary v into three mutually exclusive categories : feminine generator with grl gender classifier counterfactual generator for gender-neutral words 𝒘 𝑓 𝒘 𝒎 ෝ 𝒘 𝑓 ෝ 𝒘 𝑚 gender-pair decoder 𝐷 encoder 𝐸 decoder 𝐷 encoder 𝐸 𝒛 𝑚 𝑠 𝒛 𝑓 𝑠 𝒛 𝑚 𝑔 𝐳 𝑓 𝑔 semantic similarity 𝑔 𝑚 𝑔 𝑓 ෝ 𝒘 𝑐𝑓 ෝ 𝒘 𝑛 ෝ 𝒘 𝑛𝑒𝑢 decoding alignment 𝒛 𝑛 𝑠 𝒘 𝑛 encoder 𝐸 decoder 𝐷 𝒛 𝑛 𝑔 decoder 𝐷 ¬𝒛 𝑛 𝑔 for gender-pair words counterfactual -generator 𝐶 𝑔 decoder weight tying encoder weight tying genderclassifier 𝐶 𝑟 gen-grl 𝐶 𝑎 generator weight tying gen-grl 𝐶 𝑎 gen-grl 𝐶 𝑎 in most cases, words in v f and v m exist in pairs, so we denote ω as the set of feminine and masculine word pairs, such that (w f , w m ) ∈ ω. ## conclusions this work contributes to natural language processing society in two folds. for gender debiasing application, our model produces the debiased embeddings that has the most neutral gender latent information as well as the efficiently maintained semantics for the various nlp downstream tasks. for methodological modeling, cf-debias suggests a new method of disentangling the latent information of word embeddings with the gradient reversal layer and creating the counterfactual embeddings by exploiting the geometry of the embedding space. it should be noted that these types of latent modeling can be applied to diverse natural language tasks to control expressions on emotions, prejudices, ideologies, etc.