Learning Bias-reduced Word Embeddings Using Dictionary Definitions (2022)

Abstract: Pre-trained word embeddings, such as GloVe, have shown undesirable gender, racial, and religious biases. To address this problem, we propose DD-GloVe, a train-time debiasing algorithm to learn word embeddings by leveraging \underline{d} ictionary \underline{d} efinitions. We introduce dictionary-guided loss functions that encourage word embeddings to be similar to their relatively neutral dictionary definition representations. Existing debiasing algorithms typically need a pre-compiled list of seed words to represent the bias direction, along which biased information gets removed. Producing this list involves subjective decisions and it might be difficult to obtain for some types of biases. We automate the process of finding seed words: our algorithm starts from a single pair of initial seed words and automatically finds more words whose definitions display similar attributes traits. We demonstrate the effectiveness of our approach with benchmark evaluations and empirical analyses. Our code is available at https://github.com/haozhe-an/DD-GloVe .

## introduction word embeddings can meaningfully capture semantic and syntactic similarities between words. popular embeddings are word2vec @xcite , glove @xcite , and fast-text @xcite . although contextual word embeddings, like bert embeddings @xcite and elmo @xcite , gain increasing popularity, some recent research keeps using static word embeddings as input to their state-of-the-art algorithms in downstream natural language processing and computer vision applications @xcite . despite the effectiveness of word embeddings, biases in them reflect undesirable association between some concepts. @xcite first identify that the distance between --→ man and * work done during an internship at apple. figure 1: definitions of example gender-specific and gender-biased words. gender-specific words typically contain gendered words in their definitions, whereas gender-biased words tend to have neutral definitions. our contributions leveraging the advantages of dictionary definitions, we propose dd-glove, a train-time debiasing algorithm to learn biasreduced glove word embeddings. in summary, we make the following contributions: ## motivations we analyze the limitations in current debiasing algorithms for word embeddings and present our corresponding solutions. debiasing algorithms existing mainstream gender-debiasing algorithms are projection-based post-processing @xcite . they need a list of manually selected words (e.g. "she" and "he", "girl" and "boy", "woman" and "man") to compute a gender direction in the word embedding space. they then project the pre-trained word embeddings onto the gender direction and remove the vector component living in this direction. the resultant word vectors preserve useful semantic meanings but contain less gender information. however, these algorithms do not consider the possible usage of additional knowledge like dictionary definitions. furthermore, there is a limitation in this projective post-processing approach. the manually compiled list to approximate the bias direction might be difficult to obtain for other types of biases. it would be helpful to find an alternative that involves less human labor. our approach: using dictionary definitions using dictionary definitions to train bias-reduced word embeddings could address the above limitation and gives us additional advantages. existing dictionary debiasing algorithm a recent work makes the first attempt to debias word embeddings using dictionary definitions via postprocessing @xcite . they compute a weighted average of pre-trained word vectors as the definition embeddings. they assume these definition embeddings are the "neutral" reference points for word embeddings. however, this is a major flawed assumption in post-processing debiasing. due to the biases in pre-trained word vectors, the definition embeddings also contain biases. partially owing to this flawed assumption, their resultant embeddings show limited effectiveness in several benchmark evaluations like the word embedding association test @xcite . our approach: training from scratch training from scratch addresses the problem of biased definition embeddings computed from pre-trained, biased word vectors. as word embeddings are initialized randomly, they contain virtually no biases. correspondingly, the definition embeddings obtained at this point will contain minimal biases. as training proceeds, the debiasing algorithm can continuously apply corrections, so as to learn distributional semantics and reduce biased information simultaneously. in sec. 5.1, we empirically demonstrate that training from scratch could produce substantially more neutral definition embeddings that lead to improved debiasing. ## dd-glove we propose four dictionary-guided loss functions, namely (1) orthogonal loss, which mitigates general biases by diminishing the redundant component in word vectors that disagree with their defi-nition embeddings, (2) projection loss, which directly reduces a specific type of bias by minimizing the difference between word vectors' projection and definitions' projection onto the bias direction, (3) definition loss, which injects semantic meanings from definitions into word embeddings, and (4) bias-aware glove loss, which dynamically adjusts weights of co-occurrences for bias reduction. in addition, we introduce a novel algorithm that automatically searches seed words for bias direction approximation with only one pair of initial seed words as the input. notations we use @xmath0. we overload the symbol w to represent a word in some contexts. s(w) denotes the definition embedding of word w. a word can have multiple definitions in a dictionary. since glove does not distinguish word meanings, we choose to use all available definitions for w when computing s(w). previous works compute definition embeddings by smoothed inverse frequency @xcite . we propose a simpler but empirically effective method that averages the definitional words. therefore, our definition embedding is ## experiments we present two settings for dd-glove. (1) in dd-glove gender , we mainly mitigate gender bias, thus using "she" and "he" as the initial seed words. (2) dd-glove race , we focus on reducing racial bias. the initial seed words are "black" and "white". for each word in the vocabulary of glove, we try to find its definitions from the oxford online dictionary. if the word has multiple definitions, we simply concatenate them into one definition. stopwords are removed for pre-processing. we average the definitional words to obtain s(w) by following eqn. 1. words that are not present in the oxford dictionary are skipped. in total, we have 92,140 words with definitions. we run glove @xcite , double hard debias (dhd) @xcite , dictionarybased debiasing (dict debias) @xcite , and gn-glove @xcite as algorithm 1 find seed words automatically and approximate the bias direction input: initial seed words (v 1 , v 2 ), desired total number of seed words n for each attribute output: two sets of seed words q a 1 , q a 2 , the approximated bias direction r ← r ∪ {(w, r(w))} end for ▷ find top n most attribute-specific words and approximate the bias direction. r sorted ← sort r by r(w) in descending order for @xmath1, 2, . . . , n } do ## conclusion in this paper, we propose dd-glove, a train-time debiasing algorithm to learn word embeddings leveraging dictionary definitions. we achieve effective debiasing results while preserving semantic meanings. the bias direction in dd-glove is automatically approximated using our dictionaryguided algorithm given a single pair of initial seed words. our current implementation is based on glove, but the idea of using dictionary definitions to mitigate biases can be generalized to other word embeddings since our dictionary-guided losses are orthogonal to word embedding objectives. it is also likely that incorporating dictionary definitions can alleviate biases in contextualized word embeddings. this is out of the scope of this paper and remains an open research problem.