Unsupervised Discovery of Implicit Gender Bias (2020)

Abstract: Despite their prevalence in society, social biases are difficult to identify, primarily because human judgements in this domain can be unreliable. We take an unsupervised approach to identifying gender bias against women at a comment level and present a model that can surface text likely to contain bias. Our main challenge is forcing the model to focus on signs of implicit bias, rather than other artifacts in the data. Thus, our methodology involves reducing the influence of confounds through propensity matching and adversarial learning. Our analysis shows how biased comments directed towards female politicians contain mixed criticisms, while comments directed towards other female public figures focus on appearance and sexualization. Ultimately, our work offers a way to capture subtle biases in various domains without relying on subjective human judgements.

## introduction despite widespread documentation of the negative impacts of bias, stereotypes, and prejudice @xcite @xcite @xcite , these concepts remain difficult to define and identify, especially for non-experts. social biases appear to be a natural component of human cognition that allow people to make judgments efficiently @xcite . as a result, they are often implicit-people are unaware of their own biases @xcite -and manifest subtly, e.g., as microaggressions or condescension @xcite . much nlp literature has examined biases in data, algorithms, or model performance, and the negative pipeline between them: models absorb and amplify data biases, which impacts performance @xcite . however, little work has looked further up the pipeline and relied on the assumption that biases in data originate in human cognition. in contrast, this assumption motivates our work: an unsupervised approach to detecting implicit gender bias in text. text provides an ideal avenue for studying bias, because human cognition is closely tied to natural language. psychology studies often examine human perceptions through word associations @xcite . however, the implicit nature of bias suggests that human annotations for bias detection may not be reliable, which motivates an unsupervised approach. the goals of our work align with prior work in nlp that has examined biases in real-world data. however, prior work examines bias at a broad corpus level or relies on supervised models. while corpus-level analyses, e.g. associations between gendered words and stereotypes, can be insightful @xcite @xcite @xcite , they are difficult to interpret over short text spans. they also often rely on human-defined "known" stereotypes, such as lists of traditionally male and female occupations obtained through crowd-sourcing, which restricts analysis to a narrow surface-level domain. similarly, supervised approaches can provide insight into carefully defined types of bias @xcite @xcite ), but they rely on human annotations tasks, which are difficult to design or generalize to other domains, especially because social concepts differ across contexts and cultures @xcite . our work offers a new approach to surfacing gender bias that does not require direct supervision and is meaningful at a sentence or paragraph level. we create a model that takes text in the 2 nd -person perspective as input and predicts the gender of the person the text is addressed to. if the classifier predicts the gender of the addressee with high confidence based only on the text directed to them, we figure 1 : we train a classifier to predict the gender (w gen) of the person that text is addressed to (com txt), while demoting features that are predictive of gender but not predictive of bias. posts with similar content are matched through propensity scores; unmatched posts are discarded. latent traits of the addressee (e.g., nationality) are demoted through an adversarial objective. overtly gendered language ("bro") is substituted. comments indicative of gender despite these restrictions are likely to contain bias. in §2, we define the problem and intuition behind our approach. we describe our methods for confound demotion in §3, and we evaluate them in §5. our evaluation involves examining how confound control affects performance on in-domain and out-of-domain classification tasks, including detection of gender-based microaggressions. our results suggest that our model successfully identifies text likely to contain bias against women, allowing us to analyze how this bias differs across domains ( §6). to the best of our knowledge, this is the first work that aims to analyze bias in short text spans by learning implicit associations from data sets. ## problem formulation our primary task is to detect gender bias in a communicative domain, specifically in texts targeting an addressee (i.e., 2 nd -person) without relying on explicit bias annotations. our goals align with a causality framework in that we seek to identify content that occurs because of the gender of the addressee rather than other factors. we can define a counterfactual: would the addressee have received different text if their gender were different? while our framework is broadly applicable, in order to define consistent notation, we consider a setup where our primary text is a comment written in reply to text written by someone else. this includes domains like replies on social media posts, or comments on newspaper articles, and can be generalized other media, e.g., comments on youtube videos. we identify the following variables: for example, in figure 1 , when the comment "ur hot!" (com txt) is addressed to someone who said "i love tennis!" (o txt), it is an objectification and unsolicited reference to appearance, which could indicate bias. however, when it is addressed to someone who said "do i look ok?", it is likely not indicative of bias. if women ask "do i look ok?" more frequently than men, this naive classifier would identify "ur hot!" is likely addressed towards a woman and identify it as biased. however, we only want the model to learn that references to appearance are indicative of gender if they occur in unsolicited contexts. thus our model needs to account for the effects of o txt: because of correlations between w gen and o txt, com txt values may contain features that are predictive of w gen, but are caused by o txt, rather than by w gen. we face a similar problem with w traits. from the synthetic example in figure 1 , if our data set contains more men from canada than women, the model might learn that references to canada indicate w @xmath0. we provide additional empirical examples in §4. we refer to factors that might influence com txt as confounding variables and the artifacts that they produce in com txt as confounds. we distinguish two types: observed and latent. latent confounding variables cannot be controlled if they are entirely unknown; instead, we assume there are observed signals that can be used to infer them, but the values themselves are difficult to explicitly enumerate. in addition to confounds introduced by o txt and w traits, com txt may also contain overt signals, e.g. titles like "ma'am" or "sir", that are predictive of gender, but not indicative of bias. we thus identify 3 factors to account for: o txt, w traits, and overt signals. ## methodology our overall methodology centers on creating a classifier that predicts gender of the addressee while controlling for the effects of observed confounding variables (o txt), latent confounding variables (w traits), and overt signals. the input to the prediction model is com txt, while the output is w gen, and we aim to identify bias in com txt. ## experimental setup our primary data is the facebook subsection of the rtgender corpus @xcite . the data contains two subsections: politicians (400k posts and 13.9m replies addressed to 412 then-current u.s. members of congress), and public figures (118k posts and 10.7m replies addressed to 105 famous people such as actresses and tennis players). we can show that o txt is a confounding variable by computing log-odds scores between the words in o txt and w gen @xcite . in the politicians data, the most female-associated words are women, congresswoman, sexual, and assault. the most male-associated words are obamacare, iran, epa, and spending. it is evident that male and female politicians post about different topics, e.g., female politicians likely post more about sexual assault. a naive model may predict that comments using sexual language are addressed towards women, but increased sexual language may occur because of o txt, rather than gender bias. a similar problem occurs with w traits, e.g., the corpus has more comments addressed to female tennis players (9 players; 184k comments) than male players (1 player; 29k comments). the model can obtain high accuracy by predicting w @xmath1. unlike o txt, which is observable from the data, we have no way of enumerating every possible value in w traits. even if we could enumerate them, we do not expect propensity matching over w traits to work, because we cannot find reasonable matches, e.g., there is only one senior senator from massachusetts. additionally, w traits can be as fine-grained as names: we cannot find a male senator whom commenters call "liz warren". we divide each data set into train, dev, and test sets, enforcing no ow overlap between subsets. we perform propensity matching and derive the confound vectors to demote using only the training data. we apply word substitutions to all subsets. 2 ## evaluation we train our model to predict w gen from com txt, employing propensity matching over o txt, word substitutions over com txt, and w traits demotion. we focus on evaluating how well our model controls for confounds and whether or not it captures gendered language. successful demotion of confounds would suggest that our model learns to identify text indicative of gender bias. ## related work our work differs from prior work on bias detection in nlp in that we infer bias from data in an unsupervised way, whereas prior work relies on crowd-sourced annotations @xcite @xcite . this work typically focuses on specific types of bias, such as condescension @xcite or microaggressions @xcite and involves carefully constructed annotations schemes that are difficult to generalize to other data sets or types of bias. in contrast, our unsupervised approach is not limited to any particular domain and does not rely on human annotations, which can be subjective. less-supervised approaches focus on corpuslevel analyses, such as associations between gendered terms and occupational stereotypes @xcite @xcite @xcite @xcite . methodologies for identifying gender-related differences in text have varied, including word-embedding similarity @xcite , language model perplexity @xcite , and predictive words identified by logistic regression @xcite . these metrics are meaningful over a corpus-level, but are often difficult to interpret over short text spans. additionally, none of these methods focus on controlling for confounds. while matching is a well-established method for controlling for confounding variables in causality literature @xcite @xcite , considerably less work has drawn this methodology into nlp. most work takes one of two approaches. in the first scenario, text maybe be a confounding variable that needs to be controlled in order to measure the effect of a non-text variable @xcite . for example, @xcite examine whether or not papers written by male authors are cited more than ones by female authors, while controlling for the content of the paper. roberts et al. (2020) also offer a specific method for matching text, which relies on the output of a topic model. in this work, we use the output of an lstm, which is generally more appropriate for short text, does not make the simplifying bow assumption, and scales well to large data sets. in the second scenario, it may be desirable to control for non-text confounds before analyzing text. @xcite use matching to identify similar users on reddit before comparing the content that they post. our work requires both of these perspectives, as the variable we control for (o txt) and the outcome we analyze (com txt) are both text. @xcite do consider a similar setting where text is both an outcome and a confound. while their goals differ greatly from ours, our framework is generally consistent with their recommendations. @xcite provide a more complete overview of using text to reduce the influence of confounding variables. ## limitations and future work while our work serves as an initial approach toward unsupervised detection of comment-level gender bias, we identify several limitations and areas for future work. we first focus on limitations within our proposed framework. first, while our results in §5 suggest that adversarial training does help reduce the influence of latent confounding variables, the analysis in §6 suggests that there is scope for improvement. furthermore, while we focus on some confounds in the data, there may be additional ones that our model does not account for, such as the impact of videos, photos, or links shared with o txt. similarly, while our model uses o txt for propensity matching in the training data, thus encouraging the model to encode indicators of bias, a model to classify comments as biased or unbiased should also incorporate o txt when assessing test data. additionally, we assume that all comments are directly addressed to ow, but some comments may be addressed to other commenters. finally, our assumption that human judgements are not reliable for this task makes evaluation difficult, and this task would benefit from the development of additional evaluation metrics. there are additional avenues for future work beyond our proposed framework. notably, we focus on the perspective of ow and examine what bias social media users may be exposed to, i.e. what comments men and women might expect to receive in response to their posts. we do not examine why comments addressed toward men and women may differ, whether because the same commenters write different comments to men and women, or because men and women attract comments from different types of people. this perspective would require controlling for traits of the commenter, such as gender, age, and occupation. nevertheless, our work stands without this perspective: biased comments are harmful to the recipient, regardless of who wrote them. ## conclusions bias detection is useful for fostering civil communication on social media, as it allows recipients to screen out biased comments. further, our intention is to detect implicit bias that people may not know they have -revealing these biases to social media users could proactively prevent them from posting unintentionally biased comments. detecting and analyzing bias is a first step towards mitigating it, and we hope our work will encourage future work in this area.