Detecting Independent Pronoun Bias with Partially-Synthetic Data Generation (2020)

Abstract: We report that state-of-the-art parsers consistently failed to identify “hers” and “theirs” as pronouns but identified the masculine equivalent “his”. We find that the same biases exist in recent language models like BERT. While some of the bias comes from known sources, like training data with gender imbalances, we find that the bias is _amplified_ in the language models and that linguistic differences between English pronouns that are not inherently biased can become biases in some machine learning models. We introduce a new technique for measuring bias in models, using Bayesian approximations to generate partially-synthetic data from the model itself.

## introduction we share a negative result for the natural language processing (nlp) community as a whole: for 20 years the major part-of-speech (pos) taggers and parsers missed that "hers" and "theirs" were pronouns, but it had gone unreported until this paper. this paper also shows that biases against "hers" and "theirs" are amplified in popular language models, predicted by the models with less frequency than expected given the training data. our solution for the parser problem is a new dataset with "hers" and "theirs" used in a syntactically diverse set of contexts, released in universal dependency format @xcite . for the language model problem, we introduce a novel use of bayesian modeling for sentence generation, in our case using it to detect bias by alternating pronouns in different contexts. the contexts are suggested by the model, avoiding the problems in measuring bias that come from rare or pathological data @xcite . we conclude that this is a general technique that can be used for measuring other types of bias and for text generation more broadly. while our contribution doesn't mitigate bias in language models, it improves the ability to detect and measure bias. we test on bert @xcite because it is the most widely used pretrained model for which all the training data was also available. we find that other masked language models are also amplifying the bias in their data, but we cannot measure how much that bias is amplified without access to the training data. where data for a pretrained model can't be shared, we encourage researchers to report on the biases in their models using the techniques in this paper. the preference for "his" in language models will bias any text generation system against "hers" or "theirs", a problem that has led developers to remove gendered pronouns entirely from applications including gmail's predictive text @xcite . identifying idps as pronouns is also a necessary step for co-reference resolution, although recent shared-tasks for pronoun resolution did not include idps @xcite . unfortunately, major academic and commercial parsers including those from aws and google @xcite wrongly labeled "hers" and "theirs" as adjectives or nouns. parsers from 20 years ago also missed these pronouns @xcite , confirming that this is not a new bias that only surfaced with more recent dependency parsers. the syntactic information was also typically wrong, for example, parsers that labeled "hers" as an "adjective" wrongly classified the syntactic relationships as modifiers. this will perpetuate bias in any subj obj dep ind feminine she her hers masculine he him his neutral they them their theirs 1st person i me my mine 2nd person you your yours because "hers" and "theirs" have only one sense, a small amount of training data can fix the problem for the syntactic parsers. we solved this with the dataset introduced in section 2. we recognize that more data would be needed to solve the problem for polysemous independent pronouns like "mine" and the singular/plural distinction for "theirs". ## process for detecting bias in this section, we share our method for partiallysynthetic data generation to measure bias in pretrained models (see the workflow in figure 1 ). the code is open-source and will reproduce the results in a single command: 3 step 1: train a contextual model. we use the pretrained bert @xcite english un- ## bias analysis figure 2 compares the model predictions to the training data which is wikipedia and bookcorpus. bert is trained on cross-entropy loss @xcite , so if a token is four times more frequent than another in the data and occurs in the same contexts, then softmax should converge on a 4:1 ratio in the predictive model. while there can be a lot of variation in models that will change the ratio of a given prediction, especially for rare and pathological cases @xcite , we avoid rare tokens by having the model generate the contexts and by testing across thousands of sentences. ## accepting the responsibility themself we found an additional problem when we first shared the "hers/theirs" problem one year before this paper: some parsers didn't always recognize "themself" as a pronoun. the systems with "themself" errors did not fix the problem when they fixed "hers/theirs". our linguistic understanding of (neo)pronouns and inclusive nlp development are areas of ongoing research @xcite @xcite @xcite . however, we argue that no system should have still missed "themself" after we alerted everyone to the "hers/theirs" error and recommended that every pronoun be investigated. english is one of the simplest languages in terms of the paradigms like those in we find that pretrained models amplify biases in the data because linguistic differences that are not biases can become biases in the models. this has significant implications for bias in tasks like co-reference resolution and text generation. from @xcite , it is not clear that debiasing the model itself would solve the problem. however, it might be possible to encode the data with the grammatical categories to mitigate some bias, for example, encoding the two "his" pronouns, like "his{dep}" and "his{ind}". a pretrained model would therefore treat the two forms separately, without one amplifying the other even when a language model can't capture the syntactic differences. we recommend that creators of widely used english syntactic parsers and part-of-speech taggers ensure that all unambiguous pronouns, including "hers", "theirs", and "themself", are correctly identified as pronouns. this will support applications the rely on technologies like conference resolution. we recommend that creators of language models include identity words as full tokens. bert's tokenizer includes all pronouns in this paper except "themself", thus exhibiting unintended featuredesign bias by needing to be constructed as "thems-elf" or "them-se-lf", presumably because all other forms with a "-self" suffix are already full tokens. that might include words other than pronouns, especially for multilingual models. we recommend that creators of language models use the methods introduced in this paper for partially-synthetic data generation to diagnose potential bias in their models and that this text generation strategy is explored for other applications.