Using Natural Sentence Prompts for Understanding Biases in Language Models (2022)

Abstract: Evaluation of biases in language models is often limited to synthetically generated datasets. This dependence traces back to the need of prompt-style dataset to trigger specific behaviors of language models. In this paper, we address this gap by creating a prompt dataset with respect to occupations collected from real-world natural sentences present in Wikipedia.We aim to understand the differences between using template-based prompts and natural sentence prompts when studying gender-occupation biases in language models. We find bias evaluations are very sensitiveto the design choices of template prompts, and we propose using natural sentence prompts as a way of more systematically using real-world sentences to move away from design decisions that may bias the results.

## introduction over the past couple of years, we witness tremendous advances of language models in solving various natural language processing (nlp) tasks. most of the time, these models were trained on large datasets, each model pushing the limits of the other. with this success came a dire need to interpret and analyze the behavior of neural nlp models @xcite . recently, many works have shown that language models are susceptible to biases present in the training dataset @xcite . with respect to gender biases, recent work explores the existence of internal biases in language models @xcite @xcite . previous work uses prefix template-based prompts to elicit language models to produce biased behaviors. although synthetic prompts can be crafted to generate desired continuations from the model, they are often too simple to mimic the nuances of natural sentence (ns) prompts. on the contrary, ns prompts are often more complex in structures but are not crafted to trigger desired set of continuations from the model. in this paper, we ask the question: can synthetic datasets accurately reflect the level of biases in language models? moreover, can we design an evaluation dataset based on natural sentence prompts? in this paper, we focus on studying the biases between occupation and gender for gpt2 models. we find that biases evaluation is extremely sensitive to different design choices when curating template prompts. we summarize our contributions as: ## dataset collection we collect a new prompt dataset fashioned from real-world sentences in wikipedia, which we refer to as natural sentence (ns) prompts. for each occupation type found in wikipedia's catalog * , we scrape the list of professions and the corresponding sentences featuring the profession in text from the wikipedia page. since our goal is to measure the biases associated with each profession, we ensure that the dataset contains sentences that can be used for probing and filter out the ones that do not have this feature. for example, the sentence "theatrical production management is a sub-division of stagecraft" is a general reference to the occupation rather than an individual, therefore we consider it an inadmissible sentence. we also remove professions that are gendered by definition, such as "actress". following this methodology, there are a total of 893 professions in the dataset to be annotated. ## biases evaluation in language modelsn evaluating biases in language models is a nontrivial task. in this section, we aim to understand the role of prompts in the context of gender bias. we probe gpt-2 models and draw comparisons between ns prompts (our dataset) and template prompts used in @xcite . @xcite showcases how language models perceive occupations in a biased view using template-based dataset. we wonder if this perception still holds in the ns prompt setting. for each prompt in our dataset, we compute the probability of generating pronouns "he" and "she" as continuations. more concretely, given a prompt x, we compute p(he|x) and p(she|x) respectively. do larger models amplify gender biases? with respect to our experiment, we note that this is not exactly the case. although the capacity of the model increases, is there a difference in using ns prompts versus template prompts? as evidently shown in do gender-occupation association account for most of the biases? one assumption behind the bigger discrepancy for template prompts is that the simple sentence structure could lead the model to ignore the context and blindly follow the default behavior. in this section, we re-evaluate the discrepancy of generating both pronouns, under different perturbations of the template prompts. the perturbations involve masking, deleting, or replacing the profession in each original template prompt. we compute the stereotypical bias as the difference in output probability between he and she, i.e., |p(he|x) -p(she|x)|. we list the input prompts after different perturbation rules as follows: this also leads to the question of whether the default behavior of the model is biased even without feeding in any prompts. is the default behavior already biased? if not prompted, would the model already assign a different probability for male and female pronouns? to verify this assumption, we @xmath0. in figure 1 , we plot the probability of different pronouns as the first generated word on a log scale. for all models, the probabilities for male pronouns (he/his/him) are the highest, followed by gender-neutral pronouns, and the female pronouns ## conclusion and future work in this work, we introduce a new prompt dataset and evaluate gender-occupation biases using both natural sentence prompts and compare them with template-based prompts. we found that evaluation of occupation-gender bias is highly sensitive to the words present in the prompt templates. we posit that natural prompt is a way of more systematically using real-world sentences to move away design decisions that may bias the evaluation results. we would like to point out the biases evaluation could be highly dependable on the perspective, and it would be risky to argue that one evaluation is more accurate than the other. for future work, it would be interesting to study the relationship between the size of the model and the gender biases. for example, in figure 1 , gpt2medium has a distinct behavior compared with other models. this raises the question of whether larger models are more diverse and less susceptible to biases. another interesting direction is to study whether we can remove the effect of inherited biases of models independently from prompts.