Measuring bias in Instruction-Following models with P - AT (2023)

Abstract: Instruction-Following Language Models (IFLMs) are promising and versatile tools for solving many downstream, information-seeking tasks. Given their success, there is an urgent need to have a shared resource to determine whether existing and new IFLMs are prone to produce biased language interactions. In this paper, we propose Prompt Association Test (P-AT): a new resource for testing the presence of social biases in IFLMs. P-AT stems from WEAT (Caliskan et al., 2017) and generalizes the notion of measuring social biases to IFLMs. Basically, we cast WEAT word tests in promptized classification tasks, and we associate a metric - the bias score. Our resource consists of 2310 prompts. We then experimented with several families of IFLMs discovering gender and race biases in all the analyzed models. We expect P-AT to be an important tool for quantifying bias across different dimensions and, therefore, for encouraging the creation of fairer IFLMs before their distortions have consequences in the real world.

## introduction instruction-following language models (iflms) @xcite @xcite @xcite are promising versatile tools for solving many downstream information-seeking tasks. the instruction-following approach is widely used in natural language processing to solve complex tasks @xcite , to train the language model to carry out prompted instructions and to have more natural answers @xcite . iflms are then generally large-scale language models (llms) based on transformers @xcite @xcite specifically trained or finetuned to follow instructions in natural language. the training is usually coupled with human feedback @xcite . transformer-based language models have demonstrated effectiveness across different tasks in natural language processing, and the same is happening for iflms. to be effective, iflms are typically trained on huge amounts of text from multiple sources, such as the internet. while these powerful language models select useful patterns to follow instructions, they also learn harmful and nuanced information and, thus, may produce biased language interactions. as iflms will play a crucial role in the future, there is an urgent need for shared resources to determine whether existing and new iflms are prone to produce biased, harmful language interactions. indeed, word and sentence embedders have already tests to determine their bias factor, word embedding association test (weat) @xcite and sentence encoder association test (seat) @xcite , respectively. these two tests are one the extension of the other and are based on the implicit association test (iat) @xcite that aims to measure bias in humans, such as inherent beliefs about someone based on their racial or gender identity. social prejudices have negative implications for certain social classes, e.g., candidates perceived as black based on their name are less likely to be called back at job interviews than their white counterparts @xcite . a specific test for determining the bias or, better, prejudice @xcite of instruction-following language models is still missing. in this paper, we propose the prompt association test (p-at) foot_0 : a new resource for testing the presence of social biases in instruction-following language models (iflms). p-at stems from weat @xcite generalizing the notion of measuring social biases in word embeddings to iflms. basically, we propose two components: (1) a series of prompts obtained by casting word tests proposed in weat in promptized classification tasks; (2) an associated set of metrics to quantify the bias. our resource consists of 2310 prompts. we then experimented with language models fine-tuned on instruction-following demonstrations, in particular, alpaca @xcite , vicuna @xcite , and flan-t5 @xcite . these experiments suggest the presence of gender and race biases in the analyzed models according to our new p-at. by testing different models in the flan-t5 family, we also observe a positive correlation between growth in model size and bias increase, as previously observed in other lms @xcite 2 background and related work inductive bias in machine learning generally refers to prior information that a model is given to observe a certain phenomenon. in the last decade, the large pre-training phase of foundation models is largely acknowledged as a form of prior information useful in a variety of tasks @xcite . however, some prior information derived from aspects of human language can also result in harmful behavior of machine learning algorithms. in particular, a model could inherit stereotyped associations between social groups and professions @xcite and specific emotions @xcite . in this scenario, the model bias is a form of prejudice towards certain social groups. we consider a model to be stereotypebiased if it consistently prefers stereotypes over anti-stereotypes; in this scenario we say that the model exhibits stereotypical bias. the presence of bias in nlp models was initially observed in static word embeddings. the word embedding association test (weat) @xcite is a statistical test based on the implicit association test (iat) @xcite , which measures the presence of bias in type-level word embeddings by comparing the similarities of different embeddings. weat compares the representation of words belonging to a target category (e.g., male or female names) and the representation of attributes (e.g. careers and families) and aims to detect systematic association between target categories and attributes. weat is composed of ten tests that examine different target categories, each corresponding to a certain social group. it results in a test able to detect bias in type-level words embedding in the direction of gender, race, and age. after contextualized word embeddings were introduced, the sentence encoder association test (seat) @xcite was proposed as an extension of weat. seat measures bias by comparing encoded sentences rather than embedding words, giving a context to each target and attribute word. the context is designed to be as uninformative as possible. each word in weat will correspond to several sentences in seat, such as "this @xmath0, "<@xmath1. @xcite showed the presence of biases in pre-trained language models and their contextual word embeddings such as gpt-2 @xcite , elmo @xcite and bert @xcite . however, similarity-based methods give mixed results @xcite in quantifying biases in contextualized word embeddings. for this reason, the bias present in models has recently been assessed by testing the language model mechanism itself rather than the contextualized representations. in stereoset @xcite , and crowspairs @xcite , the probability of a word, identifying a certain social group is measured both in stereotyped and in antistereotyped context. while the definition of benchmarks and bias measures itself remains challenging @xcite @xcite , these analyses help to detect that models such as gpt-2 @xcite , roberta @xcite and bert @xcite are more likely to predict stereotyped association. in recent years, lms have become larger and more powerful: large language models (llms) such as gpt-3 (brown et al., 2020), bloom @xcite , t5 @xcite and @xcite have shown impressive performance in various natural language processing (nlp) tasks. furthermore, these models can be fine-tuned to follow human instruction and solve a number of tasks, both using humanannotated prompts and feedback @xcite and datasets augmented with instructions @xcite . these novel models are named instruction-following models, and a large num-ber of them are already available such as alpaca @xcite , flan-t5 @xcite , bloomz @xcite and vicuna @xcite . despite the increasing capabilities of this family of models, the underlying llms generate toxic or offensive content @xcite , and reproduce biases that exist in the training data @xcite @xcite : it is not clear if and to what extent instruction-following models can exhibit the same problematic behavior. despite their flexibility being potentially beneficial to a wide group of users, iflms can have a negative impact on society: when used to process resumes, for example, a biased model may prefer a candidate based on ethnicity. similarly, some gender stereotypes -which have been strongly associated with salary and professional prestige @xcite -may be reinforced by these models, as already happens with others @xcite . while some previous work quantifies the model's ability to cause no harm by interviewing human evaluators @xcite , it is necessary to develop automated approaches to easily test models before they are available to a large group of users. ## experiments we propose p-at, a resource with the aim of evaluating the presence of bias in instruction following language models (iflms) consisting of two components: (1) a dataset with explicit instructions and (2) a metric for evaluating the output bias of the iflm chosen. the rest of this section firstly describes the experimental set-up, and then the quantitative experimental results that discusses how the bias is captured in different iflms by prompting them with p-at. the bias in models is measured by the previously introduced p-at bias score. statistically significant bias presence is assessed with the usage of a fisher's exact test for contingency tables. moreover, we check whether the models seem to solve the proposed task with the entropy measure. ## conclusions in this paper, we propose instruction-following association test (p-at), a novel resource to quantifying bias of instruction-following language models (iflms) in multiple directions. p-at consists of a dataset inspired to weat @xcite and a set of metrics to capture and measure the degree of the social bias of iflms output. our experiments with different models show that, when a model seems to effectively solve the binary classification task is prompted with, it also tends to be more biased, producing stereotyped association more often than anti-stereotyped one. in particular, p-at detects bias in alpaca and flan-t5-xxl both in gender and race dimension. we expect p-at to be an important tool for quantifying the presence of bias across different dimensions and, therefore, for encouraging the creation of fairer instruction-following lms before the distortions of these models have consequences in the real world.