On Measuring Social Biases in Prompt-Based Multi-Task Learning (2022)

Abstract: Large language models trained on a mixture of NLP tasks that are converted into a text-to-text format using prompts, can generalize into novel forms of language and handle novel tasks. A large body of work within prompt engineering attempts to understand the effects of input forms and prompts in achieving superior performance. We consider an alternative measure and inquire whether the way in which an input is encoded affects social biases promoted in outputs. In this paper, we study T0, a large-scale multi-task text-to-text language model trained using prompt-based learning. We consider two different forms of semantically equivalent inputs: question-answer format and premise-hypothesis format. We use an existing bias benchmark for the former BBQ and create the first bias benchmark in natural language inference BBNLI with hand-written hypotheses while also converting each benchmark into the other form. The results on two benchmarks suggest that given two different formulations of essentially the same input, T0 conspicuously acts more biased in question answering form, which is seen during training, compared to premise-hypothesis form which is unlike its training examples. Code and data are released under https://github.com/feyzaakyurek/bbnli .

## introduction the use of pretrained language models through the canonical "pretrain, fine-tune" scheme for transfer learning gave way to a new paradigm called promptbased learning @xcite where text-based nlp problems are posed in a format that is similar to pretraining tasks. as an example, the translation task is formulated using the prompt translate 1 warning: this paper and the released dataset contain content that may be offensive or upsetting. english to german: <source @xmath0. while some self-supervised language models such as @xcite can handle prompts of this kind, @xcite demonstrated that following the pretraining stage with supervised learning where inputs are formulated as task-specific prompts further improved generalizability. @xcite scaled this idea by employing many datasets across multiple tasks and numerous prompts per task, achieving state-ofthe-art results in a wide range of nlp problems. they collect a large set of prompts for each of the 62 datasets across 12 tasks and fine-tune t5 @xcite on a subset of these tasks using prompts, holding out some of the tasks for zeroshot testing (fig. 1 ). with the power of added supervision and use of diverse prompts, t0 facilitates generalization into novel tasks such as natural language inference (nli)-the task of testing the semantic concepts of entailment and contradiction @xcite . in prompt learning, some prompts work significantly better than others @xcite suggesting that the model behavior is highly susceptible to prompt design and the form in which the input is presented @xcite . however, limited work has been done on how different formulations of semantically the same input affect models' behavior beyond known performance metrics such as social biases similar to those studied by @xcite ; @xcite and @xcite . hence, in this paper, we test whether the form in which a problem is encoded influences language model bias, independent of the content. we consider t0 @xcite given its open-sourced nature and competitive performance to flan @xcite despite its relatively smaller size (11b vs 137b). we use four datasets in our analysis: an existing bias benchmark in figure 1 : we prompt t0pp using an example from bbnli dataset in two different forms of semantically the same input. "read and extract" (created by in @xcite for quoref @xcite and "does this imply" (anli @xcite ) prompt templates (non-italicized in the above inputs) are used for qa and nli, respectively. every example in bbnli comes in pro-and anti-stereotypical versions for every form. based on the outputs, we compute the bias score. this paper thus offers two main contributions: we provide the first comparative analysis of biases exhibited by multitask language models when performing different tasks; second we release a bias benchmark for natural language inference bbnli, the first bias benchmark with hand-written hypotheses to our knowledge. bbnli measures human cognitive biases across three different domains and capturing 16 harmful stereotypes. in fig. 1 , we provide examples from our bbnli benchmark and the predictions by t0 model given two different forms of semantically the same input. ## datasets and evaluation in this section we lay out the four datasets used for a fair comparison across qa and nli forms: one existing benchmark bbq @xcite , @xmath1. ## experiments and results in all of our experiments we use the bestperforming checkpoint of t0-suite called t0pp 2 . we use 3 prompt templates 3 both for qa and nli (see bbnli in our proposed dataset bbnli, we assess various kinds of stereotypes (called subtopics) across three domains. each subtopic comes with multiple set of premises and stereotypical hypotheses all of which can be paired in forming an nli query. similar to ambiguous examples in bbq amb , the preferred answer to stereotypical hypotheses in this dataset is exclusively neutral. however, given the subtlety of hypotheses, t0 fails to remain abstained and almost always makes a choice between target and non-target groups in qa and answers with entailment or contradiction for nli forms. in bbq bbq contains two formats: ambiguous (fig. 2a ) and disambiguated (fig. 2b ). we convert the same set of examples into nli form as demonstrated in fig. 2c-d , yielding @xmath2. when the model is prompted in different ways, predictions for semantically identical examples yield vastly different distributions. similar to the case of bbnli, t0 fails to answer with neutral/unknown and points at one of the target or non-target options for the mentioned behavior (e.g. planting a bomb). in in ## analysis is nli less biased because it outputs random answers? in order to assess effectiveness of t0 to handle the premises in bbnli, we use our test hypotheses in what other factors contribute to biased answers? in fig. 3 , we observe that predictions are affected by (1) the order of names (e.g. "one muslim man and one christian man" vs "one christian man and one muslim man") as they appear in the input, also suggested by @xcite , (2) the length of the premise/context, and/or (3) details provided in the context/premise. first, we observe that in all three formats (examples shown in what causes a training input form to result in more bias than a novel form? it is highly likely that the question answering datasets (a few dozens were used in training t0) contain biases @xcite which makes it easy for t0 to exhibit stereotypical associations learned during the training when faced with this task form. in this familiar form, the model is also more likely to rely on spurious correlations when providing answers rather than generating a correct answer (e.g. "unknown"). within the scope of this paper, we argue that such associations cannot be consistently prevented by simply using novel prompt templates, however, more substantial changes such as the ones presented in @xmath4. how can the ideas presented in this study be bootstrapped from in debiasing efforts? within the scope of the problems and models considered in this work, we observe that presenting an input to a language model in a novel form results in less biased predictions. while we cannot control user-created queries in client-facing applications, we have control on the training data we use in developing our models. hence for future work, one idea that is worth testing in multi-task learning is whether limiting the set of training tasks to those that are not immediately interesting to layperson and holding out "popular" tasks for testing would result in less biased predictions in popular tasks such as question answering. in order to obtain a strong task-specific model to tackle various nlp tasks, the de facto practice has been to use a pretrained language model and fine-tune it on a downstream task @xcite @xcite . we call these specific checkpoints of language models tailored for a particular downstream task task-conditioned lms and non-conditioned versions general-purpose lms. previous work established that both types of models exhibit social biases @xcite . in the following parts, we discuss efforts aiming at systemically quantifying these biases in lms. measuring bias in task-conditioned language models several benchmarks and metrics have been proposed to measure bias in coreference resolution @xcite ), text generation @xcite @xcite )-or more specifically story completion @xcite , abusive language detection @xcite , sentiment analysis @xcite and for the tasks of interest to this work: question answering @xcite and natural language inference @xcite @xcite . these works take a step forward in bridging the gap between how biases are measured and what the model is actually been trained on and used for @xcite . measuring bias in general-purpose or multitask language models crows-pairs @xcite is a collection pairs of sequences which differ only by a single word such that one sequence is stereotypical and the other anti-stereotypical. crows-pairs can be used for measuring biases trained with the masked language modeling objective. @xcite presents an interesting self-diagnosis approach fit for both masked language modeling-style and autoregressive lms. techniques used for autoregressive lms often intersect with those used in measuring bias in text generation, described above. further, it is common to introduce a set of simple prompts such as "she works as" vs "he works as" and measure sentiment, regard @xcite or other metrics based on word occurrences @xcite . ## conclusion in this paper, we have tested whether the form in which a problem is encoded influences language model bias, independent of the content. our results highlight that in the cases while performance is not affected, biases vary significantly across different forms of the semantically same input. having demonstrated that it is extremely difficult for models like t0 to consistently escape logical fallacies and cognitive biases, alternative input formulations to those appeared in training may be used to alleviate biases without much sacrifice on performance. our results hint at how zero-shot generalization may provide some hopeful representation toward minimizing harm and bias in these large-scale language models. further, our bbnli dataset is designed to integrate detailed stereotypes and more complex logical statements that will be crucial to the accelerating advancements in natural language inference problem and measuring biases in multitask systems, more broadly. anticipated risks while this study is intended to shed a more nuanced and context-sensitive light toward various social biases in t0 as measured using two benchmarks, a potential risk lies in the models, tasks, prompt templates, domains and subtopics we were not able to exhaustively include. in bbnli, although we did our best to approach the top stereotypes and biases that appear in real-life, we were not able to include every ethnicity, gender, and religious point of view. given these limitations, the risk of using our benchmark could be that the model will show biases in social-cultural categories we did not account for. additionally, with the added complexity of skip-logic embedded within the premise and hypotheses, there may be some outputs that produce unexpected, unrelated biases that were not explicitly determined. moreover, the stereotypical hypotheses we devised are harmful social biases that have real-life consequences to certain groups of people. further, out intent is to address these highly problematic statements as clearly as possible to understand model biases. however, when these hypotheses are taken out of context and interpreted at face-value, they can cause serious damage to what a model might output or create misunderstanding of our study's purpose. lastly, we acknowledge that as human researchers ourselves, we are prone to exuding biases that we have accumulated from our personal environments. as such, this work should be expanded upon by future works and more importantly, our bias dataset can be strengthened through increased collaborative efforts with scholars from the social sciences and humanities.