Analyzing the Limits of Self-Supervision in Handling Bias in Language (2022)

Abstract: Prompting inputs with natural language task descriptions has emerged as a popular mechanism to elicit reasonably accurate outputs from large-scale generative language models with little to no in-context supervision. This also helps gain insight into how well language models capture the semantics of a wide range of downstream tasks purely from self-supervised pre-training on massive corpora of unlabeled text. Such models have naturally also been exposed to a lot of undesirable content like racist and sexist language and there is only some work on awareness of models along these dimensions. In this paper, we define and comprehensively evaluate how well such language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. We define three broad classes of task descriptions for these tasks: statement, question, and completion, with numerous lexical variants within each class. We study the efficacy of prompting for each task using these classes and the null task description across several decoding methods and few-shot examples. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation. We believe our work is an important step towards unbiased language models by quantifying the limits of current self-supervision objectives at accomplishing such sociologically challenging tasks.

## introduction transformer-based language models @xcite , pre-trained using self-supervision on unlabeled textual corpora, have become ubiquitous figure 1 : four tasks defined using natural language task descriptions for bias: diagnosis, identification, extraction & rephrasing, and performed by prompting a self-supervised generative language model. @xcite in natural language processing (nlp) due to their general applicability and compelling performance across a wide spectrum of natural language tasks, ranging from machine translation @xcite to question answering @xcite and dialogue @xcite . to further improve these models' ability to generalize, there has also been interest and success @xcite @xcite in scaling them to billions of parameters and terabytes of unlabeled data. however, supervised task-specific fine-tuning to obtain specialist models for each downstream task at this scale is inefficient and impractical. in-context prompting with natural language task descriptions has been demonstrated @xcite @xcite to be an interpretable, general-purpose technique to query these "foundation" models @xcite to solve several downstream tasks with reasonably high accuracy. while parameter-efficient techniques such as soft-prompt tuning @xcite and adapter fine-tuning @xcite have been designed to avoid fine-tuning full specialist models for each task, the original interpretable prompting paradigm can be seen as a mechanism to validate the efficacy of current selfsupervision techniques in capturing the semantics of various downstream tasks from unlabeled data. on the other hand, recent studies @xcite @xcite demonstrate that selfsupervised language models have learned inaccurate and disturbing biases such as racism and sexism against a variety of groups from the webscale unlabeled data that they were pre-trained on. hence, a critical first step toward making these "foundation" models aware and adept at handling bias is quantifying how weak/strong their foundations really are for such complex sociological tasks. in this paper, we take this step and use the natural language task-prompting paradigm to analyze how well self-supervision captures the semantics of the downstream tasks of bias diagnosis (is there bias in a piece of text?), identification (what types of bias exist?), extraction (what parts of the text are biased?) and rephrasing (rephrase biased language to remove bias). the tasks are illustrated in figure 1 with example natural language task descriptions. we define broad classes of natural language task descriptions: statement, question and completion, for the aforementioned tasks, and construct numerous lexical variants per class. we study the efficacy of prompting for each task using these classes and the null task description across several fewshot examples and decoding methods. our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation. we observe that performance on the coarse-grained bias diagnosis task is poor, achieving only 42.1% accuracy in the zeroshot setting. although we observe improvements with in-context supervision, the best performance is only slightly above random chance. we observe that fine-grained bias identification generally benefits from non-null task descriptions and few-shot examples. we also observe large disparities in performance across different bias dimensions (differences as large as 75% for exact match in a zeroshot setting), indicating a skew in internal model biases across dimensions. qualitative analysis also shows that the phrasing of a task description can have an outsized impact on accuracy of identification. we observe that bias extraction performs best with a span-based decoding strategy compared to alternatives such as unconstrained decoding. we collect our own crowdsourced annotations for bias rephrasing, with several rounds of data verification and refinement to ensure quality. we find that models generally perform poorly on this task, but that larger model size does improve performance. overall, our work indicates that self-supervised autoregressive language models are largely challenged by tasks intended to diagnose, identify, extract and rephrase bias in language when prompted with a comprehensive set of task descriptions. ## related work there has been a large body of work focused on defining and measuring social bias during natural language generation @xcite @xcite , neural toxic degeneration of language models using prompts @xcite , understanding social bias implications @xcite , and various bias mitigation strategies @xcite @xcite @xcite . recently, @xcite proposed new benchmarks and metrics to measure representational biases in text. @xcite proposed powertransformer, a language model trained with auxiliary objectives such as paraphrasing and reconstruction, and propose bias-controlled generation for rephrasing. several datasets have also been released for measuring and rephrasing social bias. @xcite introduced a dataset with crowdsourced stereotype pairs across different kinds of bias and @xcite released a large test set of online comments annotated for unintended bias. more recently, @xcite released a dataset annotated with bias labels and spans of biased text in language. brown et al. (2020) introduced gpt-3 and demonstrated that in-context few-shot learning with and without natural language task descriptions could yield close to state-of-the-art fine-tuning results for several nlp tasks. this was followed by several studies exploring language models with task descriptions and in-context examples @xcite . there is also work that discusses limitations of this approach: efrat and levy (2020) discovered that models perform poorly with task descriptions on both simple and more complex tasks and @xcite found that models do not understand the meaning of task descriptions for natural language inference and are sensitive to the choice of language model verbalizers. our work is inspired by self-diagnosis proposed by @xcite , wherein a language model is prompted to generatively predict whether or not a given piece of text contains a specific bias attribute such as a threat or sexually explicit language. the task description itself contains the bias attribute, which is derived from the perspective api 1 . while they find that this generative binary prediction approach works fairly well, it comes with a drawback that diagnosing the mere presence of bias requires pre-defining all bias attributes and prompting the language model as many times as the number of bias attributes. in contrast, our work takes the approach of decoupling self-diagnosis into diagnosis of the presence or absence of bias and finegrained generative identification of the dimension in which a piece of text is biased. we adopt and study this approach to better understand how well auto-regressive language models pre-trained using self-supervision separately capture the semantics of diagnosis and identification. our work is also broader in scope than @xcite since we also consider the tasks of bias extraction and rephrasing along with several broad classes and lexical variants of task descriptions, few-shot example sampling and decoding strategies. ## methods let m denote an auto-regressive language model with a byte-pair encoding tokenizer t m and p m (w|w 1 , . . . , w k ) denote the probability that m assigns to w being the next token, given previous tokens w 1 , . . . , w k . let x denote a document containing a sequence of k tokens. x can be unbiased or contain a biased sub-sequence x s:e defined by start and end token indices s, @xmath0, ..., k] where @xmath1. if the biased sub-sequence exists, it is labeled with a dimension of bias @xmath2. finally, x s:e is paired with a rephrased version x r s:e in which the bias targeting b has been removed. while in principle there can be multiple biased sub-sequences in x, we limit our definition to 1 sub-sequence due to 1 https://support.perspectiveapi.com/s/ about-the-api-attributes-and-languages dataset limitations. ## conclusion in this paper, we used the natural language taskprompting paradigm with popular auto-regressive language models to comprehensively analyze how well self-supervised pre-training captures the semantics of the tasks: bias diagnosis, identification, extraction and rephrasing. we performed experiments across multiple classes of task descriptions with numerous lexical variations, decoding mechanisms and different in-context examples by varying their size or sampling methods. we find that such models are largely challenged when prompted to perform these tasks and also exhibit large disparities in performance across different bias dimensions. we further demonstrate and discuss potential biases and task description sensitivities that such language models exhibit. we hope our work promotes future research on curating pre-training corpora and enhanced self-supervision during pretraining @xcite toward building language models that are more aware and adept at handling biases present in language, which would ultimately provide a path to safer adoption for downstream use-cases.