Reinforcement Guided Multi-Task Learning Framework for Low-Resource Stereotype Detection (2022)

Abstract: As large Pre-trained Language Models (PLMs) trained on large amounts of data in an unsupervised manner become more ubiquitous, identifying various types of bias in the text has come into sharp focus. Existing ‘Stereotype Detection’ datasets mainly adopt a diagnostic approach toward large PLMs. Blodgett et. al. (2021) show that there are significant reliability issues with the existing benchmark datasets. Annotating a reliable dataset requires a precise understanding of the subtle nuances of how stereotypes manifest in text. In this paper, we annotate a focused evaluation set for ‘Stereotype Detection’ that addresses those pitfalls by de-constructing various ways in which stereotypes manifest in text. Further, we present a multi-task model that leverages the abundance of data-rich neighboring tasks such as hate speech detection, offensive language detection, misogyny detection, etc., to improve the empirical performance on ‘Stereotype Detection’. We then propose a reinforcement-learning agent that guides the multi-task learning model by learning to identify the training examples from the neighboring tasks that help the target task the most. We show that the proposed models achieve significant empirical gains over existing baselines on all the tasks.

## introduction following @xcite and then @xcite , several empirically effective large pre-trained language models (plms) @xcite @xcite @xcite have been proposed. their empirical success on several existing tasks has resulted in them being ubiquitously used in many nlp applications that interact with humans on a daily basis @xcite . as they are trained in an unsupervised manner on large amounts of arbitrary web data, harmful language and bias creeps into their output. this, in turn, is reflected in the applications that are built on top of them and further propagated in society through the use of those applications @xcite . this issue motivates the need for a two-pronged solution: 1) to diagnose and de-noise the bias in the plms and 2) to identify and regulate harmful text externally at the output level. progress on the first front is necessary to de-bias the plms and prevent subtle biases that creep in through their distributed representations which can often only be recognized in retrospect. this results in deep distrust of such systems among general population. the second front is paramount to detect and potentially block harmful output and also ensure that future iterations and novel proposals of the language models don't take spurious paths to gaming the de-biasing techniques. in this work, we focus on identifying harmful text with stereotypical associations externally. s 1 : she may or may not be a jew but she's certainly cheap! (insult, stereotype) s 2 : burn in hell, you asian bastard! (abuse, stereotype) s 3 : asians are good at math. (stereotype) s 4 : my african-american friend owns a watermelon patch. (stereotype) there exist several types of harmful language such as hate-speech, misogyny, stereotypes, abuse, threats, insult etc,. each type of offensive language has subtle linguistic nuances that are specific to the type of offensive language. often, offensive text contains multiple types of offense. from the examples above, consider s 1 and s 2 . both, consist of multiple modes of offense. while s 3 is purely a stereotype, it is still undesirable to be perpetuated. @xcite defines stereotype as a "fixed, over-generalized belief about a particular group or class of people". stereotypes differ from other types of offensive text in two key aspects: (1) they require knowledge of their existence in the society to be identified, and (2) they might also often express positive sentiment about the target group. although some stereotypes ostensibly express positive sentiment towards the target group, they are still undesirable as they propagate false biases in the society and are offensive to the target group. consider sentences s 3 and s 4 from above examples. while s 3 expresses positive sentiment, it is still false and undesirable. s 4 requires knowledge of that particular stereotype's history to understand its offensive nature. requiring prior knowledge makes annotating data for the task of 'stereotype detection' harder, as annotators are unlikely to be aware of all the stereotypes that exist in the society. @xcite . two recent works have proposed pioneering diagnostic datasets for measuring stereotypical bias of large plms @xcite . @xcite has demonstrated that these datasets suffer from two major types of issues: (1) conceptual: include harmless stereotypes, artificial anti-stereotypes, confusing nationality with ethnicity etc, and (2) operational: invalid perturbations, unnatural text, incommensurable target groups etc,. in addition, diagnostic datasets also suffer from lack of sufficient coverage of subtle nuances of manifestations of stereotypes in text. this makes them less suitable for training an effective discriminative classifier. hence, we undertake a focused annotation effort to create a fine-grained evaluation dataset. we mainly aim to alleviate the conceptual issues of anti-vs. nonstereotypes, containing irrelevant stereotypes and operational issues of unnatural text, invalid perturbations. we achieve this by a mix of (1) selecting more appropriate data candidates and (2) devising a focused questionnaire for the annotation task that breaks down different dimensions of the linguistic challenge of 'stereotype identification'. collecting real-world data from the social forum reddit for annotation also results in better coverage of subtle manifestations of stereotypes in text. although stereotypes differ from other types of offensive language in multiple ways, they also overlap to a significant extent. often, various types of offensive text such as abuse, misogyny and hate speech integrally consists stereotypical associations. abundance of high-quality annotated datasets are available for these neighboring tasks. we leverage this unique nature of stereotype detection task to propose a multi-task learning framework for all related tasks. as the overlap between the tasks is only partial, we then propose a reinforcement learning agent that learns to guide the multi-task learning model by selecting meaningful data examples from the neighboring task datasets that help in improving the target task. we show that these two modifications improve the empirical performance on all the tasks significantly. then, we look more closely at the reinforcement-learning agent's learning process via a suite of ablation studies that throw light on its intricate inner workings. to summarize, our main contributions are: ## related work with the rise of social media and hate speech forums online @xcite offensive language detection has become more important that ever before. several recent works focus on characterizing various types of offensive language detection @xcite @xcite . but, works that focus solely on stereotype detection in english language are scarce. this is partly because stereotypes tend to be subtler offenses in comparison to other types are offensive languages and hence receive less immediate focus, and in part due to the challenge of requiring the knowledge of the stereotype's existence in society to reliably annotate data for the task. we approach this problem by breaking down various aspects of stereotypical text and crowd-sourcing annotations only for aspects that require linguistic understanding rather than world-knowledge. few recent works have focused solely on stereotypes, some proposing pioneering diagnostic datasets @xcite @xcite for identifying stereotypical text. computational model based works either use datasets meant for other tasks such as hate speech detection etc, or focus mainly on the available diagnostic datasets modified for classification task. but, diagnostic datasets suffer from lack of sufficient coverage of naturally occurring text due to their crowd-sourced construction procedure @xcite . we address these issues in our work by collecting natural text data from social forum reddit, by mining specific subreddits that contain mainly subtle stereotypical text. multi-task learning @xcite , can be broadly classified into two paradigms @xcite : hard parameter sharing @xcite and soft parameter sharing @xcite . we implement hardparameter sharing based multi-task model for our experiments. given the low-resource setting on stereotype detection task, semi-supervised data annotation is one plausible solution for the problem. several recent works have also been focusing on reinforcementlearning guided semi-supervision @xcite @xcite . @xcite , in particular, work with a singletask and unsupervised data to generate automatedannotations for new examples. in contrast, we use the data from neighboring tasks with different labels for multi-task learning and apply an rl agent to select examples for training the neighboring task in such a way that benefits the target task the most. ## our dataset as blodgett et al. (2021a) demonstrates, existing diagnostic datasets such as stereoset @xcite and crows-pairs @xcite suffer from several conceptual and operational issues. additionally diagnostic datasets, by nature, also suffer from lack of coverage of subtle manifestations of stereotypes in text. but, this is mandatory in the dataset used to train an effective discriminatory classifier that can generalize well to identifying stereotypical associations in open domain use cases. this is often the case in the examples encountered in downstream applications that interact with humans on a daily basis. nadeem et al. (2020) and nangia et al. (2020) both follow the following dataset construction procedure: they provide a prompt word or phrase, which is the target group of the stereotype, and ask the crowd workers to construct a stereotypical sentence which includes the provided prompt. then, they either flip the identity of the target group @xcite , the associated characteristic/action of the group or both @xcite to obtain negative (anti-stereotypes) examples. although this process results in good coverage of existing stereotypes within society due to the variety of prompts provided, it also results in data examples that either explicitly state the stereotype or are anti-stereotypes that are mostly contrived sentences and are unlikely to occur in regular discourse. an example from stereoset dataset @xcite is shown below to illustrate this point. 1 is an explicitly states stereotype where as 2 and 3 are unlikely to occur in regular discourse. ## experiments we perform experiments on six datasets in three phases. in the first phase, we experiment with plm-based fine-tuned classifiers for each task as baselines. in the second phase, we experiment with all the tasks using the multi-task learning model described in section 4.1, with each plm as a base classifier. in the third phase, we train the reinforcement-learning guided multi-task learning framework (section 4.2) for all the tasks with each of the plms as base classifier. ## results we present the results of the empirical evaluation tasks in for hate speech and offensive language detection task, the respective numbers for baseline, mtl and rl-mtl models are 66.13, 68.57 and 68.97. the models achieve 74.16, 74.40 and 75.21 on misogyny detection task, respectively. in coarse-grained stereotype detection task, they achieve 65.71, 68.29 & 74.18, which is a significant gradation over each previous class of models. on our focus evaluation set of fine-grained stereotype detection, we achieve 61.36, 65.00 & 67.94 in each class of models. the results on this dataset are obtained in a zero-shot setting as we only use this dataset for evaluation. ## analysis & discussions in the first ablation study described in subsection 7.1, we study the importance of initializing rl-mtl model with the trained parameters of mtl model. following that, we look into more detail about the usefulness of neighbor tasks on the target task via an ablation study.we describe these experiments in further detail in subsection 7.2. ## conclusion we tackle the problem of stereotype detection from data annotation and low-resource computational framework perspectives in this paper. first, we discuss the key challenges that make the task unique and a low-resource one. then, we devise a focused annotation task in conjunction with selected data candidate collection to create a fine-grained evalua-tion set for the task. further, we utilize several neighboring tasks that are correlated with our target task of 'stereotype detection', with an abundance of high-quality gold data. we propose a reinforcement learning-guided multitask learning framework that learns to select relevant examples from the neighboring tasks that improve performance on the target task. finally, we perform exhaustive empirical experiments to showcase the effectiveness of the framework and delve into various details of the learning process via several ablation studies.