Reducing Non-Normative Text Generation from Language Models (2020)

Abstract: Large-scale, transformer-based language models such as GPT-2 are pretrained on diverse corpora scraped from the internet. Consequently, they are prone to generating non-normative text (i.e. in violation of social norms). We introduce a technique for fine-tuning GPT-2, using a policy gradient reinforcement learning technique and a normative text classifier to produce reward and punishment values. We evaluate our technique on five data sets using automated and human participant experiments. The normative text classifier is 81-90% accurate when compared to gold-standard human judgements of normative and non-normative generated text. Our normative fine-tuning technique is able to reduce non-normative text by 27-61%, depending on the data set.

## introduction human societies implicitly establish codes of acceptable behavior in social contexts. normativity is behavior that conforms to expected societal norms and contracts, whereas non-normative behavior aligns to values that deviate from these expected norms. @xcite defines norms as: "...informal rules that are not written, but, when violated, result in severe punishments and social sanction upon the individuals, such as social and religious exclusions." non-normativity does not connote behavior devoid of value or immoral, but behavior that fails to conform to social standards shared by other individuals in the relevant group, organization or society. norms can also be thought of as actions taken by an entity which conform to an identity @xcite , thus allowing others to categorize behavior as in-group or out-group. different societies and groups collectively have different ideals about what actions constitute normative behavior; group members use these ideals * equal contributions to heuristically guide their actions to avoid social ostracization. for example, many societies have norms against violence, or certain behaviors being conducted in public. conflicts between individuals can arise when enacting non-normative behaviors or uttering non-normative speech. this paper examines generative language models and the frequency at which they generate descriptions of non-normative behavior. large-scale, transformer-based neural language models such as elmo @xcite , bert @xcite , gpt-2 @xcite , gpt-3, grover @xcite , ctrl @xcite , t5 @xcite , and xlnet @xcite are trained on very large corpora such as text scraped from the internet, books, or both. these language models generate text that is statistically representative of the corpora they were trained on. as such, text scraped from the internet co-mingles text produced by many groups with differing norms, as well as text produced by people intentionally using non-normative speech, like "trolling" language. models trained on these data can then produce undesirable, harmful output. stories from the internet and books also contain normative and non-normative situations (e.g., antagonists, as well as protagonists conducting conventionally non-normative behaviors). consequently, it is possible, and often likely, for language models to generate non-normative descriptions of behavior (murder, crime, suicide, racist actions, rude behavior, etc.), exhibit biases against certain demographics groups @xcite , stereotypical biases @xcite or racist text when prompted with trigger phrases @xcite . value alignment @xcite is the concept that an agent is unable to perform actions that cause harm to humans. harmful behavior is not limited to physical actions by robots, the focus of some ai value alignment research. we recognize that natural language communication can also cause harm. for example, amazon alexa, a virtual assistant ai, was reported to suggest a user commit suicide. 1 frazier et al. (2019) developed a classifier for normative behavior which exhibits strong zeroshot and few-shot transfer across a variety of text corpora. the authors speculate that their modelwhich they call a value-aligned prior-can bias model output perceived as more normative. in this paper we ask a different question: whether a valuealigned prior can be used to reduce the generation of descriptions of non-normative behavior by neural language models. the common approach to fine-tuning language models is to provide additional corpora of exemplars. if a corpus of exemplars is normative, the language model can be trained to emulate this over time. generally, in the absence of very large normative corpora, we need an alternative approach to fine-tuning language models. we use a reinforcement learning approach to fine-tuning language models, using the normative behavior classifier of @xcite as a non-differentiable reward function. our method back-propagates reward relative to the degree of non-normativity of text generated by the language model. we evaluate our reinforcement learning finetuning technique with three sets of experiments. first, we replicate the experiments by @xcite on text generated by a language model instead of originally held-out corpus text. second, we show with automated and human participant experiments that fine-tuning on reward generated by a normative classifier model can reduce the generation of non-normative text by 27 -61%. third, we ablate our technique and show with automated and human participant experiments that the fine-tuning technique works with classifiers other than the normative classifier-specifically models trained to classify negative-sentiment and toxic language. ## normative fine-tuning the gpt-2 model is trained by minimizing its cross-entropy loss given by @xcite : to punish gpt-2 for producing non-normative text, we use a normative text classifier to evaluate the model's performance and produce a reward value, which is applied to the loss and backpropagated through gpt-2. given that the normativity of a sentence can only be determined by reading the entire sentence, the classifier must therefore produce a single numeric value per sentence. specifically, we augment the cross-entropy loss computation with predictions from the pre-trained classifier. we define the sentence loss as: to prevent the model from deviating too much from the language in the original dataset, we feed the fine-tuned model with the same set of input sentences at every loop and use the output sentences to even further fine-tune the model. as the model is trained, the output sentences will differ, and the reward value c(s) may change after every iteration as the model shifts its distribution. ## experiments we conduct three sets of experiments to ( 1 ) verify the normative text classifier on generated continuations, (2) evaluate our reward-based fine-tuning with the normative text classifier, (3) evaluate our reward-based fine-tuning on other classifiers. ## discussion we demonstrate how value-aligned priors such as normative text classifiers can act as a reward provider to nudge the gpt-2 language model towards producing more normative and neutral descriptions of behaviors and events. applying this reward-based fine-tuning technique reduces the likelihood of generating sentences containing non-normative behavior by approximately ∼27-61%, depending on the dataset. some datasets are more non-normative and more resistant to reduction of non-normativity. datasets with low nonnormativity to begin with are, naturally, more resistant. beyond the numerical results, this shows evidence that policy-gradient based reinforcement learning approaches to fine-tuning can be an effective means for reducing the generation of nonnormative descriptions. by using a normative text classifier-a value aligned prior-one can finetune a language model to the desired domain and then fine-tune using the prior again to reduce nonnormative generation that stems either from gpt-2 or from the domain corpus. we provide this approach as an alternative-or in complement todebiasing techniques that attempt to correct for prejudicial bias in datasets prior to training. our approach is roughly equivalent to teaching a language model to censor itself. the policy-gradient based reinforcement learning technique using a value-aligned prior can be even more valuable with gpt-3, where fine-tuning to a domain is less necessary. gpt-3 has been demonstrated to be capable of non-normative, toxic, and prejudicially biased generation. by using the normative text classifier trained on goofus & gallant comics, our results are limited to western-and in particular americanmainstream ideals of normative behavior. we acknowledge that culture is not monolithic, even within the united states of america, and this represents only one of many possible sources of normativity. we cannot conclusively say that our results will hold if we had normative text classifiers trained from different source materials. because general sources of normative behavior are currently hard to come by, we attempt to show generalization of our technique with experiments using sentiment and toxic language. one limitation of our work is that the normative text classifier can only classify individual sentences without context. given the context-dependent nature of normativity, the normative text classifier may overlook non-normative sentences that may appear normative out-of-context. this may lead to gpt-2 still producing this sentence in its nonnormative context. another limitation is that finetuning gpt-2 on plotto, rocstories and sci-fi datasets leads to it generating both neutral and normative sentences. if a model that generates solely normative sentences is desired, one can substitute the normative classifier with a ternary classifier with labels for normative, non-normative, and neutral sentences, and adjust the reward signals accordingly. furthermore, fine-tuning classifiers requires datasets with labeled exemplars, hence, in order to replicate our work to generate texts with some other desirable characteristics, datasets with labels would be prerequisites. the motivation of our work is to show how those who are concerned with generating undesirable text language models can obtain some control over the generation process. we look at normativity, but also show how other criteria can be applied. however, as is true for most algorithms, those with malicious intent can find ways to corrupt the intentions of the work. for example, equation ( 3 ) can be trivially modified to punish normative text instead of non-normative text. ## conclusions we have shown that large-scale transformer-based neural language models can be made to generate text containing fewer descriptions of nonnormative behavior by applying data-efficient, policy-gradient reinforcement learning. as most large-scale language models are trained on datasets from the internet and from books, the potential for intentional or unintentional non-normative language persists. we see this as a first step toward decreasing the potential for unintended, unacceptable, anachronistic or harmful language. while our primary result is to show that we can decrease the generation of non-normative behavior descriptions, our normative classifier of choice is rooted in western/american norms and values. normative classifiers are rare and datasets containing normative or preference learning examples are difficult to obtain. however, our results show that even small datasets of normative examples can be converted into few-shot classifiers and applied to