Contextualizing Language Models for Norms Diverging from Social Majority (2022)

Abstract: To comprehensibly contextualize decisions, artificial systems in social situations need a high degree of awareness of the rules of conduct of human behavior. Especially transformer-based language models have recently been shown to exhibit some such awareness. But what if norms in some social setting do not adhere to or even blatantly deviate from the mainstream? In this paper, we introduce a novel mechanism based on deontic logic to allow for a flexible adaptation of individual norms by de-biasing training data sets and a task-reduction to textual entailment. Building on the popular ‘Moral Stories’ dataset we on the one hand highlight the intrinsic bias of current language models, on the other hand characterize the adaptability of pre-trained models to deviating norms in fine-tuning settings.

## introduction social norms -whether explicitly codified or just widely agreed upon -to a large degree govern the interaction of humans. in that sense they allow for assessing and making sense of everyday situations. thus, also the successful deployment of ai systems in social settings, e.g., conversational agents or decision making systems, will depend on the ability of such systems to adequately reflect existing social norms @xcite . recently, studies on transformer-based language models (lms) have shown that indeed there seems to be a 'moral dimension' to lms, as they show high accuracy in related downstream tasks such as moral reasoning and action classification @xcite @xcite . arguably, this notion of morality can be attributed to the lms' pre-training corpora containing social majority biases also exhibited by the later used benchmarks, which are often gained by general crowd-sourcing tasks. thus, throughout this paper we will understand the acquisition of the social norms by lms in a descriptive, rather than an explicit prescriptive fashion. while it is notoriously difficult to effectively remove all bias from ai systems, using known biases to fulfill some specific and clearly defined goals has been investigated, see e.g, @xcite . but what if desirable norms in some social setting do not adhere to or even blatantly deviate from norms agreed upon by the social majority, e.g., specific norms of social subgroups? is a completely new pre-training needed to override generally accepted norms in a seemingly consistent normative system in an lm's moral dimension? since the costs of building huge and necessarily well-curated pre-training corpora for each social subgroup are clearly prohibitive, this option is of a rather theoretical nature. could simpler techniques like fine-tuning then effectively create sufficient awareness in language models to allow for successful downstream tasks? in this paper we will investigate the question of how well general purpose language models can 'tune in' to norms deviating from majority society. building on deontic logic for norm inversion we perform extensive experiments allowing to remove arbitrary norms with respect to benchmark corpora or even impose contrasting norms during fine tuning. on the technical level, we show how to construct the necessary datasets for fine tuning such that models can achieve a high degree of accuracy. thus, the actual norm acquisition always stays of a strongly descriptive nature, since we impose no explicit mechanisms to guarantee that the lm will always adhere to explicitly altered norms. due to the problems of deriving adequate document sets for individual social subgroups in the real world, however, within the scope of this paper we perform only synthetic experiments on often used real world datasets (in particular social chemistry, @xcite . although this is a clear limitation of the work presented here, the paper's basic techniques and insights promise to allow for generalization. we will point out and critically assess limitations and possible problems for generalization in all parts of this work. this paper is organized as follows: in section 2 we will revisit related work and especially take a closer look at typical datasets and downstream tasks in the field of moral ai. as the goal of this paper is to investigate the stability of pre-trained language models in the face of conflicting norms, we take a closer look at the task of moral action classification in section 3. we provide a detailed description of our dataset design and creation in section 4 as a basis for later experiments. this also includes the inversion of arbitrary norms following the rules provided by deontic logic. section 5 will then present the actual experimental investigation of our hypothesis that fine-tuning may be a suitable remedy for the task of reflecting norms differing from social majority in downstream tasks. after a discussion in section 6, we close with our conclusions in section 7. ## related work there is a growing body of work concerning the development of ai/machines that behave ethically and/or are aligned with human values. for example, @xcite investigate potential applications of deontological ethics in the context of nlp and, similarly to @xcite , study the first-principles of generalization and autonomy. other works have prioritized aligning artifical agents with shared human values @xcite . value alignment has been approached from numerous angles, including preference learning @xcite , imitation @xcite and inverse reinforcement learning @xcite . additionally, several approaches concerning controllable text generation have been proposed to steer model generation towards specific attributes @xcite . however, our experimental setup focuses on classification tasks instead of generation. similarly, @xcite incorporate speaker context into language model pre-training objectives. here, we do not consider pre-training, but rather only investigate fine-tuning. several datasets of normative knowledge have been published to assess to which extent current models are able to represent specific morality or ethical rules. one important aspect of the benchmarks is their degree of implicitness of normativity. in this regard, implicit datasets usually contain examples of right and wrong behavior with according labels @xcite @xcite , whereas others rely on explicitly stating the social rules at play @xcite @xcite . @xcite introduce social chemistry 101, a large collection of so-called rules-of-thumb (rot) associated with a rich structure of human annotations. according to the authors, these rots were designed to represent social norms and moral judgment as experienced by crowd-workers. with moral stories, @xcite propose self-contained branching narratives consisting of norms, context, moral and immoral actions and their expected consequences as written by crowd-workers. the authors suggest that the roberta @xcite model exhibits a normativity bias due to pre-training, but they do not follow up with investigations. other datasets, e.g., ethics @xcite or scruples @xcite also present resources containing normative information, but only through examples, whereas explicit mentions of the appropriate norms and rules are required for our purposes. although conceivable, we leave adaptation of full paragraphs or stories to reflect contrary values or norms for future work, since current language models have been shown to lack the capabilities of dealing with the various nuances of negation and contradiction @xcite . in this paper, we focus on the action-classification task introduced by emelin et al. and propose a controlled approach for negation grounded in deontic logic. with commonsense norm bank, @xcite compile several benchmarks into a large collective of moral judgment q&a tasks. one aspect of their work is similar to ours, as they also derive augmented norms from the moral stories dataset. however, their focus is on deriving equivalent norms through morality-preserving transformations, whereas we explicitly opt for the derivation of opposite norms. finally, perhaps most similar is the work of @xcite , who investigate to which degree pre-trained language models reflect cross-cultural values according to external value surveys. their experiments employ probing techniques and provide evidence of normativity biases, but only weak alignment to the surveys. in contrast, we aim to analyze to which extent the models are able to reflect norms explicitly deviating from majority-imposed bias. ## moral action classification several works have provided evidence of pretrained language models achieving notable results in approximating human decision-making in social context. to assess to which extent the models are able to generalize, researchers frequently turn to analyses of previously unseen situations. however, in many cases, these "test" sets stem from the same population that curated the data used for priming the models in the first place (e.g., gathered from crowd-workers with high agreement levels). this connection becomes even more apparent in the case of language models utilizing pre-training, which have been suspected to contain a normativity bias @xcite . recently, @xcite found such bias in their study of cross-cultural value alignment of plms. in light of these findings, we aim to test model generalizability from a broader perspective. if plms do contain a bias towards a specific set of norms, then to which extent are they adaptable to new norms? in this paper, we focus on the case of norms explicitly contrary to what has been argued to be picked up during pre-training. the motivation is to reflect the inherent nature of social subgroups, which usually oppose certain norms imposed by majority society. however, it is not yet clear, what opposites, or inversions of norms are. we adopt deontic logic, which formalizes relations between norms, such as contrary or contradictory. deontic logic requires norms to be directly expressed, which rules out many of the published benchmarks as potential bases. to the best of our knowledge, moral stories is the only benchmark so far incorporating norms, actions and corresponding labels of adherence or violation. therefore, of the many proposed tasks to assess normative knowledge in language models, we consider action classification for its explicitness and clear-cut semantics. thus, we build on and extend definitions and data from moral stories @xcite and social chemistry 101 @xcite . more specifically, we define the action classification task following @xcite : we focus on the setting of actions grounded by corresponding norms. although moral stories also provides context as well as consequences for grounding, we omit them in this paper due to the increased complexity. for the remainder of the paper we understand moral action classification as the (norm, action)-scenario, where the task is to decide for any such pair whether the action is deemed moral or immoral with respect to the norm. models are evaluated on the accuracy metric. for further details, see @xcite . ## dataset design and creation in the following subsections we briefly introduce deontic logic as the theoretic foundation of our norm inversion procedure. then we show how moral stories and social chemistry 101 datasets relate to the theoretical considerations, and lastly we present and evaluate two automatically derived sets of opposing norms, namely anti-ms and optionalms. note that we use deontic logic only as means to derive new norms by inversion. we explicitly do not use it for logical inference, as this would require the underlying datasets to be consistent. but, as already pointed out by their authors, neither moral stories nor social chemistry 101 are designed to be free of contradictions. for example, "you shouldn't let animals suffer." and "you should not kill animals.", both from moral stories, could be mutually exclusive under certain circumstances. ## experiments we conduct several experiments based on the original moral stories (original-ms), anti-ms and optional-ms datasets over variations of the moral action classification task. we include seven models in our studies: distilbert (66m) @xcite as a rather small model, bert (110m & 336m) @xcite , since they are among the most used, roberta (355m) @xcite to ensure comparability with moral stories, al-bert (223m) @xcite for its exceptional performance on the ethics benchmark and lastly, gpt-neo (1.3b & 2.7b) @xcite as representatives of larger transformer models. ## discussion we used concepts of standard deontic logic to derive norms contrary to those of moral stories. overall, sdl can only be viewed as one of many possible frameworks that could be used. we reiterate that we explicitly do not adopt sdl for reasoning purposes, but only for its clear-cut definitions of operators, which we deem transferable to the natural language domain. here, we intuitively map definitions of moral stories to those of deontic logic. specifically, we interpret human judgment as salient indicators. experiments on polarity classification (see section 5.3) indicate that the mapping indeed results in distinct classes. one major difference of sdl and natural language is the way negation is handled. while the former provides an exhaustive operationalization of negation, the latter is much more nuanced @xcite . here, we effectively contract a multitude of textual judgments (e.g. "it is good to") into three equivalence classes. these, in turn, are then associated with sdl operators. ## conclusion we investigated the abilities of language models to simultaneously represent opposing sets of norms in the context of a moral action classification setting. based on notions from deontic logic, we derived two such sets from the moral stories benchmark and ran extensive evaluations on a range of architectures. our results suggest that fine-tuning on just one of the sets imposes a strong bias onto the models, in the sense that the left out norms are severely misrepresented. further, when subjected to highly conflicting norms, we found pre-training to play an essential role for models to adapt well. models that were not pre-trained and thus are not affected by possible bias towards specific norms were found to collapse to random guessing. however, contrary to intuition, with pre-training enabled, the models were able to reconcile even most inconsistent normative settings. finally, we propose one option to factor out the reasoning aspect of the task into textual entailment. the approach performs on par to the best fine-tuned model.