On the Machine Learning of Ethical Judgments from Natural Language (2022)

Abstract: Ethics is one of the longest standing intellectual endeavors of humanity. In recent years, the fields of AI and NLP have attempted to address issues of harmful outcomes in machine learning systems that are made to interface with humans. One recent approach in this vein is the construction of NLP morality models that can take in arbitrary text and output a moral judgment about the situation described. In this work, we offer a critique of such NLP methods for automating ethical decision-making. Through an audit of recent work on computational approaches for predicting morality, we examine the broader issues that arise from such efforts. We conclude with a discussion of how machine ethics could usefully proceed in NLP, by focusing on current and near-future uses of technology, in a way that centers around transparency, democratic values, and allows for straightforward accountability.

## introduction this paper offers a general critique of the nascent nlp task of computing moral and ethical decisions from text through reading a prominent system for moral prediction, @xcite , against the grain. we select delphi for its prominence, and because it has received significant attention and criticism from the general public. 1 in contrast to that criticism, much of which has focused on details of the particular outputs of the model, our goal is to highlight broader, general issues with the task of automatically predicting the morality of judgments of text situations, and expound on why any such nlp model should be considered unsafe at any accuracy. work that uses nlp techniques to automate morality "aims to assess the ability of @xcite models to make moral decisions in a broad set of everyday . . . situations" @xcite . delphi, is trained to emulate three conceptualizations of human moral and ethical judgments (see figure 1 ): a free-form question answering (qa) task, a yes/no qa task, and relative qa task, the latter judging how two statements rank in terms of morality. 2 the fact that the delphi project includes multiple conceptualizations of human moral and ethical judgments makes it an ideal candidate for a case study for morality models in nlp, as we will argue that no currently existing conceptualization of the morality task resolves the issues we outline in this audit. through our discussion, we intend to highlight that "ethical inquiry in any domain is not a test to be passed or a culture to be interrogated, but a complex social and cultural achievement" @xcite , and offer a critique of machine ethics from such a perspective. our critique is divided into several points of rebuttal to the task. first, we discuss issues with the conceptualization of the task and the poor fit between the task and the learning paradigms employed for it. then, we discuss issues with the training data available, as illustrated by commonsense norm bank-the corpus the authors develop to train delphi-as a foundation for training a machine learning model that makes morality judgments. for example, it contains judgments of situations that are not morality judgments. we also consider the implication of commonsense norm bank, i.e. that ethical and moral judgments can be derived from short text snippets with little context. next, we argue that, despite the authors' assertion that delphi is "the first unified model of descriptive ethics," any model developed for the task will necessarily be an inconsistent model of nor-stabbing someone with a cheeseburger stabbing someone over a cheeseburger is more morally acceptable than killing a bear to please your child note that the relative qa fragment "stabbing someone with a cheeseburger" is structurally ambiguous: either (i) someone with a cheeseburger was stabbed, or (ii) someone was stabbed using a cheeseburger. it is not clear whether (i) ought to be more morally acceptable than "stabbing someone over a cheeseburger." mative ethics. indeed, through generation, delphi outputs a prescriptive moral judgment for any input situation. given this, we also question (i) whether there ever could be sufficient diversity of moral judgment in a crowd-sourced dataset in practice, and (ii) whether aiming for a "diversity of moral perspective" is compatible with the desire for a morality model (especially one trained on an unconstrained crowd-sourced corpus). we then turn to the inherent contradictions that arise when modeling ethics by averaging over individual morality judgments. systems like delphi are at best capable of approximating the morality judgments of the population they were trained on. however, the average human judgment is not a good substitute for a system of ethics, since ethical evaluation is an open-ended, debate-based, sociopolitical process. ethics are not a static good that can be extracted from the public opinion of a given moment, but are instead continuously formed and negotiated through debate and dissent from previously accepted norms and values (see e.g., @xcite . thus, averaging over existing arguments cannot serve as a replacement for the processes of debate and negotiation. finally, we discuss some practical implications of the general prospect of utilizing delphi-like models to automate moral decision-making. systems for predicting morality like delphi, lack agency and thus cannot be held responsible for their decisions. this raises a concern over who ought to bear the responsibility for any potential infraction such systems could make if deployed in an envisioned future. we therefore question an assumption implicit in nlp projects like delphi that models ought to be ascribed the agency necessary to make moral prescriptions. we contend that, without an appropriate method of holding an agent to account, moral judgments are not of inherent utility, but dangerous: through foreclosing the possibility of debate and contestation, such models undermine the essential social foundations of ethical decision making. we conclude the paper by discussing how we believe nlp work at the intersection of ethics and machine learning could usefully proceed. we believe it is more crucial to address questions of morality or ethics in current and near-future use of technology, rather than considering hypothetical and distant-future uses @xcite . furthermore, we believe inquiries into the morality and ethics of current and near-future uses must keep actual human moral perspectives and their contradictions firmly at the forefront. we end with a word of caution: researchers in nlp and ai more broadly should not base their work on the assumption of a particular future, as delphi and others do, where the application technology must be made dependent on automated moral judgments, and humans (be they crowd-workers, researchers outside nlp, or other affected parties) have been cut out of the loop. ## background in this section, we describe and discuss relevant previous work in ethical nlp and the assumptions behind the nlp task of generating moral judgments and the creation of models like delphi. incorporating ethics into nlp work explicitly is a relatively new development @xcite . for example, the taln eternal, the first workshop on ethics in nlp, only took place seven years ago. recent works have begun to supplement tasks like stance detection with additional morality annotations @xcite , or to use nlp tools to track changes in human morality over time @xcite . other work seeks to characterize what language models already implicitly represent about morality by investigating their learned sentence representations @xcite @xcite . still other works like @xcite and @xcite focus on particular ethical theories and how they might be used in nlp to guide our modeling efforts, and @xcite foreground the importance of ethics training in nlp education. works like @xcite and @xcite go beyond this in finetuning language models to output moral prescriptions for sentential descriptions of situations. as such, @xcite and @xcite each represent one further step along an evolving trajectory in research on the intersection of nlp and ethics: a shift from measurement and classification to generation, and thus from a murky mix of descriptive and prescriptive aspects, to models producing prescriptive outputs. ## points of criticism our criticism falls into two parts that relate to the two stated contributions of jiang et al. we first describe issues with commonsense norm bank, namely the contents of data and its interpretation. second, we discuss general issues with the task formulation and training of models like delphi. ## future directions for machine ethics in this section, we discuss how accuracy improvements alone cannot mitigate the problems with work such as delphi in §4.1 and encourage a shift towards multi-disciplinary work in §4.2. ## conclusion in this paper, we have offered a general critique of the nlp task of generating moral judgments through a targeted audit of @xcite . we have highlighted issues with the operationalization of the task, with the learning paradigm, and with currently available training datasets. we have argued that the general enterprise is rooted in multiple category errors: it belies a misunderstanding of the descriptive/normative distinction, and falsely treats morality as a mere test to be passed. ultimately, automating ethical decisions forecloses possibilities for debate and contestation. since these are themselves prerequisites for the socio-political process of ethical inquiry, such a task is inherently incompatible with the social project of ethics.