Does Moral Code have a Moral Code? Probing Delphi’s Moral Philosophy (2022)

Abstract: In an effort to guarantee that machine learning model outputs conform with human moral values, recent work has begun exploring the possibility of explicitly training models to learn the difference between right and wrong. This is typically done in a bottom-up fashion, by exposing the model to different scenarios, annotated with human moral judgements. One question, however, is whether the trained models actually learn any consistent, higher-level ethical principles from these datasets – and if so, what? Here, we probe the Allen AI Delphi model with a set of standardized morality questionnaires, and find that, despite some inconsistencies, Delphi tends to mirror the moral principles associated with the demographic groups involved in the annotation process. We question whether this is desirable and discuss how we might move forward with this knowledge.

## introduction it has become obvious that machine learning nlp models often generate outputs that conflict with human moral values: from racist chatbots @xcite , to sexist translation systems @xcite , to language models that generate extremist manifestos @xcite . in response, there has been growing interest in trying to create ai models with an ingrained sense of ethics -a learned concept of right and wrong. 1 one such high-profile example is the delphi model, released simultaneously as a paper and an interactive web demo foot_1 by allenai on october 14, 2021 @xcite . almost immediately, social media users began posting examples of inputs and outputs that illustrated flaws in delphi's moral reasoning. subsequently, the researchers on the project modified the demo website to clarify the intended use of delphi strictly as a research demo, and released software updates to prevent delphi from outputting racist and sexist moral judgements. the research team also published a follow-up article online @xcite to address some of the criticisms of the delphi system. in that article, they emphasize a number of open challenges remaining to the ethical ai research community. among those questions is: "which types of ethical or moral principles do ai systems implicitly learn during training?" this question is highly relevant not only to ai systems generally, but specifically to the delphi model itself. the delphi research team deliberately take a bottom-up approach to training the system; rather than encoding any specific high-level ethical guidelines, the model learns from individual situations. indeed, it seems reasonable to avoid trying to teach a system a general ethical principle such as "thou shall not kill," and then have to add an exhaustive list of exceptions (unless, it is a spider in your house, or if it is in self-defense, or if you are killing time, etc.). however, it is also clear that at the end of the day, if the model is able to generalize to unseen situations, as claimed by @xcite , then it must have learned some general principles. so, what has it learned? here, we probe delphi's implicit moral principles using standard ethics questionnaires, adapted to suit the model's expected input format (free-text description of a situation) and output format (a three-class classification label of 'good', 'bad', or 'discretionary'). we explore delphi's moral reasoning both in terms of descriptive ethics (schweder's "big three" ethics @xcite ) and haidt's five-dimensional moral foundations theory @xcite ) as well as normative ethics, along the dimension from deontology to utilitarianism @xcite . we hypothesize that delphi's moral principles will generally coincide with what is known about the moral views of young, english-speaking, north americans -i.e., that delphi's morality will be influenced by the views of the training data annotators. however, we anticipate that due to the effects of averaging over different annotators, the resulting ethical principles may not always be self-consistent @xcite . our intention is not to assess the "moral correctness" of delphi's output. rather, we evaluate the system using existing psychological instruments in an attempt to map the system's outputs onto a more general, and well-studied, moral landscape. setting aside the larger philosophical question of which view of morality is preferable, we argue that it is important to know what -and whose -moral views are being expressed via a so-called "moral machine," and to think critically about the potential implications of such outputs. ## methods the questionnaires, the modifications we made, and the scoring methods are outlined below. the full list of prompts for each questionnaire is available online, foot_3 and an illustrative example from each questionnaire is given in appendix b. ## discussion we now discuss these results in the context of human morality, including demographic and cultural differences in moral values, individual moral consistency, and whether moral judgement can be modelled as the binary outcome of a majority vote. ## conclusion the delphi model was designed to be a descriptive model of morality. our results suggest that delphi has learned a surprisingly consistent ethical framework (though with some exceptions), primarily aligned with liberal western views that elevate autonomy over community and divinity, rank the individualizing foundations of caring and fairness above the binding foundations of loyalty, authority, and purity, and support the utilitarian principle of impartial benefience but reject the principle of instrumental harm. however, as a descriptive model, this is markedly incomplete, even when constrained to english-speaking north american society. in the discussion, we question how such a model could be deployed in a social context without potentially harming those whose moral views do not align with delphi's annotators, and by extension, the trained model.