Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domain (2024)

Abstract: In difficult decision-making scenarios, it is common to have conflicting opinions among expert human decision-makers as there may not be a single right answer. Such decisions may be guided by different attributes that can be used to characterize an individual’s decision. We introduce a novel dataset for medical triage decision-making, labeled with a set of decision-maker attributes (DMAs). This dataset consists of 62 scenarios, covering six different DMAs, including ethical principles such as fairness and moral desert. We present a novel software framework for human-aligned decision-making by utilizing these DMAs, paving the way for trustworthy AI with better guardrails. Specifically, we demonstrate how large language models (LLMs) can serve as ethical decision-makers, and how their decisions can be aligned to different DMAs using zero-shot prompting. Our experiments focus on different open-source models with varying sizes and training techniques, such as Falcon, Mistral, and Llama 2. Finally, we also introduce a new form of weighted self-consistency that improves the overall quantified performance. Our results provide new research directions in the use of LLMs as alignable decision-makers. The dataset and open-source software are publicly available at: https://github.com/ITM-Kitware/llm-alignable-dm.

## introduction llms have enabled many new applications, ranging from improved search to code assistants @xcite . however, many application areas still remain challenging for llms, due to the need to align with human values. recent work has explored how llms encode moral concepts @xcite , perform moral commonsense reasoning @xcite , and trade-off between maximizing reward and moral behavior @xcite , which are important steps towards building more safe and ethical ai systems. while the prior work has studied basic competency through use of question-answering benchmarks @xcite , we instead focus on decision-making scenarios where there may not be one right answer. in these cases, experts often disagree about the "correct" answer and their decisions may be influenced by different attributes. these decision-maker attributes may characterize an individual's moral values and preferences, such as their tendency towards fairness @xcite or utilitarianism @xcite . we test whether llms can be used as ethical and alignable decisionmakers that capture the dmas of humans. in contrast to standard alignment approaches like reinforcement learning from human feedback @xcite , alignment in our context is dynamic and may vary from individual to individual based on their personal preferences and the set of values they prioritize in a given situation. we introduce a novel decision-making dataset in the medical triage domain that contains various scenarios labeled with a set of dmas known to influence human judgments. notably, each scenario contains multiple plausible choices that are labeled with the relevant attributes. we first present these scenarios to a set of llms to understand their implicit decision-making tendencies. we then propose a zero-shot prompting strategy with weighted self-consistency, which allows us to align llms to different attributes and quantify their alignment to these attributes. our main contributions include: 1. a novel medical triage decision-making dataset, containing different scenarios labeled with dmas, which allows us to quantify model alignment using a new attributedependent accuracy metric. 2. a new zero-shot prompting approach to align llm decisions to a set of dmas, demonstrated through detailed analysis across different attributes and model types, sizes, and training techniques. 3. extension of a self-consistency module using weighted positive and negative samples, which improves model alignment. 4. a new, extensible, and versatile open-source software framework to enable research on human-aligned decision-making with llms. our work extends previous question-answering benchmarks, while relating to existing llm reasoning and alignment approaches, as described below. ## medical triage alignment dataset our dataset focuses on medical triage, which requires complex decision-making in critical lifeand-death situations where there is often no right answer. this contrasts with medical questionanswering datasets @xcite , which are often used to assess knowledge in different areas against known ground truth answers. each scenario in our dataset contains background context, a question, and multiple answer choices corresponding to decisions exhibiting a high or low value of a dma (fig. 1 ). our dataset construction method is an adaptation of prior work from the field of moral psychology, which has a longstand-high your base needs a delivery of whole blood. you estimate that your current supply is enough for 4 days, maybe a little longer if you start rationing. there are two possible courier companies with different schedules. ## approach in this section, we present our approach for creating ethical and alignable llm-based decision-makers. fig. 2 provides an overview of our approach, which is described in more detail below. ## evaluation metric the wasserstein distance was proposed as an alignment metric for the opinionqa dataset @xcite , but cannot be used here since the answers within our dataset are nominal, not ordinal. instead, we introduce an alignment accuracy that measures the selection of the correct choice(s), conditioned on a target attribute value (high or low). we calculate accuracy (ideal value: 100%) for each attribute a separately and also report accuracy across the entire dataset. for each question, the accuracy m of the generated answer g and the correct answer c given attribute g a , c a is: overall alignment accuracy is computed by averaging over the set of questions, answers, and generated responses for an attribute q a and then averaging over all attributes a: for unaligned models, alignment accuracy measures the implicit decision-making tendencies of the model. for example, a model expected to value fairness in its decisions should result in a high alignment accuracy to the high fairness target attribute value and, conversely, low alignment accuracy for the low fairness target attribute value. for aligned models, alignment accuracy measures how alignable the model is to different target attribute values based on the proposed zero-shot prompting strategy. furthermore, to provide a single metric across both the high and low target attribute values, we also report the f 1 score, which we define as the harmonic mean of the high and low alignment accuracy. ## experiments here, we report the results of our experiments across models and attributes. we study three different model configurations: 1) unaligned (sec. 4.1), 2) aligned using zero-shot prompting (sec. 4.2), and 3) aligned with the additional weighted selfconsistency (sec. 4.3). figs. 3a & 3b and tab. 2 provide the main results of this analysis. the llama2-13b aligned + self-consistency configuration generated the best results across the dataset, followed by mistral-7b aligned + self-consistency. appendices a and b provide additional quantitative and qualitative results with related insights. ## conclusions we have introduced a new medical triage alignment dataset and quantified the implicit decisionmaking tendencies of llms. we present a simple zero-shot prompting approach to align llms to a set of dmas, including different moral attributes. we also demonstrate the benefit of weighted selfconsistency, with the use of both positive and negative samples, improving overall alignment. our approach generalizes across different model types, sizes, and training techniques. while we tested our approach with open-source llms, additional experiments with proprietary models such as openai's chatgpt or gpt-4 (ope-nai, 2023) are of interest. our future work will also extend the proposed approach to alignment to multiple dmas at the same time (e.g. both high protocol focus and high fairness), as real-world decisions involve multiple attributes. we have seen early evidence of some success with promising results based on a preliminary alignment approach for this. this is closely related to work on modeling pluralistic human values @xcite . augmenting our approach with methods like retrieval-augmented generation @xcite ) may provide llms with background knowledge in other domains. while we proposed a simple prompt-based alignment strategy, other approaches that leverage (parameter-efficient) fine-tuning @xcite or few-shot learning with in-context examples @xcite could also be explored. finally, another interesting direction to pursue is to compare the decisions and explanations of llms with that of human decision-makers, to better understand potential differences in decision-making and other gaps in the alignment of these systems.