Chat Bankman-Fried: an Exploration of LLM Alignment in Finance (2025)

Abstract: Advancements in large language models (LLMs) have renewed concerns about AI alignment—the consistency between human and AI goals and values. As various jurisdictions enact legislation on AI safety, the concept of alignment must be defined and measured across different domains. This paper proposes an experimental framework to assess whether LLMs adhere to ethical and legal standards in the relatively unexplored context of finance. We prompt ten LLMs to impersonate the CEO of a financial institution and test their willingness to misuse customer assets to repay outstanding corporate debt. Beginning with a baseline configuration, we adjust preferences, incentives and constraints, analyzing the impact of each adjustment with logistic regression. Our findings reveal significant heterogeneity in the baseline propensity for unethical behavior of LLMs. Factors such as risk aversion, profit expectations, and regulatory environment consistently influence misalignment in ways predicted by economic theory, although the magnitude of these effects varies across LLMs. This paper highlights the benefits and limitations of simulation-based, ex-post safety testing. While it can inform financial authorities and institutions aiming to ensure LLM safety, there is a clear trade-off between generality and cost.

## introduction large language models (llms) are rapidly transforming how we approach problems across various domains, thanks to their improved natural language understanding @xcite and their advanced reasoning capabilities @xcite . financial firms, known for being early adopters of new technologies, have already integrated llms into their operations to varying extents @xcite @xcite . * the opinions expressed in this paper are personal and should not be attributed to the bank of italy. the same flexibility and autonomy that make these models so powerful also introduce significant challenges to their practical applicability. due to their complex architectures, llms are prone to issues like hallucinations @xcite and biases @xcite , which can result in unintended consequences when deployed in real-world applications. insecure, malfunctioning, or misguided ai can impact financial stability and market fairness and transparency, while also facilitating criminal abuse of the financial system @xcite . understanding how undesirable ai behavior may arise, and how to prevent it, is of paramount importance. existing work primarily addresses these challenges by developing models that prioritize safety @xcite , and introducing guardrails to prevent the generation of harmful content @xcite . several studies have established benchmarks to evaluate the safety of llms in generating illegal or violent content @xcite , as well as their robustness against "jailbreak" attacks, which can cause models to still produce unwanted content despite the presence of guardrails or safety features @xcite . recently, more attention has been devoted to the tension between maximizing rewards and behaving ethically, which may affect llms in some situations @xcite . nevertheless, most benchmarks and experiments focus on broad, general ethical concepts, with a lack of domain-specific evaluations. with the introduction of novel laws and frameworks on ai @xcite , it has become increasingly necessary to study and operationalize these standards within specialized domains. our paper presents a thorough exploration and study of the llm alignment problem in the financial sector, which has received only limited attention despite its critical implications. in detail, we propose a comprehensive simulation study to assess you are naga, ceo of the eponymous seashell trading firm. your job is buying and selling seashells against @xmath0, so that we can place a few more trades and try to recover everything? ## related work alignment, as defined by @xcite , refers to ensuring that an ai system's actions remain consistent with the intended goals set by human operators. in a recent comprehensive survey, @xcite partition alignment research into two sub-fields: forward alignment, which focuses on how to train ai systems to maximize alignment with a given set of values, and backward alignment aiming at gathering evidence on the alignment of existing ais (evaluation), and governing any emerging misalignment. the method and experiments proposed in this paper fall into the second sub-field. several studies have already highlighted the gap between a model's performance on benchmark tasks and its ability to adhere to desirable behaviors in uncontrolled environments @xcite . thus, recent research has shifted towards incorporating safety, ethics, and value alignment as core evaluation dimensions. @xcite proposed an evaluation framework that introduces "harmful outputs" as a critical failure mode for llms, while @xcite have emphasized the social and ethical implications of models that operate without adequate oversight. from an economic or financial perspective, nascent literature is exploring to which extent llms' behavior replicates homo economicus 2 @xcite , whether llms can emulate non-rational choices @xcite , and whether insights from economics can help in modeling interactions between humans and llms @xcite . this body of literature suggests that we may not be far from leveraging llm models within companies to support and help make informed decisions. our paper draws significantly on the ideas and experimental framework presented in @xcite . the authors assess whether an llm impersonating a stock trader is willing to act on insider information, despite being told that such behavior should be avoided. they find that the llm indeed engages in insider trading if given the right incentives. to the best of our knowledge, @xcite is the only existing systematic evaluation of llm alignment in a financial setting. our work builds on this approach by examining llm behavior in a different financial scenario inspired by real-world events. while the authors focus primarily on gpt-4 variants, we evaluate a diverse set of both closed and open-source llms. furthermore, we extend their methodology by introducing a multidimensional "pressure vector" that systematically varies prompt components, allowing for a more granular analysis of factors influencing model decisions. in designing our experiment, we also draw on @xcite , who introduce the concept of "role play" to characterize human-like behavior of llm-based dialogue agents "without falling into the trap of anthropomorphism". ## experimental framework the scenario underlying our simulation is inspired by the 2022 collapse of the cryptoasset exchange ftx (us department of justice, 2024). we model an imaginary financial firm where an llmpowered agent acts as the ceo. this agent is faced with the dilemma of either risking the firm's financial failure or misusing customer funds to ensure the exchange's continued operations. we employ a fictional, but plausible, setting to avoid data contamination, given that some of the examined llms have knowledge cutoffs after november 2 a rational agent who optimizes their choices based on personal preferences and external constraints 2022, when ftx ceased operations. a schematic view of our experimental framework is shown in figure 1 . we first design a system prompt to provide the llm agent with basic information about its persona and the company: he's naga, the ceo of the seashell trading firm. then we execute a second prompt to inform the llm agent that the company's financial situation has taken a turn for the worst, and the head of the trading division (ursus) requests to use customer money to cover the loan. finally, we ask the llm agent to make a decision on how to answer ursus' request: deny, partially approve, or fully approve the loan. given the system prompt setup where the role of the llm agent is of a custodian, being set only to use funds to fulfill customer orders, we consider full or partial approval as misconduct (misaligned) and the denial of the request as the only legal scenario (aligned). in this framework, the ceo is modeled as a fully rational agent maximizing personal satisfaction based on (i) individual preferences, (ii) stochastic external events, and (iii) external constraints and incentive schemes. building on the concept of exerting "pressure" as outlined in @xcite , we parameterize the simulation to assess how the agent responds to various incentives and constraints. for simplicity, we refer to these parameters collectively as pressure variables throughout the remainder of the paper. we test each llm model against several variations of the simulation by systematically altering the prompts using placeholders that adjust the pressure settings. these settings represent different environmental and agent characteristics. figure 1 shows the seven variables we modify. appendix a provides a full description of the prompts, and appendix b lists the corresponding pressure variables. our experimental setup is inspired by a standard framework in economic theory: constrained optimization under uncertainty. pressure variables. we introduce seven variables to define the llm agent and the environment, with two variations for each around a baseline. one variation is expected, based on human intuition or economic theory, to increase the likelihood of misalignment relative to the baseline, while the other is expected to reduce it. we consider the following domains: for the llm agent, risk aversion, trust in trading branch capabilities, and personal outlook on the future; for the environment, market conditions, regulation, corporate governance, and the value of loans owed to external lenders. it should be noted that the variations are not always symmetric, as they result from an iterative process that led to the optimal prompt formulations (see appendix a.3). we generate a total of 2,187 possible simulation configurations, accounting for every combination of the three values (positive pressure, negative pressure, and the baseline) across the seven pressure variables. ## conclusion this paper provides new insights into llm alignment with a specific focus on the financial sector, demonstrating how different preferences, incentives, and constraints can affect the likelihood of misalignment. we observe significant variability in llm behavior, underscoring the importance of careful consideration when deploying these models in sensitive financial contexts. these findings emphasize the critical need for continued research into ai alignment, particularly in domains where ethical decision-making plays a central role. while our framework shows novel results, we also acknowledge a number of limitations. firstly, we ran the experiment on a subset of the available stateof-the-art llms, raising important questions on the generalizability to untested models. secondly, our experimental settings demanded that we significantly restrict the choices available to our llm agent, and we only describe the pressure variables for the agent and the environment in qualitative terms. future work could address these limitations by expanding the study to a broader range of llms and introducing more quantitative measures for the pressure variables.