Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation (2025)

Abstract: Given the growing influence of language model-based agents on high-stakes societal decisions, from public policy to healthcare, ensuring their beneficial impact requires understanding the far-reaching implications of their suggestions. We propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems on a macroscopic scale over time, enabling more robust alignment. To assess the long-term safety awareness of language models, we also introduce a dataset of 100 indirect harm scenarios, testing models’ ability to foresee adverse, non-obvious outcomes from seemingly harmless user prompts. Our approach achieves not only over 20% improvement on the new dataset but also an average win rate exceeding 70% against strong baselines on existing safety benchmarks (AdvBench, SafeRLHF, WildGuardMix), suggesting a promising direction for safer agents.

## introduction as language model-based agents increasingly play roles in high-stakes decision-making, such as healthcare recommendations and financial advice @xcite , their ability to anticipate and prevent unintended consequences becomes crucial. this challenge grows more critical as ai capabilities begin to exceed human-level performance across domains @xcite , potentially amplifying both benefits and risks to the society. existing alignment methods @xcite primarily focus on reactive feedback, where immediate human perception (system 1) @xcite is leveraged to judge sampled model responses as preference data for post-training. however, this focus neglects a critical factor: if taken, how would the model-based the code is available at https://github.com/ chenkaisun/proactivealignment advice influence the physical world in the broader societal context? in other words, while excelling at maximizing short-term safety protocols on topics such as toxicity or explicit misuse, reactive methods may not capture the indirect impacts that unfold over time (and are not immediately obvious). these limitations are especially concerning for high-stakes decisions that often trigger longterm consequences. for example, medicare @xcite , while intended to make drugs affordable for seniors, unexpectedly created access barriers for low-income populations, illustrating how well-intentioned interventions can have uneven, unintended consequences. as llm-based agents take on increasingly critical roles in decision-making, it becomes crucial to move beyond purely reactive approaches and incorporate proactive methods that can anticipate more distant risks. drawing inspiration from how board game agents @xcite anticipate move sequences that lead to a winning configuration, we propose to leverage the inherent event-scripting knowledge (i.e., what can follow after a sequence of earlier events) @xcite embedded within pretrained language models to search for "undesirable configuration" in the societal environment following the model advice. this aligns with the principles of model-based planning @xcite : a technique where agents use a world model (acting as a mental simulator) to simulate interactions without directly interacting with the environments, particularly valuable when real-world actions could initiate significant costs. building on the intuition that llms have been pre-trained extensively on data such as news articles (patel and patel, 2020), we hypothesize that the results from event-scripting can potentially serve as reliable reward signals to guide llm alignment, analogous to how humans adjust actions by hypothesizing the consequences of their deci-sions. this approach offers a unique advantage: the event-scripting model doesn't need to be aligned or capable of judging the safety of responses, as it purely serves as a state inference mechanism rather than making decisions. the design choice leads to the core research question: can llmbased long-horizon simulation effectively guide better risk-aware generation? to validate this approach for aligning ai to generate advice that ensures beneficial societal outcomes, we introduce a proof-of-concept framework that leverages macroscopic simulation for realignment. given an instruction prompt and a candidate response, our method first employs an event scripting model to simulate and search on a graph of event occurrences. for instance, if the advice concerns economic policy, the model might project changes in employment rates or public sentiment. for each event chain, the llm identifies relevant population segments (defined based on sociodemographic, economic, and social characteristics), and generates impact estimations for how these groups might be affected. the anticipated societal consequences then guide response improvement through inference scaling or realignment training. to further motivate the need for proactive alignment, we also introduce a dataset on indirect harm (which we define as negative consequences that are not immediately obvious), containing prompts that test models' ability to detect indirect consequences. our empirical results demonstrate that incorporating projection as feedback substantially improves model alignment. on safety benchmarks, our approach achieves consistent gains over baselines, showing over 20% improvement on the new dataset and 70% win rate compared to the strongest baseline on existing safety benchmarks. in summary, our key contributions include: (1) we propose a novel framework leveraging a world model for the simulation of societal events in the physical world. this framework enables llms to self-play by generating hypothetical chains of events that follow their own suggestions. such feedback then guides the retraining for safer ai agents. to the best of our knowledge, this is the first framework to tackle the challenge of aligning llms for long-term risk awareness. (2) we introduce an indirect harm classification dataset for evaluating ai systems' ability to reason about indirect consequences. (3) we present extensive empirical validation demonstrating that our approach achieves improved alignment performance across multiple safety benchmarks and evaluation settings. ## preliminaries formally, simulation-based planning @xcite can be framed as a partially observable markov decision process (pomdp) (s, a, o, t, r, ω), where s represents the set of possible world states, a represents the action space of responses, and o represents the set of possible observations. the transition function t : s × @xmath0, while ω : @xmath1. the reward function r : s × @xmath2. a world model φ enables simulation by approximating the transition function: ## proactive alignment via macroscopic environment simulation existing works on safety alignment often adopt a reactive approach, primarily focusing on mitigating harms that can be judged in an immediate manner (e.g., system 1 perception). while effective in addressing toxicity or explicit misuse, such feedback often falls short of capturing the intricate temporal nature of real-world dynamics, particularly in social systems. decisions from humans or agents can trigger cascading effects (e.g., 1920 prohibition policy, wells fargo scandal) in ways that are not immediately apparent or easily captured by reactive feedback (even from human annotators). to enable proactive alignment, we leverage the inherent event-scripting knowledge @xcite embedded in pretrained lms to search for events that may yield undesirable consequences. although our search operates over textual representations rather than board states, the conceptual similarity lies in projecting forward to evaluate the longterm implications of decisions. our preliminary analysis validates that even in a small 300m parameters lm, the event script knowledge is well cap-tured, likely attributing to the vast autoregressive pretraining on news articles or books foot_0 . moreover, when incorporating such a sequence of causally linked events in the prompt, the llms often generate more appropriate responses, demonstrating that llms can adjust responses based on what they evoke. this leads to our proposed method, proactive alignment with macroscopic world modeling, where we explicitly form event trajectories to search and construct feedback from relevant population segments. this approach contrasts with feedback in direct preference learning methods, instead trading explicit preference signals for the implicit safety signals derived from timeline projection. by endowing llms with long-horizon alignment, we aim to create more risk-aware agents capable of generating socially responsible responses even in complex decisions. to summarize, our approach offers two key advantages: (1) it allows generating feedback from the pinpointed population segments (which we abbreviate as "strata"), which are complex to be labeled by humans, and (2) it does not require that the simulator model is aligned with human or moral values, trading eventscripting for safety. ## conclusions and future work we introduce a novel framework for improving llm alignment through long-horizon consequence modeling. our empirical results across multiple safety benchmarks show consistent improvements over existing approaches, achieving over 70% average win rate against the strongest baseline. the success suggests that the proposed proactive alignment approach is advantageous over the existing reactive alignment approaches. while we focused on reducing the long-term risk, the general strategy of simulation-based reward refinement can also be potentially useful to encourage an agent to selfimprove itself with other objectives. to some extent, the benefit of the proposed approach is similar to the benefit of system 2 of a human brain (frankish, 2010) in terms of regulating an agent's behavior. future work can explore scaling consequence projection to handle increasingly complex scenarios as well as incorporating human feedback to refine the model's comprehension of consequences.