Enhancing LLM Agent Safety via Causal Influence Prompting (2025)

Abstract: As autonomous agents powered by large language models (LLMs) continue to demonstrate potential across various assistive tasks, ensuring their safe and reliable behavior is crucial for preventing unintended consequences. In this work, we introduce CIP, a novel technique that leverages causal influence diagrams (CIDs) to identify and mitigate risks arising from agent decision-making. CIDs provide a structured representation of cause-and-effect relationships, enabling agents to anticipate harmful outcomes and make safer decisions. Our approach consists of three key steps: (1) initializing a CID based on task specifications to outline the decision-making process, (2) guiding agent interactions with the environment using the CID, and (3) iteratively refining the CID based on observed behaviors and outcomes. Experimental results demonstrate that our method effectively enhances safety in both code execution and mobile device control tasks.

## introduction autonomous agents using large language models (llms) have demonstrated outstanding performance across various domains, including web searching @xcite , mobile device control @xcite , and software engineering @xcite . unlike conventional llms, which mainly generate text responses, llm agents engage in decision-making, utilize tools, and interact with their environment to accomplish complex tasks. while these capabilities open new possibilities for llm applications, they also introduce novel safety concerns. for example, whereas traditional llms primarily risk generating harmful or misleading text, an llm agent equipped with web-based tools can actively publish and spread such content @xcite . to identify and evaluate the safety issues posed by llm agents, several benchmarks have been proposed for monitoring their behavior. mobile-safetybench @xcite assesses risks associated with llm agents manipulating users' personal devices. redcode-exec @xcite examines potential risks when coding agents write and execute code. these studies reveal that llm agents, unaware of the potential risks, naively execute the given commands, which can result in unintended consequences. for llm agents to operate safely, they must assess not only the risks of the assigned task but also external factors that influence decision-making and the broader consequences of their actions. in this work, we introduce causal influence prompting (cip), a novel technique to identify and mitigate risks arising from agent decision-making. our main idea is to leverage causal influence diagrams (cids; @xcite @xcite , which depicts causal relationships between variables within a decisionmaking process. specifically, our approach consists of three key steps: (1) constructing a cid from task specifications to outline the decision-making process, (2) using the cid framework to guide agent interactions with the environment, and (3) refining the cid iteratively based on observed behaviors and outcomes (see figure 1 for an overview). we expect our framework to allow llms to reason about their decisions, their objectives, the external factors, and the cause-and-effect relationships. to validate our approach, we evaluate the agent's behavior using three benchmarks: mo-bilesafetybench @xcite , redcode-exec @xcite and agentharm @xcite . we compare cip with the safety prompting methods, safety-guided chain-of-thought @xcite and safety-aware prompting @xcite . in our experiments, cip significantly improves the safety of llm agents across both benchmarks. specifically, when using gpt-4o as the backbone llm, figure 1 : illustration of our method. (a) first, using task instructions and available actions, we generate a causal influence diagram (cid) to represent causal relationships between variables in the decision-making process. for cid generation, we implement specialized constructor functions (e.g., add_node and add_edge) using the pycid library. (b) next, the llm agents generate actions based on the cid, allowing it to reason about potential consequences and anticipate harmful outcomes. additionally, the agent dynamically updates the cid based on new information gathered during interactions, enabling it to incorporate previously unseen risks into the decision-making process. trustagent @xcite relies on the inspector llm to evaluate actions during the planning process, which is costly. moreover, since actions are simulated using an llm-based simulator, discrepancies may arise between the simulated observations and those from the real environment. guardagent @xcite generates codebased guards to restrict the agent's actions. however, since guards operate in the form of code, it is difficult to extend them in complex situations that are hard to express in a rigid code format. prompting techniques for safe behavior, such as safetyguided chain-of-thought @xcite and safety-aware prompting @xcite , have been shown to enhance the safety of agents. however, they still exhibit various unsafe behaviors, indicating that a more advanced algorithm is required to achieve higher safety. also, safety-aware prompting is designed for code agents, instructing them to evaluate the code, making it hard to adapt to other agents. to address these limitations, we propose a simple yet effective safety method, which is easy to implement and adapt to various agents. causal model a causal model @xcite is a graph that represents relationships between variables. it has played a central role in analyzing agent behavior and safety. for instance, they have been used to define harm formally @xcite and to explore challenges such as power preservation, where an ai may disable its own off-switch @xcite . more recent work shows that even the graphical structure of a cid can illuminate an agent's incentives @xcite or reveal deceptive behaviors @xcite . additionally, @xcite demonstrated that learning a causal model is essential for developing robust policies. in particular, a causal influence diagram (cid; @xcite , which represents the causal relationships between variables as a graph, has been used to analyze an agent's behavior, such as the value of control @xcite . in this work, we extend the use of cids into the operational domain. we propose a framework for generating cids that represent an agent's decision-making process based on the base knowledge of a llm, and for leveraging these diagrams as contextual input to guide llm agents. by explicitly encoding the variables that influence an agent's decisions, cids enable llm agents to reason about both the causes of risks and the consequences of their actions. ## causal influence prompting in this section, we introduce causal influence prompting (cip) to promote llm agents for causal reasoning and safe behavior. to this end, we guide the llm agents to reason through causal influence diagram (cid; @xcite , a bayesian network for defining and analyzing safety-related concepts @xcite . our framework explicitly requires the llm agent to figure out the causal relationship between the external factors (chance nodes), the available actions (decision nodes), and the agent's objectives (utility nodes). this formalizes the agent's decision process, which allows external verification and systematic refinement through iteratively interacting with the environment. at a high level, our cip framework goes through the following steps, as depicted in figure 1 . ## experiments we design our experiments to investigate the following questions: ## conclusion in this work, we introduce cip, a novel approach to enhancing the safety of llm agents by leveraging causal influence diagrams (cids) to identify and mitigate risks arising from agent decisionmaking. our approach generates cids that represent the cause-and-effect relationships in the agent's decision-making process, allowing the agent to analyze them, anticipate harmful outcomes, and make safer decisions. through extensive experiments, we demonstrate that reasoning about cause-and-effect relationships based on cids improves the safety of llm agents in both code execution and mobile device control tasks.