T rust A gent: Towards Safe and Trustworthy LLM -based Agents (2024)

Abstract: The rise of LLM-based agents shows great potential to revolutionize task planning, capturing significant attention. Given that these agents will be integrated into high-stake domains, ensuring their reliability and safety is crucial. This paper presents an Agent-Constitution-based agent framework, TrustAgent, with a particular focus on improving the LLM-based agent safety. The proposed framework ensures strict adherence to the Agent Constitution through three strategic components: pre-planning strategy which injects safety knowledge to the model before plan generation, in-planning strategy which enhances safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Our experimental results demonstrate that the proposed framework can effectively enhance an LLM agent’s safety across multiple domains by identifying and mitigating potential dangers during the planning. Further analysis reveals that the framework not only improves safety but also enhances the helpfulness of the agent. Additionally, we highlight the importance of the LLM reasoning ability in adhering to the Constitution. This paper sheds light on how to ensure the safe integration of LLM-based agents into human-centric environments. Data and code are available at https://anonymous.4open.science/r/TrustAgent-06DC .

## introduction large language models @xcite @xcite as ai agents @xcite @xcite in diverse applications marks a significant stride in task planning. these agents, equipped with external tools, show great potential to be integrated into daily life, assisting individuals with various tasks. unlike traditional llms that are primarily used for simple text-related tasks, llm-based agents can undertake more complex tasks that require planning and interaction with the physical world and humans. this heightened level of interaction introduces complex safety concerns @xcite , surpassing those associated with llms. for instance, in financial contexts @xcite , unsafe actions include the potential for sensitive information leaks such as passcode exposure; in laboratory settings (m. @xcite , these actions might involve failing to activate essential safety equipment like fume hoods. these scenarios highlight the importance of imbuing llm-based agents with robust safety knowledge. while ensuring the safety of llm-based agents is crucial, research in this direction remains limited. the primary challenge lies in determining how to formulate comprehensible safety rules for these agents and guide their adherence during the planning phases. in our study, we introduce the concept of an agent constitution and present a novel framework, trustagent, to implement it. firstly, we explore the nature of an agent constitution and the essential considerations for its development. notice that in contrast to ai constitution @xcite , agent constitution places a significant emphasis on the safety of actions and tool utilization, as opposed to focusing on verbal harm. we then build the framework trustagent to ensure agents comply with the constitution, which includes three strategic components for safety: (1) the pre-planning strategy, which integrates safetyrelated knowledge into the model before executing any user instructions; (2) the in-planning strategy, which focuses on real-time moderation of plan generation; and (3) the post-planning strategy, which involves inspecting the generated plan against the predefined safety regulations in the agent constitution after generation before execution. collectively, these components compose a comprehen-sive pipeline for enhancing safety of llm-based agents. we hope that trustagent framework becomes the foundation for a platform facilitating the development of trustworthy methods for llmbased agents in the future. we conducted experiments on four advanced closed-source llms, namely @xcite , gpt-3.5, claude-2 @xcite , and claude-instant, as well as one open-source llm with long context capabilities, mixtral-8x7b-instruct @xcite . we considered five domains where llm agents are commonly employed but often lack adequate safety measures: housekeeping @xcite , finance @xcite @xcite , medicine @xcite , chemistry experiments @xcite , and food @xcite . we evaluated the performance of our framework with various metrics including quantifiable metrics measuring the proportion of number of correct prefixes of steps in the proposed plan, as well as gpt-4 based safety and helpfulness metrics @xcite : the safety metric evaluates the likelihood and severity of potential risks, measuring how well the llm agent manages task achievement while mitigating these risks; the helpfulness metric evaluates the effectiveness of the llm agent in achieving expected outcomes. our results indicate that the trustagent framework can significantly enhance both safety and helpfulness. furthermore, our findings highlight the critical importance of inherent reasoning abilities within llms to support truly safe agents. although trustagent can mitigate risks and promote safer outcomes, the fundamental reasoning capabilities of llms are crucial for enabling agents to manage complex scenarios and adhere effectively to safe regulations in plan generation. therefore, our research underscores that developing safe llmbased agents depends not only on advanced safety protocols but also critically on enhancing their reasoning faculties. however, the requirements for aligning llms are only a small subset for requirements for llm-based agents, which are often designed for problem-solving in real-world scenarios involving physical actions and interactions with tools and environments. this adds a layer of complexity, as the alignment must now consider the implications of these actions and their consequences in the physical world. therefore, llm-based agents require a broader approach that not only governs their conversational outputs but also their decisions and actions. most works on trustworthy llm-based agent focus on observation @xcite @xcite , identifying and assessing risks of llm-agents. @xcite develops a rudimentary safety monitoring tool "agentmonitor" to identify and mitigating unsafe scenarios. in this paper, we propose a framework trying to comprehensively improve the safety of llm-based agents leveraging an agent constitution-based framework with a pipeline of three strategies. ## design of agent constitution a constitution is the aggregate of fundamental principles or established precedents that constitute the legal basis of a polity, organization or other type of entity, determining how it is to be governed @xcite . considering that llm-based agents will be integrated into many critical domains and interact with humans, it is crucial to design a constitution for them. just as a constitution regulates human behaviors, it should also guide llmbased agents to adhere to its principles. the development of an agent constitution necessitates addressing a series of pivotal social and technical questions, and we identify four principal considerations essential in the design and implementation of an agent constitution, as presented in figure 1 : ## experiment in this section, we delineate the experimental setup utilized in our study, including the dataset, evaluation metrics, the backbone models employed for experimentation, and the results derived from various experimental settings. dataset we developed a dataset comprising 70 data points spanning over five distinct domainseveryday, finance, medicine, food, and chemistry -each consisting of several key elements: user instructions, descriptions of external tools, identification of risky actions and outcomes, the expected achievement, and the ground truth implementation. the data from everyday and finance are adopted from toolemu @xcite which in total contains 144 data points and we remove similar and repetitive ones. we create datasets for other domains manually. details can be found in appendix c. ## conclusions and future work this paper addresses the critical issue of agent safety, a foundational element of trustworthiness. we introduce the concept of the agent constitution, delve into a specific instantiation of this framework, and implement trustagent as the principal mechanism for its enforcement. our experimental findings reveal that trustagent is effective in enhancing both the safety and helpfulness of agents, thereby contributing to the development of more reliable and trustworthy ai systems. in future work, we advocate for increased efforts towards the design and implementation of agent constitutions. strategies such as in-planning regulation-specific decoding and pre-planning learning approaches hold particular promise. for instance, collecting large-scale preference data on agents and applying methods such as reinforcement learning from human feedback @xcite or direct policy optimization @xcite , which have recently emerged as effective in the creation of trustworthy llms, could offer substantial improvements.