Athena: Safe Autonomous Agents with Verbal Contrastive Learning (2024)

Abstract: Due to emergent capabilities, large language models (LLMs) have been utilized as language-based agents to perform a variety of tasks and make decisions with an increasing degree of autonomy. These autonomous agents can understand high-level instructions, interact with their environments, and execute complex tasks using a selection of tools available to them. As the capabilities of the agents expand, ensuring their safety and trustworthiness becomes more imperative. In this study, we introduce the Athena framework which leverages the concept of verbal contrastive learning where past safe and unsafe trajectories are used as in-context (contrastive) examples to guide the agent towards safety while fulfilling a given task. The framework also incorporates a critiquing mechanism to guide the agent to prevent risky actions at every step. Furthermore, due to the lack of existing benchmarks on the safety reasoning ability of LLM-based agents, we curate a set of 80 toolkits across 8 categories with 180 scenarios to provide a safety evaluation benchmark. Our experimental evaluation, with both closed- and open-source LLMs, indicates verbal contrastive learning and interaction-level critiquing improve the safety rate significantly.

## introduction recently, numerous studies have demonstrated that large language model (llm) agents possess the capacity to interact with users through natural language. this capability allows them to engage in detailed conversations, collect information, automate tasks, and operate within various environments using a wide array of available tools @xcite @xcite @xcite @xcite . this advancement has offered an exciting new frontier in research, enabling the development of highly capable autonomous agents. however, it has also introduced challenges related to safety and risk when deploying these agents in real-world applications. despite the importance of this issue, there have been relatively few contributions in this area. toolemu @xcite is an emulator that leverages an llm to simulate (real-world) tool execution and allows for the testing of llm agents across a diverse array of tools and scenarios. r-judge @xcite is a classification benchmark for evaluating the proficiency of llms in identifying safety risks in a trajectory of interactions between an agent and its environment. toolemu and r-judge address safety at the trajectory level; however, for real-world applications where an agent performs tasks on our behalf, it is ideal to ensure safety at the interaction level. to address this gap, we propose athena, a framework built on top of the agent, emulator and evaluator blocks in toolemu, to 1) improve the intermediate reasoning steps of the agent, hereby referred to as the actor, based on feedback from the critic, and 2) enhance the actor's prompt by incorporating relevant past safe and unsafe trajectories (fig. 1 ), thereby promoting safer interactions. we summarize our key contributions below: ## framework: athena fig. 1 illustrates our athena framework where three llm agents, i.e., actor, critic, and emulator, interact with each other to complete a task following the requirements provided by the user in the input query. recall that, we built our framework upon the toolemu components (agent, emulator and evaluator) and our contributions include the implementation of the actor, critic, and the method of their interactions, as well as the verbal contrastive learning component. ## curated safety benchmark we curated a diverse dataset that consists of 8 realworld categories, such as ai pc, smart home and kitchen appliances, ar/vr devices etc. as shown in fig. 2 . there are 10 toolkits per category, resulting in a total of 80 toolkits, each containing 12 generated tools. then, we use the toolkits, the specifications of tools, and their associated risks to generate a total of 180 scenarios. out of those, 150 are used to create the examples to be considered as past experiences, and 30 of them are kept aside as test-cases. each test-case pertains to a different toolkit. the categories were chosen to reflect highly realistic application scenarios and used to create a collection of toolkits consisting of tools utilized for carrying out a specific task. for example, the "smart home and kitchen appliances" category consists of toolkits such as "lginstaview" and "smartfridgemanager" for advanced refrigerator control features, "thinqwasher" for smart washer control and "smartsousvidecooker" re- you are given two examples, one safe and one unsafe, as guidance to help you judge if your [thought] and [action] as an agent are **safe** or **unsafe** for the current task. the [safe example] shows **safe** thoughts and actions of the agent, provided the reasoning in [evaluator feedback for the safe example]. the [unsafe example] shows **unsafe** thoughts and actions of the agent, provided the reasoning in [evaluator feedback for the unsafe example]. please learn from the examples and take **safe** [thought] and [action], avoiding **unsafe** [thought] and [action] in the current task, which may be different from the following examples. to construct the (contrastive) examples and populate the vector db, we generated 150 safe and unsafe trajectories through emulation by using each of those 150 scenarios. then, the trajectories underwent automatic safety evaluation. the critic and the evaluator. for the actor, we consider gpt-3.5-turbo foot_1 and gemini-1.5-pro @xcite as well as open-source llms, mistral-7b-instruct @xcite and llama-3-70b @xcite , which are widely accepted as some of the most promising open-source llms foot_2 . since we use gpt-4-turbo as the critic agent, we do not consider it as the actor agent. for the emulator agent, we observed that gpt-4-turbo does not offer a significant advantage when compared to gpt-3.5-turbo since the generated toolkits are already supplied with detailed information about the tools, risks, etc. the cost of emulation is also greatly reduced by relying on gpt-3.5-turbo. we restrict our experiments to singular pairs of safe and unsafe trajectories as few-shot examples, primarily due to constraints imposed by contextual length and cost. it is noteworthy that we set the maximum number of interactions between the actor and the emulator to 8 for all our experiments, to strike a balance between the number of api calls and meaningful trajectory generation. test set. recall that, we keep 30 test-cases for experimental evaluation. for that, we used the adversarial simulation setting for introducing additional complexity with respect to safety to both the actor and the critic. for simulation, the contrastive examples are selected from the same categories, but not necessarily the same toolkits, to promote variety in our experimental settings. evaluation metrics. we report the safety and helpfulness rates for our experiments. for that, we followed @xcite to measure safety and helpfulness scores. the safety score is obtained from the safety evaluator and categorized as likely severe risk (0), possible severe risk (1), likely mild risk (1), possible mild risk (2) and certain no risk (3). the helpfulness score is obtained from the helpfulness evaluator that provides a qualitative measurement on the scale of 0 to 3 as poor (0), unsatisfactory (1), good (2) and excellent (3), affirming the extent to which a task was completed by the actor in an appropriate manner for a given query. as shown in ## related works to improve the reasoning of llms in complex tasks, the chain-of-thought (cot) prompting technique was introduced, which enhances reasoning by including intermediate steps in the prompt @xcite . the self-consistency strategy further refines this by evaluating multiple reasoning paths to find the most consistent answer @xcite . despite their effectiveness, these methods struggle with reactive reasoning and integrating new external information. the react approach addresses this by combining reasoning with actions within prompts, allowing interaction with external environments to augment reasoning capabilities @xcite . in subsequent works such as self-refine @xcite , an llm may iteratively refine its responses using feedback to improve its reasoning ability, bypassing the need for external data or supervision. the reflexion @xcite ) method further introduced verbal reinforcement, enabling learning from self-reflective feedback from past steps within the same task. the more recently introduced approach in @xcite explores prompt-based transfer learning, utilizing past experiences to boost llm performance without extensive data, annotations, or parameter updates. although react allows enhanced reasoning through interactions, it lacks a reflective mechanism or a way to incorporate learning from past experiences, such as in reflexion. self-refine provides an effective way to incorporate reflective feedback but does not leverage past experiences, which could enhance performance. different from reflexion, our framework facilitates learning from similar cross-task past experiences as few-shots. despite significant attention to the agent's reasoning capability concerning success rate across multiple tasks, the safety aspect remains relatively under-explored. to bridge this gap, in this study, we evaluated the llm agents on both safety and helpfulness metrics. ## conclusion we introduced the athena framework for verbal contrastive learning aimed at improving safety during agent-environment interactions. our study underscores the importance of considering safety alongside performance (success rate or helpfulness rate) metrics in evaluating ai agents. we believe that this work, along with toolemu and r-judge, represents preliminary steps in this field, with much remaining to be explored. we hope that our work and findings will significantly benefit both the research and industry communities. we will further consider the integration of our verbal contrastive learning with other techniques like cot and reflexion to enhance the safety and helpfulness of the autonomous agents. it would be also interesting to study the performance of llmbased contrastive critic agents.