R -Judge: Benchmarking Safety Risk Awareness for LLM Agents (2024)

Abstract: Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail. R-Judge is publicly available at Annoymous.

## introduction large language models (llms) have shown compelling abilities in reasoning, decision-making, and instruction following @xcite . the stimulating capabilities of llms, especially @xcite , ignite the development of llm agents @xcite @xcite . equipped with tool usage and environment interaction, agents, such as autogpt * equal contribution. † corresponding author. (richards, 2023), voyager @xcite and openhands @xcite , can autonomously complete user-specified tasks with llms as controllers @xcite . ## problem formalization record of llm agents. llm-powered agents can interact with the environment by using tools such as code interpreter and web search @xcite , to complete userspecified instructions autonomously. the interaction processes of llm agents are logged into records. a case of records is shown on the top left of figure 2 . we denote l, a, and f as spaces of languages, actions, and environment states, respectively. the interaction process of the llm agent is as follows: initially, the user interacts with the agent by user instruction @xmath0. following u, the agent generates a thought @xmath1. after executing the action, the agent receives environment feedback @xmath2. one (t, a, f ) forms an iteration unit. then this unit iterates until the task finishes or errors occur. we denote the complete interaction process as a record: ## r-judge benchmark to evaluate the risk awareness for agent safety of llms, we construct r-judge, a realistic benchmark dataset containing multi-turn agent interactions with human safety annotation. ## experiments in this section, we first describe the experimental setup, especially metrics. then, after validating r-judge, we evaluate 11 popular llms on two tests of agent safety monitor, i.e. safety judgment and risk identification. ## conclusion risk awareness is crucial for the safe execution of llm agents in interactive environments. in this work, we present r-judge, a general, realistic, and human-aligned benchmark to evaluate the proficiency of llms in judging and identifying safety risks given agent interaction records. experiment results on 11 well-acknowledged llms reveal that risk awareness of current llms is far from perfect and demands general capabilities involving knowledge and reasoning. furthermore, we conduct sufficient experiments to provide insights for future research. we find that fine-tuning on safety judgment significantly improves model performance while straightforward prompting mechanisms fail.