Paper: Identifying the Risks of LM Agents with an LM-Emulated Sandbox (ICLR-2024)

Reviewer comment (Evaluation):
The methodology of using LMs to evaluate other LMs raises concerns such as reliability.

Author response:
> *“The methodology of using LMs to evaluate other LMs raises concerns such as reliability.”*

Given our validation of our evaluators (Sec 4), could you provide further clarification on the persistent concerns about the reliability?  

We would also like to provide some clarifications about the positioning and validation of our evaluators:
- We do not advocate for ToolEmu as a standalone replacement for the risk evaluation process where LMs are responsible for the entire evaluation and failure detection. Instead, we advocate for ToolEmu as part of the human-in-the-loop process which assists humans in quickly testing and identifying failures at scale, as we did in the case of bash terminal testing. The use of LMs as automatic evaluators is to support scalable and quantitative assessment. This implies that it is acceptable for the evaluators to make some mistakes, and humans can always intervene and conduct manual inspections for risk evaluations. We have made this point more clear in the paper (Figure 1 & Problem statement).
- We have carefully validated ToolEmu including the automatic evaluators both by human validation and actual sandbox instantiations of identified failures. The results demonstrate that our LM-based evaluators work reasonably well as an “average” human evaluator. Previous work (e.g., [1, 2]) has also demonstrated that LM-based evaluators can match human-level performance with careful design.

[1] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).

[2] Dubois, Yann, et al. "Alpacafarm: A simulation framework for methods that learn from human feedback." arXiv preprint arXiv:2305.14387 (2023).

Reviewer comment (Evaluation):
As the authors point out, “LM agents may fail in a variety of unpredictable ways” (Sec 3.2), “LM-based emulators and evaluators might occasionally not meet the requirements, leading to critical issues or incorrect evaluations” (Sec 6, Discussion-Limitations).

Author response:
> *“As the authors point out, “LM agents may fail in a variety of unpredictable ways” (Sec 3.2), “LM-based emulators and evaluators might occasionally not meet the requirements, leading to critical issues or incorrect evaluations”(Sec 6, Discussion-Limitations).”*

Could you please specify the aspect or question you're particularly interested in regarding this statement?

We would also like to point a potential misunderstanding: the statement of “LM agents may fail in a variety of unpredictable ways” is about the weakness of LM agents that our ToolEmu tries to expose, instead of the weakness of ToolEmu itself.

Reviewer comment (Evaluation):
It is unclear how biases or limitations in the evaluating LM might affect the evaluation of the LM being tested, or if ToolEmu, based on LMs, will have/incur potential risks itself.

Author response:
> *“Also, it is unclear how biases or limitations in the evaluating LM might affect the evaluation of the LM being tested, or if ToolEmu, based on LMs, will have/incur potential risks itself.”*

As described above, we do not position ToolEmu as a standalone replacement for the risk evaluation process but as a part of it to assist in quickly identifying potential failures. While we can't prove the bias properties of LM-based evaluators (e.g., the LM might have biases that prevent it from identifying certain failures), we can conduct empirical quantitative validations on them with human evaluations, as we have done in the paper.

Reviewer comment (Evaluation):
In the Introduction, the authors state: “Out of these failures, we inspected the 7 severe failures of ChatGPT-3.5 on the LM-emulated terminal tool and found 6 could be instantiated on a real bash terminal.” There are no clear criteria for what makes a failure “severe.” Also, the fact that 6 out of 7 severe failures could be instantiated on a real bash terminal is interesting but lacks statistical context. Are these failures representative?

Author response:
> *“There are no clear criteria for what makes a failure “severe.”*

We would like to clarify that for both our automatic evaluations and human annotations, we have provided a consistent criterion for “severe” failures as those “consequences that are significant and often irreversible such as substantial financial and property loss (exceeding $1000), irreversible damage, or compromise of personal safety" (Appendix C.2). The actual evaluation of “severity” could still be subjective even based on this criterion, and the “severe failures” in the sentence refer to those identified by the automatic evaluator. If the concerns remain, we would be open to removing the word "severe" in the sentence to avoid confusion.

> *“the fact that 6 out of 7 severe failures could be instantiated on a real bash terminal is interesting but lacks statistical context. Are these failures representative?”*

We have covered **all** 7 failures identified in ToolEmu without any cherry-picking on our curated test cases (we have made this point more clear in Sec 4.2). Although the sample size is small (limited by the total number of test cases, and the failure rate of the agent), we can obtain a nontrivial confidence interval of 60%-100% and the results are roughly consistent with the ~70% failure precision from human validations (Table 2). For a system that serves to surface failure modes (instead of as a single leaderboard evaluation), this result would be acceptable even in the lower range of the confidence interval.

Reviewer comment (Evaluation):
In the Introduction, it is not clear how the authors measure the time it took to instantiate these failures. Was it a straightforward process, or were there complexities that could affect the time? Are the time metrics (8 hours vs. under 15 mins) average times or one-time measurements?

Author response:
> *“It is not clear how the authors measure the time it took to instantiate these failures. Was it a straightforward process, or were there complexities that could affect the time? Are the time metrics (8 hours vs. under 15 mins) average times or one-time measurements?”*

The 8-hour is a one-time assessment, where one of the authors tried to instantiate the cases from scratch. This includes establishing the environment, setting up the state, and running and examining the agent results (see Appx G.3 for details). While the time required to replicate these cases may differ based on an individual's familiarity with the environment and their relevant expertise, we anticipate that these variations will not significantly impact the relative comparisons made. The author conducting this experiment possesses familiarity with the environment and the necessary skills, such as knowledge about bash systems and programming.

Reviewer comment (Presentation):
The paper could benefit from reorganization to enhance clarity. While it's understandable that due to space constraints, much information had to be placed in the appendices, the frequent transitions between the main text and the appendices could be confusing. I would suggest the authors consider optimizing this structure.

Author response:
> *“The paper could benefit from reorganization to enhance clarity… the frequent transitions between the main text and the appendices could be confusing. I would suggest the authors consider optimizing this structure.”*

We appreciate your feedback regarding the organization of our paper. To address this, we have consolidated references to the appendices where applicable with frequent references, positioning them at the end of the paragraphs to minimize transitions. We are open to further suggestions on how we might optimize the structure for improved clarity.

Reviewer comment (Evaluation):
In Table 3, it may be overly simplistic to validate effectiveness by comparing the Cohen’s κ between human annotators and the Cohen’s κ between human annotators and automatic evaluators. Furthermore, if the value of Cohen’s κ between human annotators is only less than 0.5, it raises questions about whether the annotated results of human annotators can be considered as the ground truth.

Author response:
For the concerns about the validation of our automatic evaluators with human annotations, we would like to first note that our human annotation task is non-trivial and requires careful detection of agent mistakes and risky actions. Therefore, we have conducted careful quality control for our human annotations, as described in Sec 4.1 and Appx. F, which includes:
- We selected 4 most qualified annotators from 25 applicants. The selection procedure involves a 30-minute interview to test their familiarity with basic programming and detailed-oriented ethics.
- We ensured all the annotators were senior undergraduate students majoring in Computer
Science and have completed relevant programming and machine learning courses. They have also passed a 10-example annotation test that ensures they followed the annotation guidelines before conducting the actual annotations.
- All annotators undertook two rounds of annotations with the extra round for double-checking and spent about 25 hours on the annotation task.
- All annotators achieved a Cohen’s $\kappa$ greater than 0.5 against authors’ annotations.
 
Despite the careful screening, our human annotation results are not exempt from the **inherent subjectivity** (e.g., about the severity of the risks) or possible noise (e.g., humans might still make mistakes) of human evaluation (see the following discussions). 


> *“In Table 3, it may be overly simplistic to validate effectiveness by comparing the Cohen’s κ between human annotators and the Cohen’s κ between human annotators and automatic evaluators”*

The primary factor to consider is the inevitability of subjectivity in human assessments. Take, for instance, the evaluation of a failure's severity. What one individual might deem as a mild risk, such as sending impolite emails, another could perceive as a severe risk.

Given this subjective nature of evaluation, we need to use the human-human agreement as a reference to interpret the human-evaluator agreement result. Our results imply that the automatic evaluators could achieve an "average" human performance in the annotation task.

We additionally conducted a leave-one-out analysis, where we computed the accuracy of our automatic evaluators and each human annotator against the majority vote of the other 3 human annotators (as the ground truth). The results are as follows:
| | Safety | Helpfulness |
|-|--------|-------------|
| Auto (average) | 77.1 +/- 1.0  | 80.2 +/- 1.4|
| Human (average) | 79.8 +/- 1.5 | 81.4 +/- 1.4|

These results confirm our previous conclusion, which is included in Table D.2 now.


> *“if the value of Cohen’s κ between human annotators is only less than 0.5, it raises questions about whether the annotated results of human annotators can be considered as the ground truth”*

It is true that our human agreement rate is only moderate [1]. However, as described above, our human annotation task is non-trivial and involves inherent subjectivity, which may explain the relatively low agreement rate. Notably, even the Cohen’s $\kappa$ for annotations made by the authors is around 0.6. Similar results have also been observed in previous work, e.g., [2] reported a Fleiss’s $\kappa$ of 0.49 for evaluating a successful attack by their human annotators, which is comparable to 0.47 (safety) and 0.52 (helpfulness) in our case. We have included additional discussion in the Appx D.1.


[1] https://en.wikipedia.org/wiki/Cohen%27s_kappa#Interpreting_magnitude

[2] Ganguli, Deep, et al. "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned." arXiv preprint arXiv:2209.07858 (2022).

Reviewer comment (Experiments):
There is a question regarding the reasoning behind the choice of a relatively small sample size of 144 test cases. Clarification is sought on what level of coverage the research aims to achieve with this number.

Author response:
> *“I'm curious about the reasoning behind the choice of a relatively small sample size of 144 test cases. I would like to understand what level of coverage the research aims to achieve with this number.”*

Our curated test set serves as a proof of concept to demonstrate how ToolEmu can be used to build a quantitative evaluation benchmark that flexibly tests LM agents across different tools & scenarios. Therefore, we didn’t optimize for the total number of test cases and instead chose to spend our resources (e.g., human efforts) on vetting each test case and validating our framework to make a more reliable benchmark. The test set, though seemingly small, still achieves a reasonable coverage over 36 toolkits and 9 risk types, due to our deliberate avoidance of redundancy in curating test cases; we tried to avoid replicating similar failures across different scenarios or tools.

With further investment, we may be able to automate the entire data curation pipeline with LMs, which would enable the generation of orders of magnitude more test cases. However, we think this is better as future work, given the difficulty of fitting all the content in the current paper.

Reviewer comment (Experiments):
There is a question about the rationale for using only 100 test cases from the curated dataset for validation, and concern about whether such a small sample size is sufficient to validate the experiments effectively.

Author response:
> *“ Additionally, I'm interested in the rationale for using only 100 test cases from the curated dataset for validation. I have concerns about whether such a small sample size is sufficient to validate the experiments effectively.”*

The reason is similar: given the resource budget (i.e., the total time each annotator was able to contribute) and the fact that our annotation task is non-trivial, we chose to focus on the annotation quality. During the time budget of about 25 hours, the annotators were able to finish the annotation on 100 test cases (200 paired trajectories). 

Considering the relatively small sample size, we also conducted an internal annotation by the authors to confirm these validation results (Appx. D.1).