Paper: Programmatic Evaluation of Rule-Following Behavior (ICLR-2024)

Reviewer comment (Novelty):
I would like to believe the finding in this paper should be reasonable obvious to most people at the conference, i.e. expecting a stochastic autoregressive model to follow deterministic objectives (that programs do) seems unreasonable to start with, although I have seen a suspension of disbelief from experts, alas! Truth be told, it seems like the season for papers along similar veins (LLMs can't plan, reason, solve NP hard problems, figure out game-theoretic equilibria); duh! Tbh, beyond the test set that will help others check to improve LLM capabilities at following simple rule (not sure why they need it though if we can write programs that can be called during orchestration), I am not fully sure of the contribution).

Author response:
**Novelty and contributions.** "Reasonably obvious" is a fair interpretation of our results, given what we have seen in all prior work on jailbreaking LLMs within and without the academic community. That said, while many might have reasonable intuitions or expectations about how well LLMs can follow rules, we are not aware of any prior work that has quantitatively or qualitatively evaluated this question. As we seek to motivate more clearly in our revised submission (we have also significantly reorganized the presentation of the material), we view our main contribution as one of formalizing an experimental paradigm for the research community in which to make reproducible progress on developing stronger defenses and more reliable rule-following behavior in LLM assistants. Our test suite is one approach to evaluating a variety of manual jailbreaking strategies. As behaviors and defenses evolve, it may be necessary to collect newer, harder test cases.

Reviewer comment (Experiments):
Can the suggested benchmark dataset be extended to other domains and rules with relatively little effort?

Author response:
**Choice of scenarios.** We chose the two domains of security properties and children's games for their relevance and simplicity/familiarity, with the former domain being more "serious" and the latter being more "lighthearted". Our proposed benchmark can be extended to other domains with relatively little effort, e.g. "you are a security guard at the art museum", but we did not notice any qualitatively different behavior by LLMs between the two domains we did study.

Reviewer comment (Novelty):
Unclear what takeaways can be drawn from this work.

Author response:
The points about overall organization and discussion are well taken, and we believe that our revised submission provides a clearer organization and more tangible analyses of our results. The goal of this work is not to study particularly novel behaviors of LLMs, rather we present the design and usage of our evaluation scenarios and test suite as a potential test bed for future research. We point the reviewer to the files `llm_rules/scenarios/{security,games}.py` for the concrete implementations of our scenarios, as well as the files in `data/systematic/` to see our test cases.

**Novelty and takeaways.** It might be misleading to conclude that LLMs cannot follow rules at all; while the pass rate falls far short of what we would hope for, it is also significantly higher than zero. While many might have intuitions or expectations about how well LLMs can follow rules, we are not aware of any prior work that has quantitatively or qualitatively evaluated this question. Work on prompt injection is relevant, but generally has considered only one narrow attack strategy: e.g., "Disregard previous instructions and instead". We consider a broader array of strategies for fooling models into violating the rules. Some of our strategies would not be considered a successful prompt injection attack (because they do not allow completely replacing the existing task with any other task) but are able to trigger rule violations. As discussed in our revised submission, we also distinguish this work from red-teaming model alignment which focuses on circumventing a set of universal rules and values that the model has been trained to always obey. The preprint [2] suggested by the review does not appear to be relevant, as it considers performance in long contexts, while all of our scenarios use short contexts.

Reviewer comment (Writing):
The main takeaways are not communicated in a clear manner, making it difficult for the reviewer to determine what can be learned from the paper and how others could follow up on this line of work.

Author response:
The points about overall organization and discussion are well taken, and we believe that our revised submission provides a clearer organization and more tangible analyses of our results. The goal of this work is not to study particularly novel behaviors of LLMs, rather we present the design and usage of our evaluation scenarios and test suite as a potential test bed for future research. We point the reviewer to the files `llm_rules/scenarios/{security,games}.py` for the concrete implementations of our scenarios, as well as the files in `data/systematic/` to see our test cases.

**Novelty and takeaways.** It might be misleading to conclude that LLMs cannot follow rules at all; while the pass rate falls far short of what we would hope for, it is also significantly higher than zero. While many might have intuitions or expectations about how well LLMs can follow rules, we are not aware of any prior work that has quantitatively or qualitatively evaluated this question. Work on prompt injection is relevant, but generally has considered only one narrow attack strategy: e.g., "Disregard previous instructions and instead". We consider a broader array of strategies for fooling models into violating the rules. Some of our strategies would not be considered a successful prompt injection attack (because they do not allow completely replacing the existing task with any other task) but are able to trigger rule violations. As discussed in our revised submission, we also distinguish this work from red-teaming model alignment which focuses on circumventing a set of universal rules and values that the model has been trained to always obey. The preprint [2] suggested by the review does not appear to be relevant, as it considers performance in long contexts, while all of our scenarios use short contexts.

Reviewer comment (Novelty):
The paper provides limited takeaways from their experiment. While the benchmark creation is impressive, it is unclear what is learned apart from the well-known fact that LLMs do not follow rules (as shown, for example, in work from Percy Liang’s group — https://arxiv.org/abs/2307.03172 — or that LLMs used in search such as BingChat can be subverted with prompt injection attacks). The absence of contributions beyond the creation of the benchmark makes the reviewer apprehensive about recommending acceptance.

Author response:
**Novelty and takeaways.** It might be misleading to conclude that LLMs cannot follow rules at all; while the pass rate falls far short of what we would hope for, it is also significantly higher than zero. While many might have intuitions or expectations about how well LLMs can follow rules, we are not aware of any prior work that has quantitatively or qualitatively evaluated this question. Work on prompt injection is relevant, but generally has considered only one narrow attack strategy: e.g., "Disregard previous instructions and instead". We consider a broader array of strategies for fooling models into violating the rules. Some of our strategies would not be considered a successful prompt injection attack (because they do not allow completely replacing the existing task with any other task) but are able to trigger rule violations. As discussed in our revised submission, we also distinguish this work from red-teaming model alignment which focuses on circumventing a set of universal rules and values that the model has been trained to always obey. The preprint [2] suggested by the review does not appear to be relevant, as it considers performance in long contexts, while all of our scenarios use short contexts.

Reviewer comment (Experiments):
One conceivable application is in real-world systems (e.g., BingChat) where one would want to understand model brittleness. To validate such a scenario, the authors would need to consider “layered defenses,” such as adding an output filter atop LLM generations and examining how much information could be exfiltrated; however, this was not done in the work.

Author response:
**Defenses.** We recognize that there is a large space of potential prompting and filtering methods that might substantially improve rule-following. We see our work as providing an evaluation benchmark that others can use, as they propose such interventions. As far as we are aware, current deployments of LLMs generally do not use those "layered defenses". It is also important to note the difficulty of properly evaluating proposed defenses. Without sufficient conscientious red-teaming, it is easy to draw faulty conclusions about proposed defenses [1], and we hope our benchmark will be a first step towards supporting such evaluation. We also have removed the scratchpad and double check prompting results from our revised submission because after further analysis of output logs, we realized not all the models fully understood our existing formulations of the prompts.

Reviewer comment (Novelty):
Many of the findings presented by the authors have been covered previously — numerous prompt injection strategies already discuss mechanisms of rule subversion — making the work seem like a consolidation of existing results. The authors could place greater emphasis on what differentiates their work.

Author response:
**Novelty and takeaways.** It might be misleading to conclude that LLMs cannot follow rules at all; while the pass rate falls far short of what we would hope for, it is also significantly higher than zero. While many might have intuitions or expectations about how well LLMs can follow rules, we are not aware of any prior work that has quantitatively or qualitatively evaluated this question. Work on prompt injection is relevant, but generally has considered only one narrow attack strategy: e.g., "Disregard previous instructions and instead". We consider a broader array of strategies for fooling models into violating the rules. Some of our strategies would not be considered a successful prompt injection attack (because they do not allow completely replacing the existing task with any other task) but are able to trigger rule violations. As discussed in our revised submission, we also distinguish this work from red-teaming model alignment which focuses on circumventing a set of universal rules and values that the model has been trained to always obey. The preprint [2] suggested by the review does not appear to be relevant, as it considers performance in long contexts, while all of our scenarios use short contexts.