Paper: LLM Censorship: The Problem and its Limitations (ICLR-2024)

Reviewer comment (Novelty):
From the "security" perspective, the considerations made in the paper are "obvious" and "not new," as the possibility of bypassing censorship through certain prompting strategies is already well-known and unsurprising to security researchers.

Author response:
We would like to first raise the point that while the results may seem somewhat obvious, they can carry significant implications and are not reflected in current perspectives of AI safety. For example, the recent preprint [1] suggests a path forward for controllable AGI through adaptation of formal verification, however our theoretical results show potential limitations for such approaches when many of the desired constraints are semantic. Even with formal verification, it might be possible to establish guarantees for individual model behaviors, but handling their composition would be far more challenging. While this paper reflects a forward looking perspective on AI safety, we’ve also included an overview of extant approaches. 

From a security perspective, the novelty and purpose of the paper is to expose fundamental security risks of LLMs that have been overlooked and whose significance is underappreciated. Existing LLM safety literature focuses on overly constrained threat models, such as adversaries that include adversarial tokens to prompts, with the aim of simply breaking a model’s “fine-tuned alignment”. We focus on output censorship mechanisms because, unlike alignment or intermediate representation censorship, output censorship mechanisms would lend themselves more to establishing safety guarantees, as permissibility of a model’s output would be defined in terms of the **content** of the string rather than the **internal mechanisms of the LLM** which produce it. The intent of this work was to make clear that providing guarantees that LLM outputs are not harmful through censorship mechanisms is hopeless by describing **novel threat models** not previously considered. The intent was not to demonstrate that a specific, implemented, censorship mechanism can be bypassed. The only research which has discussed bypassing censorship methods is [2], and they only studied the problem from a traditional jailbreaking perspective with a few modifications to minimize output of keywords that could trigger output filtering mechanisms.

Reviewer comment (Experiments):
From the "ML" perspective, the paper lacks a clear, concrete experiment to demonstrate even one of the scenarios described in the practical implications, relying instead on textual "thought experiments" with excessive use of speculative language such as "can," "could," and "may."

Author response:
We have added a proof-of-concept demonstration to the appendix showing how GPT-4-turbo can be bypassed using a combination of a roleplay jailbreak and instructions for performing Caesar Cipher to communicate in an encrypted manner. While these attacks are not yet easy to execute and come with challenges, we believe that it is best to be aware of these potential risks before they pose serious threats and thereby allow themselves to be rigorously, empirically, demonstrated. As such, we did not focus on developing benchmarks for these problems as such benchmarks could lead to a false sense of security from new censorship mechanisms which handle certain encryption methods, however, fail to provide guarantees as the underlying problem of censorship is inherently flawed.

Reviewer comment (Novelty):
Some statements in the paper require further evidence, particularly the claim that LLM censorship is "commonly treated as a machine learning problem" without considering security-based approaches, as only four works are cited and some already advocate security principles.

Author response:
See updated related works section

Reviewer comment (Novelty):
A more thorough analysis of current works on LLM censorship, quantifying how many truly adopt ML-only approaches without security considerations, would strengthen the paper’s contribution.

Author response:
We’ve added a subsection to related works detailing current approaches to LLM defenses. While the approach of censorship has not been formalized prior to this work, many extant defense methods are implicitly defining an instance of an input or output censorship mechanism which relies on an LLM to evaluate harmfulness of strings. There is of course awareness that these methods are imperfect, however there is no awareness that such defense mechanisms are fundamentally limited as we describe. If the reviewer could describe more specifically what they mean by a more profound analysis of current methods, we would be happy to expand further on this point.

Reviewer comment (Theory):
If censorship were applied during an intermediate process, it could potentially work; thus, evidence is needed that the a-posteriori assumption holds in real systems, or else the scope and impact of the results should be adjusted.

Author response:
Censorship mechanisms that aim to intervene on intermediate states of an LLM, e.g. Representation Engineering [3] approaches such as [4] would not be subject to the theoretical limitations. Generally, such methods may pose a practical way forward for further improving security but become subject to similar limitations as model alignment. In particular, one cannot provide guarantees of safety as permissibility of an output would not be determined by hidden representations of an LLM but rather by the string itself, i.e. we assume there is a ground truth for permissibility. Furthermore, the vulnerability of such methods to adversarial prompts, which already pose significant challenges to model alignment, is unclear and would require further study and analysis. In other words, just as one would not define a cat in terms of the internal representations of a classifier, we cannot define impermissible content through internal representations, such definitions would lead to counter-examples via adversarial perturbations. 
Once LLMs are effectively integrated with some form of memory tape or scratchpad where they can store information, the ability to monitor and audit that scratchpad may also provide options for defense, however it is difficult to draw conclusions without knowing how these mechanisms will work and be used. 
We note that the aforementioned potential avenues for defense would not affect Mosaic Prompting attacks.

Reviewer comment (Novelty):
The name "Mosaic Prompts" is presented in an excessive way for a technique that is essentially the well-known "divide and conquer" approach, which already appears in LLM applications; the naming should be toned down or acknowledged as a renaming of an existing method.

Author response:
We acknowledge that Mosaic Prompts are related to decomposed prompting strategies that have been studied for improved problem solving, however the formulation is more general than question decomposition, has a different motivation for the decomposition, and stresses decomposition across context windows. We have added related work outlining this.
We are open to renaming Mosaic Prompts to Dual-use or Dual-intent Decomposed Prompting if this would still be preferred, the latter was used by the Code LLama [6] paper when referring to attacks described in this work *(we ask the reviewers to not check the reference to so as to not deanonymize the authors)*.

Reviewer comment (Writing):
Typographical errors appear throughout, including "either nothing is be permissible," "appears" spelled as "appers," and "ASCII" misspelled as "ACII."

Author response:
We have also corrected the typo's, and thank you for spotting them.

Reviewer comment (Experiments):
Practical suggestions and examples mentioned in the text, like "users could provide [...] running the model," would be more compelling if demonstrated in practice today.

Author response:
While the aim of this work was to present fundamental limitations of LLM safety to encourage a forward looking re-evaluation of the problem, design threat modeling, and defenses, we can present a preliminary proof-of-concept example as added to the appendix to demonstrate that one can already abuse these limitations. We see our work as a foundational step towards defining strongest adversaries that our systems should ultimately attempt to address to stop manipulation and abuse.

Reviewer comment (Theory):
The authors prove the impossibility of semantic censorship using string transformation by showing how the transformed string might break the "invariance of semantic censorship." Here, I have some doubts regarding the invariance property. In my opinion, the semantics of a string often change after the transformation. Thus, it is reasonable for the transformed string to bypass semantic censorship mechanisms. Moreover, LLMs do not necessarily output harmful texts with the transformed string. Why does the invariance property hold? Is this property an important goal considered by LLM censorship developers when designing their mechanisms?

Author response:
We acknowledge the point made regarding the semantics of an output string changing upon undergoing a string transformation, which is why we specify the invariance holds under knowledge of $g^{-1}$. What this means is that harmful text may be recoverable from apparently benign text if a specific transformation is known. We’ve provided an example in the appendix demonstrating how one can communicate with GPT-4-turbo using a Caesar cipher, and thereby recover an impermissible output from encrypted text that on it’s own is semantically nonsensical and perceived as harmless when evaluated by an independent LLM. The central theme of our work is demonstrating that harmlessness and safety cannot be evaluated independent of knowledge of a broader context. From seemingly benign output strings that could be transformed into harmful content under invertible transformations to benign outputs which are part of a collection that in aggregate leads to construction of impermissible content. As we show, one can extract dangerous information from LLMs through these methods, which necessitates a re-evaluation for how AI safety is studied.

Reviewer comment (Theory):
Implications can be extended. It appears to me that the current implication discussion stops at showing LLM censorship is more of a security problem than a machine learning problem. What are the direct implications for model developers when building censorship? Are there any defensive measures against the Mosaic prompts? The authors only briefly mention that there are standard approaches, such as access controls and user monitoring, to build censorship from the security view. However, there is no further analysis showing that these approaches can indeed overcome the theoretical limitations of current external censorship mechanisms and surpass them in censorship performances.

Author response:
See the updated appendix for a more extensive discussion and explanation on the implications for developers building censorship mechanisms. The intent of such approaches isn’t to overcome the theoretical limitations of current censorship mechanisms, rather, they are intended to allow for the provision of certain types of output guarantees, clarifying what statements regarding the safety and vulnerability of a deployed LLM can and cannot be made.
Censorship mechanisms against Mosaic Prompt are infeasible, however, prompt templates result in strict constraints on the space of possible outputs and information accessible by users.

Reviewer comment (Writing):
Readability can be improved. Many sentences are too long and difficult to read. For example, "Thus, we can understand censorship as a method of determining permissibility of a string and censorship mechanisms can be described as a function, f(x), restricting the string x to the set of permissible strings P by transforming it to another string x' ∈ P if necessary, e.g. x' ='I am unable to answer.'"

Author response:
Thank you for the feedback regarding readability, we’ve updated the draft.