Paper: Automatically Eliciting Toxic Outputs from Pre-trained Language Models (ICLR-2024)

Reviewer comment (Novelty):
Limited scope: The paper only focuses on the generation of toxic content from language models and does not address other potential risks associated with their deployment, such as bias or misinformation. This could limit the paper's relevance and impact in the broader context of language model research.

Author response:
As pre-trained language models are seeing increasing use across a wide variety of applications, a lot of problems in language models have been discovered and studied, e.g. toxicity, hallucination, bias, etc. Due to the different sources of these problems in language models, it is very difficult to study them under the same framework. Most existing researches also view these problems as different vulnerabilities in language models and study them separately. In this paper, we mainly focus on the toxicity and security risks of language models.  Bias like gender and racial discrimination is also included in some toxic expression.

We have tried to automatically construct prompts to generate statements that violate common sense such as examples in [1] without changing parameters. However, all existing algorithms do not work well in this task. The possible reason is that toxic outputs of language models output are due to the toxic examples in the training data [2], while misinformation and hallucinations can be caused by architecture flaw and suboptimal training objective [3]. Therefore, most current researches divide these potential risks into different types for study. 

As discussed in Section 5.2 in our paper, the success rate of the attack is strongly correlated with the perplexity of the target output. In other words, our method can achieve a good performance if the target text has high fluency on the language model. 

We think it is meaningful to propose a framework that can elicit different kinds of potential risks at the same time. We will further study this problem in future work.

Reviewer comment (Experiments):
The paper only tests on 1/2/3-word output, which is not practical. It is unknown whether the proposed method would work for longer toxic outputs.

Author response:
Following established experimental settings of adversarial prompts [4], we construct test dataset containing 1/2/3 tokens from Civil Comments Dataset. The adaptation of the tokenization method for the test data is performed to align it with the requirements of various language models. As longer output text requires a longer prompt to elicit, relatively short test data help reduce the computational cost of the optimization. To better alleviate your concerns, we have carried out experiments on a new sentence dataset to alleviate your concerns. The experimental results are reported in the response to all reviewers.

Reviewer comment (Evaluation):
The proposed method seems a little complicated, especially the DPP procedure. I'd like to know its speed (e.g. throughput) compared with previous works.

Author response:
We adopt DPP to model the negative correlations between quality and diversity in beam selection. Calculating the determinant of the kernel matrix precisely requires a high computational cost, previous algorithm that provide exact implementation has *O($M^4$)* complexity, where $M$ denotes the total number of candidate items [1]. However, the approximate greedy algorithm used in our implementation has *O($N^2M$)* complexity when selecting $N$ items out of $M$ [2]. Therefore, actually in ASRA, the running time of DPP selection procedure is extremely short and almost negligible. We have calculated the average running time in our datasets. One iteration takes around  0.732s on GPT-J and 0.834s on LLaMA in total, while the DPP algorithm only takes 0.002s on these language models. We provide the average running time of one iteration in the algorithm and the corresponding throughput of different methods on LLaMA  in the following table:

|                         | GBDA  | AutoPrompt | ARCA  | ASRA (-) DPP | ASRA (+) DPP |
| :---------------------: | :---: | :--------: | :---: | :----------: | :----------: |
| Running Time per iter/s | 0.185 |   0.203    | 0.283 |    0.832     |    0.834     |
|  Throughput (token/s)   | 48.65 |   44.33    | 31.80 |    10.82     |    10.79     |

Reviewer comment (Experiments):
Experiments limit eliciting toxic text to up to 3 words. In practice, however, what we really care about is not generating toxic text at any length. Therefore I have a concern about this work's transferability to a more practical scenario.

Author response:
Given any length of toxic text, our algorithm will search for prompt to elicit it from the language model. Obviously, longer toxic text is harder to elicit from language models, thus requires a sequence with more tokens to be optimized as the prompt. As prompt length increases, all adversarial algorithms face a higher computational cost. 

Following established experimental settings of adversarial prompts [3], we construct test dataset containing 1/2/3 tokens from Civil Comments Dataset. The adaptation of the tokenization method for the test data is performed to align it with the requirements of various language models. As longer output text requires a longer prompt to elicit, relatively short test data help reduce the computational cost of the optimization.

To better alleviate your concerns, we have carried out experiments on a new sentence dataset to alleviate your concerns. The experimental results are reported in the response to all reviewers.

Reviewer comment (Experiments):
I want to know whether DPP could be possibly replaced by another simpler method, e.g. methods often used in the document summarization task.

Author response:
DPP model is used to model the negative correlation between quality and diversity to avoid redundant search in the algorithm. This property of algorithm is very important in solving the task. We make attempts on a famous algorithm for unsupervised summarization, Textrank [4]. The success rate of the algorithm on LLaMA is provided as follows.

|                | (+)Textrank | (+)DPP |
| -------------- | ----------- | ------ |
| 1-word dataset | 6.85%       | 91.78% |
| 2-word dataset | 1.27%       | 36.02% |
| 3-word dataset | 0%          | 10.71% |

The algorithm which mainly models similarity between different candidates to select items does not work well in this task.

Reviewer comment (Experiments):
Lack of defense models (detoxified models): While I appreciate the authors’ efforts in comparing different pretrained models, it would also be interesting to evaluate against different defense approaches/detoxification approaches, such as [2,3,4], and confirm whether the attack is still effective.

Author response:
We supplement the performance of our algorithm against two defense approaches, including DAPT [2] and DExperts [3]. We follow the implementation in the original paper and test the performance of our algorithm against the two defense approaches on GPT-2. The results are listed as follows:

| Dataset     | GPT-2  | DAPT   | DExperts |
| ----------- | ------ | ------ | -------- |
| 1-word data | 97.26% | 95.89% | 82.19%   |
| 2-word data | 69.49% | 70.76% | 18.22%   |
| 3-word data | 23.36% | 28.71% | 8.27%    |

According to the experimental results, ASRA can also elicit some toxic cases even against defense approaches. Generally, DAPT does not improve the robustness of the pre-trained language model against our adversarial attack, while DExperts achieves a relatively good performance. The attack and evaluation of defense methods need further research in the future.

Reviewer comment (Evaluation):
Validity of toxicity evaluation: The authors mention that their evaluation setup “can be applied to bridge the evaluation of toxicity in different PLMs”, and “speculate that the success rate of ASRA attack might be positively correlated with the toxicity of language models.” However, I do not see any evidence about whether the ASR here can be a good proxy to reflect model toxicity. Given that the model toxicity evaluation is conducted by evaluating model responses with a lot of different inputs and contexts, the setup of this paper is to evaluate model responses given “unnatural” prompts. I am thus suspicious about whether the test above can give an accurate evaluation of model toxicity.

Author response:
Most existing toxicity evaluation is conducted by evaluating model responses with human-written inputs. The evaluation method of safety of pre-trained language models depends on the practical scenario. Common evaluation assumes that users query language models with some questions or dialogue. However, malicious attackers can attack pre-trained language models by intentionally constructing adversarial prompts. Ensuring the safety of language models requires evaluation at different levels. This paper explores how our algorithm can assess the safety of pre-trained language models when facing intentional adversarial challenges.

Reviewer comment (Experiments):
The proposed optimization seems generalizable for different text generation tasks. Have you ever used the approach to attack other text generation tasks (such as [1])?

Author response:
Our algorithm is not specific to text generation tasks like summarization, but to test the security and toxicity of pre-trained language models. The objective of our algorithm is to elicit certain content from the language model. Therefore, it is challenging to conduct experiments on various text generation tasks like [1].  As we discuss in Section 5.2, the success rate of the attack is strongly correlated with the perplexity of the target output. In other words, our method can achieve a good performance if the target text has high fluency on the language model.