Paper: Robustness Over Time: Understanding Adversarial Examples’ Effectiveness on Longitudinal Versions of Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
My primary concern with this paper is its limited scope in terms of technical innovation. While the paper considers a range of models against existing benchmarks, albeit with some modifications and combinations, it fails to introduce new evaluation benchmarks or methodologies. Therefore, I think the paper does not meet the standards of ICLR.

Author response:
We believe we introduced a new evaluation methodology, i.e.,  adversarial robustness should be evaluated `over time`. The other three reviewers highly appreciate this new angle of research. Developing new attacks/benchmarks is of general interest but would not solve (and is not relevant to) the problem that LLMs are vulnerable even to existing attacks in existing benchmarks. We also believe that uncovering the LLMs’ vulnerability to existing (simple) attacks poses more severe threats than newly designed (advanced) attacks.

Reviewer comment (Experiments):
Another issue is the selection of datasets for evaluation. The benchmarks employed are commonly used, potentially even in the training of the GPT models under scrutiny. This compromises the conclusions of the results.

Author response:
This is a good point, and we will add a related discussion in the paper based on the following. In the LLM era, an LLM's training data may (partially and inadvertently) include the evaluation/test data. However, we point out that our evaluation data is not included in the training data of both GPT and LLaMA models because:
- Regarding GPT-3.5, the GPT-3.5 training data is up to Sep 2021 according to OpenAI's website (see https://platform.openai.com/docs/models/gpt-3-5). The `AdvGLUE` dataset, however, was published in Nov 2021, and the `PromptBench` dataset was published in 2023. Therefore, It is safe to conclude that there is no possibility of training the GPT-3.5 model using these datasets.
- Regarding LLaMA-1/2, their paper (https://arxiv.org/abs/2302.13971) states they used the GitHub dataset with `Apache`, `BSD`, and `MIT licenses`. However, the `AdvGLUE`'s license is `CC BY-SA 4.0`. Thus, Meta cannot choose `ADVGLUE` to train the model. In addition, `PromptBench` is a brand new dataset generated using different LLMs from 5 months ago, one month before the LLaMA-2 was released. Therefore, it is very unlikely for Meta to use this dataset for training the LLaMA-2 model.
More generally, adversarial data are much less likely than clean data to be included in any natural dataset because they are specifically optimized. Our finding that LLMs are vulnerable to our adversarial data also suggests that they are not used for training.

Reviewer comment (Evaluation):
Furthermore, the objective behind updating from GPT-3.5 v0301 to GPT-3.5 v0613 may not exclusively target robustness enhancement. Other factors such as reasoning ability, following prompts, and computational efficiency could also be taken into account. Thus, expecting substantial improvements in robustness may not be reasonable.

Author response:
We agree that robustness is not *exclusively* important, and we do not claim that. However, the security of LLMs is an emerging research topic, and achieving adversarial robustness is a key requirement. For example, OpenAI has established the `preparedness` team to build a robust framework for monitoring, evaluation, prediction, and protection against the dangerous capabilities of frontier AI systems [4]. Following this research trend, we provide new insights that updated LLMs are vulnerable even to existing attacks.

Reviewer comment (Experiments):
In this work, the authors primarily focus on assessing adversarial robustness exclusively within various iterations of LLMs. The authors should broaden the scope of their investigation to encompass additional thematic categories across diverse subject matter domains. Furthermore, the authors should expand their evaluation efforts to encompass various dimensions of the model iterations they are not considering. Specifically, regarding the LLaMA model family, which includes models of varying architectural sizes, the authors should investigate and provide insights into the robustness of these models in the context of different parameter sizes.

Author response:
Thanks for the valuable suggestions. Our current research is focused on the `time` dimension since we consider it the most straightforward dimension for comparing the robustness of two closely related models. We would like to consider the suggested extensions for future work, i.e., `more subject matter domains`, `dimensions`, and `model parameter settings`.

Reviewer comment (Reproducibility):
The motivation of this paper is clear. However, the authors should elaborate more on the process of generating adversarial examples by different surrogate language models.

Author response:
We have added more details on how to generate the adversarial samples by different surrogate language models in Appendix.

Reviewer comment (Experiments):
I didn't find any results of LLaMA-30B, but the authors list this model in Section 4.2. I think the authors should provide some details about that.

Author response:
This is indeed a typo, and we have removed it from the paper.

Reviewer comment (Evaluation):
Could the authors provide an explanation for the underlying reason that caused the LLMs not to exhibit heightened robustness over time? Additionally, could they discuss potential strategies to address this issue?

Author response:
The training details of LLMs are generally not available to the public. However, a simple observation is that the model trainer tends to consider model accuracy over robustness, partially because achieving adversarial robustness requires considerable additional effort. We have added the content to suggest the model trainer to use enhancing adversarial robustness methods in model upgrades in our discussion.

Reviewer comment (Presentation):
Hope the author can provide a more detailed description of "zero-shot ICL learning" and "few-shot ICL learning" in Figure 2.

Author response:
If the reviewer means the description of the two terms, we will further clarify that we follow [1,2,3] on this terminology. Zero-shot learning means that the query includes only the description and the question but **without any demonstrations**, while few-shot learning means that the query also includes a few `demonstrations`. We have added the explanation in the caption of Figure 2. If the reviewer means the `description` component in the query, please see the detailed definitions in Section 2.1.

[1]. Yongqin Xian et al. Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.

[2]. Tom B. Brown et al. Language Models are Few-Shot Learners. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.

[3]. Jason Wei et al. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations (ICLR), 2022.

Reviewer comment (Presentation):
What does the second column "Adversarial Query" in Table 1 mean? Please clarify.

Author response:
`Adversarial Query` refers to the query that contains the adversarial content in any of its three components (`description`, `question`, and `demonstrations`), as defined in *Eq. (1)* and *Eq. (2)*. Specifically, each component can be clean (denoted as C) or adversarial (denoted as A). 
Therefore, in Table 1, the adversarial query `AC` means a zero-shot learning-based query that consists of an adversarial `description` and a clean `question`, and `AAC` means a few-shot learning-based query that consists of an adversarial `description`, an adversarial `question`, and clean `demonstrations`. We will describe it more clearly in Section 5.1.

Reviewer comment (Presentation):
"For instance, on the SST2 dataset, when applying BertAttack (Li et al., 2020) to create the adversarial description, the Robust Test Scores (see Section 4.3) for both versions of GPT-3.5 are almost 0.", which table or figure is being described specifically? Please clarify.

Author response:
Sorry, we indeed made a mistake here. It should be ''For instance, on the SST2 dataset, the average result of Robust Test Scores of zero-shot learning for both versions of GPT-3.5 dropped from 85.093% and 87.390% to 37.210% and 20.652%, respectively (see Figure 3).''