Paper: Fool Your Large (Vision and) Language Models with Embarrassingly Simple Permutations (ICLR-2024)

Reviewer comment (Novelty):
I don't see any related work section. That would have made the work more sound.

Author response:
We would like to clarify that a comprehensive review of related work is indeed present in our paper, specifically in Section 5. This section discusses the development of Large Language Models (LLMs) and Vision-and-Language Models (VLLMs), their wide adoption, and popular studies using multiple-choice question answering for evaluation. Additionally, it includes an analysis of the robustness of LLMs and VLLMs. We hope this section can draw parallels with previous studies, and highlight the distinct contributions and originality of our work.

Reviewer comment (Presentation):
The comparison tables on LLMs seem to be inconsistent. For example, Table 3 has GPT-3.5 turbo but Tables 5 and 6 (that are also comparisons on LLM) do not have GPT-3.5 turbo.

Author response:
Thank you for pointing this out. We acknowledge that GPT-3.5 Turbo is included in Table 3 but not in Tables 5 and 6. The reason for this is primarily due to budgetary constraints. To address this issue and maintain the integrity of our comparative analysis, we have now conducted additional experiments with GPT-3.5 Turbo for Table 5 and 6. The results can be seen in the updated tables. Additionally, we also incorporated Llama2-70B, a strong open-source model, in the revised version of our manuscript. This addition aims to provide another comparable level of insight and draw similar conclusions to what would have been achieved with GPT-3.5 Turbo.

Reviewer comment (Reproducibility):
In the experimentation section, it would be good if the authors described some more details. For example, did they use model APIs or model weights for testing? This would give us an idea of the consistency of experiments.

Author response:
Thank you for your suggestions. All of the models (except GPT-3.5-turbo) we used are open-sourced models. Their weights can be obtained from the official Huggingface model zoo (https://huggingface.co/models) or the original Github repositories. We have added additional descriptions of these details in the Reproducibility Statement section and we have uploaded the source code to reproduce these experiments.

Reviewer comment (Reproducibility):
In Table 3 for GPT-3.5 turbo, how did the authors do the testing? Is it using OpenAI API?

Author response:
Thank you for your question. Yes, we used the official OpenAI GPT-3.5-turbo API for the experiments following their documents (https://platform.openai.com/docs/api-reference/chat).

Reviewer comment (Experiments):
While the paper focuses on permutation-based vulnerabilities, it might benefit from a broader discussion on other potential vulnerabilities or comparisons to other attack methods.

Author response:
### Q2. Other attacks
Thank you for your suggestions. In our original version, we compare our attack strategy with the position attack, i.e. always rotating the correct answers to a specific position (A/B/C/D) in the answer list. The results show that although the models also exhibit varying degrees of position bias, our adversarial permutation shows a much stronger effect (a larger proportion of predictions drop to below chance level).
To further address your concern, we conducted additional experiments to compare with three more types of attack methods. (1) ICL Permutation attack: Given a fixed set of few-shot in-context examples, permute the in-context learning examples to their worst-case order. (2) ICL Search attack: Search for the worst-case set of ICL examples to use from within the training set.  (3) Symbol attack: Consider different types of option symbols (e.g. A/B/C/D, 1/2/3/4, a/b/c/d), and choose the worst-case symbol set for each question. We present the results of these attacks on MMLU dataset below and the results of other datasets in the appendix Table 16-20.

| **Models**   	| **Original 0-shot** | **ICL** | **ICL Permutation** | **ICL Search attack** | **Symbol Attack** | **Adversarial Attack** |
|------------------|---------------------|---------|---------------------|-----------------------|-------------------|------------------------|
| Llama2-7B    	| 40.91           	| 45.67   | 35.09           	| 34.46             	| 25.70         	| **6.17**           	|
| Llama2-13B   	| 52.22           	| 54.52   | 46.65           	| 46.07             	| 30.76         	| **18.33**          	|
| Llama2-70B   	| 64.68           	| 68.25   | 59.82           	| 59.68             	| 47.40         	| **33.16**          	|
| Vicuna-v1.5  	| 48.57           	| 50.27   | 40.85           	| 41.92             	| 33.85         	| **18.09**          	|
| Vicuna-v1.5-13B  | 54.68           	| 55.68   | 54.65           	| 49.17             	| 45.40         	| **26.27**          	|
| WizardLM-13B 	| 48.60           	| 48.93   | 39.98           	| 39.90             	| 29.07         	| **15.87**          	|
| InternLM-7B  	| 45.72           	| 48.36   | 38.17           	| 38.17             	| 29.38         	| **10.45**          	|
| InternLM-20B 	| 59.14           	| 60.50   | 54.45           	| 54.45             	| 47.06         	| **29.52**          	|
| Falcon-7B    	| 31.66           	| 32.95   | 27.18           	| 26.79             	| 14.38         	| **2.49**           	|
| MPT-7B       	| 35.60           	| 38.73   | 30.51           	| 27.33             	| 21.62         	| **3.52**           	|


As can be seen from the table, while other attacks can also harm the performance, our adversarial attack has the biggest impact on the LLMs and causes the largest performance drop.

Reviewer comment (Experiments):
The experiments provided are on a specific dataset. It would be beneficial to see how the models fare on diverse datasets to ensure the findings, which are not specific to one dataset's characteristics.

Author response:
### Q3. Evaluation datasets
We appreciate the reviewer's comment on the dataset specificity of our experiments. We would like to clarify that our study encompasses a broad spectrum of datasets: 5 NLP datasets for Large Language Models (LLMs) and 4 vision-language datasets for Vision-Language Large Models (VLLMs). These datasets are meticulously chosen to evaluate a diverse range of characteristics, such as commonsense reasoning, reading comprehension, external knowledge, and visual grounding abilities. The composition and specifics of these datasets are detailed in Tables 1 and 2, while the main results are presented in Tables 3 and 4. Also, in appendix Table *9-27*, we have provided the detailed results of the other datasets.

Our analysis in Section 4, using the MMLU dataset as a case study, aims to delve deeper into the potential vulnerabilities and their remedies. MMLU is a comprehensive benchmark aggregating 57 diverse tasks, encompassing areas like mathematics, US history, computer science, and law. We believe this diversity supports the generalizability of our findings. However, to address your concern, we conducted additional analysis with another vision-language dataset A-OKVQA, and have updated the results in the appendix Table 27. The supplemental results align with our original findings, further solidifying the generalizability of our analysis.