Paper: Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR-2024)

Reviewer comment (Experiments):
The proposed solution could benefit from more details. Exploring the soft-prompt size (i.e., the variable N) would be nice. The N used is not specified in the main text either, which I believe is an important detail.

Author response:
**W1 (Soft-prompt size)**: We use a prefix length of 10 for the result reported (thank you for pointing this out!) We have explored many different settings (e.g., soft-prompt size) in our pilot studies and noticed no substantial differences, suggesting that one needs only a minimal set of free parameters to tune with the proposed objective to realize improvements. Below we reproduce results observed when using different soft-prompt sizes (from 1 to 50), and show their performance and semantic distance:

| **Soft Prompt Size** | **MMLU Obs.**  | **MMLU Unobs.** | **MMLU Ave.**  | **Ave. MMLU Distance** | **BBL Obs.**    | **BBL Unobs.**  | **BBL Ave.**    | **Ave. BBL Distance** |
|----------------------|----------------|-----------------|----------------|------------------------|-----------------|-----------------|-----------------|-----------------------|
| 1                    | **47.4 ± 0.2** | **47.8 ± 0.2**  | **47.6 ± 0.3** | 3.82                   | 55.9 ± 19.7     | 54.2 ± 17.6     | 55.1 ± 18.4     | 7.33                  |
| 5                    | 47.9 ± 0.2     | 48.4 ± 0.3      | 48.1 ± 0.3     | 4.03                   | 55.8 ± 21.2     | **52.4 ± 19.0** | **54.1 ± 19.9** | **6.88**              |
| 10                   | 47.6 ± 0.1     | 48.0 ± 0.2      | 47.8 ± 0.3     | 4.30                   | **55.0 ± 21.6** | 53.6 ± 18.0     | 54.3 ± 19.6     | 7.47                  |
| 30                   | 47.6 ± 0.1     | 48.2 ± 0.1      | 47.9 ± 0.3     | 3.84                   | 55.4 ± 20.8     | 53.8 ± 17.9     | 54.6 ± 19.3     | 7.02                  |
| 50                   | 48.0 ± 0.2     | 48.6 ± 0.2      | 48.3 ± 0.5     | **3.73**               | 55.9 ± 21.9     | 52.6 ± 18.9     | 54.2 ± 20.0     | 7.03                  |

The results indicate that the prefix length does not affect the performance for both observed and unobserved instructions. As they fluctuate around the same level. Additionally, we also calculated average distance between unobserved instructions to their closest equivalent observed instructions, and the value is highly similar between all soft prompt sizes.

The takeaway here is that soft-prompt is only a method we use to drag the equivalent instructions close together without affecting the model’s performance, and the number of parameters we dedicate to do that is seemingly insignificant.

Reviewer comment (Reproducibility):
"We then form batches by including one instance featuring the original instruction, and the rest comprising paraphrased instructions." -> what's the batch size, and consequently, what's the paraphrase size and the final dataset size altogether?

Author response:
**Q1 (Dataset size)**: The batch size we use for both models is 4, hence each batch contains one “prototype” instruction and 3 paraphrased instructions to align (this is also reported at the beginning of Appendix B.3.) We randomly sampled 1k prototype samples from the original instruction tuning datasets, hence the final dataset altogether is 4k; we report this Section 5.

Reviewer comment (Evaluation):
In Tables 3 and 7, compared to the performance difference, the variance is very large, questioning the significance of the difference between observed and unobserved instructions. A significance test should be conducted to test the significance of the difference.

Author response:
**W1 (Significance test)**: Please see our general response, (2). Briefly: The results in Tables 3 and 7 are statistically significant, despite the observed variances. We have added significance results to the manuscript. We also note that in Table 3, the variances are much larger for the unobserved instructions, which is part of the issue that we are highlighting.

Reviewer comment (Experiments):
Although the title, abstract, and motivation of the paper mention that the paper tries to evaluate the zero-shot robustness of instruction-tuned language models generally, it only focuses on models up to 11B. The tendency is likely to be unpredictable for state-of-the-art LLMs such as GPT-3.5 or GPT-4.

Author response:
**W2 (Size of LLMs)**: The reviewer is correct that our work here is focussed on modestly sized LMs. We have tried to be clear about this in the abstract and introduction. We write in the second sentence in the abstract that instruction tuning “... has shown particular strength in improving the performance of modestly sized LLMs, sometimes inducing performance competitive with much larger model variants”; this is what motivates our focus on these <= 11b models. We believe this analysis is important precisely because prior work has suggested that instruction tuning imbues such models with performance surprisingly competitive with massive LLMs; our findings call this into question.

Reviewer comment (Evaluation):
How does PT+KL on Table 7 perform on negated instructions? The alignment stage might make the LLM insensitive to most instructions even though the instructions are not relevant or negated, which is a similar finding for instruction-tuned LLMs for Webson & Pavlick (2022) Webson & Pavlick (2022) - Do Prompt-Based Models Really Understand the Meaning of their Prompts?

Author response:
**W3 (Negation)**: The reviewer suggests an interesting additional analysis: Evaluating sensitivity to instruction negation. We have now conducted such analyses and report details (experimental setup and results) in Appendix F. For convenience, below we reproduce the aggregated results on how PT + KL does on all 6 settings (including negated observed instructions) we conducted in Section 4.5

| **Method**| Closest | Incorrect | Negated | Task Designer | Collected | Nonsensical |
|-|-|-|-|-|-|-|
| - Baseline | 58.1 $\pm$ 1.8     | 52.2 $\pm$ 8.6 | 48.3 $\pm$ 3.8 | 44.2 | 55.3 $\pm$ 4.5 | 30.6 $\pm$ 4.4 |
| - PT + KL | **60.5 $\pm$ 2.4** | **54.2 $\pm$ 7.2** | 47.1 $\pm$ 4.9 | **47.1** | **56.7 $\pm$ 3.9** | 38.0 $\pm$ 4.1 |
| - PT + KL (Negation) | 58.6 $\pm$ 2.0     | 52.7 $\pm$ 9.2 | **43.0 $\pm$ 4.9** | 46.5 | 55.6 $\pm$ 4.3 | 35.7 $\pm$ 5.6 |

This table reports the results we reported in Figure 3 (first row) after applying PT + KL (second row). We observe that PT + KL yields non-trivial improvement on models performance with: 1) suitable instructions (Closest) that are observed. 2) suitable instructions (Task Designer, Collected) that are unobserved. However, results also indicate that the proposed method does not succeed fully in terms of imbuing the model with the ability to truly “understand” instructions: 
The performance of observed instructions after adding negation does not drop significantly (only by 1.2 on average)

We hypothesize that training the model to align semantically equivalent instructions does not explicitly incent the recognition of negated prompts. This is because our objective enforces that equivalent instructions should be similar, but negations rarely occur in the collected paraphrases. Moreover, learning to handle negations would more intuitively be done by encouraging the model to push negated versions of instructions away from their original forms; this differs from the form of our objective which focuses on similarity. Therefore, we conduct another experiment by modifying the objective slightly to handle negation.

Specifically, we trained the model to push the representation of the negated instructions away from the original ones. To do so, we take the same 1k seed instances from the FLAN collection and prompt (with three different prompts) GPT-4 to rewrite them such that the instruction is negated. We keep all the hyperparameters same as how we did for PT + KL, but use KL to push the representation of negated instructions away from the originals:

$\mathcal{L} = (1-\lambda) \mathcal{L}_{CE} - \lambda \mathcal{L}$$ _{KL}$

After training, we run the “closer look” experiment again to see if this slightly different objective would result in a stronger robustness (third row). The result shows a significant drop of the accuracy when negation is added to the observed instruction. However, the performance in other settings remains comparable with the baseline when applying the PT + KL (Negation).

This result provides additional evidence for our findings that robustness to instruction variants is strongly tied to the distances between learned representations of instructions: When we incentivize the model to learn similar representations for equivalent instructions, it performs better when given equivalent unobserved instructions, though it does not better handle negated instructions. But if we modify this slightly to encourage the model to induce dissimilar representations between instructions and their negated versions, it improves its responsiveness to negated instructions. In short, the proposed strategy of targeting (dis)similarity of model instruction representations is general enough to facilitate realizing robustness beyond paraphrases.

We report a full set of results for this finding in Appendix F in the updated draft.

Reviewer comment (Experiments):
Do you think that RLHF after instruction tuning might mitigate this issue?

Author response:
**Q1 (RLHF)**: RLHF is most appropriate for aligning model outputs for tasks that require open-ended responses. In our work, we are benchmarking instructions and tasks that have a set of correct answers. Collecting human feedback for training is not a good fit for our main evaluation measures, which focus exclusively on the correctness of the output for (mostly) “objective” tasks like classification.

However, the reviewer raises an interesting idea, in that it is possible that optimizing LLMs to align with human preferences generally via RLHF may implicitly require models to “understand” instructions in a more robust way. 

We perform a preliminary investigation into whether training with RLHF on an unrelated, open-ended text generation tasks affects the robustness of the instruction-following abilities. Specifically, we take an off-the-shelf Alpaca-7B trained with RLHF via PPO and run it on all the tasks and instructions we have used in this paper. We found that the output tends to do “extra things” for a given instruction (e.g., providing its explanation for question answer), we found that the overall performance is highly unstable and degrades significantly in most of the datasets. 

| **Categories** | Delta Obs. Accuracy | Delta Unobs. Accuracy | Delta Ave. Accuracy |
|-|-|-|-|
| **MMLU**| -18.5 ± 0.5 | -17.4 ± 1.7 | -18.0 ± 1.1|
| **BBL-QA**| -11.2 ± 4.0 | -9.7 ± 4.7 | -10.4 ± 4.3|
| **BBL-MC**| -0.3 ± 0.8 | -5.3 ± 0.8|  -2.8 ± 0.8 |
| **BBL-BC**| -11.8 ± 1.7 | -6.2 ± 8.3| -9.0 ± 5.0 |

Therefore, we do not have evidence to show that RLHF on modestly-sized LLMs would mitigate such issue.

Reviewer comment (Novelty):
The novelty of the robustness experiments is somewhat limited, as there is prior work that has observed variance based on prompt formatting (discussed in the work itself). Considering this work seems to examine more modern models such as Alpaca, and considers a different set of tasks, I think this is a minor weakness - the work does add a number of interesting insights, especially around representation distance.

Author response:
**W1 (Novelty)**: Regarding novelty, please see our general response, (3). In brief, our work builds upon but complements and considerably deepens these prior efforts.

Reviewer comment (Evaluation):
The claim that models perform worse at novel phrasings of tasks seems a bit strong compared to the results: examining Table 3, the variance of performance on unobserved phrasings often is greater than the difference in average performance, suggesting that the difference in performance may not be significant (although the large increase in variance is certainly not desirable and interesting to see).

Author response:
**W2 & Q2 (Significance Test)**: Please see our general response, (2). Briefly, the large variances here in part reflect the fact that we are reporting aggregated results over large benchmarks. That said, one of the main observations is that variances are considerably larger for unobserved instructions, which is—as the reviewer rightly points out—undesirable. 

To the reviewers’ question: The results in aggregate are statistically significant (p << 0.05) under a paired t-test, and if we run the t-test separately for each model (which avoids the issue of repeating datasets in observations), we observe p < 0.05 for all but T0++ (p=0.13).

Reviewer comment (Evaluation):
While the proposed method seems strong, examining the results in Appendix B.3 seems to show that in many cases, prompt-tuning alone performs within one standard deviation of the PT+KL results (e.g., for BBL unobserved and average). As such, the proposed method might not be significantly different from just prompt tuning.

Author response:
**W3 & Q3 (Efficacy of the objective)**: We observe two major differences between the performance using just prefix tuning (PT) and prefix tuning with extra objectives (PT + KL). 
The performance diff between unobserved instructions and observed instructions - i.e., robustness - is significantly higher (close to zero) for PT + KL (Table 7 in Section 6)
The average standard deviation of the instruction performance of PT is even higher than the baseline, indicating instability in its performance; whereas this high variance is significantly mitigated with PT + KL (**Table 10 in Appendix B.3**)

Intuitively, we design the extra objective to explicitly “increase the similarity” between equivalent instructions — unobserved to observed — from the model’s “perspective”, which we hypothesize to have positive correlation with the robustness of instruction following ability. 

Motivated by this, we calculate the average l2 distance between unobserved instructions and its closest observed instruction over all the examples (Same approach as Table 6) for PT and PT + KL.  (Fig 7 in Appendix B.3.2). We show that in 13 out of 14 datasets, the distance between equivalent instructions is smaller for PT + KL. The consistency of the close distance and better robustness supports our hypothesis of the correlation.

We have added these results, figures and the discussion to Appendix B.3.2 of the revised paper.

Reviewer comment (Experiments):
How does your proposed method compare to introducing the model paraphrases during initial instruction finetuning? Training on the paraphrased versions alone after the instruction tuning may cause forgetting in an undesirable way.

Author response:
**Q1 (Augment to initial instruction-tuning set)**: Due to the modest size of the paraphrases set (4k), it has minimal effect on the training performance and is very time-consuming by adding to the initial instruction finetuning. Please refer to Table 11 in Appendix B.3.1 for the detailed comparison. We reiterate that the main goal of the improvement section is to find a way to narrow down the semantic gap between the observed and unobserved, so that we validate our hypothesis (representation similarity for equivalent instructions v.s. robustness )

In Table 11, we initialized to the checkpoint weights "google/t5-v1.1-xl" and "llama-7", which are the foundation models for Flan-T5-XL and Alpaca-7B, respectively. We fine-tuned these models with the mixture of the original instruction training set and acquired paraphrases, following the exact procedure as used for Alpaca and everything but the packing technique (putting multiple instances into one forward pass) for Flan-T5-XL. This, to us, seemed like the simplest way to use the paraphrased data in instruction-tuning.


The results in Table 11 indicate that this does not provide much benefit, relative to the original instruction-tuned model (for Flan-T5; we see a slight improvement for Alpaca). We hypothesize that including the additional instructions in a standard instruction-tuning setup probably yields little benefit because the added set of new instructions/paraphrases is relatively small (e.g., 1/52 of the Alpaca dataset size); adding the KL term to explicitly pair these provides additional signal, and—as we show—induces similar representations for equivalent instructions in a way that naively training on all data does not. If one collected a much larger set of paraphrases, fine-tuning in the straightforward way may prove to yield more substantial benefits.
We hope that this answers your question, but please let us know if we can clarify further. Thank you again for your revised feedback!

Reviewer comment (Evaluation):
Is the degradation of performance on unobserved prompts statistically significant? The variance seems very high.

Author response:
**W2 & Q2 (Significance Test)**: Please see our general response, (2). Briefly, the large variances here in part reflect the fact that we are reporting aggregated results over large benchmarks. That said, one of the main observations is that variances are considerably larger for unobserved instructions, which is—as the reviewer rightly points out—undesirable. 

To the reviewers’ question: The results in aggregate are statistically significant (p << 0.05) under a paired t-test, and if we run the t-test separately for each model (which avoids the issue of repeating datasets in observations), we observe p < 0.05 for all but T0++ (p=0.13).

Reviewer comment (Evaluation):
Doing prompt tuning alone seems to often perform very similarly to PT+KL, and within the reported standard error in Appendix B.3. Do you have statistical tests for significance for your results?

Author response:
**W2 & Q2 (Significance Test)**: Please see our general response, (2). Briefly, the large variances here in part reflect the fact that we are reporting aggregated results over large benchmarks. That said, one of the main observations is that variances are considerably larger for unobserved instructions, which is—as the reviewer rightly points out—undesirable. 

To the reviewers’ question: The results in aggregate are statistically significant (p << 0.05) under a paired t-test, and if we run the t-test separately for each model (which avoids the issue of repeating datasets in observations), we observe p < 0.05 for all but T0++ (p=0.13).

Reviewer comment (Evaluation):
It would be interesting to see the results in Table 7 broken down into the BBL QA/BC/MC categories shown in Table 6 to see how much the difference in closest distance affects the performance of these different categories.

Author response:
**Q4 (Disaggregated results for Table 7)**: We appreciate the suggestion and we have now added the disaggregated result in a dataset level in Table 30 and Table 31 in Appendix J.3. For convenience, here we reproduce results disaggregated by categories:

| Categories | Alpaca Obs. Delta | Alpaca Unobs. Delta | Alpaca Ave. Delta | Flan Obs. Delta | Flan Unobs. Delta | Flan Ave. Delta |
|------------|-------------------|---------------------|-------------------|-----------------|-------------------|-----------------|
| MMLU       | 0.4               | 0.6                 | 0.5               | 0.8             | 1.1               | 1.0             |
| BBL-QA     | 1.1 $\pm$ 2.5     | 2.3 $\pm$ 2.8       | 1.7 $\pm$ 2.1     | 0.7 $\pm$ 0.4   | 4.6 $\pm$ 1.5     | 2.7 $\pm$ 0.9   |
| BBL-MC     | 0.3 $\pm$ 0.4     | 0.3 $\pm$ 0.1       | 0.3 $\pm$ 0.2     | 0.3 $\pm$ 0.0   | 3.5 $\pm$ 1.6     | 1.9 $\pm$ 0.8   |
| BBL-BC     | -1.3 $\pm$ 5.2    | 2.0 $\pm$ 2.5       | 0.4 $\pm$ 3.5     | 0.9 $\pm$ 0.2   | 3.2 $\pm$ 0.3     | 2.1 $\pm$ 0.2   |

This Table reports the average delta performances observed after applying the proposed method on Flan-T5-XL and Alpaca-7B. For BBL datasets we report standard deviations over the individual datasets (for MMLU we group all together so do not report a variance). 
Table 6 in the paper reports the average distance change after applying the objectives for Flan-T5-XL. In general, we observe a consistent but weak correlation relating the closest distance between equivalent instructions to the delta performance for unobserved instructions. The similarity between equivalent instructions changes little for MMLU, which leads to a marginal improvement. However, this pattern is also not absolute; the BBL-QA category in general receives the greatest improvement with respect to performance but not distance. We leave further analyses of instruction representations and downstream performance to future work building on our results here.

Reviewer comment (Presentation):
nit: "instruction-tuned models are not especially robust to instruction rephrasings" → this is very clear, say it first. Better yet, make it the title :)

Author response:
**W1 (Title of the paper)**: We thank the reviewer for the suggestion, and agree this is the main takeaway; we will emphasize this point earlier on (and consider using it as our title) in future version of the manuscript.

Reviewer comment (Novelty):
I don't know if just releasing prompt rephrasing is a significant enough contribution-- https://openreview.net/forum?id=9Vrb9D0WI4 and others also have datasets of different rephrasing of prompts for the same task.

Author response:
**W2 (Rephrasing Contribution)**: We agree with the reviewer that the release of this data is, on its own, not a substantial contribution. However, we do think this dataset is novel with respect to existing resources (e.g., T0/PromptBench, Flan, and others) which have released rephrasings of instructions for the same tasks intended for model **training**. By contrast, the dataset of diverse instructions we release with this work is intended to aid model **evaluation**, specifically on tasks in two major benchmarks, MMLU and BBL. Our hope is that this resource will both permit reproducibility of our work, and facilitate additional research into the robustness of instruction-tuned models.

Reviewer comment (Evaluation):
It looks like just fine tuning (or even fine tuning + KL) severely drops performance. Do you know why? This seems very surprising

Author response:
**Q1 (Fine-tuning performance)**: Please see our general response, (1). Briefly, we think this is an instance of “catastrophic forgetting” and have provided new results that support this hypothesis in Appendix G.

Reviewer comment (Experiments):
Why use NLP grad students to generate rephrasing of the prompts, when you could just use an LLM (which it seems like you do later for the eval set)?

Author response:
**Q2 (Rephrasing source)**: We were interested in evaluating the robustness of instruction tuned models to novel instructions, as they might be observed in deployment—this means we were interested not strictly in rephrasings but in novel instructions written anew by practitioners for specific tasks of interest. Collecting new instructions from NLP graduate students (who are familiar with instructing language models) better aligns with this envisioned stress-testing than using LLMs to automatically rephrase existing instructions; in particular this yields a sample from the distribution of instructions which might plausibly be written by practitioners, so we felt it best to manually collect these for evaluation. 

Note that in the proposed methods to make models more robust to instruction variations (Section 5), we did use LLMs to paraphrase existing instructions from the training data as a means to augment the data and realize the proposed objective term.