Paper: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR-2024)

Reviewer comment (Experiments):
The paper primarily substantiates its core claim that "Performance of large language models (LLMs) is highly sensitive to prompt formatting choices, particularly in few-shot settings," through experiments in classification tasks. However, the scope of the experiments does not extend to the frequently utilized capability of LLMs for long text generation. While short text generation tasks (such as identifying the second letter of a word, adding numbers, or responding with a synonym) are discussed in the appendix, these do not fully capture the important aspect of long text generation. Hence, I suggest that the authors either explicitly state that these findings are specifically under the context of classification tasks or conduct additional experiments on long-text generation to avoid any potential overclaim or misleading interpretation.

Author response:
***“I suggest that the authors either explicitly state that these findings are specifically under the context of classification tasks or conduct additional experiments on long-text generation”*** We agree with the reviewer that longer text generation is an important setting to evaluate for LLM usage, and thus we are included additional experiments for text generation using BertScore and ROUGE-L as metrics in App. B.2 (“Experiments with continuous metrics in open-ended text generation tasks”, also Fig. 19), showing that the issue of performance variance across formats still holds: we computed spread on 10 randomly sampled formats on 17 open text generation tasks with LLaMA-2-7B, and found that performance variance remains regardless of the metric and the number of n-shots considered, with LLaMA-2-7B 5-shot having 25% of tasks with a ROUGE-L spread ≥ 0.098, and a BERTScore spread ≥ 0.09. ROUGE-L is the metric used in the original Super-NaturalInstructions tasks, but is known to have a preference for lexical similarity; therefore, we complement results with BertScore, a model-based metric (which, like all model-based open generation metrics, is more robust to changes in vocabulary but that will express Bert’sinductive bias). We focus our new experiments on evaluating sentence-long text generation tasks, since longer text generation experiments are significantly more computationally intensive, given that they require processing a significantly longer input including the solved few-shot examples, plus generating a long text. We will carefully qualify the extent of our experiments across the manuscript: we also added a Limitations section (Appendix C) in this same vein.

Reviewer comment (Evaluation):
In Section 3.2, the performance spread is quantified as the difference between the maximum and minimum values. I suggest that a more comprehensive approach would be to report both the range (max - min) and the standard deviation, providing a fuller understanding of the data distribution.

Author response:
***“A more comprehensive approach would be to report both the range plus the standard deviation”*** We agree that including standard deviation gives a more comprehensive understanding of the distribution, and we have added analysis to Appendix B.2 (see “Characterizing a model’s accuracy distribution beyond spread” and Figure 20) showing the distribution of accuracy across 500 formats for 30 different tasks. Plots also include the standard deviation, which varies from stdev=0.31 to 0.02 across the tasks considered (median stdev=0.04). We would like to emphasize that reporting the spread (i.e. the distribution range) is already a much more comprehensive report than the single-format estimation that is currently the norm, and that efficiently estimating the standard deviation of a distribution is a priori a much more computationally expensive problem. This would require precisely computing the accuracy of possibly hundreds of formats, and would render our current bandit formulation unsuitable. We nonetheless believe that efficiently computing the standard deviation of such a distribution of accuracies is an exciting question for future work!

Reviewer comment (Evaluation):
In the experimental section, the authors explain that classification tasks were selected for their ease of automatic evaluation. I am curious about the challenges associated with measuring performance for text generation tasks, especially considering the benchmarks that have been proposed recently.

Author response:
***“Why classification tasks? What are the challenges associated with measuring performance in long generation tasks?”*** We focused primarily on classification tasks since they unambiguously showcase clearly that performance spread can be a serious concern, given that they are generally much more straightforward to evaluate than text generation metrics. Text generation evaluation is notoriously more difficult, noisy, and subjective than simple classification metrics. Text generation metrics remain an active area of research, and metrics are often task-specific (e.g. a summarization metric) which would hinder us from using the same metric across tasks. Moreover, evaluating text generation often requires a multifaceted approach (e.g. fluency, helpfulness, similarity with the recorded human response) and we specifically wanted to avoid conflating the establishment of the spread discussion with the open problem of text generation evaluation. We nonetheless conducted additional experiments on sentence-length text generation problems; see next question for details!

Reviewer comment (Theory):
In Algorithm 1, the formulation assumes the reward of each arm is success or failure, but in the NLP field, there are many important tasks where the output of LM cannot be determined between success or failure (for example, a task needs human alignment). Does the formulation still hold for the tasks that cannot be evaluated in discrete value?

Author response:
***“Does the formulation still hold for the tasks that cannot be evaluated in discrete value?”*** It does! Thompson Sampling still works for continuous rewards (see Algorithm 1 of Chappelle & Li, 2011; Bernoulli bandits are shown in Algorithm 2). We focused our explanation on the exact setting we used throughout the paper for ease of reading, but the only requirement to extend it to continuous rewards would be to provide a more suitable reward distribution to sample from instead of sampling from a Beta (e.g. a Gaussian).

Reviewer comment (Evaluation):
Add further explanation of which specific tasks are more susceptible to changes in prompt formatting.

Author response:
- ***“How can we analyze which types of tasks are more susceptible to prompt format influences, rather than just conducting sensitivity evaluations?”*** This is an interesting question, and one we focused on as part of our preliminary explorations. We did not find specific factors that can faithfully predict a priori the sensitivity of a model on a given task. For example, we did not find that multiple choice questions had any significant difference from regular tasks with two fields (e.g. “Input: <text> Output: <text>”) or more. However, digging more into why certain tasks result in more sensitivity is an exciting direction for future work!