Paper: In-Context Learning Learns Label Relationships but Is Not Conventional Learning (ICLR-2024)

Reviewer comment (Novelty):
Several of the outcomes of this paper are, admittedly, already examined in some of the background work described in Sec 2 (and elsewhere), e.g., Zhao et al (2021). It may be advised to add differentiating factors with previous work directly into Sec 2.

Author response:
Thanks for this suggestion! We will happily highlight differences between our submission and prior work directly in Section 2. In particular, we think there are a variety of important differences between our experiments and those of prior work, e.g. those of Zhao et al. (2021) (see below).

Reviewer comment (Evaluation):
Additional differences exist between this work and that of Zhao et al (2021; e.g., the models are entirely different) so a direct comparison in Sec 8 of this type is somewhat disingenuous.

Author response:
Apologies, but we believe this might be a misunderstanding. Our results in Section 8 are not a direct comparison between our results and those of Zhao et al (2021).
In fact, we believe that Zhao et al. look at different and complementary ways in which the distance of examples to the test query, i.e the ‘recency’, affects ICL predictions.
We already discuss this a bit at the end of Section 8 but are happy to elaborate here and in the next version of the draft.


Zhao et al. (2021) observe a ‘recency bias’ in the predictions of ICL, i.e. they find that ICL predicts those labels more frequently that appear more often close to the query.
For example, if the input is  $[..., [x_N-4, y_N-4=1], [x_N-3, y_N-3=0], [x_N-2, y_N-2=1], [x_N-1, y_N-1=1], x_N]$ the model predicts $y_N=1$ more often, regardless of $x_N$. And if the input is $[..., [x_N-4, y_N-4=0], [x_N-3, y_N-3=0], [x_N-2, y_N-2=1], [x_N-1, y_N-1=0], x_N$ the model predicts $y_N=0$ more often, regardless of $x_N$.
Crucially here, all datapoints are drawn from the same fixed distribution $(X, Y)\sim p(X, Y)$, with a single fixed label relationship. Only the order of the datapoints is changed.


In contrast, we study how ICL behaves if _multiple_ label relations are present in the input, e.g. we present the model with $[[x0, y0], …, [x_N/2, y_N/2], [x_N/2+1, y_N/2+1], …[x_N, y_N]]$, where $(x_0, y_0), \dots, (x_{N/2},  y_{N/2})\sim p_1(X, Y)$ and $(x_{N/2+1}, y_{N/2+1}), \dots, (x_{N}, y_N) \sim p_2(X, Y)$.
Imagine that some of your in-context examples actually follow a different label relationship (e.g. due to a label distribution shift or noisy labeling): does it matter where these examples are in the context? This is what we study in NH3 by presenting the model with examples from two different label relations in the in-context demonstrations.
Intuitively speaking, while Zhao et al. (2021) study if ICL has a recency bias for predicting a _particular_ label, we study if there is a recency bias for learning label _relationships_!
These are two distinct biases: in our experiments, we avoid the recency bias from Zhao et al. (2021) by averaging over the order of our examples and labels.
Lastly, we note that one cannot follow from the Zhao et al. (2021) results that a recency bias for learning label relationships exists, and the mechanisms underlying the two biases are likely different: while Zhao et al. (2021) seem to suggest that ICL uses a simple heuristic to predict (looking indiscriminately at the last few labels), this does not explain our label learning bias.

Reviewer comment (Evaluation):
Although not a direct effect of this paper, the extent to which these particular results will generalize to future models is somewhat limited. On a positive note, the methods used to obtain these results may be applicable for some time.

Author response:
Thanks, we agree this could be seen as a limitation and are happy to make this more explicit in the paper. However, we also think this is not really something that can be solved, and so, we believe, should not count too strongly against our submission.
The best we can probably do is compare performance across model sizes, where we do observe some trends (such as larger models being more susceptible to random labels and more willing to flip labels) which _might_ give insight into what behavior to expect in the future.
Regardless, given their widespread use, we do think it is important to publish work on the behavior of these particular models.

Reviewer comment (Evaluation):
Some of the arguments seem to fall to the ‘hasty generalization’ fallacy. E.g., claims are made about ICL in general (at least within the context of LLMs), but really only specific examples are applicable. This is most notable on the experimentation around NH2, which is perhaps the weakest of the three so-called null-hypothesis experiments.

Author response:
Thanks for pointing bringing this to our attention. While we believe that our experiments cover a wide variety of currently relevant ICL scenarios, it is not at all in our interests to make inappropriate claims about their generalization. We have generally tried to be quite careful about this, e.g. we preface our rejection of NH2 on page 7 with ‘For the models we study’, as, particularly here, we felt results could change with future models.
However, if you have any particular cases in mind where you felt our phrasing is too strong, we would be glad to improve these. We could also add the above discussion to our “Limitations” section, if you think this would be helpful.

Reviewer comment (Writing):
Can you please check your work for grammatical errors (e.g., ‘we rephrase these them as…’) and formatting errors (e.g., capitalization in references can be resolved by adding {} appropriately)?

Author response:
Thanks for spotting these! We will fix these errors and will make sure to thoroughly check for any other errors before the final submission.

Reviewer comment (Experiments):
Is it really necessary to examine _all possible_ numbers of demonstrations if the behaviours are so macroscopically clear?

Author response:
Apologies, but could you clarify which behaviors exactly ‘are so macroscopically clear’?


More generally, we think it _would_ be sufficient to replace our figures with ones that have only half (or a third/fourth/fifth) of the resolution of the x-axis, i.e. only half (third/…) the numbers of in-context demonstration sizes.


However, common practice is to only choose _very few_, e.g. one to three different numbers of few-shot demonstrations. For example, the Min et al. (2021) paper considers only the k=16-shot scenario. It is this practice that can be insufficient to draw general conclusions about ICL behavior. Considering multiple different numbers of demonstrations will often reveal differences in ICL behavior, with the effects of labels typically being larger at larger numbers of demonstrations. In particular, prior work often does not consider large numbers of in-context observations, and thus underestimates the effects labels can have on ICL. Lastly, note that we incur no extra cost from obtaining ICL behavior at all numbers of demonstrations due to our novel evaluation strategy that we describe briefly in Section 4 and more extensively in Appendix B.

Reviewer comment (Evaluation):
If you couch your _research questions_ as null hypotheses, why do you not do any significance testing?

Author response:
We _do_ actually perform significance testing: in Tables 1 and 2, we highlight results that are statistically significant with bold numbers. In the appendix, we give extended versions of these tables (Tables F.1 and F.2) that give the standard errors, and we also detail how we compute statistical significance in Appendix E.

Reviewer comment (Novelty):
The paper is framed as a comparison between ICL and conventional learning and claims that hypotheses 2 and 3 reveal differences between ICL and conventional learning, but for H2, the empirical observations corresponding to this hypothesis are not novel and not limited to ICL. For example, in "Making Pre-trained Language Models Better Few-shot Learners", Gao et al show that even fine-tuning the models with flipped or neutral labels degrades their performance, so even conventional learning is not able to overcome the pre-training priors.

Author response:
Thank you for your hard work and helpful feedback! We hope that our responses below and the accompanying paper updates alleviate your concerns.


> The paper is framed as a comparison between ICL and conventional learning and claims that hypotheses 2 and 3 reveal differences between ICL and conventional learning. I do not agree with this.


Apologies, but we believe your concerns here are (1) based on a misunderstanding about what we mean by ‘conventional’ learning algorithms and (2) tangential to our actual key contribution, which is to contribute to a better understanding of label learning in ICL.


(1) You point out that finetuning of LLMs also does not conform to one of our null hypotheses. We agree this is the case and interesting related work (see next reply). However, we would also not consider this an instance of ‘conventional learning’. Our concept of a conventional learner expresses how we would _want_ a learning algorithm to behave based on classical intuitions about learning, e.g. we would want the learning algorithm to eventually predict according to the training data and overcome any ‘priors’. We could also call them ‘idealized learning algorithms’ if you think this is more clear, and are happy to clarify this in the next version of the draft.


(2) Ultimately, however, we believe your concerns here are tangential to our actual key contribution, which is to address the need for a better understanding of label learning in ICL through rigorous experiments. The shortcomings of other learning algorithms, such as finetuning of LLMs, are tangential here and do not negate the need to better understand ICL. This is the case because the mechanism of ICL is different from other established learning algorithms: e.g. observations from gradient-based finetuning do not necessarily transfer to ICL. Our results are important, in particular, because various previous works have been quick to reach questionable conclusions about the behavior of ICL.


> For example, in "Making Pre-trained Language Models Better Few-shot Learners", Gao et al show that even fine-tuning the models with flipped or neutral labels degrades their performance, so even conventional learning is not able to overcome the pre-training priors.


Thanks for suggesting this interesting reference. As previously explained, we believe the behavior of other particular learning algorithms, such as finetuning of LLMs, is largely tangential to our goal of learning about the behavior of ICL, or to our definition of conventional learners.  However, analogies between ICL and finetuning of LLMs are generally interesting at a higher level, and we will be glad to add this to our discussion of related work.

Reviewer comment (Novelty):
For H3, sample order does matter in conventional learning too, which is why we shuffle the training data. In addition, these experiments seem to be a re-discovery of the recency bias in LLMs, as studied in "Calibrate Before Use: Improving Few-Shot Performance of Language Models", Zhao et al.

Author response:
>  Sample order does matter in conventional learning too, that is why we shuffle the training data.


We think this actually demonstrates our previous point well. A fundamental assumption in machine learning is that the data is independent and identically distributed (IID). Thus, we _want_ our learning algorithms to be independent to the order of the samples. Again, the fact that some learning algorithms fail to achieve this in practice, is independent to our investigations into ICL, which deserves to be studied on its own.


(Although, technically, we note that in the example that you mention, the shuffling of the data (over multiple epochs of gradient-based optimization) actually does lead to approximate invariance to example order. As the optimization routine is part of the learning algorithm, we would thus actually say that many, if not most, practical instances of learning are approximately invariant to example order.)


We hope our remarks here clarify this point and we will update the draft to include the discussion.

Reviewer comment (Evaluation):
One major claim of the paper is that although flipping/randomizing labels of ICL examples sometimes does not affect accuracy of the model much, probabilistic metrics such as log likelihood or entropy reveal larger gaps in performance. However, this claim is not quantified well, because accuracy has a clear and interpretable range but the same cannot be said about loglikelihood.

Author response:
> Lastly, one major claim of the paper is that although flipping/randomizing labels of ICL examples sometimes does not affect accuracy of the model much, but probabilistic metrics such as log likelihood or entropy reveal larger gaps in performance. But this claim is not quantified well, because accuracy has a clear and interpretable range but the same cannot be said about loglikelihood.


A key take-away from our work is that a focus on accuracy is one reason why previous work has reached undependable conclusions about the behavior of ICL. Our work shows that probabilistic quantities, such as log likelihood and entropy, can reveal significant additional behavior compared to accuracy, and thus should be part of any thorough analysis. For example, they can reveal statistically significant differences between predictions when accuracy cannot. This is independent from any potential concerns about their interpretability.


For example, in Section 5, we want to know if predictions change when labels are randomized. The average absolute difference in metrics _relative to the standard error_ determines statistical significance. This ratio is smaller for accuracy than for log likelihood and entropy. When we compute the average value of this ratio for the experiments of Section 5 (see Table F.1 for individual means and standard errors), we find it is 6.2 for accuracy, 9.8 for log likelihoods, and 23.3 for entropy. Hence, accuracy is worse at finding differences in predictions because it changes less relative to its variance. Table F.1 also shows a variety of examples where differences are only significant when looking at probabilistic metrics. Lastly, we also refer you to Figure F.10 for an intuitive example of the higher variance of accuracy.


We hope this clarifies your concern, and we will clarify our exposition of the advantages of probabilistic metrics in the draft to avoid any future misunderstandings.

Reviewer comment (Writing):
Minor writing suggestion — the phrase "expresses the sentiment" is not appropriate for describing a scientific paper's conclusion.

Author response:
> Minor writing suggestions:


Thanks for spotting these! We have improved our wording in these instances in the updated version of the draft.

Reviewer comment (Writing):
Minor writing suggestion — the sentence "This is important when the context contains diverging information about a label relationship" is unclear.

Author response:
> Minor writing suggestions:


Thanks for spotting these! We have improved our wording in these instances in the updated version of the draft.

Reviewer comment (Experiments):
Question regarding experiments — was the novel task of author identification tried in the label flipping experiments? If so, the reviewer would like to see some examples of the author-id prompts as well as results regarding all hypotheses, especially for novel tasks, since there is less chance of contamination in the models' pretraining data.

Author response:
> Did you try the novel task of author identification in the label flipping experiments as well? 


Thanks for this interesting suggestion! Following your feedback we have performed the label flipping experiments (NH2) with the author-id dataset across all LLaMa-2 models and added them to the revised version of the draft in Appendix G.3. 


In summary, we do still find that ICL treats different label setups differently, although differences are much less pronounced for the author-id task. 
This provide additional support for our intuitions behind Section 7: we would expect the model to only suffer significantly from flipped or replaced in-context labels if the pre-training data actually suggest to prefer a particular label relation. For the author-id task, which is _guaranteed_ to not be part of the training data, it seems likely the model should not have a strong preference for a particular label relationship.
We refer you to Appendix G.3 for further discussion of these results.


Thanks again for suggesting we perform these experiments, which we believe strengthen our submission!

> If so, can we see some examples of the author-id prompts 


Thanks for this suggestion. We have added some examples from our novel author identification task to Appendix C.

Reviewer comment (Experiments):
For section 5, it would be nice to show how the model performance degrades as randomization increases. Similarly in section 7, you could try gradually mixing in the flipped labels or the arbitrary labels. I think this could help us study the robustness of ICL (in contrast to deep learning models or conventional learning algorithms), as in-context information can often be noisy in general. For instance, I am thinking of an example where the human prompter is giving examples of their preferences to align a model, and fails to accurately articulate one or two examples.

Author response:
Thanks for this great suggestion! We agree these are interesting experiments and now provide results for gradually increasing proportions of random labels in Appendix G.1. We observe that the degradation in model performance increases consistently with increased label noise for LLaMa-2-70B on SST-2. We would like to refer you to Appendix G.1 for further discussion of these results.

We are currently working on providing these results for our usual selection of tasks and models. Lastly, we will also provide gradual label flipping results for the experiments in Section 7 as soon as possible. Given the compute requirements of our experiments and timeline of the discussion period, we hope for your understanding in this matter.

Reviewer comment (Experiments):
Another experiment I'd be interested in seeing: can you study the Lipschitzness/label consistency of ICL? For example, you can come up with a setting like "x1 y1 x2 y2 .... xtest ?" where xtest is a perturbation of one of the x_i's (e.g. a slight rewording, or very close by in embedding space). Will the model predict ytest to be equal to y_i? And will this happen even if we scramble the labels? How different from x_i does x_test need to be in order for it to no longer be significantly influenced by a flipped y_i? Overall I think extending your study to consider the relationships among the x_i's and the x_test w.r.t. the label is interesting, since this is a property of conventional learning algorithms (inputs with similar feature vectors have similar labels).

Author response:
Thanks for this suggestion! We agree this is an interesting experiment relevant to understanding how ICL learns label relationships. As a first step, in Appendix G.4, we have implemented an initial version where xtest is directly equal to one (or multiple) of the xi for the SST-2 dataset. In other words, we investigate to what extent the LLMs copy labels if there are exact matches between the in-context examples and the test query.
We perform these experiments across three label setups: default labels, flipped labels, and arbitrary replacement labels.
We find that one repetition improves performance for arbitrary or flipped replacement labels, but that robust copying behavior, i.e.~perfect accuracy, emerges only for multiple repetitions of the test query.
Again, we would like to refer you to Appendix G.4 for an extended discussion of these results.
We are currently working on extending these experiments to slight rewordings of the x_i and to our usual selection of tasks.

Reviewer comment (Writing):
Minor nit: I'd prefer some more signposting about how a conventional learning algorithm is defined, or some notation of (x_ICL, y_ICL) versus conventional (x_train, y_train). Is a deep learning model considered a conventional learner if it can fit to arbitrary noise?

Author response:
Thanks for bringing this to our attention! We well gladly add more detail and examples to our introduction of conventional learning algorithms. 

Thanks, this is a good question. For the purposes of this draft, we would consider a deep learning model with appropriate optimization a conventional learning algorithm, even if it can fit to arbitrary noise. This is because we would expect this learner to conform to all the expectations we express in the null hypotheses (learns the label distribution, overcomes prior, considers all datapoints (approximately) equally).

Reviewer comment (Experiments):
In section 5 I'd appreciate an example of what randomizing the labels exactly means. I had it confused with the experiment in section 7, whose arbitrary (A, B) experiment could be thought of as randomizing the label space but not the label relationships.

Author response:
Thanks for highlighting these. We will improve the draft here to make sure this is clear!

Reviewer comment (Experiments):
While I found the experiment in section 6 to be quite clever, I feel like "novel task we create" can be made a bit more explicit by specifying it's based on your private data. Without any elaboration, I was surprised about how you were so certain about the task.

Author response:
Thanks for highlighting these. We will improve the draft here to make sure this is clear!

Reviewer comment (Evaluation):
This observation that larger models are more sensitive to label randomization is concerning. I would have expected that with more pre-training data and parameters, the model would be more confident in its prediction and less influenced by random noise in the in-context examples. On the other hand, I do buy the argument that for small models and hard tasks, changing the in-context examples significantly will still result in the model doing poorly, thus appearing to not be as impacted by the random labels. Do you have any thoughts on how to interpret this?

Author response:
Thanks for your thoughts here. We agree it is potentially concerning that larger models are more sensitive to randomized labels. It seems that, in general, larger models are more sensitive to the context, for better or for worse.

We agree with the intuition that more relevant pre-training data should make a model more confident in its predictions, and thus less susceptible to random in-context noise. However, the LLaMa models are actually all trained on similar amounts of tokens (1T for the small models, 1.4 T for the large models). Further, it is unclear how ‘relevant’ this (additional) pre-training data is towards informing ICL predictions on our selection of tasks.

However, our paper does actually contain some results that support the intuition that more relevant data leads to more confident/less flexible predictions. For the Falcon models, we report results for both the base and instruction-tuned variants. We find that instruction-tuned Falcon is somewhat more robust against label randomization, e.g. compare SST-2 for Falcon-40B in Figure F.11 and SST-2 for Falcon-40B Instruct in Figure F.12. One could speculate that the instruction tuning, which exposes the model to many tasks with non-randomized labels, reinforces the models belief to predict default label relationships confidently.

Again, thank you very much for your review. We hope that our reply has addressed any remaining concerns. Please let us know if there are any further changes you would like to see or if there is anything else that we can clarify.

Reviewer comment (Evaluation):
The experiments in the paper do not provide enough evidence or analysis to explain how ICL utilizes label information. While results under the probability metric show large differences between true labels and random labels, this seems more like an ablation study of Min et al. rather than providing deeper insight.

Author response:
> people do not know how ICL utilizes this label information.. The experiment results in this paper are not enough to tell people how ICL utilizes this label information.


While we agree that investigating how ICL uses utilizes label information is a highly important topic of investigation, we disagree that the contributions of this paper are ‘not enough’ for publication for two reasons. Firstly, we believe that our results _do_ contribute a variety of significant novel insights that will be of significant interest to the community: we resolve the controversies around NH1 as mentioned above, we show LLMs can learn truly novel label relationships in-context, the ‘v. interesting’ findings around NH2 show ICL struggles to overcome pre-training preference, and NH3 shows ICL does not aggregate in-context label information like conventional learners. Secondly, we completely agree with you that there is far more to investigate on this topic. However, the paper is already struggling to stay within the page limit without any redundant content to cut out. We thus feel that it is perhaps a strength of this work to provide the foundations that open up new research avenues for future work to make further progress on this important problem.


> [Additional Insights]


We would also like to bring to your attention to two of the four new experiments that we have performed for the rebuttal, which provide even more novel insights into label learning in ICL.


(1) The ‘Progressive Randomization’ experiment in Appendix G.1 studies how robust ICL is to more moderate levels of label noise, an important practical scenario. (2) The ‘Answer in Context’ experiment in Appendix G.4 studies the decision function of ICL by purposefully adding _test_ queries to the in-context _training_ examples. This gives us further insight into how ICL uses label information to update its predictions.


We would like to refer you to Appendices G.2 and G.4 for a detailed discussion of these results. Please let us know if there are any other particular experiments you would like to see, and we will happily add them.

Reviewer comment (Novelty):
Regarding the claim about the positional importance of different ICL examples, the reviewer notes that it is already well-known that changing the order of ICL examples can significantly alter prediction accuracy, as seen in prior work (e.g., Xie et al.).

Author response:
> I don’t think the experiment in [Section 8] deepens our understanding of how/why ICL treats different in-context examples differently. It seems to use new experiments to support some known or folklore claims.


While we agree that the results of this section are perhaps consistent with some of the current ‘folklore’, we believe that our experiments and conclusions provide concrete evidence for something that has not been demonstrated in the existing literature. In particular, our conclusions go beyond the findings in previous work that ‘ICL does not treat each in-context example equally’. Concretely, we show that ICL prefers to learn the _label relationships_ of examples closer to the query. In other words, our work provides insights into _how_ ICL depends on order that are not evident from prior work.

Reviewer comment (Novelty):
The paper does not deepen understanding of how or why ICL treats different in-context examples differently. It instead uses new experiments to support claims that are already known or considered folklore.

Author response:
> I don’t think the experiment in [Section 8] deepens our understanding of how/why ICL treats different in-context examples differently. It seems to use new experiments to support some known or folklore claims.


While we agree that the results of this section are perhaps consistent with some of the current ‘folklore’, we believe that our experiments and conclusions provide concrete evidence for something that has not been demonstrated in the existing literature. In particular, our conclusions go beyond the findings in previous work that ‘ICL does not treat each in-context example equally’. Concretely, we show that ICL prefers to learn the _label relationships_ of examples closer to the query. In other words, our work provides insights into _how_ ICL depends on order that are not evident from prior work.

Reviewer comment (Experiments):
In relation to NH2 (ICL cannot overcome pre-trained preferences), the reviewer asks whether the authors tried the calibrated version of ICL proposed by Zhao et al., 2021, and whether NH2 would still hold after such calibration.

Author response:
In addition, in the NH2 --- the ICL cannot overcome pre-trained preferences-- did you try the calibrated version of ICL, proposed by [1]? I am not sure if NH2 is still rejected if you calibrate the output.


Thanks, this is an excellent suggestion. Our appendix already contains some results for calibration according to Zhao et al. (2021) with default and randomized labels (cf. Figure E.2), where we find that calibration does not significantly improve ICL across tasks. However, given your explicit request, we are happy to also provide results with calibration for the experiments in Figure 5 regarding NH2. You can find these new experiments in Appendix G.2. We again find, that the effects of calibration are negligible and do not change our conclusions.


We do not necessarily find these results surprising.  Zhao et al. (2021) focus on scenarios with small in-context dataset sizes (up to 16). They show that this is where calibration is most important. Already for 16 in-context demonstrations, the gains from calibration are insignificant, see, e.g. their Figure 1. We study ICL at large numbers of in-context demonstrations (20-150 examples). Thus, we are not surprised that calibration does not change our conclusions.