Paper: Understanding In-Context Learning from Repetitions (ICLR-2024)

Reviewer comment (Writing):
The paper remains somewhat unclear regarding the overall contributions and implications. The paper argues that the results help understand both limitations of ICL and the inner workings of ICL. To the extent that the paper aims to illuminate the "inner workings" of ICL, the contribution is left somewhat unclear. Do the authors argue that co-occurrence reinforcement is implicated in LMs' ability to pick up input-label mappings? And what "inner workings" are responsible for the co-occurrence effect? A range of recent research discussed in Section 5 aims to explain how ICL works, and observations on the self-reinforcement effect could help shed light -- and indeed the paper hints at this repeatedly (e.g., end of page 4, "In the context of ICL, this pattern corresponds to demonstrations like" -- it seems that the experiment is understood as some kind of simple prompt-like structure, but this idea and its implications are then not made explicit).

Author response:
**Q: “unclear contributions”**

A: Thanks for your advice. Here, we restate our contributions as follows:

- We propose a novel perspective to understand icl with repetitive text generations, where the key effect causing repetitions appears to be an important feature in ICL.
- We empirically establish the existence of token reinforcement across various LLMs, alongside with the inevitable reason behind learning process and inner workings responsible for it.
- We show that token reinforcement plays crucial role in ICL for both beneficial and detrimental effects. It constrains output space and desired patterns, but it is also responsible for spurious connections and possible failure of ICL. Our findings provide a novel research basis for analyze the mechanism of in-context learning.

**Q: “what inner-working are responsible for token reinforcement”**

A: We conduct more analysis over the attention weights and find that in our reinforcement experiments, the LLMs tend to attend the preceding adjacent token with high attention weights. Thus, we hypothesize that the attention to adjacent token is crucial for token reinforcement. To validate our hypothesis, we mask the each token’s attention to its preceding adjacent token and find the strength of token reinforcement is reduced by ~10 times. 

Hence, the reinforcement is not simply realized by attending to similar tokens. Rather, the information is propagated through a process wherein each token iteratively attends to its adjacent token.

For more details about our experiments, we refer you to Appendix C. 

**Q: “clearer implications”**

A: Thanks for your advice about implications and it helps improve our paper! Instead of the traditional thinking of ICL as a mapping from input to label, in this paper, we think of ICL as a combination of reinforcement of tokens, even though some of them only appear once and not get strongly reinforced. Currently, in the last paragraph of Section 3 and the conclusion of our revised paper, we highlight this implication, and we are making more efforts to make the paper clearer.

Reviewer comment (Experiments):
Clarity: Section 3.1 uses the term "sentence" for the examples, but in the experiment they are "randomly generated". How are they sampled -- just as random sequences of symbols sampled i.i.d. from the vocabulary? Are all symbols in the vocabulary equally likely to appear? The same question about how masked subsequences are resampled also applies to the other experiments.

Author response:
**Q1 “random sentences”:** 

A: Yes. All random sentences are generated by uniformly sampled from the vocabulary. All the symbols are equally likely to appear, except the special tokens. We add the description to Appendix B.1.

Reviewer comment (Experiments):
Section 4.1, "Learning to follow patterns" -- when masking the Question, does this mean that the questions in the demonstrations are masked (i.e., replaced with random word sequences) but the question in the final element of the prompt (the one to respond to) is not masked?

Author response:
**Q2 “mask question”:** 

A: Yes. In this experiment, we are trying to test which reinforcement is helpful for following patterns. Thus, we do not touch the test query, and only modify the demonstrations. 

We add the above mentioned experiments to the paper and promise to update the main content accordingly. We hope these results resolve your concern and will be really appreciated if you re-consider the score of our paper.

Reviewer comment (Theory):
This work tries to understand the inherent ICL behavior of LLMs, yet is in lack of theoretical analysis. For example, how is such token co-occurrence reinforcement established? This may involve the detailed interactions between prompts and self-attention mechanism, etc., which I would like the authors to delve into.

Author response:
**Q: “how is such token co-occurrence reinforcement established?”**

A: We provide three additional experiments, demonstrating that (1) The similar token reinforcement lies in the pretraining dataset and thus it is learned by LLMs; (2) repeating features responsible for token reinforcement are effective in the pre-training stage, and thus would be utilized by LLMs; (3) Attending preceding token is responsible for reinforcement.

Based on these experiments, we explain the inevitable reason of token reinforcement is the models’ effort to maximize training likelihood.

For more details, we refer you to the general response and Appendix C in our revised version of paper.

Reviewer comment (Experiments):
As an experiment-oriented work, the authors should examine their assumptions on more LLMs; otherwise, it's hard to reach a common conclusion.

Author response:
**Q: “examine assumptions on more LLMs”**

A: Thanks for your advice! We further consolidate our findings on LLaMA-2(7,13,70b), GPT-J and Mistral. We now examine 16 LLMs in total. All results (Appendix D.2, marked as blue) accord with our findings in main content.

Reviewer comment (Novelty):
The findings of this work are not completely new. As far as I'm concerned, the findings are based on the distributional bias in the demonstrations. The impact of spurious correlations are widely discussed in out-of-distribution generalization literature. In this regard, more insights are welcome. Also, the authors could discuss on how to address such (inevitable) distributional bias in the demonstrations.

Author response:
**Q: “distributional bias in the demonstrations”:** 

A: This is an insightful perspective to understand our work. 

In our understanding, our findings provide an interesting perspective of distributional bias. 

If we consider the distributional bias between the pre-training stage and testing, in-context learning itself should be regarded as a result of distributional bias. In this regard, we show that this distributional bias is helpful through token reinforcement, where the distributional bias studied in OOD is generally harmful. The beneficial part of the bias should be enhanced, and the detrimental part should be mitigated. 

We would like to discuss more related work of distributional bias in our next version.

Reviewer comment (Experiments):
As mentioned above, how could we mitigate the biased effect in ICL demonstrations?

Author response:
**Q: “How to mitigate such bias?”** 

A: Currently, we do not have an experiment showing how to resolve this. Our intuition is that such reinforcement indicates concentration in the representation space at certain direction. If we can detect the direction, we can selectively enhance or mitigate the beneficial and detrimental parts of the bias.