Paper: Do Pre-trained Transformers Really Learn In-context by Gradient Descent? (ICLR-2024)

Reviewer comment (Theory):
The authors make several arguments that do not prove the main thesis. The reviewer’s understanding of the main thesis of this work is that the ICL setup presented in [0] is not equivalent to gradient descent. However, the setup in [0] considers linear self-attention and not general transformers trained with causal masking. To make their point, the authors should explicitly say what is wrong in the work or setup of [0].

Author response:
Please refer to our general response that clarifies our main thesis. As we clarify in the general response, our intention is **not** to prove the existing line of work wrong. Rather, we are studying the curious problem of whether the existing results stretch beyond the boundaries of their assumptions. This is clarified in our updated draft and we request the reviewer to take a look at it.

Reviewer comment (Presentation):
The font of some of the figures is a bit too small, making it hard to read.

Author response:
We have tried to make the font larger and more consistent. Please let us know if anything else catches your eye.

Reviewer comment (Presentation):
The font across the figures and the text is not consistent.

Author response:
We have tried to make the font larger and more consistent. Please let us know if anything else catches your eye.

Reviewer comment (Theory):
**Hypothesis under study in section 4:** In this section, the authors assume equivalence between ICL and GD on the same model. However, Akyurek et al.'22, and Oswald et al.'23 argue that transformers train a small auxiliary model (different from the parent transformer) inside. Hence, the setting that the authors consider aligns more with the result of [1], which claims in-context learning as implicit optimization of the same parent model. So, if I understand correctly, the authors are simply refuting [1]'s hypothesis. Can the authors comment on this discrepancy? If the authors want to refute the hypothesis completely, then maybe they need to search for all possible sets of auxiliary models that can fit inside.

Author response:
We thank the reviewer for their detailed review and request them to see our general response highlighting the changes that were made to the initial draft. 

> Hypothesis under study in section 4

> the authors assume equivalence between ICL and GD on the same model. 

We have revised our draft so that the connection of Hypothesis 2 with prior work is clearer and we have included experiments on sub-model GD. Section 4 contains experimental results that involve (implicit) sub-models that are more comparable with the setup of Akyurek et al., and Oswald et al., i.e. hypothesis 2. Because we can not search over all possible subsets of the parameters, our experimental setup specifically focused on “intuitive subsets”` that were inspired by the construction of the earlier works: Von Oswald et al. hypothesized implicit model lies in $W_V$ of Transformers, while the probing experiments of Akyürek et al. suggest that this optimization happens in top layers (suggesting that initial layers focus more on representation learning). 

We conducted an experiment where we only fine-tuned $W_V$ of a single layer, for different choices of higher layers (with ablations) in the LLaMa model in hopes of fine-tuning only the implicit model. Based on our notation (Figure 1), call this model $\widehat{\text{GD}}$. We then compare $\widehat{\text{GD}}$ to ICL on our three metrics. We found that the _differences between the two remain significant_. We have added this result in the updated draft (Figure 2).

Reviewer comment (Theory):
**Arguments against sparsity:** I believe the argument that trained transformers are not sparse, as given by the constructions of Akyurek et al.'22, and Oswald et al.'23, isn't a valid argument for the discrepancy between the two algorithms because the previous works simply aim to give an expressivity result on transformers. It is certainly possible that transformers find a dense and more compressed solution to simulate gradient descent. To completely refute the argument, the authors need to refute the probing experiments that the previous works did to search for traces of gradient descent inside these trained models (which I believe is a herculean task).

Author response:
> Arguments against sparsity ... previous works simply aim to give an expressivity result on transformers

As we clarify in the general response, we are curious about the generalizability of these expressivity results beyond the scope of their assumptions. The previous “expressivity” works’ definition of ICL ($\widehat{\text{ICL}}$: our notation) is different from the emergent ICL in pre-trained transformers. Moreover, these works contain experiments and statements that imply **stronger** conclusions. 
- In Von Oswald et al., there are a variety of experiments comparing ICL vs. GD and emphasizing the transformer weights with the hand-constructed weights. They conclude that: `"... when training multi-layer self-attention-only Transformers on simple regression tasks, we provide **strong evidence that the construction is actually found**"`. 
- You can see their stretching of the claims in the title of the published work: “Transformers learn in-context by gradient descent,” though a more accurate statement is “Transformers **have the capacity to** learn in-context by gradient descent”
- A similar weight comparison (implicit linear weight difference; section 4.1) is done by Akyürek et al, that implies these weights do emerge by training transformers. Although we agree that real models might find some dense counterparts of the simple constructions, more efforts should be made to find those families of weights. 

These stronger conclusions, stretching beyond the boundaries of “expressivity,” have motivated our work. We provide a variety of empirical and theoretical arguments that interpreting ICL as [implicit] GD is not a foregone conclusion, and the community should view these results with a grain of salt. 

Just to be clear, our intention is not to dismiss prior work such as Akyürek et al., Von Oswald et al., Dai et al., etc. They are important results, but we feel that the community would benefit from additional clarity about their implications.

Reviewer comment (Experiments):
Instead of simply looking at the movement of the transformer weights across training to argue that the model doesn't stabilize to a single sparse solution, maybe the authors can come up with experiments to suggest that the model changes its internal mechanism across training, instead of simply using different weight matrices to represent the same internal mechanism across time (which again is a herculean task).

Author response:
> ... come up with experiments to suggest that the model changes its internal mechanism across training ...

We do **not** claim that ICL is equivalent to any mechanism (GD or otherwise). With our evolution of weights result, we just want to highlight that the ability of LMs to perform ICL remains stable even with evolving parameters. This just means that any claim of equivalence should be with a mechanism that elicits a family of weights, rather than fixed sparse constructions.

Reviewer comment (Experiments):
Instead of simply optimizing the cross-entropy loss with the entire vocabulary being the candidate set, maybe the authors can put more weight on the relevant logits during training and inference.

Author response:
Our choice to fine-tune the model with a cross-entropy loss over the entire vocabulary was deliberate. This decision aligns with standard fine-tuning practices. While it's valid that placing more weight on relevant logits during training and inference could potentially refine the model's predictions, our primary goal was to evaluate the models in a setting that closely mirrors how they are commonly fine-tuned and deployed in real-world applications (for example, most of the works in the “instruction tuning” literature).

Reviewer comment (Presentation):
What do an overlap rate and cosine similarity of 1.0 mean and < 1.0 mean for ICL in 1 demonstration and >2 demonstration settings in Figure 4?

Author response:
> About overlap rate and cosine similarity?

In the figure, the purple dashed line represents the gap between two instances of ICL that _vary solely in the order of demonstrations_. 

When the demonstration number equals 1, the overlap rate and cosine similarity metrics equal 1. This is because there is inherently only one possible order for a single demonstration. On the other hand, multiple orders are possible when the number of demonstrations exceeds one, introducing variability in the model’s output. In Figure 5, we include this comparison to show the impact that varying orders of demonstrations have on the model's ICL performance, as quantified by our three metrics.

Reviewer comment (Experiments):
Overall, I believe the paper attempts to take a deep dive into a very difficult question using simple experiments, which is impressive in itself. However, as far as I understand, these experiments don't refute the hypothesis of equivalence between GD and ICL completely. Instead, they simply ask the community to make small changes to the hypothesis (e.g. SGD in place of GD, auxiliary internal model isn't the same parent model, etc.). Hence, I have a slightly lower score but am happy to discuss it during the rebuttal period.

Author response:
Thanks for going through our detailed rebuttal!

> ... concerns about the exact hypothesis that the paper aims to verify in the experiments

As the reviewer already understands, previous works' claims are about the equivalence of $\widehat{\text{ICL}}$ and  $\text{GD}$/$\widehat{\text{GD}}$. What we wanted to test with our experiments was whether this implies any equivalence between $\text{ICL}$ and $\text{GD}$/$\widehat{\text{GD}}$. Therefore, we performed two sets of experiments (Section 4) for comparing $\text{ICL}$ with $\text{GD}$ and $\widehat{\text{GD}}$.

1. In practice, $\text{GD}$ is implemented by fine-tuning the whole model on task-specific data, which is what we did. We found that _$\text{ICL}$ and $\text{GD}$ behaved differently on our three evaluation metrics aimed at looking at functional equivalence_. In fact, Dai et. al.  in their experiments look at the same setting (whole model tuning) but only base their equivalence on raw performance which does not paint the whole picture about functional equivalence.
2. For $\widehat{\text{GD}}$, because it is hard to find where exactly would this hypothetical implicit model lie in a big "transformer" like LLaMa, we relied on the suggestion of reviewer **uBmS** to look for "intuitive subsets" of the model to consider as implicit models. We _based our intuition on previous works which hypothesize the existence of these implicit models_. Particularly, we used von Oswald et al.'s construction, which says that the implicit model lies in $W_V$ of the transformer, and Akyürek et al.'s probing experiments to guide us about which layers we should be looking at (deeper layers). Using these intuitions, we performed $\widehat{\text{GD}}$ updates in three different studies:
   - Fine-tuning $W_V$ in one of the deep layers (randomly choosing one of the last 4 LLaMa layers).
   - Fine-tuning $W_V$ in one of the middle layers (randomly choosing one of the layers in layers 16-20 of the 32 in LLaMa).
   - Fine-tuning all weights in 8 layers randomly selected out of the 32 in LLaMa. 

In our updated section 4, we have presented results from all these experiments for $\text{GD}$ and our intuitive understanding of $\widehat{\text{GD}}$ when compared with $\text{ICL}$ and found that they _function differently in all cases_. 

We agree with the reviewer that these comparisons for $\widehat{\text{GD}}$ are not exhaustive and do not invalidate the claim of equivalence completely, but as mentioned in our future work section, identifying implicit models used for GD in LLMs in a computationally feasible manner could be an interesting avenue of future research.

Reviewer comment (Novelty):
I am a bit worried about the significance. The proposed order sensitivity is interesting and yet requires more in-depth analysis (see questions). However, I am not an expert in this field so I will defer to other reviewers regarding this point.

Author response:
Beyond the comparison of the order sensitivity of ICL vs GD, in the appendix, we had results on the order sensitivity of variants of GD (like SGD and Adam) which provide a broader context. As suggested by another reviewer, we have brought these to the main text. 

We also did ablations on the SGD comparison with varying batch sizes (added in the appendix of the updated draft), although it is unclear how the model would decide which order to use in-context examples in the formation of Akyürek et al., our comparisons show that _ICL is still substantially more sensitive to order than SGD or Adam_.

Reviewer comment (Writing):
The writing can be improved. E.g., there are typos such as "We know that both if ..." and "This is a relative metric is computed based...".

Author response:
We regret that the writing of the paper was not up to the mark in the initial draft. We have fixed the mistakes the reviewer pointed out and hope they find the revised version much improved.

Reviewer comment (Theory):
As the author mentioned, the construction of Akyurek et al. ¨ (2022) allows for order sensitivity in GD by update on samples one by one. Do we still have the difference in terms of order sensitivity in that setting?

Author response:
We also did ablations on the SGD comparison with varying batch sizes (added in the appendix of the updated draft), although it is unclear how the model would decide which order to use in-context examples in the formation of Akyürek et al., our comparisons show that _ICL is still substantially more sensitive to order than SGD or Adam_.

Reviewer comment (Experiments):
What if we use specific types of positional encoding to make ICL agnostic to the order of demonstrations? Would the performance increase or not?

Author response:
We want to clarify that our objective with this paper is **not** to make ICL order-stable (although that’s an interesting problem to focus on). ICL does **not** necessitate order-stability as it emerges in real-world models like LLaMa or GPT3 that are shown to be sensitive to input order. 

The LLaMA model (which we use for our experiments) uses relative (specifically, rotary embedding (RoPE, Su et al. 2022) positional embedding, which tends to be more robust to reordering context content. However, as our results in section 4 show, they remain order-sensitive. Whether there will be any future way of encoding information that is robust to order is an interesting goal for future research.

Reviewer comment (Experiments):
Alternatively, we can use the average prediction of many random orders of demonstrations to make ICL agnostic. Is that setting explored?

Author response:
We note that our results, shown in section 4, have all been averaged across multiple runs where demonstrations and their order are randomly sampled. This is to make sure that any order-sensitivity behavior is not a one-off incident but average behavior over multiple runs.

Reviewer comment (Theory):
The GD is an ambiguous algorithm and it is unlucky to be GD in the title of Oswal and hence propagated to this paper. And this has important implications for the experiments presented in this paper. A proper learning algorithm for GD can be specified together with a loss function and a neural network. So, I believe the proper claim to refute should be in the form of “LMs implement internal GD on X neural network with Y loss function to achieve ICL for all X, Y” (some X, Y can also be meaningful but doesn’t refute the possibility fully).

Author response:
We are **not** suggesting that hypothesis 1 is what should be or can be proven. There are two hypotheses in our paper, one which is a more general notion of equivalence between ICL and GD, while the other is what previous works look at. We are just investigating whether the second implies the first in any way. In our general response and the updated draft, we have made the distinction very clear and request the reviewer to kindly read it.

Reviewer comment (Theory):
**Definition-1 and its relation to the strong claim:** The strong claim **cannot be** “LMs implement GD on cross-entropy on themselves” because the Transformer can only implement internal algorithms on a strictly smaller model. For example, Akyürek and Von Oswald show Transformers can implement GD on a linear model way smaller than the actual Transformer that does ICL. However, In Figure-1, authors compare ICL on the same model to GD on the same model with cross-entropy which is inherently impossible to be equal. The same issue of evaluating ICL of a model to GD of the same model exists in Token Overlap experiments as well. And all of these issues arise from Definition-1 which seeks for equivalence of ICL with some fine-tuned version of the same model. On the other hand, authors also propose Definition-2 which is the proper version of the strong claim, however, do not present experiments where only some parts of an LM are finetuned. This unfortunately requires a search over what parameters to finetune which might be computationally expensive. But authors can search over intuitive subsets of all possible parameters.

Author response:
As the reviewer pointed out, it is infeasible to search over all subsets of sub-models to find if one corresponds to the hypothesized implicit model. At their suggestion, we have added experiments involving `“intuitive subsets” in Section 4. (Thank you for the suggestion!) 

Our experimental setup specifically focused on subsets that were inspired by the construction of the earlier works: Von Oswald et al. hypothesized implicit model lies in $W_V$ of Transformers, while the probing experiments of Akyürek et al. suggest that this optimization happens in top layers (suggesting that initial layers focus more on representation learning). 

We conducted an experiment where we only fine-tuned $W_V$ of a single layer, for different choices of higher layers (with ablations) in the LLaMa model in hopes of fine-tuning only the implicit model. Based on our notation (Figure 1), call this model $\widehat{\text{GD}}$. We then compare $\widehat{\text{GD}}$ to ICL on our three metrics. We found that the _differences between the two remain significant_. We have added this result in the updated draft (Section 4).

Reviewer comment (Experiments):
The GD part of the claim is a bit strong to be meaningful. For example, order sensitivity experiments are not related to **SGD** (online GD with some batch size < number of examples) which you also mentioned in the end of P4 “… construction of Akyürek allows for in GD as the update is performed on samples one-by-one instead …”. A better experiment is to look at order-sensitivity of SGD. Those experiments are left to the Appendix, I suggest moving them to the body and displaying them together with GD.

Author response:
We looked at the order-sensitivity for vanilla GD because that is what is studied in Von Oswald et al. We agree with the reviewer’s suggestion about highlighting the sensitivity study with SGD and Adam to show that even if Akyürek et al. allow for order sensitivity, ICL exhibits a lot more sensitivity than SGD/Adam. We have updated the order sensitivity figure (**Figure 2**) in the main text.

Reviewer comment (Theory):
Akyürek et al. does not make the strong claim that LMs implement GD to achieve ICL, and does not even imply this. The paper argues that Transformers can discover learning algorithms to achieve ICL if it’s trained for ICL. Their main result is that the size of the Transformers changes the learned internal algorithm to achieve ICL. Smaller models are more close to SGD whereas large models implement Bayes optimal **Ridge Regression** solution to the linear regression problem. Even if I assume this paper implies something, it cannot be GD from reading these results. Because it suggests Bayesian learning, we expect ICL to have a prior or a regularization that is learned during training time. On the other hand, yes, Von Oswald et al. implies the strong claim in their intro: “We find and describe one possible realization of this concept and hypothesize that the in-context learning capabilities of language models emerge through mechanisms similar to the ones we discuss here.”

Author response:
While Akyürek et al. are more precise in their claims, there are phrasings that may lead to confusing conclusions of hypothesis 1. Most importantly, their use of ICL (training Transformers with ICL objective) is critically different from the emergent ICL in Transformers (the CLM objective on natural text). These two objectives elicit different families of transformer models ($\text{ICL}$ vs $\widehat{\text{ICL}}$, in our notation in Fig.1), as pointed out in our *updated Background (section 2)*. 

The misleading nature of conclusions is in fact evident in your own phrasing. As you suggested, Akyürek et al. argue `“that Transformers can discover learning algorithms to achieve ICL **if it’s trained for ICL**”`. The key assumption *“... if it is trained for ICL”* is easy to miss. Furthermore, if a model “is trained for ICL” is it really capable of ICL?! There is no work that shows the equivalence of this to the emergent ICL in real-world models like LLaMa. We have tried to clarify this distinction in our revised description in “background”. This is one of the first works that adopted the use of ICL objective and its framework has since been used in the several follow-up works that blindly equate $\widehat{\text{ICL}}$ with $\text{ICL}$. 

Moreover, even Akyürek et al. seem to over-extend their conclusions' scope. For example, their section 4, which `“aim[s] to explain ICL at the computational level by identifying the kind of algorithms to regression problems that transformer-based ICL implement,”` is premised by saying that `“it is these iterative algorithms that capture the behavior of **real learners**”`, for which we did not find any concrete evidence. Or the title their published work is: “What learning algorithm is in-context learning?” though a more accurate title is “What learning algorithm **can be simulated** in-context by Transformers?”

Just to be clear, our intention is **not to dismiss** prior works. These are excellent and innovative works. Our hope is to bring in more clarity between the actual claims and the extend to which they can be generalized beyond their assumptions.