Paper: What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization (ICLR-2024)

Reviewer comment (Theory):
Following the above point, the authors should explain the reasons from one step to the next step more concretely. For instance, in the proof of Proposition 5.4, the assumptions 5.2 and F.3 do not appear. If there are some steps obtained with these assumptions implicitly, it is encouraged to point it out explicitly. There may be more areas throughout the paper where such improvements in clarity are needed, and the authors are encouraged to address them to enhance overall presentation.

Author response:
**Where Assumptions 5.2 and F.3 are used**

We point to where Assumptions 5.2 and F.3 are used in the revised manuscript. In fact, we construct $\phi^\ast$ and $\rho^\ast$ with the help of Lemma I.9, which requires Assumption F.3. In Step 3, we bound the ratio between $\mathbb{P}(x|S)$ and $\mathbb{P} _{\theta^{\ast}}(x|S)$, which requires the denominator to be larger than $0$. We use Assumption 5.2 here to derive the bound.

Reviewer comment (Experiments):
With all these bounds obtained, it is recommended to validate them empirically, even if through synthetic experiments. Such empirical verification would not only enhance the credibility of the theoretical results but also provide an additional perspective on their soundness.

Author response:
**Added Experimental Results**

We conduct five experiments to verify our theoretical findings, including the Bayesian view (Proposition 4.1 and Eqn. (4.7)), the regret upper-bounded in Corollary 4.2 and  Theorem 6.2, and the constant ratio between $\mathtt{attn}_{\dagger}$ and $\mathtt{attn}$ in Proposition 4.3. The results and implementation details are provided in Appendix C and D in the revised manuscript. We will list a part of numerical results as table below.

* **Verification of the Bayesian View**
...
[Full content from R1 retained, as in original, covering all experimental details, verification of Bayesian view, regret bound, and constant ratio check, exactly as in provided rebuttal text.]

Reviewer comment (Theory):
According to the definition of the assumed model on \(r_t\), is it not related to the true hidden concept? Is this a common assumption on realistic data?

Author response:
**Explanations of (4.1)**

We highlight that our model in (4.1) indicates that $r _t$ **depends on the true hidden concept $z _\ast$** through $h _t$, but $c _t$ does not depend on $z _\ast$. In fact, $r _t$ directly depends on some hidden variable $h_t$, whose distribution is determined by $z _\ast$. When $h _{t}=z$ for $t\in[T]$ degenerates to the hidden concept, this recovers the casual graph model in [7] and the model in [8].

Reviewer comment (Theory):
In the proof of Proposition 4.1, how do the authors obtain the first and the third equalities formally? For the current first equality, does it mean \(h_{t+1}\) is independent of \(c_{t+1}\)? Even though the authors mention these equalities come from model (4.2) (by the way, should it be model (4.1) instead?), more concrete explanations are expected.

Author response:
**Explanations of The Proof of Proposition 4.1**

We thank the reviewer for the suggestions and modified the revised version accordingly. We provide detailed calculations of the first and the third equality here. 

* For the first equality, we have
\begin{align*}
\mathbb{P}(r _{t+1}| \text{pt} _t) &= \int \mathbb{P}(r _{t+1}| h _{t+1},\text{pt} _t) \mathbb{P}(h _{t+1}| \text{pt} _t)\mathrm{d}h _{t+1} \\
&= \int \mathbb{P}(r _{t+1}| h _{t+1},\tilde{c} _{t+1}) \mathbb{P}(h _{t+1}\,|\, S _t)\mathrm{d}h _{t+1},
\end{align*}
where the first equality results from the Bayes’ rule, and the second equality results from the fact that $r _{t+1}$ is **conditionally independent** with the previous history given $h _{t+1},\tilde{c} _{t+1}$ and the fact that $h _{t+1}$ only **parameterizes the transition kernel** of $r _{t+1}$ given $c _{t+1}$. Thus, $h _{t+1}$ and $c _{t+1}$ are independent. 

* For the third equality,
\begin{align*}
&\int \mathbb{P}(r _{t+1} | c _{t+1}, h _{t+1}) \mathbb{P}(h _{t+1}  | S _t, z)\mathbb{P}(z  | S _t) \mathrm{d} h _{t+1} \mathrm{d} z \\
&\quad= \int \mathbb{P}(r _{t+1} | c _{t+1}, h _{t+1},S _t, z) \mathbb{P}(h _{t+1}  | S _t, z) \mathrm{d} h _{t+1} \mathbb{P}(z  | S _t) \mathrm{d} z \\
&\quad= \int\mathbb{P}(r _{t+1}|c _{t+1}, S _t, z)\mathbb{P}(z | S _t) \mathrm{d} z,
\end{align*}
where the first equality results from the fact that $r _{t+1}$ is conditionally independent with the other variables given $h _{t+1},\tilde{c} _{t+1}$, and the second equality results from the Bayes’ rule.

Reviewer comment (Theory):
Regarding the proof of Corollary 4.2, in the first formula, why \(P(S_t|z)\) only relates to responses \(r_{i}\)? In the third formula, should the most outer sum be \(\sum_{t=0}^{T}\), or \(\sum_{t=1}^{T}\)? If the second equality is derived through telescoping, it is not immediately clear why only one term remains. Also, how is the last equality obtained?

Author response:
**Explanations of the Proof of Corollary 4.2**

*In the first formula, since the hidden variable $z$ only parameterizes the **conditional probability** of $r _t$ given $c _t$, $c _t$ and $z$ are independent. 

* In the third formula, the most outer sum is $\sum _{t=0}^{T}$. For the telescoping argument, we follow the convention that $\sum _{t=1}^{0}\cdot = 0$. In fact, when $t=0$, 
\begin{align*}
\mathbb{P}(r _{t+1}|c _{t+1},S _{t})= \frac{\int \mathbb{P}(r _1|c _1,z)\mathbb{P}(z)\mathrm{d}z}{1}.
\end{align*}

Since $\log 1 =0$, **this term just disappears**. Thus, the telescoping argument only leaves one term. 

* The last equality follows from the direct calculations... [Full verbatim continuation of last equality derivation from R4 as given in rebuttal text.]

Reviewer comment (Theory):
Regarding the proof of Theorem 5.3, in the last inequality of (F.2), do we have an expectation over \(S_t^n\) for the first term? According to lemma I.5, there is an expectation over \(X\sim\nu\) for the squared TV. Also, the authors note the bound of \(P_{\hat{\theta}}(x|S)\) in (F.6), but how is this obtained? How is the TV bound located between (F.6) and (F.7) satisfied? Why \(2\epsilon/b_y\) is in \(\mathcal{O}(\frac{1}{NT})\)?

Author response:
**Explanations of The Details in the Proof of Theorem 5.3**

* In the last inequality of (F.2), we note that there should be no expectation for the first term. The reason is that the left-hand side of (F.1) takes conditional expectation with respect to $\tilde{\mathcal{D}}$ conditioned on $\mathcal{D}$. Since **$S _t^n$ is a part of $\mathcal{D}$**, taking the conditional expectation with respect to $\tilde{\mathcal{D}}$ conditioned on $\mathcal{D}$ of the first term is trivial. Thus, there is no explicit expectation over the first term.

* The bound in (F.6) originates from the last **softmax layer** in transformers, which is defined below (5.1). Here $B_A$ is the bound of each component of the softmax input. To derive the lower bound of the output probability of transformers, we calculate the minimal output component of softmax given the bound of each component.

* The TV bound located between (F.6) and (F.7) originates from Proposition F.1 and the distribution assignment (F.5). As discussed in the revised manuscript, we set the distribution $P$ as the uniform distribution on the neighborhood on a parameter with **radius proportional to $1/NT$**. The corresponding approximation error measured in $l_1$ distance,i.e., the TV distribution, is bounded by $1/NT$. Thus, Proposition F.1 shows that the approximation error is $O(1/NT)$ a.s. We also added a figure in the appendix of the revised manuscript to highlight this.

Reviewer comment (Theory):
In the proof of Proposition 5.4, the authors obtain the bound for the difference in L1 norm to be in terms of \(B_{A,1}^{D''}\), which is an exponential growth term, but how does this later become exponentially decay in terms of \(D\)? What does it mean for the sentence “We take that D’=… in Lemma I.9”? Do we let D’ and D’’ be these quantities or C in D’ and D’’ be these quantities? How is the TV bound in step 3 satisfied? Please clean up the proof to make it clearer.

Author response:
**Explanations of Details in the proof of Proposition 5.4**

We would like to provide detailed calculations for the exponential decaying error on $D$, and we have modified the revised manuscript accordingly. The main idea is that: (1) the later modules will **amplify** the approximation error in the previous modules (2) we will **balance the depths** of different modules to handle the amplification.

* For the problems regarding Step 2:

 Lemma I.9 indicates the approximation error $\varepsilon$ of a fully-connected network will depth $D$ can be upper bounded as
\begin{align*}
	\varepsilon\leq \exp\bigg(-\sqrt{\frac{D-\log B}{B}}\bigg).
\end{align*}
We denote the depth of $\Psi _{\phi^\ast}$  and $\Psi _{\rho^\ast}$ as $D^\prime$ and $D^{\prime\prime}$, respectively. Then the total approximation error is
\begin{align*}
	\text{approx err}&\leq d _y \exp\bigg(-\sqrt{\frac{D^{\prime\prime}-\log B}{B}}\bigg) +d _y B _{A,1}^{D^{\prime\prime}}\exp\bigg(-\sqrt{\frac{D^{\prime}-\log B}{B}}\bigg)\\
\end{align*}
... [Full continued derivation exactly as in R5, concluding with Step 3 TV bound explanation.]

Reviewer comment (Theory):
Some of the assumptions made for the theoretical analysis are not justified with respect to their applicability in practical scenarios. The paper does not sufficiently discuss how these assumptions can be relaxed without losing the theoretical results.

Author response:
**Practical Relevance Of Assumptions**

We would like to discuss the relevance of our models and assumptions for the real-world applications:

* Assumption 5.1:

Assumption 5.1 requires that the norm of the token is bounded. It is satisfied in real applications, where the token is a finite-dimensional vector with bounded components. For example, the tokenizers used in GPT-NeoX-20B [15] and Llama2 [16] both satisfy this assumption.

* Assumption 5.2:

We remark that Assumption 5.2 applies to real data. The assumption comes from the ambiguity of language, that is, a sentence can take lots of words as its next word. Similar regularity assumptions are also widely adopted in ICL literature [9,17].

* Assumption 6.1:
This assumption states the coverage of the pretraining data. We note that there will be an **information-theoretic barrier** without this assumption, the ICL learning problem presents an information-theoretic barrier. For example, if the pretraining data does not contain any material about a specific mathematical symbol in the ICL prompt. In this case, it will be extremely difficult for the LLM to derive the correct prediction, since the meaning of this math symbol is unclear to LLM. 

* Assumption G.1:
This assumption states that the input-output pairs are independent when conditioned on the hidden concept $z$. This assumption is relevant in many applications, since the inputs of demonstrations are **independently sampled**, and the output only depends on the input when the concept (task) $z$ is clear.

* Assumption G.2:
This assumption states the coverage of the hidden concept $z _\ast$. If $P(z _\ast)=0$, this concept never appears in the pretraining dataset. It will be very difficult for LLMs to complete this task as explained in the previous response. Compared with Assumption 6.1, this assumption is stronger since the coverage is in the hidden space, and it will be used to handle the wrong outputs.

* Assumption G.3:
This assumption states the distinguishability of the current concept $z _\ast$. When there are only wrong outputs in the prompt, the concept $z _\ast$ itself should be distinguishable enough to **identify from the wrong input-outputs pairs**. This assumption is also stated in [9,17].

Reviewer comment (Novelty):
While the paper attempts to address important questions, its insights as compared to the existing works seem very limited at this point.

Author response:
**Comparison with Existing Works**

We note that there is a large body of empirical work explaining the ICL mechanism of LLMs [6,7,8]. Compared with these works, our paper provides the **theoretical justifications** for some claims in them. 

* In [6], the authors said that “we find that a small number of attention heads transport a compact representation of the demonstrated task”, which is well supported by Proposition 4.3. 

* In [7], the authors claimed that “randomly replacing labels in the demonstrations barely hurts performance”, which is supported by Proposition 6.3 under the distinguiability assumption. 

* In [8], the authors claimed that “TabPFN is a Prior-Data Fitted Network (PFN) and is trained offline once, to approximate Bayesian inference on synthetic datasets drawn from our prior.”, which is instantiation of our Bayesian view in Proposition 4.1.


We note that theoretical papers mainly study ICL from two perspectives. 

* The first one adopts the Bayesian view, where LLMs infer the hidden concept for the prediction  [9,10]. The results in [9] are all based on the hidden Markov model assumption. These papers assume that LLMs are **perfectly pretrained**. In contrast, we do not need the strong hidden Markov model assumption, and we analyze the pretraining error of LLMs. 
* The other kinds of papers focus on the approximation ability of LLMs, where LLMs run optimization algorithms for prediction[1,2,11]. They analyze how transformers can approximate gradient descent. In contrast, our work proposes a totally different mechanism. We note that [11] proposes that LLMs implement natural gradient descent for ICL. Theorem 2 in [12] and Theorem 1 in [13] show that implementing natural gradient descent in the exponential family can be viewed as the Bayesian inference. This view **agrees with Proposition 4.1** in our paper.

Reviewer comment (Experiments):
How can the theoretical findings be empirically validated in the absence of practical experiments within the paper?

Author response:
**Added Experimental Results**

We conduct five experiments to verify our theoretical findings, including the Bayesian view (Proposition 4.1 and Eqn. (4.7)), the regret upper-bounded in Corollary 4.2 and  Theorem 6.2, and the constant ratio between $\mathtt{attn}_{\dagger}$ and $\mathtt{attn}$ in Proposition 4.3. The results and implementation details are provided in Appendix C and D in the revised manuscript. We will list a part of numerical results as the table below.

* **Verification of the Bayesian View**

Table 1:

|                 | zs-hc | ICL-0 | ICL-1 | ICL-5 | ICL-10 |
|-----------------|-------|-------|-------|-------|--------|
| antonym         | 0.736 | 0.000 | 0.296 | 0.651 | 0.675  |
| country-capital | 0.825 | 0.048 | 0.881 | 0.905 | 0.952  |
| present-past    | 0.847 | 0.082 | 0.820 | 0.967 | 0.967  |

To verify the Bayesian view that we adopt in the paper, we implemented two experiments. In the first experiment, we explicitly construct the hidden concept vectors that are found by LLMs. Motivated by (4.7) and Proposition 4.3, we construct the hidden concept vector as the average sum over prompts of the values of twenty selected attention heads, i.e., we compress the hidden concept into a vector with dimension $4096$. To demonstrate the effectiveness of the constructed hidden concepts, we add these hidden concept vectors at a layer of LLMs when the model resolves the prompt with zero-shot. In Table 1, "zs-hc" refers to the results of LLMs that infers with learned hidden concept vectors and zero-shot prompt, and "ICL-$i$" refers to the results of LLMs prompted with $i$ examples. This table report the accuracy of LLMs under different prompts. We consider the tasks of finding antonyms, finding the capitals of countries, and finding the past tense of words. The results indicate that the LLMs with learned hidden concept vectors have **comparable performance** with the LLMs prompted with several examples. This indicates that the learned hidden concept vectors are indeed efficient **compression of the hidden concepts**, which proves that LLMs deduce hidden concepts for ICL. This result strongly corroborates with (4.7). 

Table 2:

|          | 2 examples | 6 examples | 10 examples | 14 examples | 18 examples |
|----------|------------|------------|-------------|-------------|-------------|
| accuracy | 0.000      | 0.917      | 0.917       | 1.000       | 1.000       |


In the second experiment, we aim to verify that LLMs implement inference with the Bayesian framework, not with gradient descent [1,2,3] on some tasks. We prompt the LLMs with the history data of a set of similar multi-armed bandit instances with $100$ arms, and let LLMs indicate which arm to pull in a similar new bandit instance. In these similar bandit instances, there is an informative arm, whose reward is exactly the index of the arm with the highest rewards. We also provide the side information that "Some arm may directly tell you the arm with the highest reward, even itself does not have the highest reward". In each example provided in the prompt, there are the rewards of six arms, including the informative arm and the best arm, in one bandit instance. As shown in Table 2, the LLMs can efficiently implement ICL even with only $6$ examples. We note that the gradient descent algorithms in the previous works cannot explain this performance, since the gradient descent algorithms need **at least $100$ data points**, where each data point is the reward of one arm, to learn. In contrast, the Bayesian view can clearly explain Table 2, where LLMs make use of the side information to calculate **a better posterior** for the bandit instance. Thus, LLMs have good ICL performance in this setting.

Reviewer comment (Writing):
Is there a possibility to simplify the presentation to make the paper more accessible to non-experts?

Author response:
**Simplification of the Presentation**

We simplified the presentation of our revised manuscript in the following perspectives: 
* We provide detailed proof guidelines of our theoretical results.
* We add the figures to illustrate the important steps in our proof.
* We provide the empirical simulations to verify our theoretical findings.

Reviewer comment (Theory):
Could the authors clarify the process of deriving the posterior distributions within the context of ICL and how they relate to the attention weights?

Author response:
**The Process of Deriving the Posterior Distributions**

From the theoretical perspective, in the Gaussian linear model, we first show that the hidden concept can be estimated via the attention defined in (4.8), which is the **expectation of the posterior distribution**.  Proposition 4.3 shows that the softmax attention and the attention in (4.8) is the same up to a constant when $T$ is large. The weights of the modules before this attention can be viewed as the parameters of the representation. In summary, our theoretical results in Section 4.1 show that the softmax attention module is able to estimate the hidden concept in the Gaussian linear model.

From the practical perspective, in our first experiment, we extract the hidden concept vector as the average sum over prompts of the values of twenty selected attention heads. This hidden concept vector is the **mean embedding of the attention heads** with respect to the likelihood, which contains the information of the posterior distributions. The experimental results show that the LLMs with hidden concept vectors in the zero-shot setting have comparable performance with the LLMs prompted with several examples. This verifies that the constructed hidden concept vector indeed contains the essential information about the posterior distribution of the hidden concept. Similar experiments can be found in [6] and [14] for more settings.

Reviewer comment (Theory):
Can the authors provide more details on the assumptions made for the error analysis and how these assumptions can be justified from a practical standpoint?

Author response:
**Practical Relevance Of Assumptions**

We would like to discuss the relevance of our models and assumptions for the real-world applications:

* Assumption 5.1:

Assumption 5.1 requires that the norm of the token is bounded. It is satisfied in real applications, where the token is a finite-dimensional vector with bounded components. For example, the tokenizers used in GPT-NeoX-20B [15] and Llama2 [16] both satisfy this assumption.

* Assumption 5.2:

We remark that Assumption 5.2 applies to real data. The assumption comes from the ambiguity of language, that is, a sentence can take lots of words as its next word. Similar regularity assumptions are also widely adopted in ICL literature [9,17].

* Assumption 6.1:
This assumption states the coverage of the pretraining data. We note that there will be an **information-theoretic barrier** without this assumption, the ICL learning problem presents an information-theoretic barrier. For example, if the pretraining data does not contain any material about a specific mathematical symbol in the ICL prompt. In this case, it will be extremely difficult for the LLM to derive the correct prediction, since the meaning of this math symbol is unclear to LLM. 

* Assumption G.1:
This assumption states that the input-output pairs are independent when conditioned on the hidden concept $z$. This assumption is relevant in many applications, since the inputs of demonstrations are **independently sampled**, and the output only depends on the input when the concept (task) $z$ is clear.

* Assumption G.2:
This assumption states the coverage of the hidden concept $z _\ast$. If $P(z _\ast)=0$, this concept never appears in the pretraining dataset. It will be very difficult for LLMs to complete this task as explained in the previous response. Compared with Assumption 6.1, this assumption is stronger since the coverage is in the hidden space, and it will be used to handle the wrong outputs.

* Assumption G.3:
This assumption states the distinguishability of the current concept $z _\ast$. When there are only wrong outputs in the prompt, the concept $z _\ast$ itself should be distinguishable enough to **identify from the wrong input-outputs pairs**. This assumption is also stated in [9,17].

Reviewer comment (Experiments):
How big is the ICL regret in practice? It is hard to tell how tight the bound is in practice. How can it be measured for LLMs? Since there is no application of the results, it is hard to tell how far the bound is really in practice.

Author response:
We conduct five experiments to verify our theoretical findings, including the Bayesian view (Proposition 4.1 and Eqn. (4.7)), the regret upper-bounded in Corollary 4.2 and  Theorem 6.2, and the constant ratio between $\mathtt{attn}_{\dagger}$ and $\mathtt{attn}$ in Proposition 4.3. The results and implementation details are provided in Appendix C and D in the revised manuscript. We will list a part of numerical results as table below.

* **Verification of the Bayesian View**

Table 1:

|                 | zs-hc | ICL-0 | ICL-1 | ICL-5 | ICL-10 |
|-----------------|-------|-------|-------|-------|--------|
| antonym         | 0.736 | 0.000 | 0.296 | 0.651 | 0.675  |
| country-capital | 0.825 | 0.048 | 0.881 | 0.905 | 0.952  |
| present-past    | 0.847 | 0.082 | 0.820 | 0.967 | 0.967  |

To verify the Bayesian view that we adopt in the paper, we implemented two experiments. In the first experiment, we explicitly construct the hidden concept vectors that are found by LLMs. Motivated by (4.7) and Proposition 4.3, we construct the hidden concept vector as the average sum over prompts of the values of twenty selected attention heads, i.e., we compress the hidden concept into a vector with dimension $4096$. To demonstrate the effectiveness of the constructed hidden concepts, we add these hidden concept vectors at a layer of LLMs when the model resolves the prompt with zero-shot. In Table 1, "zs-hc" refers to the results of LLMs that infers with learned hidden concept vectors and zero-shot prompt, and "ICL-$i$" refers to the results of LLMs prompted with $i$ examples. This table report the accuracy of LLMs under different prompts. We consider the tasks of finding antonyms, finding the capitals of countries, and finding the past tense of words. The results indicate that the LLMs with learned hidden concept vectors have **comparable performance** with the LLMs prompted with several examples. This indicates that the learned hidden concept vectors are indeed efficient **compression of the hidden concepts**, which proves that LLMs deduce hidden concepts for ICL. This result strongly corroborates with (4.7). 

Table 2:

|          | 2 examples | 6 examples | 10 examples | 14 examples | 18 examples |
|----------|------------|------------|-------------|-------------|-------------|
| accuracy | 0.000      | 0.917      | 0.917       | 1.000       | 1.000       |


In the second experiment, we aim to verify that LLMs implement inference with the Bayesian framework, not with gradient descent [1,2,3] on some tasks. We prompt the LLMs with the history data of a set of similar multi-armed bandit instances with $100$ arms, and let LLMs indicate which arm to pull in a similar new bandit instance. In these similar bandit instances, there is an informative arm, whose reward is exactly the index of the arm with the highest rewards. We also provide the side information that "Some arm may directly tell you the arm with the highest reward, even itself does not have the highest reward". In each example provided in the prompt, there are the rewards of six arms, including the informative arm and the best arm, in one bandit instance. As shown in Table 2, the LLMs can efficiently implement ICL even with only $6$ examples. We note that the gradient descent algorithms in the previous works cannot explain this performance, since the gradient descent algorithms need **at least $100$ data points**, where each data point is the reward of one arm, to learn. In contrast, the Bayesian view can clearly explain Table 2, where LLMs make use of the side information to calculate **a better posterior** for the bandit instance. Thus, LLMs have good ICL performance in this setting.


* **Verification of the Regret Bound**

Table 3:
|                        | 0 examples | 10 examples | 20 examples | 30 examples | 40 examples |
|------------------------|------------|-------------|-------------|-------------|-------------|
| $t\cot\text{regret}$   | 1.006      | 8.204       | 10.505      | 10.529      | 10.535      |
| squared error of query | 1.006      | 0.528       | 0.0217      | 0.001       | 0.001       |

To verify Corollary 4.2 and Theorem 6.2, we implement experiments to evaluate the regret in two settings. In the first setting, the LLMs is trained for the linear regression task from scratch, which is a representative setting studied in [1,4]. The examples in the prompt are $\lbrace(x _i,y _i)\rbrace _{i=1}^{N}$, where $x _i\in\mathbb{R}^d$, $d=20$ and $y _{i}=w^{T}x _{i}$ for some $w$ sampled from Gaussian distribution. Given the Gaussian model, we adopt the squared error to approximate the logarithm of the probability. Then the $t\times \text{regret}$ of the LLMs can be well approximated by the sum of the squared error till time $t$.

Reviewer comment (Writing):
What is sequentially mean? Are we not really doing sequential but all at once prompting, which is different from iterative prompting?

Author response:
**Explanation of Word "Sequentially"**

The word "sequentially" comes from the fact that we view the ICL from the **online learning perspective**. We would like to understand how the ICL performance scales with an increasing number of examples in the prompt. For example, in Corollary 4.2, we prompt the LLMs with the $\text{prompt}_{t}$ that has $t$ examples in time $t$. We evaluate the regret in this setting. This is the iterative prompting.

Reviewer comment (Theory):
What is the "wrong" input-ouput mapping? There is no real wrong or right here. How can you get $z_*$?

Author response:
**Explanation of Wrong Input-Output Mapping**

The word "wrong" refers to the fact that **input-output pairing is not correct**. We would like to give an example of the wrong input-output pairing. Assume that we want LLMs to predict the classification of biological categories. The prompts with correct input-output mapping can be "Cats are animals, pineapples are plants, mushrooms are", while the prompt with wrong input-output mapping can be "Cats are plants, pineapples are animals, mushrooms are". Empirically, LLMs are able to derive the **correct answer even with random labels**, and the input-output pattern in the prompt is essential for efficient ICL[6]. Here we provide the **theoretical understanding** for these phenomena under the distinguishability assumption. We show that the pretrained LLMs can perform efficient ICL even with wrong labels, including the random label as a special case, when the distinguishability assumption holds.

Reviewer comment (Evaluation):
Is KL divergence the best to measure distribution gap, which are key in measuring the generalization gap? What is the embedding space it has to use? Is there a better divergence metrics to measure that?

Author response:
**Why we use KL divergence**

Since we adopt the cross-entropy loss to train the LLMs, KL divergence is a natural choice for it. When quantifying the training performance, we measure it via the total variation distance of the conditional probability in the **token space**, not in the embedding space. We leave the exploration of other divergence metrics as the future works.

Reviewer comment (Experiments):
The analysis only involves pretraining in relation to the ICL performance. However, large language models also have fine-tuning distribution involved as an "intermediate" between the two aforementioned stages. Should that introduce some +/- gaps in divergence?

Author response:
**The Discussion of Finetuning**

We note that the LLMs learned from the pretraining are powerful enough. For example, LLMs are able to efficiently implement the time-series tasks in a zero-shot manner[5]. Then we will discuss the influence of two kinds of finetuning. 

The first kind of finetuning is to optimize the parameters of LLMs on a specific task. We note that our performance guarantee in Theorem 5.3 takes expectation with respect to the pretraining distribution. After the fine-tuning process, the distribution here will be closer to the **token distribution of the desired task**. This will improve the performance of LLMs on this task. 

The other kind of finetuning is instruction finetuning, which aims to align the behavior of LLMs to the human preference. People first learn a reward function from the human feedback and use the learned reward to tune LLMs with reinforcement learning techniques. The analysis of this process requires the additional techniques about reinforcement learning, and we leave this as future work.

Reviewer comment (Theory):
Where do we get the hidden concept in the LLM?

Author response:
**How to get the hidden concept in the LLMs**

First, we would like to specify the reasons why LLMs can generate the hidden concepts. There are two main reasons: 1. The pretraining data of LLMs comes from the multiple tasks. There are a sufficient number of concepts presented in the pretraining data. 2. The LLMs are large enough to extract the hidden concepts from the prompt. In Proposition 4.3, we theoretically show that the **attention head** contains the hidden concept information in Gaussian linear model.

Second, we would like to discuss how to get the hidden concept in the LLM in experiments. In our experiment section in Appendix C, we extract the **hidden concept vector as the average sum over prompts of the values of twenty selected attention heads**, i.e., we compress the hidden concept into a vector. This is motivated by Proposition 4.3. To demonstrate the effectiveness of the constructed hidden concepts, we add these hidden concept vectors at a layer of LLMs when the model resolves the prompt with zero-shot. The experimental results show that the LLMs with hidden concept vector in the zero-shot setting have comparable performance with the LLMs prompted with several examples. This verifies that the constructed hidden concept vector indeed **contains the essential information about the hidden concept**. This method is recently proposed and tested in [7]. We note that there are other methods to extract the hidden concept vectors. According to [8], we can also learn the hidden concept deduced by LLMs as token values. We can represent the hidden concept as some special tokens. The value of these tokens are trained by maximizing the probability of the correct answer when the LLMs are prompted with some examples. These learned tokens are shown to be effective in the prompt design.