Paper: Understanding In-context Learning with a Pelican Soup Hypothesis (ICLR-2024)

Reviewer comment (Evaluation):
The claim related to the knowledge base needs more clarification. The experiments in [1] demonstrate that input-output mapping is not very important to the ICL. If the label space is correct, LLMs can even implement efficient ICL given wrong mapping. However, this wrong mapping conflicts with the knowledge base. More discussions are needed here.

Author response:
We consider one main contribution of our work to be explaining the ICL ability in the *semantically-unrelated label ICL (SUL-ICL)* setting [8], which is a step forward from the theorem by [7]. Indeed, our work can not explain the random label setting in [1]. However, we would like to point out that as far as we know, there has been no theorem able to explain it. We suggest that explaining it requires making more assumptions on the relationship between the input-label mapping and the domain/topic/theme distribution in the training data.

Reviewer comment (Theory):
In Section 5.1, some assumptions are presented, but there is a notable absence of justification for these assumptions within the paper. This absence makes it challenging to ascertain the realism of these assumptions.

Author response:
We are sorry for the confusion. The justification mainly follows the argument in Section 3. We will make the connection clearer in our later revision. For now, please allow us to elaborate it below:
    - Assumption 1: It follows the efforts made by early linguistics on linguistic formalism and cognitive psychology theories such as language of thought.
    - Assumption 2: This assumption follows the argument in Section 3.1 that general text is usually organized in a similar way as logical induction processes. We assume that “a paragraph is a traversal of the nodes of the proving tree in the topological ordering” because in general text, statements usually follow some causal and/or chronological order, so it can convince readers or be easy for readers to understand.
    - Assumption 3: In other words, this assumption assumes that every paragraph in an article mentions the topic of the article.
    - Assumption 4: In other words, this assumption assumes that some pronouns are used in the article and each pronoun in the article is associated with a single entity.
    - Assumption 5: This assumption follows the argument in Section 2.

    In our opinion, these assumptions are relatively mild compared with previous theoretical works. We are happy to discuss more if you still have some concerns about the realism of these assumptions.

Reviewer comment (Novelty):
I would greatly appreciate further elucidation on the distinction between the hypothesis presented in this paper and that discussed in [2]. Specifically, the variance between the "atomic elements of NLP tasks" and "a set of atom concepts" requires additional clarification.

Author response:
If we understand [2] correctly, [2] does not specify what characteristics the “atomic skills” of NLP need to have. We think our work could be an instantiation of the “atoms” proposed in [2]. 

It seems that defining a set of “atomic skills” is not trivial and can’t be done arbitrarily. “Atomic skills” must be defined in a way such that the skills that work for the training data can generalize to the downstream task. To understand why it is non-trivial to define a set of atomic skills, we can probably think about a trivial case where we define the set “atomic skill” as the 26 skills to predict the words starting with A-Z. With this setting, the argument in [2] is greatly simplified. It seems that Theorem 14 only tells us that for most of the letters, the model can do the cloze problems in the training set well if the answer starts with that letter. It doesn’t seem to be guaranteed that the model can do the cloze problems in the testing set well.

On the other hand, defining the atomic skills with our atomic concepts seems to be more generalizable. Still, we posit it is necessary to discuss the discrepancies between the training set and the downstream task. That’s why in this work we explicitly characterize the distribution shifts in Section 4 and demonstrate that LMs are able to generalize under these distribution shifts in Section 5 and 7. Surely our settings do not fully replicate the real-world data distribution, but we think it still complements some aspects of [2]. 

Please let us know if you still have any questions. We are more than happy to discuss further.

Reviewer comment (Novelty):
It is advantageous to include more highly relevant works in the related works. For example, besides HMM, implicit Bayesian inference is modeled for ICL in many different data assumptions [3,4,5]. [6] also studies the optimization side of ICL.

Author response:
Thank you for the list of relevant works. We will include some of them in our later revision (as we have some concerns about the technical soundness of some of them).

Reviewer comment (Evaluation):
In general, I am not an expert in this line of work, but I have a strong feeling that the hypothesis and the experiment are more about mimicking, or more precisely, producing an environment with which in-context learning still works, rather than explaining how/why in-context learning works in language models. Intuitively, for me, they are different things or at least an insufficient explanation.

Author response:
Thank you for your positive review. 

We would like to point out that producing an environment, with which in-context learning still works, is a practice used in most (if not all) theoretical analyses such as [1] and [2] and even some empirical analyses such as [3]. We consider one of our main contributions to be proposing an environment that is closer to the real-world scenario than previous works.

Reviewer comment (Writing):
Some of the reasonings are hard for me to follow, for example, why the yes/no questions are similar to the demonstrations for in-context learning? Or, such similarities had already been considered distribution shifts?

Author response:
- The yes/no questions in a Pelican Soup riddle have a similar function as the demonstrations for in-context learning, because based on the answer of those yes/no questions, the participants are able to figure out the latent story and thus are able to answer other following questions. Note that we use Pelican Soup riddles just to motivate the importance of the role of commonsense.
- Our argument is that solving these two problems (ICL and modeling coreference) requires similar capabilities. This argument is to draw the intuition for the framework/assumptions in Section 5.1. Surely the two problems are not identical, thus we characterize the difference between these two problems with the distribution shifts discussed in Section 4. To show that LMs may be able to generalize under these distribution shifts, we replicate these distribution shifts in our experiments, as discussed in Section 5.3 and results in Section 5.5 are aligned with our hypothesis.

Reviewer comment (Theory):
Some of the reasonings are hard for me to follow, for example, (in page 3) how do you know the process of figuring out that "she" may be a person to whom something unexpected happened is similar to recovering z for class y? I understand the outcome would be similar, but why also the process?

Author response:
- Our argument is that solving these two problems (ICL and modeling coreference) requires similar capabilities. This argument is to draw the intuition for the framework/assumptions in Section 5.1. Surely the two problems are not identical, thus we characterize the difference between these two problems with the distribution shifts discussed in Section 4. To show that LMs may be able to generalize under these distribution shifts, we replicate these distribution shifts in our experiments, as discussed in Section 5.3 and results in Section 5.5 are aligned with our hypothesis.

Reviewer comment (Writing):
Some of the reasonings are hard for me to follow, for example, if the above one is the actual process, I somehow feel that this suggests that LM should be good at handling anaphora but not catephora, which intuitively, is different from in-context learning.

Author response:
- We understand ICL as generating verbalizers based on the meaning of the verbalizers, which is recovered according to the preceding context, so we think it is similar to modeling anaphora. Could you please elaborate more on why you think ICL is more like handling catephora?

Reviewer comment (Experiments):
Many claims are made without citing prior sources or supporting evidence. For example, in 3.1, the author claims that “language models may be able to acquire the KB by modeling general text.” However, no clear evidence is provided via citations or experiments, and frankly, this is still an ongoing question the community aims to answer; it would be an important work itself to show these claims.

Author response:
Please let us know what are the arguments that you think are not substantiated. For the example you provided, we apologize for not including proper citations. We think that LMs are able to acquire some knowledge about commonsense (though imperfectly) has been widely accepted. Many studies can be found in previous works, e.g. [1][2].

Reviewer comment (Theory):
The main assumption of this paper seems to be that the text in pretraining corpora for LLMs consists of clear reasoning steps (potentially with some intermediate steps dropped). However, this assumption normally requires structured and domain-specific training data such as math text or academic papers. On the other hand, data like dialogues or other internet content may contain completely implicit reasoning steps that are hidden in the text space. So, the proposed pretraining data here, which includes some reasoning steps explicitly in the sequence, is not very representative of the overall LLM pretraining setting.

Author response:
We justify our assumptions made in Section 5.1 in Section 3. As argued in Section 3.1, we posit that training data usually contain text that has some logical structure and/or follows chronicle order, which can contribute to LMs’ reasoning capability. The dataset Calcutec is a formal abstraction of such structure.

Reviewer comment (Experiments):
The experiments are poorly designed, and the implementation details are generally missing, but the main experiment on Calcutec: dataset design is too complicated, while the experimental design and analysis are too simple, although the fact that it can do in-context learning is interesting. In addition, what is discussed in section 6 as real-world evidence does not directly support their main hypothesis.

Author response:
Could you be more specific on what details are missing and how we should improve? We will appreciate your suggestion. The construction of Calcutec is more complicated than the toy data used in previous theoretical works such as [3] because Calcutec relies on milder assumptions and is more realistic.

Reviewer comment (Theory):
The logic of the paper is weak, and the paper is poorly organized. The arguments are not supported by rigorous experimental evidence. Almost all arguments around using the word "**Therefore**" are not rigorous (either the conclusion is not supported by the evidence or the things after "therefore" are logically irrelevant to things before). A large number of arguments are based on the author's thinking that A is **similar** to B, where first the similarity is poorly defined, and how from such similarity the conclusion is reached is usually unclear. For example, in section 3.2, the author claims that predicting the correct pronoun in the next token completion using the information in the context is "**similar**" to inferring the class description z_y for y in text classification. "**Therefore**" modeling general text is similar to performing in-context learning. This may "explain" the linkage between in-context learning and emergent abilities of LLMs.

Author response:
We apologize that we didn’t make the purpose of Section 3 clearer. We treat Section 3 as the justification/intuition for the abstraction/assumptions/framework we made in Section 5.1. We then instantiate the abstraction/framework with Calcutec and show that LMs trained with it can do in-context learning.

Reviewer comment (Theory):
The title of the work or the main motivation: human solving Pelican Soup riddles is similar to LLM doing in-context learning is based on some poorly defined subjective similarity.

Author response:
We use Pelican Soup riddles only to motivate the intuition that commonsense plays an important role for ICL. We would not say our hypothesis is *based on* this similarity.

Reviewer comment (Theory):
What's your view on the mesa-optimization view of in-context learning based on your Pelican Soup Hypothesis? Do they complement each other, and can one explain the other one?

Author response:
We think mesa-optimization mainly explains why models of Transformer architecture trained with autoregressive loss have the capacity to model data that involves ICL-like phenomena. However, in works such as [4], they only study on some toy data. It is unclear how their toy data is relevant to natural language data. Our work starts from another direction, aiming to understand what characteristics of training data lead to the ICL ability.

Reviewer comment (Theory):
While I found the Calcutec experiment in particular to be innovative, the theoretical arguments in Sections 2--3 are quite hand-wavy and unspecific. There is no rigorous theoretical statement of the assumptions and conclusions made in the theoretical framework and the reasoning of how language modeling may lead to ICL.

Author response:
In Section 2, we propose a formalism for NLP classification tasks while the main purpose of Section 3 is to justify the assumptions made in Section 5. We see the assumptions in Section 5 as the formal statement/abstraction of our theorem in Section 3. We are sorry that we didn’t describe it more clearly. We will improve this part in our later revision. Please let us know if we need to make more clarifications.

Reviewer comment (Experiments):
While I believe the Calcutec toy dataset is an interesting contribution and a strength of the paper, it is limited in that the training dataset appears to bake in the repetitive nature of prompts by assuming that each "paragraph" in a document is about one of two latent concepts ("topics"), as in the prompting downstream tasks. A potential concern about the CoT evaluation is mentioned as a Question.

Author response:
As far as we know, all theoretical analyses on ICL depend on some repetitive nature of prompts or training data, e.g. Xie et al and Hahn & Goyal. One difference in our work is that we focus on the repetitive usage and the consistent meaning of pronouns, which is very realistic in real-world data. In Figure 2, we also show that the model can still do ICL when the task description is more complicated than the meaning of the pronouns seen in the training set, i.e., when the task description is composed of 3 atomic concepts (the green lines for “triple” in Figure 2).

Reviewer comment (Evaluation):
In the Digit Addition Task, the ability of the LM to complete the task in one go, whereas the training set usually had intermediate steps, is interpreted as an ICL ability representing a domain shift. However, as the training set also had intermediate steps stochastically dropped (independently, as far as I got from the paper -- so it is possible for all steps to be dropped simultaneously), it is not clear in which sense the test examples are out-of-domain relative to the training distribution. The same concern applies to the Calcutec dataset.

Author response:
We discuss this point below:
    - The digit addition task: Please refer to Figure 8 for the distribution of the number of steps in the datasets. For the 5-digit addition task, when the dropping rate is 0.2, there are only 265 (0.265%) training samples where all the intermediate steps are dropped. We think it is reasonable to say the testing samples are out-of-domain in this case. (And GPT-2-sized model can achieve ~80% accuracy with only 265 training samples in the testing domain.) Meanwhile, considering that the real-world data also has a small fraction of text that drops all the reasoning steps, we think this setting is not unrealistic.
    - Calcutec: Please refer to Figure 5. It shows that even when no steps are dropped in the training set, the model still can do ICL, though the performance is not as good.

Reviewer comment (Evaluation):
How exactly is Chain-of-thought evaluated in Calcutec? Does the prompt only include the first step in the chain? And under what circumstances is the LM's answer counted as correct -- are predictions rolled out until the ";" paragraph appears? This question is crucial for assessing the meaningfulness of the CoT results.

Author response:
The prompt is as the one in Table 1. We only include the reasoning steps of some (3) randomly selected examples and the premise of the testing input in the prompt. (For the example in Table 1, the testing input is `x55 x76 x84 x99`). We unroll until the first verbalizer is generated.

Reviewer comment (Theory):
The explanations are conceptual. More formal theoretical analysis could better elucidate the mechanisms.

Author response:
While Section 3 is purely conceptual, we provide rigorous characterization in Section 5.1. We consider this as one of our main contributions, an instantiation for the assumptions made in previous theoretic works, such as [1] and [2]. For [1], our framework constituted with the assumptions we made in Section 5.1 satisfies the requirements of Corollary 4.2, therefore we can use their analysis to get an $O(1/T)$ regret bound ($T$ is the number of examples in the demonstration). For [2], please refer to our reply to reviewer rNJR.

Reviewer comment (Experiments):
More analysis could be done on how different pretraining corpora impact in-context abilities.

Author response:
Thanks for the suggestion! We can run more experiments. Could you let us know what experiments you think are important are missing?

Reviewer comment (Experiments):
The hypothesis focuses on classification; generative tasks may involve additional factors.

Author response:
We admit that we only focus on classification tasks in this work. However, we would like to stress that even for the classification setting, the mechanism of ICL is not yet clear. Future work may also extend our framework to generation tasks.

Reviewer comment (Theory):
How well does the formalism proposed capture more complex real-world reasoning?

Author response:
The formalism in Section 2 can handle arbitrarily difficult reasoning tasks as long as there is a logic system that can handle it and the induction searching process is computationally feasible, e.g. not NP-hard. We believe that most human-solvable tasks are in this class.

Reviewer comment (Experiments):
Is it possible to design pretraining objectives to better acquire commonsense and reasoning?

Author response:
By preprocessing the training data for LLMs, we think it’s possible. However, training an LLM requires lots of computational resources. We leave it for future work.

Reviewer comment (Evaluation):
How can we test if models learn explicit commonsense rules and reasoning versus pattern matching?

Author response:
Our conjecture would be that for many commonsense rules, LLMs are trained with many instances for each of them. Because the training data may cover a large portion of common surface forms, even though it’s possible that LLMs are doing no more than pattern matching, they are still able to do “reasoning” following the commonsense rules.