Paper: On Task Description of In-context Learning: A Study from Information Perspective (ICLR-2024)

Reviewer comment (Novelty):
The authors state that our understanding of in-context learning in the meta-learning setting (which they study) is limited. This is not accurate, as we already have a strong theoretical understanding of in-context learning in meta-learned neural networks: they implement (under ideal conditions) Bayesian inference for the distribution of tasks they were trained on (Ortega et al. 2019). Given this correspondence, the main results from the present paper are unsurprising — if the mutual information between the task and its description is high, no hidden variable needs to be inferred, leading to high performance and no learning from samples; if it is low, hidden variables must be inferred to make good predictions, which is easier with more samples.

Author response:
$\bullet$ We would like to clarify that the word "limited"  here refers to that relatively less exploration has been spent on the impact of task descriptions on in-context learning performance, especially compared to extensive investigation on the influence of in-context samples, such as the works Ortega et al.[1]. We are regret that our writing has led to your misunderstanding, and promise to modify the corresponding presentation in the paper.

[1] Ortega, Pedro A., et al. "Meta-learning of sequential strategies." arXiv preprint arXiv:1905.03030 (2019).

Reviewer comment (Writing):
The notation is inconsistent and unclear. For example, in the equations on page 1 the meaning of y and r is not provided; letters are used inconsistently; \(q_{\theta}\) is not defined; Figure 1 uses r to denote targets without explanation. These issues made it difficult to follow the formulation and motivation section, even though the general method could be understood from its broader framing.

Author response:
$\bullet$ Thank you for your detailed feedback. We are regret that our notation is not clear enough. We have thoroughly revised and polished the paper for clarity and readability. We have added the meaning of y and r in intro section (highlighted in red). $q_theta$ is the predicted distribution of target $r$, and we have added the definition in method section (highlighted in red) and unified the notation of targets as r.

Reviewer comment (Reproducibility):
Important details about the training task distribution are missing, specifically how the parameters are sampled.

Author response:
$\bullet$ Thank you for questions regarding parameter sampling.
We adopt the same parameter sampling with GROKKING[2], parameters are sampled uniformly, which makes the calculation of mutual information easy.

[2] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small al-gorithmic datasets, 2022

Reviewer comment (Presentation):
Clarity: For Figure 3A, is the heatmap reflective of performance profile when adding the task prediction loss? The caption led me to believe that I'd be comparing two sets of data, one with task prediction loss and one without.

Author response:
$\bullet$ Thank you for your feedback. We are regret that Figure 3A is not clear enough, and we have modified the figure and caption for clarity.
The heatmap reflects the accuracy difference given two experiment settings: with or without task prediction. The "accuracy" here refers to the predicting accuracy of the answer of query equation, and "difference" means simple abstraction between accuracy values given same training data but under different training strategy. The two experiment settings use the same dataset, only differ in loss backward strategy. Only under task prediction setting, the loss between correct task and predicted task description will be backward. We will re-write the caption for a clearer expression.

Reviewer comment (Presentation):
Clarity: It seems Figure 3B would be better suited to be a part of Figure 2 and 3A for Figure 4 (if it is indeed a figure representing performance profile after adding the task prediction loss). Maybe this would make the figures have too many panels or introduce size constraints, but I encourage the authors to think about how each figure can have a consistent theme with panels that are related and support each other.

Author response:
$\bullet$ We appreciate your suggestion. You are correct that figures need reorganization for consistent themes.
We have made modifications to figures following your suggestion.

Reviewer comment (Presentation):
Clarity: For Figure 4, the y-axis is the accuracy in predicting the task, right? If so, maybe label it as "task prediction accuracy." I had some brief confusion on whether this meant accuracy in performing the current task correctly or predicting what the current task is. Also would be good to label the color bar in Figure 4C.

Author response:
$\bullet$ Thank you for your suggestion.
The y-axis for plots in Figure 3B (Figure 4A in previous version) and Figure 3C (Figure 4B in previous version) refers to "task accuracy", which means accuracy in predicting what the current task is. We appreciate your insight and have made the necessary changes as per your suggestion.

Reviewer comment (Evaluation):
Interpretation: The interpretation of the task description suppressing the model's in-context learning ability puzzles me. To me, it seems like, when the task description does not have enough information, the model relies on the examples to learn the task. But when the task description has enough information, the model does not need to rely on examples because it has already learned the task, so additional examples don't lead to performance gains. In both cases, the task description is not necessarily hindering the model's in-context learning ability, but maybe is the "preferred" method of learning the task over examples. Maybe this is what the authors meant when they said "suppress", but to me, suppression of in-context learning would mean that the model is unable to learn the task using more in-context examples.

Author response:
$\bullet$ Thank you for your valuable feedback. We are regret that our writing has led to your confusion.
You are correct that the task description is more likely leading the model to ignore in-context examples. We have revisited and revised the paper to optimize interpretation.

Reviewer comment (Evaluation):
Related to above: the results actually remind me of work in cognitive science on teaching with language vs demonstrations (Sumers et al. 2023). Teaching with language can be more effective because a language description of a task can effectively transmit abstract concepts needed to perform the task, whereas demonstrations give specific instances of concepts needed to perform the task and requires the learner to infer the abstract concepts given the demonstration. I'd be interested to hear the authors' thoughts on if their results can be interpreted in this light.

Author response:
$\bullet$ Thank you for your suggestion and this is a very interesting point. Inspired by your comment, we have explored the relation between our results and  work in cognitive science on teaching with language vs demonstrations, and cited the related work.

The experiments carried by Sumers et al.[1] are excellent reflection of in-context learning in real life. Teaching with language acts as adding task description, while teaching with demonstrations acts as giving in-context examples. The experimental results are very consistent with our synthesis experiments, that higher information of task description (teaching by language in a more informative and efficient way) will increase the lower bound of performance. And we offer theoretical derivation for this phenomenon in Eq.3. The phenomenon that language relies on shared abstractions to efficiently transmit complex concepts can also be illustrated by mutual information theory. As teaching languages concluding better shared abstractions are more informative and efficient, resulting in higher task description mutual info value. 

[1] Sumers, T. R., Ho, M. K., Hawkins, R. D., & Griffiths, T. L. (2023). Show or Tell? Exploring when (and why) teaching with language outperforms demonstration. Cognition, 232, 105326.

Reviewer comment (Evaluation):
The authors mention real world scenarios as a limitation. What specifically about real world scenarios could make things different? It would be good to have specific examples. One I could think of: if a task requires grounding in another non-linguistic domain (e.g. utilizing images or motor programs for robots), then the task description can supply the abstractions needed to perform the task, but the in-context examples will be needed to ground those abstractions in the new domain. In this case, the learning dynamics after the phase transition may be different from the current results, because more in-context learning examples may still help the model even after the task description has enough information.

Author response:
$\textbf{Question}$: This is a very interesting point. Our experiments use simplified settings for mutual information quantization. 
One difference is that, in real world, the mutual information can be hard to calculated accurately, as the task description can be implicit and complex. We do believe the observation in this paper can inspire following works on the task description for in-context learning.

Reviewer comment (Evaluation):
For Figure 2A, why the blue curve achieves the best performance after the threshold? In my understanding, blue curve stands for the least number of ICL expamples.

Author response:
$\textbf{Q1}$: Thank you for your question regarding Figure 2A. You are correct in your understanding that blue curve stands for the least number of ICL expamples. The phenomenon depicted in Fig 2A, that an increase in the number of ICL examples doesn't improve performance accordingly, is caused by relatively sparse sampling in in-context examples in Figure 2A. Please refer to Fig 2B, which demonstrates in-context examples' impact in continuous curves. We have listed the numerical results in the table below for clarity.  
It can be seen in Fig 2B that before the threshold (task info less than 3.2 nats), the number of in-context examples significantly impact in-context learning(the top one), while after that, it has almost no influence (the bottom one)}. And the bottom sub-figure in Fig 2B corresponding to the right most point in Fig 2A (i.e., mutual Info 4.02 nats), which clearly depicts that the accuracy stays relatively consistent across a large range of in-context example numbers. However, in-context examples do help as an accuracy gain can be observed, such as by increasing the in-context number from 0 to 4.  

|Accuracy (Mutual Info 0 nats) | 0.1017 | 0.1052 | 0.1117 | 0.1234 | 0.1320 | 0.2094 | 0.1876 | 0.2414 | 0.2955 | 0.3670 | 0.5367 |  
|----------------------------------------|-----------|-----------|----------|-----------|-----------|-----------|----------|-----------|-----------|-----------|-----------|  
|Accuracy (Mutual Info 4.02 nats) | 0.3530 | 0.4267 | 0.5052 | 0.4067 | 0.4695 | 0.4687 | 0.4656 | 0.4573 | 0.4678 | 0.4548 | 0.4674 |

Reviewer comment (Writing):
There seems to be 3 phases in Figure 2A so there should be 2 transitions. All the transition I mention in the comment is the latter one. I think the word transition refers different ones in different contexts, which makes me a bit confused. (For instance, the "phase transition" in the comment for Figure 2B in section 5.1 seems to be different from the transition in the 2nd paragraph in page 2.)

Author response:
$\textbf{Q2}$: Thank you for your question about the use of word "transition". All "phase transition" in our work should refer to the latter one(i.e., the turning point at task info threshold around 3.2 nats), and we will check the presentation for clarity. The "phase transition" in Fig 2B also refers to the turning point at task info threshold around 3.2 nats.  
Special Attention is paid to this transition as it demonstrates a switch in task description's impact on in-context examples.

Reviewer comment (Presentation):
I don't see the reason behind the sharp transition in Figure 2A and the provided interpretations for both phase (i) and (ii) fail to explain that. Is the sharp transition caused by the lack of points used in the plot?

Author response:
$\textbf{Q3}$: Thank you for your question regarding the phase transition.  
first, We would also like to clarify that we have sampled 12 points in Figure 2A, and we have added the detailed value of mutual info and corresponding accuracy in the appendix. And there are around 3 points for phase (ii), and more than 8 points for phase (i). All these points with different number of in-context examples demonstrates the phase transition phenomenons. All these indicate that the reason behind the phase transition is not due to the lack of points used in the plot.  

It is an interesting point to study the phase transition phenomenons for in context learning, however it may be hard to provide an thorough explanation for the phase transition. For example, GROKKING[1] only presents the phase transition without explanation. And another work[2] observed a phase transition in investigating ICL given varying task diversity seen by pretrained model, and they suppose the sharp transition indicates a sharp crossover from the theoretical optimal of pre-trained task to that of true task.  

We would like to present some of our thoughts on the phase transition for discussion.  
We suppose the main reason behind this phase transition may related to the optimization.  
In phase (i), the optimization of the model is trapped into an ambiguous state when the model try to learn from both the context examples and insufficient task info.  
The task info, itself, can not provide sufficient guidance towards the right solution, and meanwhile distract the attention that paid on the examples. Even though the task info and the context examples are not inconsistent,  the model need to consider both factors, which may make it hard to be optimized. On the other hand, with sufficient task info in phase (ii), the optimization can mainly rely on the task info, leading to successful optimization and better performance.  

[1] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small al-gorithmic datasets, 2022  

[2] Allan Ravent ́os, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression, 2023.

Reviewer comment (Evaluation):
Phase (i) needs more detailed explaination as increasing the number of ICL examples barely improves ICL.

Author response:
$\textbf{Q4}$: Thank you for your inquiry regarding Phase (i). We would like to clarify that increasing the number of ICL examples still improves model's performance, ad as an accuracy gain can be observed from in-context number 2 to in-context number 4 in Fig 2B. However, this improvement is soon interfered by high task info. Such phenomenon is mainly caused by ignorance to in-context examples led by task description, and can be illustrated more clearly by model's attention mechanism. As depicted in Fig 3B, a decline in the attention ratio of in-context examples can be observed when more task info added. This indicates that adding task description info will distract model's attention in in-context examples.  
In this phase, the accuracy stays relatively constant within a large range of in-context example numbers after threshold.

Reviewer comment (Theory):
Section 5.2 gives the interpretation for phase (ii) by saying "Higher information of task description will increase the lower bound of performance." What is the lower bound here? Should it be stated in a more formal way?

Author response:
$\textbf{Q5}$: Thank you for pointing out the omission of explanation of "lower bound". We regret that the interpretation here is not clear enough, and we have modified the text in the paper (highlighted in red). The interpretation of "lower bound" is formulated in Eq.3, and the "lower bound of performance" refers to the right part of the equation. The first term signifies the task label prediction, whereas the subsequent term corresponds to the loss function employed in the in-context training for the GPT model. This equation, therefore, demonstrates that accurate task label prediction contributes to the maximization of the log-likelihood. And this explanation matches the experimental results given in Fig 2 and Table 3 as under task label prediction setting, a performance gain can be observed with task info gain.

Reviewer comment (Writing):
Is the first sentence in the 3rd paragraph of section 5.3 finished? ("As the task label prediction can also reflect whether the model understand what the task is.")

Author response:
$\textbf{Q7}$: We regret that the interpretation here is not clear enough, we have modified the text in the paper (highlighted in red) as follows:  
The performance of task label prediction can also reflect whether the model understand what the  
task is.

Reviewer comment (Novelty):
Section 2 paragraph 2 says "However, despite these valuable contributions, all these explorations tend to overlook the influence of task descriptions on the in- context learning process." Is that true? For the linear regression ICL papers, I don't think they have such task decriptions.

Author response:
$\textbf{Q9}$: Thank you for your valuable feedback. You are correct that linear regression ICL papers even don't have task descriptions. We regret that the expression here is inaccurate, and we have revised the text in the paper.

Reviewer comment (Writing):
At the beginning of page 4 there seems to be a leftover comment from a co-author: “There are several grammar errors and issues with clarity in the given passage.”. Unfortunately I think that this applies to the whole paper except for Sections 1 and 2. The current paper is very cumbersome to chew through, and seems as if it was drafted and maybe reorganized multiple times in a hurry. Many conference papers have a paragraph here or there that is a bit rushed due to deadline pressure, but the current manuscript needs a major revision and rewrite in terms of clarity and readability. I personally suggest in the future to not give in too much to deadline pressure, if a paper is clearly not ready yet, take the time to polish it and turn it into the strongest possible version. This is doubly unfortunate because I really like the idea behind the main experiment; but it is currently not in a presentable state. I have left many concrete passages for improvement under ‘Questions’.

Author response:
$\bullet$ Thank you for your detailed feedback. We have thoroughly revised and polished the paper for clarity and readability.

Reviewer comment (Evaluation):
Besides clarity, I do have some concerns regarding the main experiment and its interpretation. For Figure 2: the task info (mutual info) is given in the task description by a lower and an upper bound. I assume these values can only be integers and the mutual info can thus only be changed in discrete steps. The results in Fig. 2 A, C look like the task description is only helpful for the setting with maximal mutual information (which means the upper and lower bounds are equal), but is not helpful in any other setting. The main text describing the results of the figure says “after this information threshold the accuracy grows rapidly with information gain“ and the paper suggests similar, somewhat “gradual”, results on multiple occasions: it looks to me that this is incorrect; if the information in the task description is not maximal it seems to be detrimental regarding of the amount of information, and only when it is maximal does it actually help (i.e. only for a single value of the x-axis in Fig 2A, corresponding to the rightmost column in Fig 2C). Is this correct? If yes, I think this needs an explanation (why can the network not use partial info about the task; is this a general phenomenon or is it only for modular arithmetic, etc.).

Author response:
$\bullet$  We understand your concern about the main experiment. We would like to clarify that the medium task description also helps to improve performance.

| Task Info(nats) | 0 | 0.21 | 0.45 | 0.73 | 1.02 | 1.39 | 1.61 | 1.83 | 2.30 | 3.22 | 3.40 | 3.91 | 4.02 | 4.27 | 4.61 |
| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | 
|Accuracy(0 ex) | 0.1017 | 0.1020 | 0.1027 | 0.1030 | 0.1034 | 0.1030 | 0.1042 | 0.1048 | 0.1062 | 0.1119 | 0.1164 | 0.2104 | 0.2834| 0.4267 | 0.8641 |
|Accuracy(4 ex) | 0.1118 | 0.1071 | 0.1062 | 0.1036 |  0.1023 | 0.1020 | 0.1030 | 0.1062 | 0.1083 | 0.1174 | 0.1202 | 0.2216 | 0.5052 |0.5877 | 0.9323 |
|Accuracy(32 ex) | 0.3670 | 0.2083 | 0.1670 | 0.1046 | 0.1062 | 0.1070 | 0.1084 | 0.1097 | 0.1101 | 0.1211 | 0.1314 | 0.2137 | 0.4548 | 0.7926 | 0.9688 |

We list the exact val accuracy under some experiment settings in Table above,  Accuracy(0 ex), Accuracy(4 ex), Accuracy(32 ex) refer to val accuracy of models trained with no in-context example, 4 in-context examples, 32 in-context examples. The results can provide a clearer demonstration of changes in accuracy, that accuracy basically grows with mutual info increasing. Before a certain information threshold (around 3.2 nats concerning results sampled in Table, the accuracy gain remains relatively subtle to be 0.001 between neighbouring info settings. And after this information threshold, the accuracy grows rapidly with information gain. The small accuracy increase before info threshold 3.2 nats is too small to be observed in Fig 2A and Fig 2C. Also, we actually sample several info value sampled after that information threshold (also see Table, referring to task description helps not only at maximal level. We are regret that our notations are not clear enough.

We also explore the reason for model's relative low accuracy and subtle accuracy gain given under threshold task description, and find that the task description will lead the model to ignore the information from in-context examples. As depicted in Fig 2D, the ratio of in-context examples in attention keeps declining with more task description information. On the contrary, the attention ratio of task description increases when more task-related information are given. This indicates that adding task description info will distract model's attention in in-context examples.