From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes a novel meta-learning algorithm. An interesting phase transition between memorization and generalization is observed as the number of tasks is massively scaled up in a controlled (and artificial) manner. 

All reviewers found the phase transition interesting and notable. However, the experiments are primarily conducted on simple datasets like MNIST and it is not clear at all that the approach will generalize to real-world problems. Without further exploration of this phenomenon and without theoretical analysis, our understanding is incomplete. 
justification_for_why_not_higher_score: By considering synthetic tasks on MNIST (and FashionMNIST, etc.), they were able to create millions of tasks but these are somewhat artificial and it is known that some results that hold on MNIST do not generalize to other datasets. There is also no other significant justification (like a theoretical analysis or something). Basically, it was not clear whether the most interesting aspect--the phase transition--was a real phenomenon or something particular to their setup and datasets.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: Everyone found the phase transition interesting, but the paper did not convince people that it was a general phenomenon or specific to the setup in the current paper and the particular datasets like MNIST and FashionMNIST. This is a genuine borderline paper. The pros of accepting it are that it could inspire others to see if they can find ways to test out whether this is a real phenomenon. The cons are that it hasn't fully established its claim due to the particulars of the setup.

From: ICLR.cc/2023/Conference/Paper3648/Authors
title: General Response & Rebuttal Revision
comment: We thank all reviewers for their time and valuable feedback. In addition to our individual responses addressing all reviewers concerns, we have updated our paper submission. We have made several improvements, including an updated table characterizing algorithms in their learning and generalization ability. Further, we added an algorithm box describing our meta-training process. Finally, based on the reviewers’ feedback, we have added a new experiment that demonstrates one path to scaling up our meta-learner to more difficult datasets.

**Description of the new experiment**  
In the paper, we demonstrated the feasibility of meta-learning in-context learning algorithms that are general-purpose. This aligns well with the goal of automatically discovering novel principles of learning that can be applied to a wide range of problems. An even more useful learning algorithm would be capable of both generalizing, as well as leveraging domain-specific information for learning when it is available. This would allow for considerably more efficient in-context learning, scaling to more difficult datasets without very long input sequences. Toward this goal, we investigate a simple scheme that leverages pre-trained neural networks as features to learn upon. This could be from an unsupervised learner or a frozen large language model. Here, we first project the inputs $\bar x_i$ of a base-dataset $\bar D$ into some latent space using a pre-trained network, and then proceed with meta-training and meta-testing as before, randomly projecting these alternative features. For the pre-trained network, we use a ResNet trained on ImageNet and remove its final layer. In [this new figure](https://ibb.co/47JtZ4s) we have meta-trained GPICL on MNIST either with the randomly transformed raw inputs or randomly transformed embedded features.
At meta-test time the learning algorithm generalizes to a wide range of datasets, measured by the meta-test accuracy of the 100th example. At the same time, the pre-trained ImageNet helps to accelerate learning on datasets that have a matching domain, such as CIFAR10. We observe that with only 100 examples, the learning algorithm meta-trained on MNIST, can achieve about 45% accuracy on CIFAR10.

We also plan on renaming the paper to ‘General-Purpose In-Context Learning by Meta-Learning Transformers’ to highlight the connection to in-context learning in Transformer models.

From: ICLR.cc/2023/Conference/Paper3648/Authors
title: Response to Reviewer RW6w (2/2)
comment: ... In Figure 5 we increased the number of tasks, observing multiple distinct phases in terms of the algorithm the model implements. In the first phase, the model purely memorizes instances, not resulting in any within-sequence improvement (learning). In the second phase, there is within-sequence improvement (learning) only on seen tasks, whereas unseen tasks and other datasets do not learn at all. The model can ‘identify’ previously seen tasks. This is followed by the third phase that occurs after a rapid transition where the meta-learned algorithm now generalizes to unseen tasks but also other datasets. If the model would implement the same ‘task memorization’ solution strategy in all three phases, we would see an initially large within-sequence improvement for seen tasks which slowly declines due to limited capacity of the model. At the same time, the within-sequence improvement on unseen tasks and unseen datasets would smoothly increase as a function of the number of training tasks.
Also note that for meta-testing we always use the test set of the respective base dataset, whereas meta-training only relies on the training set. Thus, the sequence of inputs at meta-test-time is always different from meta-training.
More generally, the role of interpolation vs extrapolation is an open question in deep neural networks, especially with very large models.

> No improvements over state-of-the-art (explain how the proposed method could be interesting nevertheless)

Our aim in this paper was not to build a SOTA algorithm, it is to better understand under what conditions black-box models (such as Transformers) learn-to-learn general learning algorithms. These conditions include the data distribution (the number of tasks), model capacity, model state size, and meta optimization hyper-parameters.
We are interested in using less inductive bias in the model and algorithm (eg no backprop, coordinate-wise parameter sharing, or permutation invariance) than prior work in favor of more diverse training task distributions. This allows us to use more expressible function classes (neural network architectures) by introducing the ‘bias’ into the meta-learned learning algorithm through the data distribution. We hope that in future work this enables the discovery of novel and useful general-purpose learning algorithms via meta-learning purely by training across a wide task distribution. We have seen evidence of that recently with in-context learning approaches that can learn-to-learn purely by being trained on internet-scale datasets. Nevertheless, these in-context learning approaches do not generalize in the same way human-engineered learning algorithms such as backpropagation and gradient-descent do. Here, we explore how general-purpose learning algorithms can be meta-learned in-context in black-box neural networks like Transformers.

> How does the work relate to TabPFN: https://arxiv.org/pdf/2207.01848.pdf

TabPFNs are quite related to our work (also a current ICLR submission). Similarly, the authors are interested in discovering general learning algorithms that are reusable across a wider range of tasks. Their focus is on discovering learning algorithms for tabular tasks, whereas we focus on non-tabular image classification. In their work, they are able to define processes (priors) to generate tabular tasks from scratch to train their Transformer on. In contrast, we leverage existing task regularities by generating new tasks from a base dataset via data augmentation (random transformations). This avoids the difficulty of generating realistic non-tabular learning problems from scratch. In addition to showing generalization, we also focus on studying the conditions necessary for general-purpose meta-learning: We characterize phase transitions between algorithms that generalize, algorithms that memorize, and algorithms that fail to meta-train at all, induced by changes in model size, number of tasks used during meta-training, and meta-optimization hyperparameters. We further show that the capabilities of meta-trained algorithms are bottlenecked by the accessible state size determining the next prediction, unlike standard models which are thought to be bottlenecked by parameter count. Finally, we propose practical interventions such as biasing the training distribution that improve the meta-training and meta-generalization of general purpose learning algorithms.

From: ICLR.cc/2023/Conference/Paper3648/Authors
title: Response to Reviewer RW6w (1/2)
comment: We thank the reviewer for their comments. We believe there is strong motivation for our research question and a timely interest in the community, in particular regarding in-context learning. In the following, we further contextualize this motivation and address your other concerns.

> No motivation provided (explain why no bias is useful and provide an example where this actually led to something more useful)
The authors tackle the ambitious problem of creating a black-box meta-learning algorithm. However, they provide neither theoretical, empirical or anecdotal evidence that this might be of any use. Therefore, it appears that this is a challenging problem but might be of no relevance.

Generally, the last decade of ML research has seen gradual removal of inductive biases in favor of more data. Large language models are just a recent example of using black-box architectures to solve a wide range of tasks where the capabilities exceed previous domain-specific approaches with more inductive bias. In a similar vein, we aim to investigate meta-learning learning algorithms which reduce their inductive bias in the architecture and method in favor of more data. In our paper, we aim to characterize the behavior of this limiting case when generating diverse tasks, as opposed to designing one particular algorithm. It is important to understand the fundamental limits of meta-learning and of in-context/sequence learning. This is especially relevant given the advent of transformer-based large language models, for which in-context learning has played a major role and is an extremely active area of investigation. We are interested in the extent to which in-context learning can generalize to a wider range of problems, in a similar way how learning via backpropagation and gradient descent can generalize. Although our experimentation in this paper is on smaller-scale problems only, the results are highly encouraging about this research direction. 

> Experimental setup at times unclear (how do you create the different tasks exactly?)
I have problems interpreting the results since it is nowhere described what a random transformation is. Are these rotations, shifts, crops, everything together?

The tasks are created by sampling the weights of a linear projection matrix from a Gaussian and uniformly selecting one of the $N_y!$ class permutations where $N_y$ is the number of classes. This is described at the end of section 3.1 in the submission:
“A task or dataset $D$ is then defined by its corresponding base dataset $\bar D = \\{ \bar{x}\_i, \bar{y}\_i \\}$, (linear) projection $A \in \mathbb{R}^{N_x \times N_x}$, with $A_{ij} \sim \mathcal{N}\left(0, \frac{1}{N_x} \right)$, and output permutation $\rho$, $D = \\{ A \bar{x}_i, \rho(\bar{y}_i) \\}$. Unless noted otherwise, the distribution over output permutations $p(\rho)$ is uniform.” Output permutations correspond to the assignment of each class to their respective one-hot index in the output layer. To further clarify the experimental setup, we added an algorithm box ([which can be found here](https://ibb.co/Rpv0csL); we are preparing a paper revision to be uploaded in the coming days). First, $K$ datasets $D^{(k)}$ with $k \in 1, \ldots, K$ are generated. Then, we meta-train our neural network on mini-batches of sequences drawn from these tasks as in Equation 2. If any experimental details are still unclear, please let us know.

> Maybe the effects observed are related to the fact that given enough tasks, we just have seen every possible transformation? So maybe we keep seeing memorization where test tasks are simply very close to another train test. There is no discussion on how task similarity between train and test tasks changes with growing number of tasks and whether that might have an impact on the results.

The goal of a general-purpose learning algorithm is to be applicable to a wide range of tasks. In this work, we approach this by generating a set of tasks via random transformations (as described above). Even when generating these random transformations on just one base dataset, eg MNIST, the resulting meta-learned learning algorithm can be applied (generalizes) to other datasets (see Table 1). Indeed, generating many tasks aims at generating enough diversity to make the Transformer implement a general learning algorithm. Arguably, in machine learning, training on more diverse data usually makes the test data more similar to the training data in some (latent) feature space. That said, our results suggest that the increase in training data is very different from task memorization/identification... (1/2, continued in next comment)

From: ICLR.cc/2023/Conference/Paper3648/Authors
title: Response to Reviewer momp
comment: We are glad that the reviewer received our paper so positively and would like to address the remaining concerns in our response.

> One weakness of the paper is that experiments are primarily conducted on simple datasets like MNIST. This work would be more significant if experiments could be conducted on more complex datasets.

As discussed in our limitations, human-engineered learning algorithms would usually process $> 10^4$ data points. While we would expect a meta-learned general-purpose learning algorithm to be somewhat more sample efficient, it would still need to process an incredibly large sequence of data points. Thus, it is currently still difficult to observe such general-purpose phenomena for complex datasets. We believe that advances both in sequence modeling, as well as learning to dynamically switch between general-purpose learning and leveraging domain-specific knowledge can progress this forward, but should not be the scope of this paper.
Due to our method mainly relying on the data distribution, we hypothesize that scaling is one of if not the only remaining barrier to achieving truly general-purpose black-box meta-learning algorithms. Doing so in a principled way will be a larger effort, and we plan to pursue this in follow-up work.
That said, we think such complexity of tasks is not necessary to analyze the possibility and behavior of these general learning phenomena on a smaller scale, as we have done in the paper. In addition to MNIST and FashionMNIST, and given the limited in-context dataset size, we have seen promising results compared to SGD on CIFAR10 and SVHN in Table 1 in our submission.

> Since the paper's main focus is on explaining the properties of transformers as meta-learners, it would be helpful to have an expanded related work section explaining in more detail the previously known properties of transformers.

We agree that this would be quite beneficial and will extend our related work for the camera ready version. In particular, we will focus on previous contributions regarding in-context learning, which is an emerging subfield on the phenomenological study of transformer models.

> Finally, the distinction between learning and generalization in the introduction of Section 3 is a little unclear to me. It would be helpful if the authors could provide examples of the 4 algorithms in the table here to make this more concrete.

We have updated our table with a [visualization which we display here](https://ibb.co/HCG0RhV). If this is still unclear, please let us know.

We are preparing a paper revision to be uploaded in the coming days.

From: ICLR.cc/2023/Conference/Paper3648/Authors
title: Response to Reviewer Ujps
comment: We thank the reviewer for their thoughtful comments and their general positivity about the paper. We address all concerns in the following.

> Although I find the observed phenomena and their explanations very interesting, I am concerned about how general they are, or whether they are very specific to the very limited setup that was chosen. This, in my view, limits the scope and impact of the findings, and I think there should have been more effort to relate to this point.
This is especially true, as the setup is different from the most standard one in (non-RL) meta-learning, which is few-shot learning. It is closely related, but focuses on the sequential version, rather than a support/query split.

As you point out, our setup is closer to the sequential reinforcement learning setting. The support/query split can be seen as a special case where the support set has a fixed size, instead of being scaled from a single input/label pair to some maximum size. We believe that our chosen setting is quite useful, as it also allows us to look at learning as it progresses to larger and larger datasets, observing the performance improvement. Early works in meta-learning have used a similar setting [eg 1]. Due to being closely related, we don’t think the chosen setting has a significant impact on the generality of our results. Also note that our sequential setting fits well with the recent in-context learning literature. In this literature, the support and query are similarly part of a sequence as formulated in our setup. We will extend our related work to discuss the relationship with previous in-context learning in more detail.

> In that regard too, the choice of producing tasks by random projections and orderings of MNIST, rather than the more common subsets of classes of a data-set with large variety of classes like imagenet should be justified.

Memory-based meta-learning [eg 2] (closely related to in-context learning) has been commonly used on subsets of classes of a larger image classification benchmark. Our contribution lies not in proposing a new architecture for this setting, but changing the problem itself: Can we meta-learn learning algorithms (here supervised classification) that can be applied to a wide range of classification tasks, even those that look very different from normal images? When using existing image classification datasets, the meta-learner can rely significantly on feature reuse [eg 3]. Instead, we are more interested in the algorithmic aspect of a meta-learned learning algorithm that can learn on very diverse tasks. Known learning algorithms such as gradient descent are of such general-purpose quality. Thus, we wanted to generate tasks that are so diverse such that feature reuse is not helpful.
Random projections discard the specific representation of a datapoint, focusing on the correlations between datapoints and the underlying structure of the dataset for meta-learning.
We are not claiming that this task generation approach is going to be most suitable for general-purpose learning-to-learn on a larger scale, but we propose it as a tool for studying learning-to-learn behavior in a more general setting.

> Also, would these results and conclusions generalize to non-black-box settings? to the transductive one?

We are not entirely sure what you refer to as the transductive setting, but this is our interpretation:
Currently, we meta-learn black-box neural network functions of the form
$\left( \\{x_i, y_i\\}_{i=1}^{N_D}, x' \right) \mapsto y'.$

In the transductive setting, the neural network would also have access to a larger query set
$\\{x_j'\\}_{j=1}^{N_Q}$

such that
$\left( \\{x_{i}, y_{i}\\}\_{i=1}^{N_D}, \\{x_{j}'\\}\_{j=1}^{N_Q} \right) \mapsto \\{y_{j}'\\}_{j=1}^{N_Q}.$

We would still consider this a ‘black-box’ setting and our method could be applied to this in a very similar way. We would expect our results and conclusions to be similar in the transductive setting but perhaps the meta-learned learning algorithm would also make use of the query set in an interesting way. We chose the ‘sequential’ setting due to its close relationship to prior in-context learning.
If you think we misunderstood your concern, we would appreciate clarification.

[1] Hochreiter, S., Younger, A.S., Conwell, P.R. (2001). Learning to Learn Using Gradient Descent. In: Dorffner, G., Bischof, H., Hornik, K. (eds) Artificial Neural Networks — ICANN 2001  
[2] Santoro, Adam, et al. "Meta-learning with memory-augmented neural networks." International conference on machine learning. PMLR, 2016.  
[3] Raghu, Aniruddh, et al. "Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML." International Conference on Learning Representations. 2020.

From: ICLR.cc/2023/Conference/Paper3648/Reviewer_Ujps
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work performs a detailed investigation into the training process and qualities of a black-box meta-learning algorithm in a 'general-purpose learning' framework.
The focus is on a setup where the data is a collection of 'tasks', that consist of a sequence of (input, label) pairs for which the learner needs to predict the current label, given the prefix of labeled pairs. Hence, a transformer was chosen as a model that can process the task sequentially.
The tasks are generated as randomly projected and label-permuted MNIST images, and the dynamics of the standard meta-training scheme are studied empirically, mainly by monitoring the accuracy signal over training (seen), testing (unseen) and other-domain (Fashion-MNIST) data. 
The training is claimed to go through stages of 'memorization' (first of instances, then tasks) and finally 'generalization' or 'learning-to-learn' to unseen or cross-domain tasks.
The main emphasis is on the transition between memorization and generalization - when does it occur (if at all) and how it is influenced by the properties of training and structure of the network. Mainly, the data size (number of tasks) and model state-size (rather than parameter count). 
The phase transition is also studied in terms of the loss function, which reaches a plateau before it occurs (or doesn't), and several improvements in training are suggested in order to promote the transition at an earlier stage.
strength_and_weaknesses: Strengths:
* The paper presents a very interesting purely-empirical study of the learning dynamics in 'general purpose black box' meta-learning. 
* The experiments are well designed to demonstrate in a simple and clear way how learning progresses through the different stages, towards being able to learn-to-learn. Details are all super clear, very nicely presented and well explained, with nice insights about memorization and generalization.
* The two findings that are most interesting, in my opinion: (i) How the existence of the phase shift (ability to generalize) is determined, very sharply, by the transformer model size and by the number of tasks. (ii) That the ability to learn is this setting is highly correlated to the state (rather than model) size.
* I also find very interesting the in-depth look into what is happening during the (previously observed) training plateau, that is typical before moving to generalization. The 3 suggested 'interventions' in the process are well motivated and demonstrated.

Weaknesses:
* Although I find the observed phenomena and their explanations very interesting, I am concerned about how general they are, or whether they are very specific to the very limited setup that was chosen. This, in my view, limits the scope and impact of the findings, and I think there should have been more effort to relate to this point.
* This is especially true, as the setup is different from the most standard one in (non-RL) meta-learning, which is few-shot learning. It is closely related, but focuses on the sequential version, rather than a support/query split.
* In that regard too, the choice of producing tasks by random projections and orderings of MNIST, rather than the more common subsets of classes of a data-set with large variety of classes like imagenet should be justified. 
* Also, would these results and conclusions generalize to non-black-box settings? to the transductive one? 


clarity,_quality,_novelty_and_reproducibility: As mentioned, the paper is very clear written, analysis and presentation are of great level of detail and all the information needed is provided for reproducibility.
summary_of_the_review: Overall, I am very positive about this paper. I think that it very nicely demonstrates interesting phenomena in meta-learning. The new insights are very interesting and might have an impact on related areas as well.
My main concern is about the generality of the findings, given the very specific setting used for the experiments, as I described above.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3648/Reviewer_momp
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper performs an extensive empirical analysis of transformers used as general-purpose meta-learning algorithms in which transformers take a sequence of training data and a test points to output a prediction. The authors make a number of interesting findings including phase transitions when a transformer is able to generalize to unseen tasks and when a transformer is able to generalize to unseen instances within a task. The authors also demonstrate the importance of a large state to be able to learn a task. Finally, the authors propose a number of interventions to increase the ease of meta-optimization including increasing meta-batch sizes.
strength_and_weaknesses: **Strengths**
The paper's empirical analysis is extensive and comprehensive. The authors make a number of interesting observations that to my knowledge have not been made by prior literature. Moreover, the suggestions on interventions to improve meta-optimization are insightful and may be valuable to the community.

**Weaknesses**
One weakness of the paper is that experiments are primarily conducted on simple datasets like MNIST. This work would be more significant if experiments could be conducted on more complex datasets.

Since the paper's main focus is on explaining the properties of transformers as meta-learners, it would be helpful to have an expanded related work section explaining in more detail the previously known properties of transformers.

Finally, the distinction between learning and generalization in the introduction of Section 3 is a little unclear to me. It would be helpful if the authors could provide examples of the 4 algorithms in the table here to make this more concrete.

clarity,_quality,_novelty_and_reproducibility: **Quality**
The empirical analysis in this paper is thorough. The experiments conducted by the authors are well justified, and all claims made by the authors are sufficiently backed up by experiments. As mentioned above, conducting experiments on more complex datasets would increase the significance of the paper.

**Clarity**
The paper is generally well written and figures are well illustrated. As mentioned above, more sharply defining the distinction between learning and generalization early in the paper would be helpful.

**Originality**
To my knowledge, the insights found by the authors in the paper are novel. However, it would be helpful if the authors could more explicitly separate their new contributions from prior knowledge in the literature; as mentioned above the authors may wish to expand their related works section.
summary_of_the_review: Overall, the paper makes a number of interesting observations on transformers used as meta-learning algorithms. To my knowledge, these insights are novel, and thus the paper could have a significant impact in the field in its current form. The paper's significance could be further improved by conducting experiments on more complex datasets.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3648/Reviewer_RW6w
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose to use transformers to meta-learn a general-purpose learning algorithm. They train a transformer on many related tasks such that it is able to make a prediction for x given an entire dataset and x as an input to the transformer. They analyze when under which circumstances the transformer memorizes the data seen during meta-training and when it generalizes and note that many tasks and large models are a requirement for generalization.
strength_and_weaknesses: **Strengths**
- Method is well-described
- Interesting problem

**Weaknesses**
- No motivation provided (explain why no bias is useful and provide an example where this actually led to something more useful)
- Experimental setup at times unclear (how do you create the different tasks exactly?)
- No improvements over state-of-the-art (explain how the proposed method could be interesting nevertheless)

The authors tackle the ambitious problem of creating a black-box meta-learning algorithm. However, they provide neither theoretical, empirical or anecdotal evidence that this might be of any use. Therefore, it appears that this is a challenging problem but might be of no relevance.

I have problems interpreting the results since it is nowhere described what a random transformation is. Are these rotations, shifts, crops, everything together? Maybe the effects observed are related to the fact that given enough tasks, we just have seen every possible transformation? So maybe we keep seeing memorization where test tasks are simply very close to another train test. There is no discussion on how task similarity between train and test tasks changes with growing number of tasks and whether that might have an impact on the results.

How does the work relate to TabPFN: https://arxiv.org/pdf/2207.01848.pdf
clarity,_quality,_novelty_and_reproducibility: The methodology is well-described but the work lacks a clear motivation. The problem setting is novel but not justified. Reproducibility is not guaranteed since key aspects of the experiments are not described, e.g., how the different tasks for each dataset are generated.
summary_of_the_review: The authors address a challenging research question without providing a motivation for solving it. There are some missing details which prohibit reproducibility and ability for the reader to interpret the results themselves. Finally, the method provides no empirical advantage over other methods as well.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold