Original: The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence
Revised: The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence

Original: Abstract
Revised: Abstract

Original: Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks – thus raising important questions about when and how meta-learning algorithms should be deployed.
Revised: Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks – thus raising important questions about when and how meta-learning algorithms should be deployed.

Original: In this paper, we seek to clarify these questions by 1. proposing a novel metric – the diversity coefficient – to measure the diversity of tasks in a few-shot learning benchmark and 2.
Revised: In this paper, we seek to clarify these questions by 1. proposing a novel metric – the diversity coefficient – to measure the diversity of tasks in a few-shot learning benchmark and 2.

Original: by comparing Model-Agnostic Meta-Learning (MAML) and transfer learning under fair conditions (same architecture, same optimizer, and all models trained to convergence).
Revised: by comparing Model-Agnostic Meta-Learning (MAML) and transfer learning under fair conditions (same architecture, same optimizer, and all models trained to convergence).

Original: Using the diversity coefficient, we show that the popular MiniImageNet and CIFAR-FS few-shot learning benchmarks have low diversity.
Revised: Using the diversity coefficient, we show that the popular MiniImageNet and CIFAR-FS few-shot learning benchmarks have low diversity.

Original: This novel insight contextualizes claims that transfer learning solutions are better than meta-learned solutions in the regime of low diversity under a fair comparison.
Revised: This novel insight contextualizes claims that transfer learning solutions are better than meta-learned solutions in the regime of low diversity under a fair comparison.

Original: Specifically, we empirically find that a low diversity coefficient correlates with a high similarity between transfer learning and MAML learned solutions in terms of accuracy at meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD).
Revised: Specifically, we empirically find that a low diversity coefficient correlates with a high similarity between transfer learning and MAML learned solutions in terms of accuracy at meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD).

Original: To further support our claim, we find this meta-test accuracy holds even as the model size changes.
Revised: To further support our claim, we find this meta-test accuracy holds even as the model size changes.

Original: Therefore, we conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly.
Revised: Therefore, we conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly.

Original: We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.
Revised: We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.

Original:  Introduction
Revised:  Introduction

Original: The success of deep learning in computer vision [Krizhevsky et al., 2012, He et al., 2015], natural language processing [Devlin et al., 2018, Brown et al., 2020], game playing [Silver et al., 2016, Mnih et al., 2013, Ye et al., 2021], theorem proving [Rabe et al.
Revised: The success of deep learning in computer vision [Krizhevsky et al., 2012, He et al., 2015], natural language processing [Devlin et al., 2018, Brown et al., 2020], game playing [Silver et al., 2016, Mnih et al., 2013, Ye et al., 2021], theorem proving [Rabe et al.

Original: Polu and Sutskever, 2020, Han et al.], code [Chen et al.] and more keeps motivating a growing body of applications of deep learning on an increasingly wide variety of domains.
Revised: Polu and Sutskever, 2020, Han et al.], code [Chen et al.] and more keeps motivating a growing body of applications of deep learning on an increasingly wide variety of domains.

Original: In particular, deep learning is now routinely applied to few-shot learning – a research challenge that assesses a model’s ability to learn to adapt to new tasks, new distributions, or new environments.
Revised: In particular, deep learning is now routinely applied to few-shot learning – a research challenge that assesses a model’s ability to learn to adapt to new tasks, new distributions, or new environments.

Original: This has been the main research area where meta-learning algorithms have been applied – since such a strategy seems promising in a small data regime due to its potential to learn to learn or learn to adapt .
Revised: This has been the main research area where meta-learning algorithms have been applied – since such a strategy seems promising in a small data regime due to its potential to learn to learn or learn to adapt .

Original: However, it was recently shown [Tian et al., 2020] that a transfer learning model with a fixed embedding can outperform many modern sophisticated metalearning algorithms on numerous few-shot learning benchmarks [Chen et al., 2019, 2020, Dhillon et al., 2019, Huang and Tao, 2019].
Revised: However, it was recently shown [Tian et al., 2020] that a transfer learning model with a fixed embedding can outperform many modern sophisticated metalearning algorithms on numerous few-shot learning benchmarks [Chen et al., 2019, 2020, Dhillon et al., 2019, Huang and Tao, 2019].

Original: This growing body of evidence raises the question if researchers are applying meta-learning with the right inductive biases [Mitchell, 1980, Shai Shalev-Shwartz, 2014] and designing appropriate benchmarks for meta-learning.
Revised: This growing body of evidence raises the question if researchers are applying meta-learning with the right inductive biases [Mitchell, 1980, Shai Shalev-Shwartz, 2014] and designing appropriate benchmarks for meta-learning.

Original: Our evidence suggests this is not the case.
Revised: Our evidence suggests this is not the case.

Original: Our work is motivated by the inductive bias that when the diversity of tasks in a benchmark is low, then a meta-learning solution should provide no advantage compared to a minimally meta-learned algorithm – e.g. only fine-tuning the final layer.
Revised: Our work is motivated by the inductive bias that when the diversity of tasks in a benchmark is low, then a meta-learning solution should provide no advantage compared to a minimally meta-learned algorithm – e.g. only fine-tuning the final layer.

Original: Therefore, in this work, we quantitatively show that when the task diversity – a novel measure of variability across tasks – is low, then MAML (Model-Agnostic Meta-Learning)
Revised: Therefore, in this work, we quantitatively show that when the task diversity – a novel measure of variability across tasks – is low, then MAML (Model-Agnostic Meta-Learning)

Original: [Finn et al., 2017] learned solutions have the same accuracy as a popular transfer learning algorithm – a supervised learned model with a fine-tuned final linear layer.
Revised: [Finn et al., 2017] learned solutions have the same accuracy as a popular transfer learning algorithm – a supervised learned model with a fine-tuned final linear layer.

Original: We want to emphasize the importance of doing such an analysis fairly: with the same architecture, same optimizer, and all models trained to convergence.
Revised: We want to emphasize the importance of doing such an analysis fairly: with the same architecture, same optimizer, and all models trained to convergence.

Original: We hypothesize this was lacking in previous work
Revised: We hypothesize this was lacking in previous work

Original: [Chen et al., 2019, 2020, Dhillon et al., 2019, Huang and Tao, 2019].
Revised: [Chen et al., 2019, 2020, Dhillon et al., 2019, Huang and Tao, 2019].

Original: This empirical equivalence remained true even as the model size changed – thus further suggesting this equivalence is more a property of the data than of the model.
Revised: This empirical equivalence remained true even as the model size changed – thus further suggesting this equivalence is more a property of the data than of the model.

Original: Therefore, we suggest taking a problem-centric approach to meta-learning and suggest applying Marr’s level of analysis [Hamrick and Mohamed, 2020, Marr, 1982] to few-shot learning – to identify the family of problems suitable for meta-learning.
Revised: Therefore, we suggest taking a problem-centric approach to meta-learning and suggest applying Marr’s level of analysis [Hamrick and Mohamed, 2020, Marr, 1982] to few-shot learning – to identify the family of problems suitable for meta-learning.

Original: We do this by analyzing the intrinsic diversity of the data.
Revised: We do this by analyzing the intrinsic diversity of the data.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: We propose a novel metric that quantifies the intrinsic diversity of the data of a few-shot learning benchmark .
Revised: We propose a novel metric that quantifies the intrinsic diversity of the data of a few-shot learning benchmark .

Original: We call it the diversity coefficient .
Revised: We call it the diversity coefficient .

Original: It enables analysis of meta-learning algorithms through a problem-centric framework .
Revised: It enables analysis of meta-learning algorithms through a problem-centric framework .

Original: We analyze the two most prominent few-shot learning benchmarks – MiniImagenet and Cifar-fs – and show that their diversity is low.
Revised: We analyze the two most prominent few-shot learning benchmarks – MiniImagenet and Cifar-fs – and show that their diversity is low.

Original: With this context, we clarify the previous results from [Tian et al., 2020] and show that MAML and transfer learning have statistically equivalent test performance under a fair comparison.
Revised: With this context, we clarify the previous results from [Tian et al., 2020] and show that MAML and transfer learning have statistically equivalent test performance under a fair comparison.

Original: We define a fair comparison when the two methods are compared using the same architecture (backbone), same optimizer, and all models trained to convergence.
Revised: We define a fair comparison when the two methods are compared using the same architecture (backbone), same optimizer, and all models trained to convergence.

Original: This holds even as the model size increases.
Revised: This holds even as the model size increases.

Original: As an actionable conclusion, we provide a metric that can be used to analyze the intrinsic diversity of the data in a few-shot learning benchmarks and if it measures to a low value then the suggestion is to use simple transfer learning methods .
Revised: As an actionable conclusion, we provide a metric that can be used to analyze the intrinsic diversity of the data in a few-shot learning benchmarks and if it measures to a low value then the suggestion is to use simple transfer learning methods .

Original: We hope that this line of work inspires a problem-centric first approach to meta-learning – especially a quantitative approach to benchmark creation instead of focusing on making huge data sets.
Revised: We hope that this line of work inspires a problem-centric first approach to meta-learning – especially a quantitative approach to benchmark creation instead of focusing on making huge data sets.

Original:  Background
Revised:  Background

Original: Model-Agnostic Meta-Learning (MAML): The MAML algorithm
Revised: Model-Agnostic Meta-Learning (MAML): The MAML algorithm

Original: [Finn et al., 2017] attempts to meta-learn an initialization of parameters for a neural network so that it is primed for fast gradient descent adaptation.
Revised: [Finn et al., 2017] attempts to meta-learn an initialization of parameters for a neural network so that it is primed for fast gradient descent adaptation.

Original: It consists of two main optimization loops: 1) an outer loop used to prime the parameters for fast adaptation, and 2) an inner loop that does the fast adaptation.
Revised: It consists of two main optimization loops: 1) an outer loop used to prime the parameters for fast adaptation, and 2) an inner loop that does the fast adaptation.

Original: During meta-testing, only the inner loop is used to adapt the representation learned by the outer loop.
Revised: During meta-testing, only the inner loop is used to adapt the representation learned by the outer loop.

Original: Transfer Learning with Union Supervised Learning (USL): Previous work [Tian et al., 2020] shows that an initialization trained with supervised learning, on a union of all tasks, can outperform many sophisticated methods in meta-learning.
Revised: Transfer Learning with Union Supervised Learning (USL): Previous work [Tian et al., 2020] shows that an initialization trained with supervised learning, on a union of all tasks, can outperform many sophisticated methods in meta-learning.

Original: In particular, their method consists of two stages: 1) first they use a union of all the labels in the few-shot learning benchmark during meta-training and train with standard supervised learning (SL), then 2) during the meta-testing, they use an inference method common in transfer learning: extract a fixed feature from the neural network and fully fine-tune the final classification layer (i.e., the head) with LBGFS (Limited-memoryBroyden–Fletcher–Goldfarb–Shanno algorithm).
Revised: In particular, their method consists of two stages: 1) first they use a union of all the labels in the few-shot learning benchmark during meta-training and train with standard supervised learning (SL), then 2) during the meta-testing, they use an inference method common in transfer learning: extract a fixed feature from the neural network and fully fine-tune the final classification layer (i.e., the head) with LBGFS (Limited-memory algorithm).

Original: Task2Vec Embeddings for Distance computation between Tasks:
Revised: Task2Vec Embeddings for Distance computation between Tasks:

Original: The diversity coefficient we propose is the expectation of the distance between tasks (details in section 3).
Revised: The diversity coefficient we propose is the expectation of the distance between tasks (details in section 3).

Original: Therefore, it is essential to define the distance between different pairs of tasks.
Revised: Therefore, it is essential to define the distance between different pairs of tasks.

Original: We focus on the cosine distance between
Revised: We focus on the cosine distance between

Original: Task2Vec (vectorial) embeddings as in [Achille UCLA et al., 2019].
Revised: Task2Vec (vectorial) embeddings as in [Achille UCLA et al., 2019].

Original: The vectorial representation of tasks provided by Task2Vec
Revised: The vectorial representation of tasks provided by Task2Vec

Original: [Achille UCLA et al., 2019] is the vector of diagonal entries of the
Revised: [Achille UCLA et al., 2019] is the vector of diagonal entries of the

Original: Fisher Information Matrix (FIM) given a fixed neural network as a feature extractor – also called a probe network – after fine-tuning the final classification layer to the task.
Revised: Fisher Information Matrix (FIM) given a fixed neural network as a feature extractor – also called a probe network – after fine-tuning the final classification layer to the task.

Original: The authors explain this is a good vectorial representation of tasks because 1.
Revised: The authors explain this is a good vectorial representation of tasks because 1.

Original: It approximately indicates the most informative weights for solving the current task (up to a second order approximation) 2.
Revised: It approximately indicates the most informative weights for solving the current task (up to a second order approximation) 2.

Original: For rich probe networks like CNNs, the diagonal is more computationally tractable.
Revised: For rich probe networks like CNNs, the diagonal is more computationally tractable.

Original: The Task2Vec embedding of task τ is the diagonal of the following matrix:
Revised: The Task2Vec embedding of task τ is the diagonal of the following matrix:

Original: [Equation]
Revised: [Equation]

Original: where f w is the neural network used as a feature extractor with architecture f and weights w , ˆ p ( x | τ ) is the empirical distribution defined by the training data
Revised: where f w is the neural network used as a feature extractor with architecture f and weights w , ˆ p ( x | τ ) is the empirical distribution defined by the training data

Original: D τ = { ( x i , y i ) } ni =1 for task τ , and p ( y | x, f w ) is a deep neural network trained to approximate the (empirical) posterior ˆ p ( y | x, τ ) .
Revised: D τ = { ( x i , y i ) } ni =1 for task τ , and p ( y | x, f w ) is a deep neural network trained to approximate the (empirical) posterior ˆ p ( y | x, τ ) .

Original:  The Diversity Coefficient
Revised:  The Diversity Coefficient

Original: The diversity coefficient aims to measure the intrinsic diversity (or variability) of tasks in a few-shot learning benchmark.
Revised: The diversity coefficient aims to measure the intrinsic diversity (or variability) of tasks in a few-shot learning benchmark.

Original: 3.1 Ground Truth Distribution Based Diversity
Revised: 3.1 Ground Truth Distribution Based Diversity

Original: We propose that the diversity coefficient should measure some aggregation of the distance of tasks from a few-shot learning benchmark.
Revised: We propose that the diversity coefficient should measure some aggregation of the distance of tasks from a few-shot learning benchmark.

Original: If the tasks are distributions then the measure should aggregate the distance between probability distributions.
Revised: If the tasks are distributions then the measure should aggregate the distance between probability distributions.

Original: Therefore when the ground truth distributions are available, we propose the diversity coefficient to be the expected value of distances between the task probability distribution (see section 3.2, 2 for notation details):
Revised: Therefore when the ground truth distributions are available, we propose the diversity coefficient to be the expected value of distances between the task probability distribution (see section 3.2, 2 for notation details):

Original: [Equation]
Revised: [Equation]

Original: 3.2 Task Based Diversity Coefficient (with Task2Vec)
Revised: 3.2 Task Based Diversity Coefficient (with Task2Vec)

Original: For real data the diversity coefficient is the expected distance between embeddings of different tasks given a fixed probe network .
Revised: For real data the diversity coefficient is the expected distance between embeddings of different tasks given a fixed probe network .

Original: In this work, we choose the distance to be the cosine distance between vectorial representations of tasks according to Task2Vec as described in section 2.
Revised: In this work, we choose the distance to be the cosine distance between vectorial representations of tasks according to Task2Vec as described in section 2.

Original: We define the diversity coefficient of a few-shot learning benchmark B as follows:
Revised: We define the diversity coefficient of a few-shot learning benchmark B as follows:

Original: [Equation]
Revised: [Equation]

Original: where f w is the neural networks used as a feature extractor with architecture f and weights w , ˆ p ( x | τ ) is the empirical distribution defined by the training data
Revised: where f w is the neural networks used as a feature extractor with architecture f and weights w , ˆ p ( x | τ ) is the empirical distribution defined by the training data

Original: D τ = { ( x i , y i ) } ni =1 for task τ , τ 1 , τ 2 are tasks sampled from the empirical distribution of tasks ˆ p ( τ | B ) for the current benchmark
Revised: D τ = { ( x i , y i ) } ni =1 for task τ , τ 1 , τ 2 are tasks sampled from the empirical distribution of tasks ˆ p ( τ | B ) for the current benchmark

Original: B (i.e. a batch of tasks with their data sets D =
Revised: B (i.e. a batch of tasks with their data sets D =

Original: ( τ i , D τ i )
Revised: ( τ i , D τ i )

Original: Ni =1 ), a task τ
Revised: Ni =1 ), a task τ

Original: i is the probability distribution p ( x, y | τ ) of the data, d is a distance metric (for us cosine), f w is the neural networks used as a feature extractor with architecture f and weights w , and ˆ p ( x | τ ) is the empirical distribution defined by the training data D τ = { ( x i , y i ) } ni =1 for task τ .
Revised: i is the probability distribution p ( x, y | τ ) of the data, d is a distance metric (for us cosine), f w is the neural networks used as a feature extractor with architecture f and weights w , and ˆ p ( x | τ ) is the empirical distribution defined by the training data D τ = { ( x i , y i ) } ni =1 for task τ .

Original: We’d also like to recall to the reader that the definition of a task in this setting is of an n-way, k-shot few-shot learning task.
Revised: We’d also like to recall to the reader that the definition of a task in this setting is of an n-way, k-shot few-shot learning task.

Original: Therefore, each task has n classes sampled with k examples used for the adaptation.
Revised: Therefore, each task has n classes sampled with k examples used for the adaptation.

Original: 3.3 Justification for Task Based Diversity Coefficient with Task2Vec
Revised: 3.3 Justification for Task Based Diversity Coefficient with Task2Vec

Original: Previous work showed Task2Vec is a solid task embedding method because: 1.
Revised: Previous work showed Task2Vec is a solid task embedding method because: 1.

Original: it’s able to predict task similarities that match human semantic relations between different visual tasks [Achille UCLA et al., 2019] 2.
Revised: it’s able to predict task similarities that match human semantic relations between different visual tasks [Achille UCLA et al., 2019] 2.

Original: Task2Vec distances correlates positively with taxonomical distances [Achille UCLA et al., 2019] 3.
Revised: Task2Vec distances correlates positively with taxonomical distances [Achille UCLA et al., 2019] 3.

Original: importantly, Task2Vec is based on the FIM – which is a Riemannian metric on the space of probability distributions [Amari and Nagaoka, 2007].
Revised: importantly, Task2Vec is based on the FIM – which is a Riemannian metric on the space of probability distributions [Amari and Nagaoka, 2007].

Original: This implies that distances based on Task2Vec are good proxy to ground truth distances between task distribution.
Revised: This implies that distances based on Task2Vec are good proxy to ground truth distances between task distribution.

Original: We confirm this in figure 11 where we should a strong correlation (with Pearson r = 0 . 990 ) of Task2Vec diversity compared to ground truth diversity (using the Hellinger distance).
Revised: We confirm this in figure 11 where we should a strong correlation (with Pearson r = 0 . 990 ) of Task2Vec diversity compared to ground truth diversity (using the Hellinger distance).

Original: We want to emphasize that one of our main contributions is the novel use of Task2Vec in non-trivial ways to analyze transfer and meta-learning algorithms.
Revised: We want to emphasize that one of our main contributions is the novel use of Task2Vec in non-trivial ways to analyze transfer and meta-learning algorithms.

Original:  Experiments 4.1 Experiments: Low Diversity Correlates with Equivalence of MAML and Transfer Learning when the diversity is low
Revised:  Experiments 4.1 Experiments: Low Diversity Correlates with Equivalence of MAML and Transfer Learning when the diversity is low

Original: We show that the diversity coefficient of the popular MiniImagenet and Cifar-fs benchmarks are low with good confidence intervals using four different probe networks in the left most plot of figure 1.
Revised: We show that the diversity coefficient of the popular MiniImagenet and Cifar-fs benchmarks are low with good confidence intervals using four different probe networks in the left most plot of figure 1.

Original: We argue it’s low because the diversity values are in the interval [0 . 06 , 0 .
Revised: We argue it’s low because the diversity values are in the interval [0 . 06 , 0 .

Original: 117] – and the minimum and maximum values of the cosine distance are 0 . 0 and 1 . 0 .
Revised: 117] – and the minimum and maximum values of the cosine distance are 0 . 0 and 1 . 0 .

Original: The rightmost plot in figure 1 shows the failure of transfer learning (with USL) to outperform MAML.
Revised: The rightmost plot in figure 1 shows the failure of transfer learning (with USL) to outperform MAML.

Original: Finally, we study the role of model size.
Revised: Finally, we study the role of model size.

Original: An alternative hypothesis to explain the equivalence of transfer learning (with USL) and MAML could be due to the capabilities of large neural networks to be better meta-learners in general – inspired by the impressive ability of large language models to be few-shot learners [Brown et al., 2020, Bommasani et al., 2021, Radford et al., 2021, Devlin et al., 2018].
Revised: An alternative hypothesis to explain the equivalence of transfer learning (with USL) and MAML could be due to the capabilities of large neural networks to be better meta-learners in general – inspired by the impressive ability of large language models to be few-shot learners [Brown et al., 2020, Bommasani et al., 2021, Radford et al., 2021, Devlin et al., 2018].

Original: In particular, we expect a meta-learned solution like MAML to help more for smaller-size models.
Revised: In particular, we expect a meta-learned solution like MAML to help more for smaller-size models.

Original: We tested this and surprisingly the equivalence between MAML and USL seems to hold even as the model increased – strengthening our hypothesis that the low task diversity might be a bigger factor explaining our observations 2. trained on ImageNet (indicated as “pt" on table) and randomly initialized (indicated as “rand" on table).
Revised: We tested this and surprisingly the equivalence between MAML and USL seems to hold even as the model increased – strengthening our hypothesis that the low task diversity might be a bigger factor explaining our observations 2. trained on ImageNet (indicated as “pt" on table) and randomly initialized (indicated as “rand" on table).

Original: We used 95% confidence intervals.
Revised: We used 95% confidence intervals.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We would like to acknowledge the work and the authors of Anatome, Torchmeta, Learn2Learn and
Revised: We would like to acknowledge the work and the authors of Anatome, Torchmeta, Learn2Learn and

Original: Higher [Hataya, 2020, "Arnold et al., 2020, Deleu et al., 2019, Grefenstette et al., 2019] for making their code available and answering ours questions in their project’s GitHub repository.
Revised: Higher [Hataya, 2020, "Arnold et al., 2020, Deleu et al., 2019, Grefenstette et al., 2019] for making their code available and answering ours questions in their project’s GitHub repository.

Original: We’d like to especially thank Sebastien Arnold from the great Learn2Learn meta-learning PyTorch library for great code discussions and for providing a demo of how to use his library for distributed training.
Revised: We’d like to especially thank Sebastien Arnold from the great Learn2Learn meta-learning PyTorch library for great code discussions and for providing a demo of how to use his library for distributed training.

Original: We want to acknowledge the weights and biases (wandb) framework for powerful tracking of experiments [Biewald, 2020].
Revised: We want to acknowledge the weights and biases (wandb) framework for powerful tracking of experiments [Biewald, 2020].

Original: We especially acknowledge Pedro Cisneros-Velarde’s insightful feedback on earlier versions of this manuscript, which revealed important subtleties that when clarified improved the paper’s message and readability.
Revised: We especially acknowledge Pedro Cisneros-Velarde’s insightful feedback on earlier versions of this manuscript, which revealed important subtleties that when clarified improved the paper’s message and readability.

Original: We also acknowledge Shovik Guha for their feedback to improve earlier versions of this paper.
Revised: We also acknowledge Shovik Guha for their feedback to improve earlier versions of this paper.

Original: This research was funded in part by a Sloan Fellowship, NSF 2046795, 1909577, and NIFA award 2020-67021-32799.
Revised: This research was funded in part by a Sloan Fellowship, NSF 2046795, 1909577, and NIFA award 2020-67021-32799.

Original: A Synthetic Experiments showing closeness of MAML and Transfer Learning as Diversity Changes
Revised: A Synthetic Experiments showing closeness of MAML and Transfer Learning as Diversity Changes

Original: In this section, we show the closeness of MAML and transfer learning (with USL) for synthetic experiments for low and high diversity regimes in Figure 3.
Revised: In this section, we show the closeness of MAML and transfer learning (with USL) for synthetic experiments for low and high diversity regimes in Figure 3.

Original: In the low regime, the two methods are equivalent in a statistically significant way – which supports the main claims of our paper.
Revised: In the low regime, the two methods are equivalent in a statistically significant way – which supports the main claims of our paper.

Original: As the diversity increases, however, the difference between USL and MAML increases (in favor of USL).
Revised: As the diversity increases, however, the difference between USL and MAML increases (in favor of USL).

Original: This will be explored further in future work.
Revised: This will be explored further in future work.

Original: The tasks are the usual n-way, k-shot tasks, but the data comes from Gaussian classes and the meta-learners are tasked with classifying from which Gaussian class the data points came from in a few-shot learning manner.
Revised: The tasks are the usual n-way, k-shot tasks, but the data comes from Gaussian classes and the meta-learners are tasked with classifying from which Gaussian class the data points came from in a few-shot learning manner.

Original: Benchmarks are created by sampling the parameters of individual classes from a Gaussian benchmark distribution, where class means move farther away from each other and the origin as the diversity of the benchmark increases.
Revised: Benchmarks are created by sampling the parameters of individual classes from a Gaussian benchmark distribution, where class means move farther away from each other and the origin as the diversity of the benchmark increases.

Original: Therefore, the Gaussian benchmark with the highest diversity coefficient has Gaussians that are the furthest from each other and the origin.
Revised: Therefore, the Gaussian benchmark with the highest diversity coefficient has Gaussians that are the furthest from each other and the origin.

Original: We computed the task diversity coefficients using the Task2Vec method as outlined in Section 3, using a random 3-layer fully connected probe network described in Section F.
Revised: We computed the task diversity coefficients using the Task2Vec method as outlined in Section 3, using a random 3-layer fully connected probe network described in Section F.

Original: B Can we trust the computed diversity coefficient indeed reflects the diversity of a benchmark?
Revised: B Can we trust the computed diversity coefficient indeed reflects the diversity of a benchmark?

Original: In this section, we argue that of all our low diversity computations are and valid because: 1.
Revised: In this section, we argue that of all our low diversity computations are and valid because: 1.

Original: the task based diversity using Task2Vec reflects the true diversity as argued in section 3.3 2.
Revised: the task based diversity using Task2Vec reflects the true diversity as argued in section 3.3 2.

Original: our results are robust to the choice of the probe network because Task2Vec depends solely on the task, and ignores interactions with the model
Revised: our results are robust to the choice of the probe network because Task2Vec depends solely on the task, and ignores interactions with the model

Original: [Achille UCLA et al., 2019] – implying our results are valid regardless of the probe network 3.
Revised: [Achille UCLA et al., 2019] – implying our results are valid regardless of the probe network 3.

Original: despite the evidence provided by the authors of Task2Vec
Revised: despite the evidence provided by the authors of Task2Vec

Original: [Achille UCLA et al., 2019], we nevertheless tested this dependence and used 4 probe networks to compute the task based diversity coefficient 4.
Revised: [Achille UCLA et al., 2019], we nevertheless tested this dependence and used 4 probe networks to compute the task based diversity coefficient 4.

Original: the choice of using the expectation was empirically based on the bell shape of the histogram that shows the distribution of task distances (via Task2Vec) in figures 15, 16 17.
Revised: the choice of using the expectation was empirically based on the bell shape of the histogram that shows the distribution of task distances (via Task2Vec) in figures 15, 16 17.

Original: They were Gaussian shaped, so the expectation of task distances was a sufficient statistic.
Revised: They were Gaussian shaped, so the expectation of task distances was a sufficient statistic.

Original: Although there are a large number of possible tasks based on combinatorial arguments (e.g. C 645 = 7624512 ), 500 samples are enough to make strong statistical inferences about the population.
Revised: Although there are a large number of possible tasks based on combinatorial arguments (e.g. C 645 = 7624512 ), 500 samples are enough to make strong statistical inferences about the population.

Original: If we assume the distribution of the data is Gaussian, then we expect to see a single mode with an approximate bell curve.
Revised: If we assume the distribution of the data is Gaussian, then we expect to see a single mode with an approximate bell curve.

Original: If we plot the histogram of task pair distances of the 500 tasks and see this, then we can infer our Gaussian assumption is approximately correct.
Revised: If we plot the histogram of task pair distances of the 500 tasks and see this, then we can infer our Gaussian assumption is approximately correct.

Original: Given that the histograms in figure 15, 16appear Gaussian, we can infer our assumption is approximately correct.
Revised: Given that the histograms in figure 15, 16appear Gaussian, we can infer our assumption is approximately correct.

Original: This implies we can make strong statistical assumptions about the population – in particular, that we have a good estimate of the diversity coefficient using 500 samples.
Revised: This implies we can make strong statistical assumptions about the population – in particular, that we have a good estimate of the diversity coefficient using 500 samples.

Original: The heat maps for MiniImagenet and Cifar-fs were homogeneous (uniform color) as shown in figures 14,13, 12.
Revised: The heat maps for MiniImagenet and Cifar-fs were homogeneous (uniform color) as shown in figures 14,13, 12.

Original: C Discussion
Revised: C Discussion

Original: In this work, we presented a problem-centric framework when comparing transfer learning methods with meta-learning algorithms – using USL and MAML as the representatives of transfer and metalearning methods respectively.
Revised: In this work, we presented a problem-centric framework when comparing transfer learning methods with meta-learning algorithms – using USL and MAML as the representatives of transfer and metalearning methods respectively.

Original: We showed that the diversity coefficient of the popular MiniImagenet and Cifar-fs benchmark is low and that under a fair comparison – MAML is very similar to transfer learning (with USL) at test time.
Revised: We showed that the diversity coefficient of the popular MiniImagenet and Cifar-fs benchmark is low and that under a fair comparison – MAML is very similar to transfer learning (with USL) at test time.

Original: This was also true even when changing the model size – removing the alternative hypothesis that the equivalence of MAML and transfer learning with USL held due to large models.
Revised: This was also true even when changing the model size – removing the alternative hypothesis that the equivalence of MAML and transfer learning with USL held due to large models.

Original: Instead, this strengthens our hypothesis that the diversity of the data might be the driving factor.
Revised: Instead, this strengthens our hypothesis that the diversity of the data might be the driving factor.

Original: The equivalence of MAML and USL was also replicated in synthetic experiments.
Revised: The equivalence of MAML and USL was also replicated in synthetic experiments.

Original: Therefore, we challenge the suggestions from previous work Tian et al.
Revised: Therefore, we challenge the suggestions from previous work Tian et al.

Original: [2020] that only a good embedding can beat more effective than sophisticated meta-learning – especially in the low diversity regime, and instead suggest this observation might be due to lack of good principles to design metalearning benchmarks.
Revised: [2020] that only a good embedding can beat more effective than sophisticated meta-learning – especially in the low diversity regime, and instead suggest this observation might be due to lack of good principles to design metalearning benchmarks.

Original: In addition, our synthetic experiments show a promising scenario where we can systematically differentiate meta-learning algorithms from transfer learning algorithms – which supports our actionable suggestion to use the diversity coefficient to effectively study meta-learning and transfer learning algorithms.
Revised: In addition, our synthetic experiments show a promising scenario where we can systematically differentiate meta-learning algorithms from transfer learning algorithms – which supports our actionable suggestion to use the diversity coefficient to effectively study meta-learning and transfer learning algorithms.

Original: In addition, this problematizes the observations that fo-MAML in meta-data set [Triantafillou et al., 2019] is better than transfer learning solutions – since our synthetic experiments show MAML is not better than USL in the high diversity regime.
Revised: In addition, this problematizes the observations that fo-MAML in meta-data set [Triantafillou et al., 2019] is better than transfer learning solutions – since our synthetic experiments show MAML is not better than USL in the high diversity regime.

Original: To further problematize, we want to point out that meta-learning methods are not better than transfer learning as observed by [Guo et al., 2019] – as observed in our synthetic experiments.
Revised: To further problematize, we want to point out that meta-learning methods are not better than transfer learning as observed by [Guo et al., 2019] – as observed in our synthetic experiments.

Original: Meaning that further research is needed in both data sets – especially from a problem-centric perspective with quantitative methods like the ones we suggest.
Revised: Meaning that further research is needed in both data sets – especially from a problem-centric perspective with quantitative methods like the ones we suggest.

Original: We also have theoretical results from a statistical decision perspective in the supplementary section Q that inspired this work and suggest that when the distance between tasks is zero – then the predictions of transfer learning, meta-learning, and even a fixed model with no adaptation are all equivalent (with respect to the l2 loss).
Revised: We also have theoretical results from a statistical decision perspective in the supplementary section Q that inspired this work and suggest that when the distance between tasks is zero – then the predictions of transfer learning, meta-learning, and even a fixed model with no adaptation are all equivalent (with respect to the l2 loss).

Original: We hope this work inspires the community in meta-learning and machine learning to construct benchmarks from a problem-centric perspective – that go beyond only large scale data sets – and instead use quantitative metrics for the construction of such research challenges.
Revised: We hope this work inspires the community in meta-learning and machine learning to construct benchmarks from a problem-centric perspective – that go beyond only large scale data sets – and instead use quantitative metrics for the construction of such research challenges.

Original: We’d like to emphasize that our synthetic experiments are promising because we can systematically differentiate meta-learning algorithms from transfer learning algorithms – which supports our actionable suggestion to: 1.
Revised: We’d like to emphasize that our synthetic experiments are promising because we can systematically differentiate meta-learning algorithms from transfer learning algorithms – which supports our actionable suggestion to: 1.

Original: use the diversity coefficient to effectively study meta-learning and transfer learning algorithms, and 2.
Revised: use the diversity coefficient to effectively study meta-learning and transfer learning algorithms, and 2.

Original: to use the diversity coefficient to design better benchmarks.
Revised: to use the diversity coefficient to design better benchmarks.

Original: In addition, this problematizes the observations that fo-proto-MAML in meta-data set [Triantafillou et al., 2019] is better than transfer learning solutions – since our synthetic experiments show MAML is not better than USL in the high diversity regime.
Revised: In addition, this problematizes the observations that fo-proto-MAML in meta-data set [Triantafillou et al., 2019] is better than transfer learning solutions – since our synthetic experiments show MAML is not better than USL in the high diversity regime.

Original: To further problematize, we want to point out that metalearning methods are not better than transfer learning as observed by [Guo et al., 2019] – as observed in our synthetic experiments.
Revised: To further problematize, we want to point out that metalearning methods are not better than transfer learning as observed by [Guo et al., 2019] – as observed in our synthetic experiments.

Original: We hypothesize however that the two scenarios in [Triantafillou et al., 2019] are different [Guo et al., 2019].
Revised: We hypothesize however that the two scenarios in [Triantafillou et al., 2019] are different [Guo et al., 2019].

Original: The first one focuses on the same meta-training and metatesting conditions, while the latter focuses on a cross-domain.
Revised: The first one focuses on the same meta-training and metatesting conditions, while the latter focuses on a cross-domain.

Original: We hypothesize that the cross-domain scenario might benefit from a meta-learning which lower variance (e.g., a fixed embedding [Tian et al., 2020]) – which might explain why sophisticated meta-learning solutions might perform worse on the cross-domain setting as observed in [Guo et al., 2019].
Revised: We hypothesize that the cross-domain scenario might benefit from a meta-learning which lower variance (e.g., a fixed embedding [Tian et al., 2020]) – which might explain why sophisticated meta-learning solutions might perform worse on the cross-domain setting as observed in [Guo et al., 2019].

Original: Further research is needed in both benchmarks – especially from a problem centric perspective with quantitative methods like the ones we suggest.
Revised: Further research is needed in both benchmarks – especially from a problem centric perspective with quantitative methods like the ones we suggest.

Original: In addition, we hypothesize that diversity might be a good proxy to predict the difference between meta-learning and transfer learning methods.
Revised: In addition, we hypothesize that diversity might be a good proxy to predict the difference between meta-learning and transfer learning methods.

Original: More precisely, we conjecture that in a low diversity setting meta-learning methods are equivalent at meta-test time to transfer learning methods but their difference increases as the diversity of tasks in a benchmark increases.
Revised: More precisely, we conjecture that in a low diversity setting meta-learning methods are equivalent at meta-test time to transfer learning methods but their difference increases as the diversity of tasks in a benchmark increases.

Original: In the high diversity regime, we conjecture that the difference between meta-learning and transfer learning methods increases as the diversity increases.
Revised: In the high diversity regime, we conjecture that the difference between meta-learning and transfer learning methods increases as the diversity increases.

Original: We are optimistic that meta-learning algorithms might outperform transfer learning methods, once we start comparing them in more thoughtfully designed benchmarks.
Revised: We are optimistic that meta-learning algorithms might outperform transfer learning methods, once we start comparing them in more thoughtfully designed benchmarks.

Original: It is possible that despite our efforts, meta-learning algorithms – as currently designed – are too sophisticated and in fact lead to meta-overfitting, as shown in previous work [Miranda et al., 2021].
Revised: It is possible that despite our efforts, meta-learning algorithms – as currently designed – are too sophisticated and in fact lead to meta-overfitting, as shown in previous work [Miranda et al., 2021].

Original: We’d like to emphasize, that up until now, the meta-learning community has evaluated meta-learning algorithms in benchmarks that might not be the most appropriate.
Revised: We’d like to emphasize, that up until now, the meta-learning community has evaluated meta-learning algorithms in benchmarks that might not be the most appropriate.

Original: We conjecture high diversity benchmarks are more appropriate, since they might capture the meta-learning inductive prior: high diversity means that adaptation is required by construction.
Revised: We conjecture high diversity benchmarks are more appropriate, since they might capture the meta-learning inductive prior: high diversity means that adaptation is required by construction.

Original: Thus, we conjecture that previous conclusions should be taken with a grain of salt until a more in depth study can be made in the high diversity regime – especially with benchmarks with real world data that have been analyzed extensively with metrics like the diversity coefficient that we propose.
Revised: Thus, we conjecture that previous conclusions should be taken with a grain of salt until a more in depth study can be made in the high diversity regime – especially with benchmarks with real world data that have been analyzed extensively with metrics like the diversity coefficient that we propose.

Original: We conjecture that we can finally do meta-learning research effectively – given that a regime where meta-learning and transfer learning methods can be differentiated has been discovered, and previous low diversity benchmarks have been understood.
Revised: We conjecture that we can finally do meta-learning research effectively – given that a regime where meta-learning and transfer learning methods can be differentiated has been discovered, and previous low diversity benchmarks have been understood.

Original: We also conjecture that meta-learning research is different from classical machine learning research.
Revised: We also conjecture that meta-learning research is different from classical machine learning research.

Original: Historically, a seminal paper is the one where AlexNet was proposed [Krizhevsky et al., 2012].
Revised: Historically, a seminal paper is the one where AlexNet was proposed [Krizhevsky et al., 2012].

Original: In that time we had low performance on a fixed task e.g., Imagenet and couldn’t even interpolate the data (i.e., reach zero train error).
Revised: In that time we had low performance on a fixed task e.g., Imagenet and couldn’t even interpolate the data (i.e., reach zero train error).

Original: We conjecture that meta-learning is different because if we have a diversity so large where all possible tasks are incorporated, then we should reach the no-free lunch theorem regime
Revised: We conjecture that meta-learning is different because if we have a diversity so large where all possible tasks are incorporated, then we should reach the no-free lunch theorem regime

Original: [Wolpert and Macready, 1997] – where all algorithms should perform the same on average.
Revised: [Wolpert and Macready, 1997] – where all algorithms should perform the same on average.

Original: Therefore, we hypothesize that a deliberate and quantitative efforts to design benchmarks is essential.
Revised: Therefore, we hypothesize that a deliberate and quantitative efforts to design benchmarks is essential.

Original: A great example of such an attempt is the Abstraction and Reasoning Corpus (ARC) benchmark
Revised: A great example of such an attempt is the Abstraction and Reasoning Corpus (ARC) benchmark

Original: [Chollet, 2019] – which was made very thoughtfully with Artificial General Intelligence (AGI) in mind.
Revised: [Chollet, 2019] – which was made very thoughtfully with Artificial General Intelligence (AGI) in mind.

Original: We conjecture meta-learning is the most promising path in that direction, and hope this work inspires the design of benchmarks that lead to actionable and deliberate attempts to make progress to build such AGI technologies.
Revised: We conjecture meta-learning is the most promising path in that direction, and hope this work inspires the design of benchmarks that lead to actionable and deliberate attempts to make progress to build such AGI technologies.

Original: D Related Work
Revised: D Related Work

Original: Our work proposes a problem-centric framework for the analysis of meta-learning algorithms inspired from previous puzzling results [Tian et al., 2020].
Revised: Our work proposes a problem-centric framework for the analysis of meta-learning algorithms inspired from previous puzzling results [Tian et al., 2020].

Original: We propose the use of a pair-wise distance between tasks and analyze how this metric might correlate with meta-learning.
Revised: We propose the use of a pair-wise distance between tasks and analyze how this metric might correlate with meta-learning.

Original: The closest line of work for this is the long line of work by [Achille UCLA et al., 2019] where they suggest methods to analyze the complexity of a task, propose unsymmetrical distance metrics for data sets, reachability of tasks with SGD, ways to embed entire data sets and more [Achille UCLA et al., 2019, Achille et al., 2018, 2019, 2020].
Revised: The closest line of work for this is the long line of work by [Achille UCLA et al., 2019] where they suggest methods to analyze the complexity of a task, propose unsymmetrical distance metrics for data sets, reachability of tasks with SGD, ways to embed entire data sets and more [Achille UCLA et al., 2019, Achille et al., 2018, 2019, 2020].

Original: We hypothesize this line of work to be very fruitful and hope that more people adopt tools like the ones they suggest and the tools we propose in this paper before researching or deploying meta-learning algorithms.
Revised: We hypothesize this line of work to be very fruitful and hope that more people adopt tools like the ones they suggest and the tools we propose in this paper before researching or deploying meta-learning algorithms.

Original: We hope this helps meta-learning methods succeed in practice – since cognitive science suggests meta-learning is a powerful method humans use to learn [Lake et al., 2016].
Revised: We hope this helps meta-learning methods succeed in practice – since cognitive science suggests meta-learning is a powerful method humans use to learn [Lake et al., 2016].

Original: In the future, we hope to compare [Achille UCLA et al., 2019]’s distance metrics between tasks with ours to provide a further unified understanding of meta-learning and transfer learning.
Revised: In the future, we hope to compare [Achille UCLA et al., 2019]’s distance metrics between tasks with ours to provide a further unified understanding of meta-learning and transfer learning.

Original: A contrast between their work and ours is that we focus our analysis from a meta-learning perspective applied to few-shot learning – while their focus is understanding transfer learning methods between data sets.
Revised: A contrast between their work and ours is that we focus our analysis from a meta-learning perspective applied to few-shot learning – while their focus is understanding transfer learning methods between data sets.

Original: Our analysis of the feature extractor layer is identical to the analysis by [Raghu et al., 2020].
Revised: Our analysis of the feature extractor layer is identical to the analysis by [Raghu et al., 2020].

Original: They showed that MAML functions mainly via feature re-use than by rapid learning, i.e., that a model trained with MAML changes very little after the MAML adaptation.
Revised: They showed that MAML functions mainly via feature re-use than by rapid learning, i.e., that a model trained with MAML changes very little after the MAML adaptation.

Original: The main difference of their work with our is: 1) that we compare MAML trained models against union supervised learned models (USL) instead of only comparing MAML against adapted MAML, and 2) that we explicitly analyzed properties of the data sets.
Revised: The main difference of their work with our is: 1) that we compare MAML trained models against union supervised learned models (USL) instead of only comparing MAML against adapted MAML, and 2) that we explicitly analyzed properties of the data sets.

Original: In addition, we use a large set of distance metrics for our analysis including: SVCCA, PWCCA, LINCKA and OPD as proposed by [Raghu et al., 2017, Morcos et al., 2018, Kornblith et al., 2019, Ding et al., 2021].
Revised: In addition, we use a large set of distance metrics for our analysis including: SVCCA, PWCCA, LINCKA and OPD as proposed by [Raghu et al., 2017, Morcos et al., 2018, Kornblith et al., 2019, Ding et al., 2021].

Original: Our work is most influenced by previous work suggesting modern meta-learning requires rethinking [Tian et al., 2020].
Revised: Our work is most influenced by previous work suggesting modern meta-learning requires rethinking [Tian et al., 2020].

Original: The main difference of our work with theirs is that we analyzed the internal representation of the meta-learning algorithms and contextualize these with quantifiable metrics of the problem being solved.
Revised: The main difference of our work with theirs is that we analyzed the internal representation of the meta-learning algorithms and contextualize these with quantifiable metrics of the problem being solved.

Original: Unlike their work, we focused on a fair comparison between meta-learning methods by ensuring the same neural network backbone was used.
Revised: Unlike their work, we focused on a fair comparison between meta-learning methods by ensuring the same neural network backbone was used.

Original: Another difference is that they gained further accuracy gains by using distillation – a method we did not analyze and leave for future work.
Revised: Another difference is that they gained further accuracy gains by using distillation – a method we did not analyze and leave for future work.

Original: Another related line of work is the predictability of adversarial transferability and transfer learning.
Revised: Another related line of work is the predictability of adversarial transferability and transfer learning.

Original: They show this both theoretically and with extensive experiments [Liang et al., 2021].
Revised: They show this both theoretically and with extensive experiments [Liang et al., 2021].

Original: The main difference between their work and ours is that they focus their analysis mainly on transfer learning, while we concentrated on meta-learning for few-shot learning.
Revised: The main difference between their work and ours is that they focus their analysis mainly on transfer learning, while we concentrated on meta-learning for few-shot learning.

Original: In addition, we did not consider adversarial transferability – while that was a central piece of their analysis.
Revised: In addition, we did not consider adversarial transferability – while that was a central piece of their analysis.

Original: In terms of benchmarks, we’d like to start with the ARC benchmark [Chollet, 2019].
Revised: In terms of benchmarks, we’d like to start with the ARC benchmark [Chollet, 2019].

Original: ARC was designed with AGI in mind – arguably the ultimate meta-learner.
Revised: ARC was designed with AGI in mind – arguably the ultimate meta-learner.

Original: Its focus is primarily on visual reasoning using program synthesis techniques.
Revised: Its focus is primarily on visual reasoning using program synthesis techniques.

Original: We hypothesize that it’s a very promising path, but our work inspires extension that go beyond program synthesis approaches.
Revised: We hypothesize that it’s a very promising path, but our work inspires extension that go beyond program synthesis approaches.

Original: The meta-data set benchmark is an attempt to make the data set for few-shot learning at a larger scale and more diverse [Triantafillou et al., 2019].
Revised: The meta-data set benchmark is an attempt to make the data set for few-shot learning at a larger scale and more diverse [Triantafillou et al., 2019].

Original: The main difference of their work and ours is that we propose a quantitative metric to measure the intrinsic diversity in the data and go beyond data set size or number of classes.
Revised: The main difference of their work and ours is that we propose a quantitative metric to measure the intrinsic diversity in the data and go beyond data set size or number of classes.

Original: They also showed that a meta-learning algorithms – fo-Proto-MAML – is capable of beating transfer learning.
Revised: They also showed that a meta-learning algorithms – fo-Proto-MAML – is capable of beating transfer learning.

Original: However, they also showed transfer learning baselines are in fact quite difficult to beat.
Revised: However, they also showed transfer learning baselines are in fact quite difficult to beat.

Original: The
Revised: The

Original: IBM Cross-Domain few-shot learning benchmark [Guo et al., 2019] is a fascinating benchmark to evaluate meta-learning algorithms.
Revised: IBM Cross-Domain few-shot learning benchmark [Guo et al., 2019] is a fascinating benchmark to evaluate meta-learning algorithms.

Original: Their central premise however is to transfer from a source domain to a different target domain – instead of our setting where tasks are created from the same metadistribution.
Revised: Their central premise however is to transfer from a source domain to a different target domain – instead of our setting where tasks are created from the same metadistribution.

Original: This is why their paper is considered, in addition to few-shot learning, a cross-domain benchmark .
Revised: This is why their paper is considered, in addition to few-shot learning, a cross-domain benchmark .

Original: We believe this is an essential scenario to think about, but consider it different from our setting or the setting of meta-data set.
Revised: We believe this is an essential scenario to think about, but consider it different from our setting or the setting of meta-data set.

Original: We’d also like to emphasize that they do not employ a metric like our diversity coefficient that quantitatively assesses the diversity of their benchmarks.
Revised: We’d also like to emphasize that they do not employ a metric like our diversity coefficient that quantitatively assesses the diversity of their benchmarks.

Original: These two last benchmarks, although fascinating, are missing the essential quantitative analysis of the data itself we are trying to propose.
Revised: These two last benchmarks, although fascinating, are missing the essential quantitative analysis of the data itself we are trying to propose.

Original: The work by [Chen et al., 2021] gives, to the best of our knowledge – the first non-vacuous generalization bounds for the (supervised) meta-learning setting.
Revised: The work by [Chen et al., 2021] gives, to the best of our knowledge – the first non-vacuous generalization bounds for the (supervised) meta-learning setting.

Original: Their statements apply to a non-convex loss function and use stability theory at the task level.
Revised: Their statements apply to a non-convex loss function and use stability theory at the task level.

Original: The bound depends on the mutual information on the input data vs the output data of the meta-learner.
Revised: The bound depends on the mutual information on the input data vs the output data of the meta-learner.

Original: The results, although fascinating, are not built to separate classes of meta-learning – like our work attempts to do empirically.
Revised: The results, although fascinating, are not built to separate classes of meta-learning – like our work attempts to do empirically.

Original: The work by [Wang et al., 2021] proposes the idea of global labels as a way to indirectly optimize for the meta—learning objective for a fixed feature extractor.
Revised: The work by [Wang et al., 2021] proposes the idea of global labels as a way to indirectly optimize for the meta—learning objective for a fixed feature extractor.

Original: Global labels are equivalent to the concept we call USL in this paper.
Revised: Global labels are equivalent to the concept we call USL in this paper.

Original: They show that pre-training (i.e. using USL/global labels) provides excellent meta-test results – including with their method (named MeLa) that can infer global labels given only local labels provided at in episodic meta-training.
Revised: They show that pre-training (i.e. using USL/global labels) provides excellent meta-test results – including with their method (named MeLa) that can infer global labels given only local labels provided at in episodic meta-training.

Original: Their theoretical analysis depends on a fixed feature extractor, instead of considering the whole end-to-end meta-learner as a whole – meaning two different deep learning models cannot be used in their analysis.
Revised: Their theoretical analysis depends on a fixed feature extractor, instead of considering the whole end-to-end meta-learner as a whole – meaning two different deep learning models cannot be used in their analysis.

Original: Therefore, their analysis fails to separate how different feature extractors might be trained, e.g. comparing USL vs MAML directly in an end-to-end fashion.
Revised: Therefore, their analysis fails to separate how different feature extractors might be trained, e.g. comparing USL vs MAML directly in an end-to-end fashion.

Original: In contrast, we instead tackle this question head on theoretically Q (with limited results) but instead show that the feature extractors indeed are different empirically G.
Revised: In contrast, we instead tackle this question head on theoretically Q (with limited results) but instead show that the feature extractors indeed are different empirically G.

Original: The work by [Denevi et al., 2020] proposes a theoretical treatment of meta-learning using metalearners with closed-form equations derived from ridged regularization using fixed features.
Revised: The work by [Denevi et al., 2020] proposes a theoretical treatment of meta-learning using metalearners with closed-form equations derived from ridged regularization using fixed features.

Original: They formulate the conditional and unconditional formulation using side information for the task (e.g. the support set) and show the conditional method is superior.
Revised: They formulate the conditional and unconditional formulation using side information for the task (e.g. the support set) and show the conditional method is superior.

Original: In relation to our work, they do not provide characterizations of the role of a neural network doing end-to-end meta-learning (in their empirical or theoretical analysis).
Revised: In relation to our work, they do not provide characterizations of the role of a neural network doing end-to-end meta-learning (in their empirical or theoretical analysis).

Original: In contrast, our findings make an explicit effort in understanding the role of the neural network in meta-learning in an end-to-end fashion through emperical analysis.
Revised: In contrast, our findings make an explicit effort in understanding the role of the neural network in meta-learning in an end-to-end fashion through emperical analysis.

Original: Another contrast is that their results are highly theoretical, while ours focus on empirical results.
Revised: Another contrast is that their results are highly theoretical, while ours focus on empirical results.

Original: In addition, their results are on synthetic experiments and do not explore their findings in the context of modern few-shot learning benchmarks like MiniImagenet or Cifar-fs.
Revised: In addition, their results are on synthetic experiments and do not explore their findings in the context of modern few-shot learning benchmarks like MiniImagenet or Cifar-fs.

Original: The work by [Goldblum et al., 2020] provide strong evidence that adaptation at test time is best done when the meta-trained model matches the adaptation it was meta-trained with.
Revised: The work by [Goldblum et al., 2020] provide strong evidence that adaptation at test time is best done when the meta-trained model matches the adaptation it was meta-trained with.

Original: This is shown because their classically pre-trained nets cannot perform better than the MetaOpt models with any fine-tuning method.
Revised: This is shown because their classically pre-trained nets cannot perform better than the MetaOpt models with any fine-tuning method.

Original: However, their results cannot beat [Tian et al., 2020] and thus does not help separate the role of meta-training and union supervised learning (USL).
Revised: However, their results cannot beat [Tian et al., 2020] and thus does not help separate the role of meta-training and union supervised learning (USL).

Original: Their Resnet12 results do provide further support to our hypothesis that large enough neural networks all perform the same, since 78.63 (Goldblum) vs 79.74 (RFS) have very close errors, in line with our findings.
Revised: Their Resnet12 results do provide further support to our hypothesis that large enough neural networks all perform the same, since 78.63 (Goldblum) vs 79.74 (RFS) have very close errors, in line with our findings.

Original: However, we hypothesize it is not due to the model size in accordance with our experiments 2.
Revised: However, we hypothesize it is not due to the model size in accordance with our experiments 2.

Original: The work by [Gao and Sener, 2020] provides theoretical bounds of when the expected risk of MAML and DRS (Domain Randomized Search) by bounding the gradient norm.
Revised: The work by [Gao and Sener, 2020] provides theoretical bounds of when the expected risk of MAML and DRS (Domain Randomized Search) by bounding the gradient norm.

Original: DRS attempts to model
Revised: DRS attempts to model

Original: USL but fails to do so completely, because USL is capable of modeling adaptation because the final layer is capable of adaption.
Revised: USL but fails to do so completely, because USL is capable of modeling adaptation because the final layer is capable of adaption.

Original: Thus, it does not address the capabilities of the feature extractor being able to learn all the information needed to meta-learn.
Revised: Thus, it does not address the capabilities of the feature extractor being able to learn all the information needed to meta-learn.

Original: Concisely, their analysis is not capable of separating performance of MAML and USL.
Revised: Concisely, their analysis is not capable of separating performance of MAML and USL.

Original: Even if it hypothetically could, their analysis remains an upper bounds (with assumptions).
Revised: Even if it hypothetically could, their analysis remains an upper bounds (with assumptions).

Original: This raising the question if their method truly explain the observations that transfer learning methods – like USL – beat meta-learning methods.
Revised: This raising the question if their method truly explain the observations that transfer learning methods – like USL – beat meta-learning methods.

Original: In addition, they do not provide in depth empirical analysis with respect to any real few-shot learning benchmarks like MiniImagenet or Cifar-fs.
Revised: In addition, they do not provide in depth empirical analysis with respect to any real few-shot learning benchmarks like MiniImagenet or Cifar-fs.

Original: The work by [Kumar et al., 2022] provides an exploration of the effects of diversity in meta-learning.
Revised: The work by [Kumar et al., 2022] provides an exploration of the effects of diversity in meta-learning.

Original: The main difference with our work is that they focus mostly on sampling strategies, and it’s effect on diversity, while we focused on the intrinsic diversity in the benchmarks themselves.
Revised: The main difference with our work is that they focus mostly on sampling strategies, and it’s effect on diversity, while we focused on the intrinsic diversity in the benchmarks themselves.

Original: The work by [Rosenfeld et al., 2021] provides a theoretical analysis on the difference between interpolation and extrapolation in transfer learning (and domain generalization).
Revised: The work by [Rosenfeld et al., 2021] provides a theoretical analysis on the difference between interpolation and extrapolation in transfer learning (and domain generalization).

Original: We believe this type of theory may be helpful as an inspiration to explore why in the high diversity regime there seems to be a difference between the performance of meta-learning and transfer learning methods.
Revised: We believe this type of theory may be helpful as an inspiration to explore why in the high diversity regime there seems to be a difference between the performance of meta-learning and transfer learning methods.

Original: A related line of work [Miranda, 2020b,a] first showed that there exist synthetic data sets that are capable of exhibiting higher degrees of adaptation as compared to the original work by [Raghu et al., 2020].
Revised: A related line of work [Miranda, 2020b,a] first showed that there exist synthetic data sets that are capable of exhibiting higher degrees of adaptation as compared to the original work by [Raghu et al., 2020].

Original: The difference is that they did not compare MAML models against transfer learning methods like we did here.
Revised: The difference is that they did not compare MAML models against transfer learning methods like we did here.

Original: Instead, they focused on comparing adapted MAML models vs. unadapted MAML models.
Revised: Instead, they focused on comparing adapted MAML models vs. unadapted MAML models.

Original: E Convergence of Learning Curves for Fair Comparison E.1 Convergence of Learning Curves for MiniImagenet and Cifar-fs
Revised: E Convergence of Learning Curves for Fair Comparison E.1 Convergence of Learning Curves for MiniImagenet and Cifar-fs

Original: In this section, we show that learning curves for the models used in figure 1 achieve convergence.
Revised: In this section, we show that learning curves for the models used in figure 1 achieve convergence.

Original: Note that the learning curves for the models trained with MAML look noisier because the distributed training reduces the (meta) batch size for logging purposes.
Revised: Note that the learning curves for the models trained with MAML look noisier because the distributed training reduces the (meta) batch size for logging purposes.

Original: In addition, due to episodic meta-training, (meta) batch sizes have to be smaller compared to batch sizes used in USL.
Revised: In addition, due to episodic meta-training, (meta) batch sizes have to be smaller compared to batch sizes used in USL.

Original: on MiniImagenet.
Revised: on MiniImagenet.

Original: F Experimental Details F.1 Experimental Details on MiniImagenet and Cifar-fs
Revised: F Experimental Details F.1 Experimental Details on MiniImagenet and Cifar-fs

Original: We will explain the details for the four models we trained on figure 1.
Revised: We will explain the details for the four models we trained on figure 1.

Original: Summary: We trained a five layer CNN (5CNN) and Resnet12 on both MiniImagenet and Cifarfs to convergence.
Revised: Summary: We trained a five layer CNN (5CNN) and Resnet12 on both MiniImagenet and Cifarfs to convergence.

Original: We used the Adam optimizer with learning rate 1e-3.
Revised: We used the Adam optimizer with learning rate 1e-3.

Original: We used the standard
Revised: We used the standard

Original: MiniImagenet and Cifar-fs data augmentations as provided in ["Arnold et al., 2020] matching [Tian et al., 2020].
Revised: MiniImagenet and Cifar-fs data augmentations as provided in ["Arnold et al., 2020] matching [Tian et al., 2020].

Original: Experimental Details for 5CNN on MiniImagenet: We used the five layer CNN from [Finn et al., 2017, Ravi and Larochelle, 2017].
Revised: Experimental Details for 5CNN on MiniImagenet: We used the five layer CNN from [Finn et al., 2017, Ravi and Larochelle, 2017].

Original: We used 32 filters as used in previous work.
Revised: We used 32 filters as used in previous work.

Original: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL.
Revised: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL.

Original: We used no scheduler.
Revised: We used no scheduler.

Original: We trained the
Revised: We trained the

Original: USL model for 1000 epochs.
Revised: USL model for 1000 epochs.

Original: We trained the MAML model for 100,000 episodic iterations (outer loop iterations).
Revised: We trained the MAML model for 100,000 episodic iterations (outer loop iterations).

Original: We used a batch size of 128 for USL and a (meta) batch size of 8 for MAML.
Revised: We used a batch size of 128 for USL and a (meta) batch size of 8 for MAML.

Original: For
Revised: For

Original: MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.
Revised: MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.

Original: We did not use first order
Revised: We did not use first order

Original: MAML.
Revised: MAML.

Original: It took 3 hours 5 minutes 5 seconds to train USL to convergence with a single GPU.
Revised: It took 3 hours 5 minutes 5 seconds to train USL to convergence with a single GPU.

Original: It took 1 day 6 hours 21 minutes 8 seconds to train MAML to convergence with 4 NVIDIA GeForce GTX
Revised: It took 1 day 6 hours 21 minutes 8 seconds to train MAML to convergence with 4 NVIDIA GeForce GTX

Original: TITAN X GPUs.
Revised: TITAN X GPUs.

Original: Experimental Details for Resnet12 for MiniImagenet:
Revised: Experimental Details for Resnet12 for MiniImagenet:

Original: We used the Resnet12 provided by [Tian et al., 2020].
Revised: We used the Resnet12 provided by [Tian et al., 2020].

Original: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL.
Revised: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL.

Original: We used the same cosine scheduler as in [Tian et al., 2020] for USL and no cosine scheduler for MAML.
Revised: We used the same cosine scheduler as in [Tian et al., 2020] for USL and no cosine scheduler for MAML.

Original: We trained the USL model for 186 epochs.
Revised: We trained the USL model for 186 epochs.

Original: We trained the MAML model for 37,800 episodic iterations (outer loop iterations).
Revised: We trained the MAML model for 37,800 episodic iterations (outer loop iterations).

Original: We used a batch size of 512 for USL and a (meta) batch size of 4 for MAML.
Revised: We used a batch size of 512 for USL and a (meta) batch size of 4 for MAML.

Original: For MAML we used an inner learning rate of 1e-1 and 4 inner learning steps.
Revised: For MAML we used an inner learning rate of 1e-1 and 4 inner learning steps.

Original: We did not use first order MAML.
Revised: We did not use first order MAML.

Original: It took 1 day 17 hours 2 minutes 41 seconds to train USL to convergence with a single dgx A100-SXM4-40GB GPU.
Revised: It took 1 day 17 hours 2 minutes 41 seconds to train USL to convergence with a single dgx A100-SXM4-40GB GPU.

Original: The MAML model was trained with Torchmeta [Deleu et al., 2019] which didn’t support multi gpu training when we ran this experiment, so we estimate it took 1weeks to train on a single GPU.
Revised: The MAML model was trained with Torchmeta [Deleu et al., 2019] which didn’t support multi gpu training when we ran this experiment, so we estimate it took 1weeks to train on a single GPU.

Original: In addition, it was ran with an earlier version of our code, so we unfortunately did not record the type of GPU but suspect it was either an A100, A40 or Quadro RTX 6000.
Revised: In addition, it was ran with an earlier version of our code, so we unfortunately did not record the type of GPU but suspect it was either an A100, A40 or Quadro RTX 6000.

Original: Experimental Details for 5CNN for Cifar-fs:
Revised: Experimental Details for 5CNN for Cifar-fs:

Original: We used the five layer CNN from [Finn et al., 2017,
Revised: We used the five layer CNN from [Finn et al., 2017,

Original: Ravi and Larochelle, 2017] provided by ["Arnold et al., 2020].
Revised: Ravi and Larochelle, 2017] provided by ["Arnold et al., 2020].

Original: But we used 1024 filters instead of(to speed up convergence).
Revised: But we used 1024 filters instead of(to speed up convergence).

Original: We used the Adam optimizer with learning rate 1e-3 for both MAML and
Revised: We used the Adam optimizer with learning rate 1e-3 for both MAML and

Original: USL.
Revised: USL.

Original: We used the same cosine scheduler as in [Tian et al., 2020] for MAML and no cosine scheduler for USL.
Revised: We used the same cosine scheduler as in [Tian et al., 2020] for MAML and no cosine scheduler for USL.

Original: We trained the USL model for 1000 epochs.
Revised: We trained the USL model for 1000 epochs.

Original: We trained the MAML model for 100, episodic iterations (outer loop iterations).
Revised: We trained the MAML model for 100, episodic iterations (outer loop iterations).

Original: We used a batch size of 256 for USL and a (meta) batch size of 8 for MAML.
Revised: We used a batch size of 256 for USL and a (meta) batch size of 8 for MAML.

Original: For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.
Revised: For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.

Original: We did not use first order MAML.
Revised: We did not use first order MAML.

Original: It took 10 hours 43 minutes 31 seconds to train USL to convergence with a single GPU dgx A100-SXM4-40GB.
Revised: It took 10 hours 43 minutes 31 seconds to train USL to convergence with a single GPU dgx A100-SXM4-40GB.

Original: It took 2 days 9 hours 26 minutes 27 seconds to train
Revised: It took 2 days 9 hours 26 minutes 27 seconds to train

Original: MAML to convergence with 4 Quadro RTX 6000 GPUs.
Revised: MAML to convergence with 4 Quadro RTX 6000 GPUs.

Original: Experimental Details for Resnet12 for Cifar-fs:
Revised: Experimental Details for Resnet12 for Cifar-fs:

Original: We used the Resnet12 provided by [Tian et al., 2020].
Revised: We used the Resnet12 provided by [Tian et al., 2020].

Original: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL.
Revised: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL.

Original: We used the cosine scheduler used in [Tian et al., 2020] for both USL and MAML.
Revised: We used the cosine scheduler used in [Tian et al., 2020] for both USL and MAML.

Original: We trained the USL model for 200 epochs.
Revised: We trained the USL model for 200 epochs.

Original: We trained the MAML model for 75,500 episodic iterations (outer loop iterations).
Revised: We trained the MAML model for 75,500 episodic iterations (outer loop iterations).

Original: We used a batch size of 1024 for USL and a (meta) batch size of 8 for MAML.
Revised: We used a batch size of 1024 for USL and a (meta) batch size of 8 for MAML.

Original: For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.
Revised: For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.

Original: We did not use first order MAML.
Revised: We did not use first order MAML.

Original: It took 45 minutes 54 seconds to train USL to convergence with a single GPU.
Revised: It took 45 minutes 54 seconds to train USL to convergence with a single GPU.

Original: It took 1 day 19 hoursminutes 31 seconds to train MAML to convergence with 4 dgx A100-SXM4-40GB GPUs.
Revised: It took 1 day 19 hoursminutes 31 seconds to train MAML to convergence with 4 dgx A100-SXM4-40GB GPUs.

Original: Why the Adam optimizer?
Revised: Why the Adam optimizer?

Original: We hypothesize that the Adam optimizer is the most appropriate optimizer for a fair comparison for various reasons.
Revised: We hypothesize that the Adam optimizer is the most appropriate optimizer for a fair comparison for various reasons.

Original: First, the Adam optimizer is widely used – making our results most relevant and broadly applicable.
Revised: First, the Adam optimizer is widely used – making our results most relevant and broadly applicable.

Original: Adam is generally a stable optimizer – especially for sophisticated meta-learning algorithms like MAML.
Revised: Adam is generally a stable optimizer – especially for sophisticated meta-learning algorithms like MAML.

Original: It is not uncommon to have SGD result in exploding gradients or end up diverging – especially for MAML.
Revised: It is not uncommon to have SGD result in exploding gradients or end up diverging – especially for MAML.

Original: Most importantly however, we hope to stay faithful whenever possible to how modern transformer models are trained – because they have been shown to be good meta-learners, e.g., GPT-3 is often cited as a zero-shot learner [Brown et al., 2020].
Revised: Most importantly however, we hope to stay faithful whenever possible to how modern transformer models are trained – because they have been shown to be good meta-learners, e.g., GPT-3 is often cited as a zero-shot learner [Brown et al., 2020].

Original: These type of models do use more complicated learning schemes besides only
Revised: These type of models do use more complicated learning schemes besides only

Original: Adam (e.g., warm-ups, decay rates etc.)
Revised: Adam (e.g., warm-ups, decay rates etc.)

Original: but we hypothesize using Adam is a good first step.
Revised: but we hypothesize using Adam is a good first step.

Original: We conjecture that the small benefits that SGD might provide are negligible compared to the stability that Adam provides, especially as the scale of the data sets starts to increase.
Revised: We conjecture that the small benefits that SGD might provide are negligible compared to the stability that Adam provides, especially as the scale of the data sets starts to increase.

Original: Without Adam, we conjecture it would be hard to even perfectly fit the data for large scale data sets as it’s usually done in Deep Learning.
Revised: Without Adam, we conjecture it would be hard to even perfectly fit the data for large scale data sets as it’s usually done in Deep Learning.

Original: This was definitively true in our own experiments.
Revised: This was definitively true in our own experiments.

Original: Therefore, we decided to use Adam for our experiments, since it would be too hard to use SGD reliably at scale or with sophisticated meta-learning algorithms.
Revised: Therefore, we decided to use Adam for our experiments, since it would be too hard to use SGD reliably at scale or with sophisticated meta-learning algorithms.

Original: Details for left most plot on figure 1: Results on the left most plot used a (meta) batch-size oftasks and 95% confidence intervals 1.
Revised: Details for left most plot on figure 1: Results on the left most plot used a (meta) batch-size oftasks and 95% confidence intervals 1.

Original: All MAML models were trained with 5 inner steps during meta-training.
Revised: All MAML models were trained with 5 inner steps during meta-training.

Original: MiniImagenet is abbreviated as “MI" in the figure.
Revised: MiniImagenet is abbreviated as “MI" in the figure.

Original: We used 20 shots (number of examples per class) since we can use the whole task data to compute the diversity coefficient (no splitting of support and query set required for the diversity coefficient).
Revised: We used 20 shots (number of examples per class) since we can use the whole task data to compute the diversity coefficient (no splitting of support and query set required for the diversity coefficient).

Original: We used Resnet18 and
Revised: We used Resnet18 and

Original: Resnet34 networks as probe networks – both pre-trained on ImageNet (indicated as “pt" on table) and randomly initialized (indicated as “rand" on table).
Revised: Resnet34 networks as probe networks – both pre-trained on ImageNet (indicated as “pt" on table) and randomly initialized (indicated as “rand" on table).

Original: We observe that both type of networks and weights give similar diversity results.
Revised: We observe that both type of networks and weights give similar diversity results.

Original: Details for right most plot on figure 1: “MAML5" and “MAML10" in the bar plot indicates the adaptation method used at test time i.e. we used 5 inner steps and 10 inner steps at test time.
Revised: Details for right most plot on figure 1: “MAML5" and “MAML10" in the bar plot indicates the adaptation method used at test time i.e. we used 5 inner steps and 10 inner steps at test time.

Original: MiniImagenet is abbreviated as “MI" in the figure.
Revised: MiniImagenet is abbreviated as “MI" in the figure.

Original: All confidence intervals were at 95%.
Revised: All confidence intervals were at 95%.

Original: We used 500 few-shot learning tasks.
Revised: We used 500 few-shot learning tasks.

Original: This results in (500 2 − 500) / 2 = 124 , 750 pair-wise distances used to compute the diversity coefficient.
Revised: This results in (500 2 − 500) / 2 = 124 , 750 pair-wise distances used to compute the diversity coefficient.

Original: We used 95% confidence intervals.
Revised: We used 95% confidence intervals.

Original: F.2 Experimental Details on N-way Gaussian Tasks
Revised: F.2 Experimental Details on N-way Gaussian Tasks

Original: We used a custom 3-layer fully connected network derived from Learn2Learn’s OmniglotFC model ["Arnold et al., 2020], with parameters input _ size = 1 , output _ size = 5 , and hidden layer sizes sizes =
Revised: We used a custom 3-layer fully connected network derived from Learn2Learn’s OmniglotFC model ["Arnold et al., 2020], with parameters input _ size = 1 , output _ size = 5 , and hidden layer sizes sizes =

Original: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL, but did not use a cosine scheduler for either USL or MAML.
Revised: We used the Adam optimizer with learning rate 1e-3 for both MAML and USL, but did not use a cosine scheduler for either USL or MAML.

Original: We trained the USL model forepochs.
Revised: We trained the USL model forepochs.

Original: We trained the MAML model for 14,000 episodic iterations (outer loop iterations).
Revised: We trained the MAML model for 14,000 episodic iterations (outer loop iterations).

Original: We used a batch size of 100 for USL and a (meta) batch size of 100 for MAML.
Revised: We used a batch size of 100 for USL and a (meta) batch size of 100 for MAML.

Original: For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.
Revised: For MAML we used an inner learning rate of 1e-1 and 5 inner learning steps.

Original: We did not use first order MAML.
Revised: We did not use first order MAML.

Original: It took 19 minutes 24 seconds to train USL to convergence with a single Titan X. It took 2 days and 13 hours 6 minutes 27 seconds to train MAML to convergence with a single Titan X.
Revised: It took 19 minutes 24 seconds to train USL to convergence with a single Titan X. It took 2 days and 13 hours 6 minutes 27 seconds to train MAML to convergence with a single Titan X.

Original: F.2.1 Experimental details of empirical equivalence figure in main body
Revised: F.2.1 Experimental details of empirical equivalence figure in main body

Original: Experimental details for figure 11.
Revised: Experimental details for figure 11.

Original: MAML models were trained with 5 inner steps.
Revised: MAML models were trained with 5 inner steps.

Original: MAML5 and
Revised: MAML5 and

Original: MAML10 indicate the adaptation procedure at test time.
Revised: MAML10 indicate the adaptation procedure at test time.

Original: Results used a (meta) batch-size oftasks and 95% confidence intervals.
Revised: Results used a (meta) batch-size oftasks and 95% confidence intervals.

Original: As the diversity of the benchmark increases, the Gaussian tasks are sampled further away from the origin.
Revised: As the diversity of the benchmark increases, the Gaussian tasks are sampled further away from the origin.

Original: Note, as the diversity increases, the difference between
Revised: Note, as the diversity increases, the difference between

Original: USL and MAML increases (in favor of USL).
Revised: USL and MAML increases (in favor of USL).

Original: F.3 Experimental Details on 5CNN test performance equivalence on MiniImagenet
Revised: F.3 Experimental Details on 5CNN test performance equivalence on MiniImagenet

Original: For figure 1 we used Logistic Regression (LR) with LBFGS with the default value for the l regularization parameter given by Python’s Sklearn.
Revised: For figure 1 we used Logistic Regression (LR) with LBFGS with the default value for the l regularization parameter given by Python’s Sklearn.

Original: Note that an increase in inner steps from 5 towith the MAML5 trained model does not provide an additional meta-test accuracy boost, consistent with previous work [Miranda, 2020a].
Revised: Note that an increase in inner steps from 5 towith the MAML5 trained model does not provide an additional meta-test accuracy boost, consistent with previous work [Miranda, 2020a].

Original: Note that the fact that the MAML5 representation matches the USL representation when both use the same adaptation method is not surprising – given that: 1) previous work has shown that the distance between the body of an adapted MAML model is minimal compared to the unadapted MAML (which we reproduce in 9 in the green line) and 2) the fact that a
Revised: Note that the fact that the MAML5 representation matches the USL representation when both use the same adaptation method is not surprising – given that: 1) previous work has shown that the distance between the body of an adapted MAML model is minimal compared to the unadapted MAML (which we reproduce in 9 in the green line) and 2) the fact that a

Original: MAML5 adaptation is only 5 steps of MAML while LR fully converges the prediction layer.
Revised: MAML5 adaptation is only 5 steps of MAML while LR fully converges the prediction layer.

Original: We want to highlight that only the MAML5 model achieved the maximum meta-test performance of 0.
Revised: We want to highlight that only the MAML5 model achieved the maximum meta-test performance of 0.

Original: with the MAML5 adaptation – suggesting that the USL and MAML5 meta-learning algorithms might learn different representations.
Revised: with the MAML5 adaptation – suggesting that the USL and MAML5 meta-learning algorithms might learn different representations.

Original: For USL to have a fair comparison during meta-test time when using the MAML adaptation, we provide the MAML final layer learned initialization parameters to the
Revised: For USL to have a fair comparison during meta-test time when using the MAML adaptation, we provide the MAML final layer learned initialization parameters to the

Original: USL model (but any is fine due to convexity when using a fixed feature extractor).
Revised: USL model (but any is fine due to convexity when using a fixed feature extractor).

Original: This is needed since during meta-training, USL is trained with a union of all the labels (64) – so it does not even have the right output size of 5 for few-shot prediction.
Revised: This is needed since during meta-training, USL is trained with a union of all the labels (64) – so it does not even have the right output size of 5 for few-shot prediction.

Original: Meta-testing was done in the standard 5-way, 5-shot regime.
Revised: Meta-testing was done in the standard 5-way, 5-shot regime.

Original: G Feature Extractor Analysis of USL and MAML
Revised: G Feature Extractor Analysis of USL and MAML

Original: G.1 MAML learns a different base model compared to Union Supervised Learned models –
Revised: G.1 MAML learns a different base model compared to Union Supervised Learned models –

Original: even in the presence of low task diversity
Revised: even in the presence of low task diversity

Original: The first four layers of figure 10 shows how large the distance is of a MAML representation compared to a SL representation.
Revised: The first four layers of figure 10 shows how large the distance is of a MAML representation compared to a SL representation.

Original: In particular, it is much larger than the distance value in the range [0 , 0 . 1] from previous work that compared MAML vs. adapted MAML [Raghu et al., 2020].
Revised: In particular, it is much larger than the distance value in the range [0 , 0 . 1] from previous work that compared MAML vs. adapted MAML [Raghu et al., 2020].

Original: We reproduced that and indeed MAML vs. adapted MAML has a small difference (smaller for us) – supporting our observations that a MAML vs. a USL learned representation show a statistically significant difference at the feature extractor layer, even when the diversity is low.
Revised: We reproduced that and indeed MAML vs. adapted MAML has a small difference (smaller for us) – supporting our observations that a MAML vs. a USL learned representation show a statistically significant difference at the feature extractor layer, even when the diversity is low.

Original: In this section, we discuss figure 9 that suggests that the feature extractor of MAML trained model is markedly different from a USL model – even when the diversity is low.
Revised: In this section, we discuss figure 9 that suggests that the feature extractor of MAML trained model is markedly different from a USL model – even when the diversity is low.

Original: Figure 9 shows that the initial difference is already much larger than previous work [Raghu et al., 2020] were, 0 . 

Revised: Figure 9 shows that the initial difference is already much larger than previous work [Raghu et al., 2020] were, 0 .

Original: < 0 .
Revised: < 0 .

Original: 1 and statistically significant.
Revised: 1 and statistically significant.

Original: In fact, we reproduce their results from [Raghu et al., 2020] in figure 9 with the green curve.
Revised: In fact, we reproduce their results from [Raghu et al., 2020] in figure 9 with the green curve.

Original: In addition, their original distance remains much larger than our reproduction of the distance of previous work with the green line.
Revised: In addition, their original distance remains much larger than our reproduction of the distance of previous work with the green line.

Original: This suggests that the adaptation makes even a smaller difference in MAML according to our experiments.
Revised: This suggests that the adaptation makes even a smaller difference in MAML according to our experiments.

Original: This is true for all four metrics: SVCCA,
Revised: This is true for all four metrics: SVCCA,

Original: PWCCCA, LINCKA and OPD.
Revised: PWCCCA, LINCKA and OPD.

Original: H Classification Layer Analysis of USL and MAML show predictions are similar
Revised: H Classification Layer Analysis of USL and MAML show predictions are similar

Original: The main contribution of this section is to show the similarity of the predictions of USL and MAML through figure 10.
Revised: The main contribution of this section is to show the similarity of the predictions of USL and MAML through figure 10.

Original: This is additional evidence to our claims – through a different metric than accuracy difference – that USL and MAML are empirically equivalent.
Revised: This is additional evidence to our claims – through a different metric than accuracy difference – that USL and MAML are empirically equivalent.

Original: For the background on SVCCA,
Revised: For the background on SVCCA,

Original: PWCCA, CKA, and, OPD see section O.
Revised: PWCCA, CKA, and, OPD see section O.

Original: I Task2Vec embeddings are also related to the Complexity of a Task
Revised: I Task2Vec embeddings are also related to the Complexity of a Task

Original: In addition, Task2Vec exhibits impressive properties that are strongly suggestive that it also captures the amount of information about a task.
Revised: In addition, Task2Vec exhibits impressive properties that are strongly suggestive that it also captures the amount of information about a task.

Original: Some of these properties are: 1.
Revised: Some of these properties are: 1.

Original: the ability to use Task2Vec to select a pre-trained features extractor close to the ground truth optimal expert – while costing substantially less [Achille UCLA et al., 2019].
Revised: the ability to use Task2Vec to select a pre-trained features extractor close to the ground truth optimal expert – while costing substantially less [Achille UCLA et al., 2019].

Original: the correlation of the (nuclear) norm of the
Revised: the correlation of the (nuclear) norm of the

Original: Task2Vec embedding with the difficulty of the task and test performance.
Revised: Task2Vec embedding with the difficulty of the task and test performance.

Original: In addition, Task2Vec (through FIM) is related to the (Kolmogorov) complexity of a task [Achille et al., 2018].
Revised: In addition, Task2Vec (through FIM) is related to the (Kolmogorov) complexity of a task [Achille et al., 2018].

Original: J Background of Few-shot Learning Basics
Revised: J Background of Few-shot Learning Basics

Original: The goal of few-shot learning is to learn to classify from a limited set of training samples.
Revised: The goal of few-shot learning is to learn to classify from a limited set of training samples.

Original: A few-shot benchmark is utilized to evaluate few-shot learning algorithms and typically contains many classes and a smaller number of samples per class.
Revised: A few-shot benchmark is utilized to evaluate few-shot learning algorithms and typically contains many classes and a smaller number of samples per class.

Original: Typically, few-shot learning algorithms learn in episodes, where in each episode, a task consisting of a train (or support) set and a held-out validation (or query) set is sampled.
Revised: Typically, few-shot learning algorithms learn in episodes, where in each episode, a task consisting of a train (or support) set and a held-out validation (or query) set is sampled.

Original: In particular, a task is a n -way k -shot classification problem, means that the support and query sets each consist of n classes sampled from the benchmark, and each of the n classes are represented by k shots or examples.
Revised: In particular, a task is a n -way k -shot classification problem, means that the support and query sets each consist of n classes sampled from the benchmark, and each of the n classes are represented by k shots or examples.

Original: The learner uses the support set to adapt to the task, and the query set to evaluate
Revised: The learner uses the support set to adapt to the task, and the query set to evaluate

Original: K Synthetic Gaussian Benchmark and N-way Gaussian Tasks
Revised: K Synthetic Gaussian Benchmark and N-way Gaussian Tasks

Original: We create a series of synthetic few-shot benchmarks, where each Gaussian benchmark B is defined by four parameters B = ( µ m , σ m , µ s , σ s ) .
Revised: We create a series of synthetic few-shot benchmarks, where each Gaussian benchmark B is defined by four parameters B = ( µ m , σ m , µ s , σ s ) .

Original: To form the dataset of our benchmark, we first samplemeta-train, 100 meta-test, and 100 meta-validation classes, where class 1 ≤ i ≤ 300 is a Gaussian parameterized by ( µ class i , σ class i ) where
Revised: To form the dataset of our benchmark, we first samplemeta-train, 100 meta-test, and 100 meta-validation classes, where class 1 ≤ i ≤ 300 is a Gaussian parameterized by ( µ class i , σ class i ) where

Original: [Equation]
Revised: [Equation]

Original: Then, for each class i , we sample 1000 data points ( x i, 1 , i ) . . .
Revised: Then, for each class i , we sample 1000 data points ( x i, 1 , i ) . . .

Original: ( x i, 1000 , i ) where each datapoint ( x, y ) is composed of a input value x ∈ R and class label 1 ≤ y ≤ 300 .
Revised: ( x i, 1000 , i ) where each datapoint ( x, y ) is composed of a input value x ∈ R and class label 1 ≤ y ≤ 300 .

Original: The input values x i, 1 . . .
Revised: The input values x i, 1 . . .

Original: x 1 ,are each sampled from class
Revised: x 1 ,are each sampled from class

Original: i ’s class distribution:
Revised: i ’s class distribution:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Having defined our dataset underlying our benchmark, we may now sample individual tasks from our benchmark.
Revised: Having defined our dataset underlying our benchmark, we may now sample individual tasks from our benchmark.

Original: Each task in our benchmark is 5-way, 10-shot - that is, each task is formed by first sampling 5 ways from the benchmark dataset, then sampling 10 shots from each of the 5 ways.
Revised: Each task in our benchmark is 5-way, 10-shot - that is, each task is formed by first sampling 5 ways from the benchmark dataset, then sampling 10 shots from each of the 5 ways.

Original: The goal of each task is to correctly predict which of the 5 ways an input value x ∈ R falls into.
Revised: The goal of each task is to correctly predict which of the 5 ways an input value x ∈ R falls into.

Original: We conducted experiments using 7 different benchmarks, with each benchmark defined by four parameters and its corresponding Hellinger distribution diversity coefficient and Task2Vec task diversity coefficient, as listed in Table 1:
Revised: We conducted experiments using 7 different benchmarks, with each benchmark defined by four parameters and its corresponding Hellinger distribution diversity coefficient and Task2Vec task diversity coefficient, as listed in Table 1:

Original: Benchmark Parameters
Revised: Benchmark Parameters

Original: [Table]
Revised: [Table]

Original: Hellinger
Revised: Hellinger

Original: Diversity
Revised: Diversity

Original: Task2Vec
Revised: Task2Vec

Original: [Table]
Revised: [Table]

Original: Figure 12 visualizes
Revised: Figure 12 visualizes

Original: L Distribution-based Diversity Metrics
Revised: L Distribution-based Diversity Metrics

Original: In addition to the task diversity methods (such as Task2Vec) that we chose as a measure of diversity across our experiments in our main paper, we would like to introduce an additional class of diversity metrics that we call distribution diversity .
Revised: In addition to the task diversity methods (such as Task2Vec) that we chose as a measure of diversity across our experiments in our main paper, we would like to introduce an additional class of diversity metrics that we call distribution diversity .

Original: Unlike task diversity, which quantifies diversity through the expected distance between any two distinct tasks sampled from the benchmark, distribution diversity quantifies diversity through the expected distance between any two distinct distributions that underlie the benchmark.
Revised: Unlike task diversity, which quantifies diversity through the expected distance between any two distinct tasks sampled from the benchmark, distribution diversity quantifies diversity through the expected distance between any two distinct distributions that underlie the benchmark.

Original: In our synthetic Gaussian experiments, we define the distribution diversity of our Gaussian benchmark as the expected Hellinger distance between two distinct Gaussian class distributions sampled from the benchmark - we describe the calculation of the distribution diversity of our synthetic Gaussian benchmark in more detail in Section M.
Revised: In our synthetic Gaussian experiments, we define the distribution diversity of our Gaussian benchmark as the expected Hellinger distance between two distinct Gaussian class distributions sampled from the benchmark - we describe the calculation of the distribution diversity of our synthetic Gaussian benchmark in more detail in Section M.

Original: M Hellinger Diversity Coefficient and Hellinger Distance
Revised: M Hellinger Diversity Coefficient and Hellinger Distance

Original: An alternative metric to the Task2Vec-based task diversity metric is the Hellinger-based distribution diversity metric.
Revised: An alternative metric to the Task2Vec-based task diversity metric is the Hellinger-based distribution diversity metric.

Original: The Hellinger-based distribution diversity of our Gaussian benchmark is obtained by computing the expected Hellinger distance between any two classes sampled from the benchmark.
Revised: The Hellinger-based distribution diversity of our Gaussian benchmark is obtained by computing the expected Hellinger distance between any two classes sampled from the benchmark.

Original: That is, for some benchmark parameterized by B = ( µ m , σ m , µ s , σ s ) , the diversity coefficient is given by
Revised: That is, for some benchmark parameterized by B = ( µ m , σ m , µ s , σ s ) , the diversity coefficient is given by

Original: [Equation]
Revised: [Equation]

Original: where H 2 denotes the squared Hellinger distance metric and N ( µ 1 , σ 1 ) , N ( µ 2 , σ 2 ) denote the distributions of the two classes sampled from the benchmark.
Revised: where H 2 denotes the squared Hellinger distance metric and N ( µ 1 , σ 1 ) , N ( µ 2 , σ 2 ) denote the distributions of the two classes sampled from the benchmark.

Original: The Hellinger-based distribution diversity metric provides an intuitive, model-agnostic characterization of the diversity of a benchmark - the larger the diversity, the less similar any two classes within the benchmark are, and the easier it is to distinguish between two classes.
Revised: The Hellinger-based distribution diversity metric provides an intuitive, model-agnostic characterization of the diversity of a benchmark - the larger the diversity, the less similar any two classes within the benchmark are, and the easier it is to distinguish between two classes.

Original: Conversely, the lower the diversity, the more similar any two classes within the benchmark are, and the harder it is to distinguish between two classes due to a larger overlap between the two classes’ distributions.
Revised: Conversely, the lower the diversity, the more similar any two classes within the benchmark are, and the harder it is to distinguish between two classes due to a larger overlap between the two classes’ distributions.

Original: Note that the closed-form equation for the Hellinger distance between the two class distributions
Revised: Note that the closed-form equation for the Hellinger distance between the two class distributions

Original: N ( µ 1 , σ 1 ) , N ( µ 2 , σ 2 ) is given by
Revised: N ( µ 1 , σ 1 ) , N ( µ 2 , σ 2 ) is given by

Original: [Equation]
Revised: [Equation]

Original: However, there is no simple closed-form equation for computing the diversity div ( B ) itself.
Revised: However, there is no simple closed-form equation for computing the diversity div ( B ) itself.

Original: As a result, we computed the diversity coefficient as a numerical approximation by repeatedly sampling two classes from the benchmark distribution and calculating the Hellinger distance between the two classes.
Revised: As a result, we computed the diversity coefficient as a numerical approximation by repeatedly sampling two classes from the benchmark distribution and calculating the Hellinger distance between the two classes.

Original: These samples ultimately provide a 95% confidence interval that represents the expected
Revised: These samples ultimately provide a 95% confidence interval that represents the expected

Original: Hellinger distance between two classes sampled from the benchmark.
Revised: Hellinger distance between two classes sampled from the benchmark.

Original: We also compared our Hellinger-based distribution diversity coefficient with the Task2Vec-based task diversity coefficient for each of the synthetic Gaussian benchmarks tested in Table 1.
Revised: We also compared our Hellinger-based distribution diversity coefficient with the Task2Vec-based task diversity coefficient for each of the synthetic Gaussian benchmarks tested in Table 1.

Original: We observe a strong positive correlation between the Hellinger-based distribution diversity and Task2Vec-based task diversity coefficients according to Figure 11, indicating that the Hellinger-based distribution diversity serves as an effective proxy for task diversity when the number of ways and shots of all tasks are fixed.
Revised: We observe a strong positive correlation between the Hellinger-based distribution diversity and Task2Vec-based task diversity coefficients according to Figure 11, indicating that the Hellinger-based distribution diversity serves as an effective proxy for task diversity when the number of ways and shots of all tasks are fixed.

Original: N Analysis of distribution of task distances in few-shot learning benchmarks N.1 Heat Maps show Low Diversity and Homogeneity of tasks from MiniImagenet and Cifar-fs
Revised: N Analysis of distribution of task distances in few-shot learning benchmarks N.1 Heat Maps show Low Diversity and Homogeneity of tasks from MiniImagenet and Cifar-fs

Original: In this section, we show the heat maps showing the distances between 5-way, 20-shot few-shot learning tasks from MiniImagenet and Cifar-fs in figure 13 and 14.
Revised: In this section, we show the heat maps showing the distances between 5-way, 20-shot few-shot learning tasks from MiniImagenet and Cifar-fs in figure 13 and 14.

Original: We used 20-shots because we do not need to separate the data into support and query set to compute the diversity coefficient.
Revised: We used 20-shots because we do not need to separate the data into support and query set to compute the diversity coefficient.

Original: We show that tasks sampled from these benchmarks create not only a low diversity coefficient on average, but also at the level of individual distances between pairs of tasks.
Revised: We show that tasks sampled from these benchmarks create not only a low diversity coefficient on average, but also at the level of individual distances between pairs of tasks.

Original: In addition, the heat map’s uniform coloring reveals that it is also justifiable to call the tasks from these benchmarks homogeneous .
Revised: In addition, the heat map’s uniform coloring reveals that it is also justifiable to call the tasks from these benchmarks homogeneous .

Original: Low diversity is shown because the distances are between 0.07-0.12 given that max is 1.0 and minimum is 0.0.
Revised: Low diversity is shown because the distances are between 0.07-0.12 given that max is 1.0 and minimum is 0.0.

Original: N.
Revised: N.

Original: 2 Histograms of distances of tasks in the synthetic Gaussian
Revised: 2 Histograms of distances of tasks in the synthetic Gaussian

Original: Benchmark, MiniImagenet and Cifar-fs
Revised: Benchmark, MiniImagenet and Cifar-fs

Original: In this section, we show the histograms of the cosine distances between pairs of tasks for the Gaussian
Revised: In this section, we show the histograms of the cosine distances between pairs of tasks for the Gaussian

Original: Benchmark, MiniImagenet and Cifar-fs.
Revised: Benchmark, MiniImagenet and Cifar-fs.

Original: The main purpose of this is to argue that a (relatively small) sample of the tasks is sufficient to estimate population statistics – like the expected distance between tasks i.e. diversity coefficient.
Revised: The main purpose of this is to argue that a (relatively small) sample of the tasks is sufficient to estimate population statistics – like the expected distance between tasks i.e. diversity coefficient.

Original: For ease of exposition for the argument, consider the case where we have 500 distance from a large population of size (cid:0) 645 (cid:1)
Revised: For ease of exposition for the argument, consider the case where we have 500 distance from a large population of size (cid:0) 645 (cid:1)

Original: = 7 , 624 , 512 .
Revised: = 7 , 624 , 512 .

Original: The goal is to argue that 500 samples are enough to make strong statistical inferences about the population – even if it’s as large as 7 , 624 , 512 .
Revised: The goal is to argue that 500 samples are enough to make strong statistical inferences about the population – even if it’s as large as 7 , 624 , 512 .

Original: If we assume the distribution of the data is Gaussian, then we expect to see a single mode with an approximate bell curve.
Revised: If we assume the distribution of the data is Gaussian, then we expect to see a single mode with an approximate bell curve.

Original: Therefore, if we plot the histogram of task pair distances of the 500 tasks and see this, then we can infer our Gaussian assumption is approximately correct.
Revised: Therefore, if we plot the histogram of task pair distances of the 500 tasks and see this, then we can infer our Gaussian assumption is approximately correct.

Original: Given that the 15, 16, 17 appear
Revised: Given that the 15, 16, 17 appear

Original: Gaussian, we can infer our assumption is approximately correct.
Revised: Gaussian, we can infer our assumption is approximately correct.

Original: This implies we can make strong statistical assumptions about the population – in particular, that we have a good estimate of the diversity coefficient using 500 samples.
Revised: This implies we can make strong statistical assumptions about the population – in particular, that we have a good estimate of the diversity coefficient using 500 samples.

Original: Additionally, in histograms also discard the presence of outlier tasks.
Revised: Additionally, in histograms also discard the presence of outlier tasks.

Original: O Background on Deep Neural Network Distance metrics
Revised: O Background on Deep Neural Network Distance metrics

Original: Distances for Deep Neural Network Feature Analysis: To compute the distance between neural networks, we use the distance versions of Singular Value Canonical Correlation Analysis (SVCCA)
Revised: Distances for Deep Neural Network Feature Analysis: To compute the distance between neural networks, we use the distance versions of Singular Value Canonical Correlation Analysis (SVCCA)

Original: [Raghu et al., 2017], Projection Weighted Canonical Correlation (PWCCA)
Revised: [Raghu et al., 2017], Projection Weighted Canonical Correlation (PWCCA)

Original: [Morcos et al., 2018],
Revised: [Morcos et al., 2018],

Original: Linear Centered Kernel Analysis (LINCKA)
Revised: Linear Centered Kernel Analysis (LINCKA)

Original: [Kornblith et al., 2019] and Orthogonal Procrustes
Revised: [Kornblith et al., 2019] and Orthogonal Procrustes

Original: Distance (OPD)
Revised: Distance (OPD)

Original: [Ding et al., 2021].
Revised: [Ding et al., 2021].

Original: These distances are in the interval [0 , 1] and are not necessarily a formal distance metric but are guaranteed to be zero when their inputs are equal and nonzero otherwise.
Revised: These distances are in the interval [0 , 1] and are not necessarily a formal distance metric but are guaranteed to be zero when their inputs are equal and nonzero otherwise.

Original: This is true because SVCCA, PWCCA, LINCKA are based on similarity metrics and
Revised: This is true because SVCCA, PWCCA, LINCKA are based on similarity metrics and

Original: OPD is already a distance.
Revised: OPD is already a distance.

Original: Note that we use the formula d ( X, Y )
Revised: Note that we use the formula d ( X, Y )

Original: = 1 − sim ( X, Y ) for our distance metrics where sim is one either SVCCA, PWCCA, LINCKA similarity metric and X, Y are matrices of activations (called layer matrices).
Revised: = 1 − sim ( X, Y ) for our distance metrics where sim is one either SVCCA, PWCCA, LINCKA similarity metric and X, Y are matrices of activations (called layer matrices).

Original: The distance between two models is computed by choosing a layer and then comparing the features/activations after adaptation for that layer given a batch of tasks represented as a support and query set.
Revised: The distance between two models is computed by choosing a layer and then comparing the features/activations after adaptation for that layer given a batch of tasks represented as a support and query set.

Original: O.1 Neuron Vectors
Revised: O.1 Neuron Vectors

Original: The representation of a neuron d in layer l is the vector z ( l ) d ( X ) ∈ R N of activations for a set of N examples, where X ∈ R N,D is the data matrix with N examples.
Revised: The representation of a neuron d in layer l is the vector z ( l ) d ( X ) ∈ R N of activations for a set of N examples, where X ∈ R N,D is the data matrix with N examples.

Original: O.2 Layer Matrix
Revised: O.2 Layer Matrix

Original: A layer matrix L for layer l is a matrix of neuron vectors z ( l ) d ( X ) ∈ R N with shape, [ N, D i ] i.e.
Revised: A layer matrix L for layer l is a matrix of neuron vectors z ( l ) d ( X ) ∈ R N with shape, [ N, D i ] i.e.

Original: L ∈ R N,D i .
Revised: L ∈ R N,D i .

Original: In other words, the layer matrix L is the subspace of R N spanned by its neuron vectors z ( l ) d ( X ) .
Revised: In other words, the layer matrix L is the subspace of R N spanned by its neuron vectors z ( l ) d ( X ) .

Original: In short, L is the layer matrix [ z ld ; . . .
Revised: In short, L is the layer matrix [ z ld ; . . .

Original: ; z lD 1 ] ∈ R N,D i with neuron vector z ld .
Revised: ; z lD 1 ] ∈ R N,D i with neuron vector z ld .

Original: O.3 CCA
Revised: O.3 CCA

Original: Canonical Correlation Analysis (CCA) is a well established statistical technique for comparing the (linear) correlation of two sets of random variables (or vectors of random variables).
Revised: Canonical Correlation Analysis (CCA) is a well established statistical technique for comparing the (linear) correlation of two sets of random variables (or vectors of random variables).

Original: In the empirical
Revised: In the empirical

Original: True distribution based Canonical Correlation Analysis (CCA): What we call true distribution based CCA is the standard CCA measure using the true but known distribution of the data p ∗ ( x ) and p ∗ ( y ) .
Revised: True distribution based Canonical Correlation Analysis (CCA): What we call true distribution based CCA is the standard CCA measure using the true but known distribution of the data p ∗ ( x ) and p ∗ ( y ) .

Original: In this case, CCA searches for a pair of linear combinations a ∗ , b ∗ of two set of random variables (or vectors of random variables) x
Revised: In this case, CCA searches for a pair of linear combinations a ∗ , b ∗ of two set of random variables (or vectors of random variables) x

Original: = [ X 1 , . . .
Revised: = [ X 1 , . . .

Original: , X D 1 ] and y =
Revised: , X D 1 ] and y =

Original: [ Y 1 , . . .
Revised: [ Y 1 , . . .

Original: , Y D 2 ] that maximizes the Pearson correlation coefficient:
Revised: , Y D 2 ] that maximizes the Pearson correlation coefficient:

Original: [Equation]
Revised: [Equation]

Original: where Σ X,Y , Σ X,X Σ Y,Y are the (true) covariance and variance matrices respectively (e.g.
Revised: where Σ X,Y , Σ X,X Σ Y,Y are the (true) covariance and variance matrices respectively (e.g.

Original: Σ X,Y [ i, j ] =
Revised: Σ X,Y [ i, j ] =

Original: Cov [ X i , X j ] =
Revised: Cov [ X i , X j ] =

Original: [ X i Y j ] for centered random variables).
Revised: [ X i Y j ] for centered random variables).

Original: All of these can be replaced by empirical data matrices in the obvious way.
Revised: All of these can be replaced by empirical data matrices in the obvious way.

Original: O.4 SVCCA
Revised: O.4 SVCCA

Original: At a high level, SVCCA is a similarity measure of two matrices that aims in removing redundant neurons (i.e. redundant features) with the truncated SVD by keeping 0.99 of the variance and then measure the overall similarity by averaging the top C CCA values.
Revised: At a high level, SVCCA is a similarity measure of two matrices that aims in removing redundant neurons (i.e. redundant features) with the truncated SVD by keeping 0.99 of the variance and then measure the overall similarity by averaging the top C CCA values.

Original: SV: Given two matrices L 1 ∈ R N,D 1 , L 2 ∈ R N,D 2 (e.g. layer matrices) first reduce the effective dimensionality of the matrix via a low rank approximation L ′ 1 ∈ R N,D ′ 1 , L 2 ′ ∈ R N,D ′ 2 by choosing the top k singular values that keeps 0.99 of the variance.
Revised: SV: Given two matrices L 1 ∈ R N,D 1 , L 2 ∈ R N,D 2 (e.g. layer matrices) first reduce the effective dimensionality of the matrix via a low rank approximation L ′ 1 ∈ R N,D ′ 1 , L 2 ′ ∈ R N,D ′ 2 by choosing the top k singular values that keeps 0.99 of the variance.

Original: In particular, for each layer matrix, L i keep the top D ′ i singular values (and vectors) such that
Revised: In particular, for each layer matrix, L i keep the top D ′ i singular values (and vectors) such that

Original: (cid:80) D ′ i
Revised: (cid:80) D ′ i

Original: j =1 | σ j | ≥ 0 .
Revised: j =1 | σ j | ≥ 0 .

Original: (cid:80)
Revised: (cid:80)

Original: rank ( L
Revised: rank ( L

Original: i
Revised: i

Original: j
Revised: j

Original: = | σ j | .
Revised: = | σ j | .

Original: SVCCA: SVCCA is a statistical technique for the measuring the (linear) similarity of two sets of data sets L 1 ∈ R N,D 1 , L 2 ∈ R N,D 2 (e.g. data matrices, layer matrices) by first reducing the effective dimensionality of the matrix via a low rank approximation L ′ 1 ∈ R N,D ′ 1 , L ′ 2 ∈ R N,D ′ 2 (e.g. by choosing the top k singular values that keeps 0.99 of the variance) and then applying the standard empirical CCA to the resulting matrices.
Revised: SVCCA: SVCCA is a statistical technique for the measuring the (linear) similarity of two sets of data sets L 1 ∈ R N,D 1 , L 2 ∈ R N,D 2 (e.g. data matrices, layer matrices) by first reducing the effective dimensionality of the matrix via a low rank approximation L ′ 1 ∈ R N,D ′ 1 , L ′ 2 ∈ R N,D ′ 2 (e.g. by choosing the top k singular values that keeps 0.99 of the variance) and then applying the standard empirical CCA to the resulting matrices.

Original: This is repeated C = min( D ′ 1 , D ′ 2 ) times and the overall similarity of the two matrices is computed as the average CCA: svcca = sim
Revised: This is repeated C = min( D ′ 1 , D ′ 2 ) times and the overall similarity of the two matrices is computed as the average CCA: svcca = sim

Original: ( L ′ 1 , L ′ 2 )
Revised: ( L ′ 1 , L ′ 2 )

Original: = 1 C (cid:80) C c =

Revised: = 1 C (cid:80) C c =

Original: ρ c
Revised: ρ c

Original: Concretely:
Revised: Concretely:

Original: Get the D ′
Revised: Get the D ′

Original: i components that keep 0 .
Revised: i components that keep 0 .

Original: 99 of the variance (i.e. D ′ i
Revised: 99 of the variance (i.e. D ′ i

Original: such that (cid:80) D ′ i
Revised: such that (cid:80) D ′ i

Original: j =1 | σ j | ≥ 0 .
Revised: j =1 | σ j | ≥ 0 .

Original: 99 (cid:80) rank ( L i )
Revised: 99 (cid:80) rank ( L i )

Original: j = | σ j | ).
Revised: j = | σ j | ).

Original: Get the SVD: U 1 , Σ 1 , V ⊤ 1 = SV D ( L 1 ) and U 2 , Σ 2 , V ⊤ 2 = SV D ( L 2) 3.
Revised: Get the SVD: U 1 , Σ 1 , V ⊤ 1 = SV D ( L 1 ) and U 2 , Σ 2 , V ⊤ 2 = SV D ( L 2) 3.

Original: Then produce the SVD dimensionality reduction by L ′ 

Revised: Then produce the SVD dimensionality reduction by L ′

Original: = L 1 V 1 [1 : k i ] ∈ R N,D 1 and L ′ 

Revised: = L 1 V 1 [1 : k i ] ∈ R N,D 1 and L ′

Original: =
Revised: =

Original: L 2 V 2 [1 : k ] ∈ R N,D 2 where V i [1 : D i ] gets the top D i columns of a layer matrix i .
Revised: L 2 V 2 [1 : k ] ∈ R N,D 2 where V i [1 : D i ] gets the top D i columns of a layer matrix i .

Original: Get the CCA of the reduced layer matrix:
Revised: Get the CCA of the reduced layer matrix:

Original: [ ρ c ] Cc =1 = CCA ( L ′ 1 , L ′ 2 ) where, C = min( D ′ 1 , D ′ 2 ) 5.
Revised: [ ρ c ] Cc =1 = CCA ( L ′ 1 , L ′ 2 ) where, C = min( D ′ 1 , D ′ 2 ) 5.

Original: Finally return the mean CCA: svcca = 1 C (cid:80) Cc =1 ρ c , where is the k-th CCA value of the reduced layer matrix.
Revised: Finally return the mean CCA: svcca = 1 C (cid:80) Cc =1 ρ c , where is the k-th CCA value of the reduced layer matrix.

Original: O.5 PWCCA
Revised: O.5 PWCCA

Original: At a high level
Revised: At a high level

Original: PWCCA was developed to increase the robustness (to noise) of SVCCA in the context of deep neural networks.
Revised: PWCCA was developed to increase the robustness (to noise) of SVCCA in the context of deep neural networks.

Original: In particular, Maithra et al.
Revised: In particular, Maithra et al.

Original: [Morcos et al., 2018] noticed that when the performance of the neural networks stabilized, so did the set of CCA vectors (or principle neuron vectors) related to the network stabilized on the data set in question.
Revised: [Morcos et al., 2018] noticed that when the performance of the neural networks stabilized, so did the set of CCA vectors (or principle neuron vectors) related to the network stabilized on the data set in question.

Original: Thus, they suggest to give higher weighting to the canonical correlation ρ c of these stable CCA vectors – in particular to the ones that are similar to the final output layer matrix, e.g. L 1 .
Revised: Thus, they suggest to give higher weighting to the canonical correlation ρ c of these stable CCA vectors – in particular to the ones that are similar to the final output layer matrix, e.g. L 1 .

Original: Note this is simpler than trying to track the stability of these CCA vectors during training and then give those higher weighting.
Revised: Note this is simpler than trying to track the stability of these CCA vectors during training and then give those higher weighting.

Original: PWCCA: Formally let L 1 be the layer matrix [ z ld ; . .
Revised: PWCCA: Formally let L 1 be the layer matrix [ z ld ; . .

Original: ; z lD 1 ] ∈ R N,D i with neuron vectors z ld for some layer l .
Revised: ; z lD 1 ] ∈ R N,D i with neuron vectors z ld for some layer l .

Original: Recall that the k-th left CCA vector for layer matrix L 1 is defined as follows,
Revised: Recall that the k-th left CCA vector for layer matrix L 1 is defined as follows,

Original: ˜ x c =
Revised: ˜ x c =

Original: L 1 a c =
Revised: L 1 a c =

Original: L 1 (Σ − 12 u c ) where a c is the cth CCA direction and u c is the c-th left singular value from the matrix M = Σ − 12 L 1 Σ L 1 ,L 2 Σ − 12 L 2 = U Λ V ⊤ .
Revised: L 1 (Σ − 12 u c ) where a c is the cth CCA direction and u c is the c-th left singular value from the matrix M = Σ − 12 L 1 Σ L 1 ,L 2 Σ − 12 L 2 = U Λ V ⊤ .

Original: Then, PWCCA can be computed as follows:
Revised: Then, PWCCA can be computed as follows:

Original: Calculate the CCA vectors ˜ x c
Revised: Calculate the CCA vectors ˜ x c

Original: =
Revised: =

Original: L 1 a c =
Revised: L 1 a c =

Original: L 1 (Σ − 12 L 1 u c ) and explicitly orthonormalize with Gram-Schmidt for numerical stability.
Revised: L 1 (Σ − 12 L 1 u c ) and explicitly orthonormalize with Gram-Schmidt for numerical stability.

Original: Compute the weight ˜
Revised: Compute the weight ˜

Original: α c of how much the layer matrix L 1 is account for by each CCA vector ˜ x k with equation ˜
Revised: α c of how much the layer matrix L 1 is account for by each CCA vector ˜ x k with equation ˜

Original: α c ( h c , L 1 ) =
Revised: α c ( h c , L 1 ) =

Original: (cid:80) Cc =1 |⟨
Revised: (cid:80) Cc =1 |⟨

Original: ˜ x c , z lc ⟩ R N | where z lc is the c-th column of the layer matrix L3.
Revised: ˜ x c , z lc ⟩ R N | where z lc is the c-th column of the layer matrix L3.

Original: Normalize the weight indicating how much each CCA vector h c accounts for L 1 and denote it with, α c (˜ x k , L 1 ) =
Revised: Normalize the weight indicating how much each CCA vector h c accounts for L 1 and denote it with, α c (˜ x k , L 1 ) =

Original: ˜
Revised: ˜

Original: α c (˜ x k ,L 1 ) (cid:80)
Revised: α c (˜ x k ,L 1 ) (cid:80)

Original: Cc =1 α c (˜ x k ,L 1 ) 4.
Revised: Cc =1 α c (˜ x k ,L 1 ) 4.

Original: Finally return the mean CCA weighted by α c (˜ x k , L 1 ) : pwcca = (cid:80)
Revised: Finally return the mean CCA weighted by α c (˜ x k , L 1 ) : pwcca = (cid:80)

Original: Cc =1 α c (˜ x k , L 1 ) ρ c where C = min( D ′ 1 , D ′ 2 ) .
Revised: Cc =1 α c (˜ x k , L 1 ) ρ c where C = min( D ′ 1 , D ′ 2 ) .

Original: The original authors could have used the right CCA vectors,
Revised: The original authors could have used the right CCA vectors,

Original: i.e. ˜ y c
Revised: i.e. ˜ y c

Original: =
Revised: =

Original: L 2 b k
Revised: L 2 b k

Original: =
Revised: =

Original: L 2 (Σ − 12 v k ) and in fact the details of their code suggest they choose the one that would have lead to fewer values removed by SVD.
Revised: L 2 (Σ − 12 v k ) and in fact the details of their code suggest they choose the one that would have lead to fewer values removed by SVD.

Original: This choice seems to already be robust to noise, as shown in [Morcos et al., 2018].
Revised: This choice seems to already be robust to noise, as shown in [Morcos et al., 2018].

Original: Note that the CCA vectors ˜ x k , ˜
Revised: Note that the CCA vectors ˜ x k , ˜

Original: y
Revised: y

Original: k are of size R N and thus could be viewed as the principle neuron vectors that correlate two layers L 1 , L 2 .
Revised: k are of size R N and thus could be viewed as the principle neuron vectors that correlate two layers L 1 , L 2 .

Original: With this view, PWCCA computes the mean CCA normalized by of the c principle neuron vectors are account for the output layer matrix the most.
Revised: With this view, PWCCA computes the mean CCA normalized by of the c principle neuron vectors are account for the output layer matrix the most.

Original: O.6 CCA for CNNs
Revised: O.6 CCA for CNNs

Original: The input to CCA are two data matrices, but CNNs have intermediate representations that are 4D tensors.
Revised: The input to CCA are two data matrices, but CNNs have intermediate representations that are 4D tensors.

Original: Therefore, some justification is needed in how to create the data matrices needed for computing CCA for CNNs.
Revised: Therefore, some justification is needed in how to create the data matrices needed for computing CCA for CNNs.

Original: Note that it’s the same reasoning for both SVCCA and PWCCA.
Revised: Note that it’s the same reasoning for both SVCCA and PWCCA.

Original: Each channel as the dimensionality of the data matrix: One option is to get the intermediate representation of size [ M, C, H, W ] and get a layer matrix of size [ MHW, C ] .
Revised: Each channel as the dimensionality of the data matrix: One option is to get the intermediate representation of size [ M, C, H, W ] and get a layer matrix of size [ MHW, C ] .

Original: Thus, MHW is the effective number of data points and the channels (or number of filters) is the effective dimensionality of the (layer) matrix.
Revised: Thus, MHW is the effective number of data points and the channels (or number of filters) is the effective dimensionality of the (layer) matrix.

Original: In this view, each patch of an image processed by the CNN is effectively considered a data point.
Revised: In this view, each patch of an image processed by the CNN is effectively considered a data point.

Original: This view is very natural because it also considers each filter as its own “neuron" – which seams reasonable considering that each filter uniquely responds to each stimulus (e.g., data patch).
Revised: This view is very natural because it also considers each filter as its own “neuron" – which seams reasonable considering that each filter uniquely responds to each stimulus (e.g., data patch).

Original: This view results in HW images for every sample in the data set (or batch) of size
Revised: This view results in HW images for every sample in the data set (or batch) of size

Original: M and C effective neurons.
Revised: M and C effective neurons.

Original: Although the original authors suggest this metric as a good metric mainly for comparing two layers that are the same – we hypothesize it is also good for comparing different layers (as long as the effective number of data points match for the two layer matrices).
Revised: Although the original authors suggest this metric as a good metric mainly for comparing two layers that are the same – we hypothesize it is also good for comparing different layers (as long as the effective number of data points match for the two layer matrices).

Original: The reason is that CCA tries to compute the maximum correlation of two data sets (or sets of random variables) and assumes no meaning in the ordering of the data points and assumes no process for generating each individual sample for the set of random variables, thus meaning that this metric (CCA) can be used for any two layers in a matrix.
Revised: The reason is that CCA tries to compute the maximum correlation of two data sets (or sets of random variables) and assumes no meaning in the ordering of the data points and assumes no process for generating each individual sample for the set of random variables, thus meaning that this metric (CCA) can be used for any two layers in a matrix.

Original: Overall, in this view, we are comparing the representation learned in each channel.
Revised: Overall, in this view, we are comparing the representation learned in each channel.

Original: Each activation as the dimensionality of the data matrix: One option is to get the intermediate representation of size [ M, C, H, W ] and get a layer matrix of size [ M, CHW ] .
Revised: Each activation as the dimensionality of the data matrix: One option is to get the intermediate representation of size [ M, C, H, W ] and get a layer matrix of size [ M, CHW ] .

Original: Thus, M is the effective number of data points (which matches the number of samples in the data set or batch) and therefore each activation value is the effective dimensionality of the (layer) matrix.
Revised: Thus, M is the effective number of data points (which matches the number of samples in the data set or batch) and therefore each activation value is the effective dimensionality of the (layer) matrix.

Original: In this view, each activation is viewed as a neuron of size M and we have CHW effective neurons for each activation.
Revised: In this view, each activation is viewed as a neuron of size M and we have CHW effective neurons for each activation.

Original: The authors suggest this metric for comparing different layers (potentially at different depths).
Revised: The authors suggest this metric for comparing different layers (potentially at different depths).

Original: However, because CCA assumes no correspondence between the data points nor the same dimensionality in the data matrix – we hypothesize this way to define the data matrix is as valid as the previous definition for comparisons between any models at any layer.
Revised: However, because CCA assumes no correspondence between the data points nor the same dimensionality in the data matrix – we hypothesize this way to define the data matrix is as valid as the previous definition for comparisons between any models at any layer.

Original: One disadvantage however is that it will often result in data matrices that are very large due to CHW being very large – which results in artificially high CCA similarity values.
Revised: One disadvantage however is that it will often result in data matrices that are very large due to CHW being very large – which results in artificially high CCA similarity values.

Original: Potential ways to deal with it are noticing that there is no correspondence between the data matrices, so a cross comparison of every data point with every other data point in CCA is possible (resulting is O
Revised: Potential ways to deal with it are noticing that there is no correspondence between the data matrices, so a cross comparison of every data point with every other data point in CCA is possible (resulting is O

Original: ( M 2 ) comparisons for the empirical covariance matrix).
Revised: ( M 2 ) comparisons for the empirical covariance matrix).

Original: Alternatively one can pool in the spatial dimensions [ H, W ] resulting in potential smaller layer matrices e.g. of shape [ M, C ] with a pool over the entire spatial dimension.
Revised: Alternatively one can pool in the spatial dimensions [ H, W ] resulting in potential smaller layer matrices e.g. of shape [ M, C ] with a pool over the entire spatial dimension.

Original: For these reasons and the fact that we hypothesize an image patch being its own image – we prefer to interpret the number of channels as the natural way to compare CNNs so that the layer matrices results of size [ MHW, C ] .
Revised: For these reasons and the fact that we hypothesize an image patch being its own image – we prefer to interpret the number of channels as the natural way to compare CNNs so that the layer matrices results of size [ MHW, C ] .

Original: Subsampling of representations for channels as dimensionality: In this section, we review the subsampling we did when comparing the representations learned in each channel, i.e. the layer matrix has size [ MHW, C ] .
Revised: Subsampling of representations for channels as dimensionality: In this section, we review the subsampling we did when comparing the representations learned in each channel, i.e. the layer matrix has size [ MHW, C ] .

Original: The effective number of data points MHW will often be much larger than needed (e.g. for 16 data samples M = 16 and H = W = 84 results in MHW = 112 , 896 ), especially compared to the number of filters/channels (e.g. C = 64 ).
Revised: The effective number of data points MHW will often be much larger than needed (e.g. for 16 data samples M = 16 and H = W = 84 results in MHW = 112 , 896 ), especially compared to the number of filters/channels (e.g. C = 64 ).

Original: Previous work [Raghu et al., 2017, Morcos et al., 2018] suggest using the number of effective data points to be from 5-10 times the size of the dimensionality in a layer matrix of size [ N ′ , D ′ ] that means N ′ = 10 D ′ .
Revised: Previous work [Raghu et al., 2017, Morcos et al., 2018] suggest using the number of effective data points to be from 5-10 times the size of the dimensionality in a layer matrix of size [ N ′ , D ′ ] that means N ′ = 10 D ′ .

Original: Based on our reproductions of that number, we choose N ′ = 20 D ′ which results in NHW = 20 C
Revised: Based on our reproductions of that number, we choose N ′ = 20 D ′ which results in NHW = 20 C

Original: O.7 Centered Kernel Alignment (CKA)
Revised: O.7 Centered Kernel Alignment (CKA)

Original: At a high level, CKA is based on the insight that one can first measure the similarity between every pair of examples in each representation separately and then use the similarity structure to compute an overall similarity metric.
Revised: At a high level, CKA is based on the insight that one can first measure the similarity between every pair of examples in each representation separately and then use the similarity structure to compute an overall similarity metric.

Original: In our case, we can treat the examples as the neuron vectors and compare all neuron vectors using some kernel function.
Revised: In our case, we can treat the examples as the neuron vectors and compare all neuron vectors using some kernel function.

Original: Usually this will end in a kernel matrix of size M, M ′ where M and M ′ are the number of examples.
Revised: Usually this will end in a kernel matrix of size M, M ′ where M and M ′ are the number of examples.

Original: In our case, they would be D, D ′ for the number of neurons of each layer matrix.
Revised: In our case, they would be D, D ′ for the number of neurons of each layer matrix.

Original: Note, the layers matrices can correspond to neurons of different layers in a neural network.
Revised: Note, the layers matrices can correspond to neurons of different layers in a neural network.

Original: Linear CKA: We use the linear kernel function as used in previous work [Kornblith et al., 2019, Ding et al., 2021].
Revised: Linear CKA: We use the linear kernel function as used in previous work [Kornblith et al., 2019, Ding et al., 2021].

Original: Given two layer matrices X 1 ∈ R N,D l and X 2 ∈ R N,D l ′ for layers l, l ′ , we compute the linear kernel X ⊤ 1 X 2 to get the D l by D l ′ kernel matrix indicating the (linear) similarity per neuron vector for the two layers.
Revised: Given two layer matrices X 1 ∈ R N,D l and X 2 ∈ R N,D l ′ for layers l, l ′ , we compute the linear kernel X ⊤ 1 X 2 to get the D l by D l ′ kernel matrix indicating the (linear) similarity per neuron vector for the two layers.

Original: Then to obtain a single distance value we compute the Frobenius norm of the kernel matrix and subtract by one after normalization:
Revised: Then to obtain a single distance value we compute the Frobenius norm of the kernel matrix and subtract by one after normalization:

Original: [Equation]
Revised: [Equation]

Original: Note that depending on how the examples in matrices A, B are organized the cross-product could be computed with AB ⊤ instead.
Revised: Note that depending on how the examples in matrices A, B are organized the cross-product could be computed with AB ⊤ instead.

Original: Other kernel functions have been tested (e.g., the RBF kernel) for
Revised: Other kernel functions have been tested (e.g., the RBF kernel) for

Original: CKA but similar results are obtained, resulting in linear CKA being the most popular CKA method [Ding et al., 2021, Kornblith et al., 2019] to the best of our knowledge.
Revised: CKA but similar results are obtained, resulting in linear CKA being the most popular CKA method [Ding et al., 2021, Kornblith et al., 2019] to the best of our knowledge.

Original: O.8 Orthogonal Procrustes Distance (OPD)
Revised: O.8 Orthogonal Procrustes Distance (OPD)

Original: At a high level, the orthogonal Procrustes distance computes the distances between two matrices after using for the best orthogonal matrix that tries to match the two.
Revised: At a high level, the orthogonal Procrustes distance computes the distances between two matrices after using for the best orthogonal matrix that tries to match the two.

Original: Usually this is done after centering and dividing by the Frobenious the matrices, i.e. normalizing the matrices.
Revised: Usually this is done after centering and dividing by the Frobenious the matrices, i.e. normalizing the matrices.

Original: In addition, previous work [Ding et al., 2021] finds that OPD is a better metric at detecting changes that matter functionally and robust against changes that do not matter.
Revised: In addition, previous work [Ding et al., 2021] finds that OPD is a better metric at detecting changes that matter functionally and robust against changes that do not matter.

Original: OPD: Formally, the Orthogonal Procrustes Distance is the smallest distance between two matrices X and Y (with columns as the vectors in question) found by finding the orthogonal matrix Q which most closely maps A to B .
Revised: OPD: Formally, the Orthogonal Procrustes Distance is the smallest distance between two matrices X and Y (with columns as the vectors in question) found by finding the orthogonal matrix Q which most closely maps A to B .

Original: Therefore, the OPD distance is the distance value from solving the orthogonal
Revised: Therefore, the OPD distance is the distance value from solving the orthogonal

Original: Procrustes problem:
Revised: Procrustes problem:

Original: [Equation]
Revised: [Equation]

Original: 5) where ∥ · ∥ F is the Frobenius norm.
Revised: 5) where ∥ · ∥ F is the Frobenius norm.

Original: When matrices are normalized (centered and divided by their
Revised: When matrices are normalized (centered and divided by their

Original: Frobenious norm) this is called the general Procrustes problem.
Revised: Frobenious norm) this is called the general Procrustes problem.

Original: However, the closed for equation we used is the following:
Revised: However, the closed for equation we used is the following:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ∥ · ∥ ∗ is the nuclear norm, i.e. the sum of singular values (cid:80) i σ ( A ) i =
Revised: where ∥ · ∥ ∗ is the nuclear norm, i.e. the sum of singular values (cid:80) i σ ( A ) i =

Original: ∥
Revised: ∥

Original: A ∥ ∗ .
Revised: A ∥ ∗ .

Original: The division by 2 is to guarantee that the OPD distance is between [0 , 1] instead of [0 , 2] .
Revised: The division by 2 is to guarantee that the OPD distance is between [0 , 1] instead of [0 , 2] .

Original: We do the standard normalization of the matrices before computing the OPD distance – by centering and dividing by the Frobenious norm of the matrix.
Revised: We do the standard normalization of the matrices before computing the OPD distance – by centering and dividing by the Frobenious norm of the matrix.

Original: This is done because the orthogonal matrix in the orthogonal
Revised: This is done because the orthogonal matrix in the orthogonal

Original: Procrustes problem does not allow for translation or rescaling of the matrices.
Revised: Procrustes problem does not allow for translation or rescaling of the matrices.

Original: Therefore, this normalization enforces invariance to this type of transformations – i.e. we don’t want large OPD values due to rescaling or translation (and even if present, the orthogonal matrix wouldn’t be able to reflect it).
Revised: Therefore, this normalization enforces invariance to this type of transformations – i.e. we don’t want large OPD values due to rescaling or translation (and even if present, the orthogonal matrix wouldn’t be able to reflect it).

Original: Therefore, the final equation for OPD we use is:
Revised: Therefore, the final equation for OPD we use is:

Original: [Equation]
Revised: [Equation]

Original: Why OPD?
Revised: Why OPD?

Original: We use OPD due to the findings of [Ding et al., 2021].
Revised: We use OPD due to the findings of [Ding et al., 2021].

Original: They find that OPD is a more robust metric (compared to SVCCA, PWCCA, and CKA) because it is sensitive to changes that affect real functional behavior (so it detects changes to behavior that “matter") and it’s specific against changes that do not.
Revised: They find that OPD is a more robust metric (compared to SVCCA, PWCCA, and CKA) because it is sensitive to changes that affect real functional behavior (so it detects changes to behavior that “matter") and it’s specific against changes that do not.

Original: As a summary, some of the evidence that they provide for this is that OPD is able to detect when 0.75 of the principal components are removed, while CKA cannot detect removal of principal components until 0.97 are removed.
Revised: As a summary, some of the evidence that they provide for this is that OPD is able to detect when 0.75 of the principal components are removed, while CKA cannot detect removal of principal components until 0.97 are removed.

Original: CCA like metrics on the other hand are not specific – even random initialization noise overwhelms the distances it reports, while OPD is more robust to this random noise.
Revised: CCA like metrics on the other hand are not specific – even random initialization noise overwhelms the distances it reports, while OPD is more robust to this random noise.

Original: For the last point, this means that even if we compare two different layers with
Revised: For the last point, this means that even if we compare two different layers with

Original: CCA, the noise will dominate the distance reported instead of the difference caused by comparing different layer.
Revised: CCA, the noise will dominate the distance reported instead of the difference caused by comparing different layer.

Original: O.9 Correctly using Feature Based Distances
Revised: O.9 Correctly using Feature Based Distances

Original: When comparing two layers of a neural network using two layer matrices, one needs to be careful with the number of data points (or batch size) being used.
Revised: When comparing two layers of a neural network using two layer matrices, one needs to be careful with the number of data points (or batch size) being used.

Original: This is because metrics like CCA intrinsically are formulated as an optimization and if the number of examples is not larger than the number of dimensionality of the examples – then the similarity can be pathologically be perfect (e.g., the distance is zero when it’s actually not zero).
Revised: This is because metrics like CCA intrinsically are formulated as an optimization and if the number of examples is not larger than the number of dimensionality of the examples – then the similarity can be pathologically be perfect (e.g., the distance is zero when it’s actually not zero).

Original: Therefore, we follow the suggestions by the original authors of SVCCA [Raghu et al., 2017] and always use at least 10 times more examples than there are features for our feature based comparisons.
Revised: Therefore, we follow the suggestions by the original authors of SVCCA [Raghu et al., 2017] and always use at least 10 times more examples than there are features for our feature based comparisons.

Original: We call this value the safety margin .
Revised: We call this value the safety margin .

Original: To illustrate this idea, we produce two random matrices and compare how the similarity (SVCCA) values varies as a function of the dimensionality of the data and the number of points.
Revised: To illustrate this idea, we produce two random matrices and compare how the similarity (SVCCA) values varies as a function of the dimensionality of the data and the number of points.

Original: Since the two matrices are completely random, we know they should not be very similar and thus SVCCA should report a high similarity value (or low distance value).
Revised: Since the two matrices are completely random, we know they should not be very similar and thus SVCCA should report a high similarity value (or low distance value).

Original: Therefore, we can see in figure 18 how as the dimensionality increases, the similarity value approaches a perfect similarity of 1.0.
Revised: Therefore, we can see in figure 18 how as the dimensionality increases, the similarity value approaches a perfect similarity of 1.0.

Original: In figure 19 we can see how as the number of points increases, we approach a smaller similarity – closer to the true similarity for random matrices.
Revised: In figure 19 we can see how as the number of points increases, we approach a smaller similarity – closer to the true similarity for random matrices.

Original: In general, given two matrices X, Y ∈ R M ′ ,D ′ with the number of (effective) data points M ′ and (effective) dimensionality (number of features) D ′ – we want the number of points to be larger by a safety factor s .
Revised: In general, given two matrices X, Y ∈ R M ′ ,D ′ with the number of (effective) data points M ′ and (effective) dimensionality (number of features) D ′ – we want the number of points to be larger by a safety factor s .

Original: Formally, it must satisfy this inequality to avoid the pathological case for feature based distances:
Revised: Formally, it must satisfy this inequality to avoid the pathological case for feature based distances:

Original: [Equation]
Revised: [Equation]

Original: where we suggest to use s ≤ 10 (as used in previous work [Raghu et al., 2017]).
Revised: where we suggest to use s ≤ 10 (as used in previous work [Raghu et al., 2017]).

Original: Note the effective number of data points used and dimensionality can be different depending on how one reshapes the
Revised: Note the effective number of data points used and dimensionality can be different depending on how one reshapes the

Original: CNN tensors to produce layer matrices as explained in section O.6.
Revised: CNN tensors to produce layer matrices as explained in section O.6.

Original: For example, if one uses the channel as the dimensionality (i.e., use the image patches as an effective data point) then one has to obey the following inequality:
Revised: For example, if one uses the channel as the dimensionality (i.e., use the image patches as an effective data point) then one has to obey the following inequality:

Original: [Equation]
Revised: [Equation]

Original: where M is the batch size, H, W is the height and width of the images, and C is the number of filters/channels for the current layer.
Revised: where M is the batch size, H, W is the height and width of the images, and C is the number of filters/channels for the current layer.

Original: This means that for a given architecture processing images of a given size that the only parameter we can change to make the above inequality true is the batch size
Revised: This means that for a given architecture processing images of a given size that the only parameter we can change to make the above inequality true is the batch size

Original: M .
Revised: M .

Original: P Summary of Compute Required
Revised: P Summary of Compute Required

Original: We used an internal compute cluster with wide varied of GPUs.
Revised: We used an internal compute cluster with wide varied of GPUs.

Original: We used Titan X GPUs for most five layer CNN experiments.
Revised: We used Titan X GPUs for most five layer CNN experiments.

Original: We used A40 and dgx-A100 GPUs for Resnet12 experiments, with
Revised: We used A40 and dgx-A100 GPUs for Resnet12 experiments, with

Original: GB and 40 GB GPU memory respectively.
Revised: GB and 40 GB GPU memory respectively.

Original: We did notice that the Resnet12 architecture we used from previous work [Tian et al., 2020] required more memory than Resnet18 and Resnet34 used in
Revised: We did notice that the Resnet12 architecture we used from previous work [Tian et al., 2020] required more memory than Resnet18 and Resnet34 used in

Original: Task2Vec
Revised: Task2Vec

Original: [Achille UCLA et al., 2019].
Revised: [Achille UCLA et al., 2019].

Original: By requiring more memory, we mean we did not have many memory out of bound issues with Resnet18/Resnet34, but did have memory issues with Resnet12.
Revised: By requiring more memory, we mean we did not have many memory out of bound issues with Resnet18/Resnet34, but did have memory issues with Resnet12.

Original: In addition, our episodic meta-learning training for MAML used Learn2Learn’s
Revised: In addition, our episodic meta-learning training for MAML used Learn2Learn’s

Original: ["Arnold et al., 2020] distributed training to speed up experiments.
Revised: ["Arnold et al., 2020] distributed training to speed up experiments.

Original: Experiments took 1-2 weeks with MAML in a single
Revised: Experiments took 1-2 weeks with MAML in a single

Original: GPU to potentially 2-3 days with multiple GPUs (we used 2, 4 to 8 GPUs depending on availability).
Revised: GPU to potentially 2-3 days with multiple GPUs (we used 2, 4 to 8 GPUs depending on availability).

Original: For synthetic experiments, we used Titan X GPUs with 16GB of GPU memory.
Revised: For synthetic experiments, we used Titan X GPUs with 16GB of GPU memory.

Original: Experiments took around 1-2 days on average with a single GPU.
Revised: Experiments took around 1-2 days on average with a single GPU.

Original: For more precision, check the experimental details section F.
Revised: For more precision, check the experimental details section F.

Original: Q A Statistical Decision view of the differences between Supervised Learning and Meta-learning
Revised: Q A Statistical Decision view of the differences between Supervised Learning and Meta-learning

Original: Recent work in meta-learning implies that feature-reuse might be all we need to solve modern fewshot learning benchmarks [Tian et al., 2020].
Revised: Recent work in meta-learning implies that feature-reuse might be all we need to solve modern fewshot learning benchmarks [Tian et al., 2020].

Original: However, what it also reveals is our poor understanding of meta-learning algorithms.
Revised: However, what it also reveals is our poor understanding of meta-learning algorithms.

Original: Therefore, in this section, we take the most foundational perspective to formulate and analyze meta-learning algorithms by analyzing them from an optimal statistical decision theory perspective?
Revised: Therefore, in this section, we take the most foundational perspective to formulate and analyze meta-learning algorithms by analyzing them from an optimal statistical decision theory perspective?

Original: We hope that this can help clarify the results from [Tian et al., 2020] and therefore help meta-learning researchers design better meta-learning benchmarks and meta-learning algorithms.
Revised: We hope that this can help clarify the results from [Tian et al., 2020] and therefore help meta-learning researchers design better meta-learning benchmarks and meta-learning algorithms.

Original: Q.1 Supervised Meta-Learning problem set-up
Revised: Q.1 Supervised Meta-Learning problem set-up

Original: In this section, we introduce the notation for supervised meta-learning.
Revised: In this section, we introduce the notation for supervised meta-learning.

Original: Intuitively, we seek to find a function that minimizes the expected risk over tasks.
Revised: Intuitively, we seek to find a function that minimizes the expected risk over tasks.

Original: To formalize it, we will use three formulations:
Revised: To formalize it, we will use three formulations:

Original: Monolithic meta-learner : for a monolithic decision rule g (or meta-learner), we want to find the optimal g by minimizing the supervised meta-learning expected risk :
Revised: Monolithic meta-learner : for a monolithic decision rule g (or meta-learner), we want to find the optimal g by minimizing the supervised meta-learning expected risk :

Original: [Equation]
Revised: [Equation]

Original: where g is a single monolithic function, p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ
Revised: where g is a single monolithic function, p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ

Original: and ( x, y ) is the data pair of input and target value sampled from a task.
Revised: and ( x, y ) is the data pair of input and target value sampled from a task.

Original: Meta-learned meta-learner : for a meta-learned decision rule we usually have an adaptation rule A (e.g. SGD in MAML) and a function approximator h
Revised: Meta-learned meta-learner : for a meta-learned decision rule we usually have an adaptation rule A (e.g. SGD in MAML) and a function approximator h

Original: (e.g. a neural network) and minimize the follow over both:
Revised: (e.g. a neural network) and minimize the follow over both:

Original: [Equation]
Revised: [Equation]

Original: (11) p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ
Revised: (11) p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ

Original: and ( x, y ) is the data pair of input and target value sampled from a task.
Revised: and ( x, y ) is the data pair of input and target value sampled from a task.

Original: Fixed representation meta-learner without adaptation : one can also solve 10 using a single decision rule f that does not take the task τ as input as follows:
Revised: Fixed representation meta-learner without adaptation : one can also solve 10 using a single decision rule f that does not take the task τ as input as follows:

Original: [Equation]
Revised: [Equation]

Original: where f is a function to be adapted (e.g. a neural network), p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ
Revised: where f is a function to be adapted (e.g. a neural network), p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ

Original: and ( x, y ) is the data pair of input and target value sampled from a task.
Revised: and ( x, y ) is the data pair of input and target value sampled from a task.

Original: Fixed representation meta-learner with a final adaptation layer : one can also solve 10 using a single feature extractor g that does not take the task τ as input with a feature extractor g :
Revised: Fixed representation meta-learner with a final adaptation layer : one can also solve 10 using a single feature extractor g that does not take the task τ as input with a feature extractor g :

Original: [Equation]
Revised: [Equation]

Original: where g is the feature extractor from the raw inputs (e.g. a neural network), f the final layer adapted (e.g. a linear layer), p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ
Revised: where g is the feature extractor from the raw inputs (e.g. a neural network), f the final layer adapted (e.g. a linear layer), p ( τ ) is the true but unknown distribution of tasks, p ( x, y | τ ) is the true, but unknown distribution of data pair given a task τ

Original: and ( x, y ) is the data pair of input and target value sampled from a task.
Revised: and ( x, y ) is the data pair of input and target value sampled from a task.

Original: Remark Q.1 .
Revised: Remark Q.1 .

Original: Note that in practice, the meta-learner does not usually take the full task τ as input, but instead a train and test set (often referred to as support set and query set) sampled from the task τ .
Revised: Note that in practice, the meta-learner does not usually take the full task τ as input, but instead a train and test set (often referred to as support set and query set) sampled from the task τ .

Original: The goal of this work is to clarify the difference between 12 and 11 under the framework of statistical decision theory.
Revised: The goal of this work is to clarify the difference between 12 and 11 under the framework of statistical decision theory.

Original: Arguably the most important comparison between 11 and 13 is left for future work.
Revised: Arguably the most important comparison between 11 and 13 is left for future work.

Original: Q.2 Main Result: Difference between the Supervised Learned and Meta-learned decision rule
Revised: Q.2 Main Result: Difference between the Supervised Learned and Meta-learned decision rule

Original: The proof sketch is as follows: we first show the optimal decision rules for both supervised learning and meta-learning when minimizing the expected meta-risk from equations 12 and 11 and then highlight that the main difference between them is that the meta-learned solution can act optimally if it identifies the task τ while the supervised learned solution has no capabilities of this since it learns an average based on tasks instead.
Revised: The proof sketch is as follows: we first show the optimal decision rules for both supervised learning and meta-learning when minimizing the expected meta-risk from equations 12 and 11 and then highlight that the main difference between them is that the meta-learned solution can act optimally if it identifies the task τ while the supervised learned solution has no capabilities of this since it learns an average based on tasks instead.

Original: Theorem Q.2.
Revised: Theorem Q.2.

Original: The minimizer to equation 11 is:
Revised: The minimizer to equation 11 is:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ¯ y ∗ y | x,τ =
Revised: where ¯ y ∗ y | x,τ =

Original: E y ∼ p ( y | x,τ ) [ y ] and l is the squared loss l (ˆ y, y ) =
Revised: E y ∼ p ( y | x,τ ) [ y ] and l is the squared loss l (ˆ y, y ) =

Original: (ˆ y − y ) 2 .
Revised: (ˆ y − y ) 2 .

Original: Proof.
Revised: Proof.

Original: The proof is the same as the standard decision rule textbook proof but instead of minimizing it point-wise w.r.t.
Revised: The proof is the same as the standard decision rule textbook proof but instead of minimizing it point-wise w.r.t.

Original: x we minimize it point-wise w.r.t.
Revised: x we minimize it point-wise w.r.t.

Original: ( x, τ ) .
Revised: ( x, τ ) .

Original: In particular, we have:
Revised: In particular, we have:

Original: [Equation]
Revised: [Equation]

Original: without loss of generality (WLOG) and for clarity of exposition consider the special case for discrete variables:
Revised: without loss of generality (WLOG) and for clarity of exposition consider the special case for discrete variables:

Original: [Equation]
Revised: [Equation]

Original: At this point we notice we can minimize the above point-wise w.r.t ( x, τ ) and ignore h .
Revised: At this point we notice we can minimize the above point-wise w.r.t ( x, τ ) and ignore h .

Original: To do that, take the derivative of R ( A, h ) with respect to A ( h, τ )( x ) because that A ( h, τ )( x ) ∈ R and set it to zero:
Revised: To do that, take the derivative of R ( A, h ) with respect to A ( h, τ )( x ) because that A ( h, τ )( x ) ∈ R and set it to zero:

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: Corollary Q.3.
Revised: Corollary Q.3.

Original: For a monolithic meta-learner defined in section Q.1 the solution to the meta supervised learning problem is the same as in equation 14 for the squared loss l (ˆ y, y )
Revised: For a monolithic meta-learner defined in section Q.1 the solution to the meta supervised learning problem is the same as in equation 14 for the squared loss l (ˆ y, y )

Original: =
Revised: =

Original: (ˆ y − y )
Revised: (ˆ y − y )

Original: 2 i.e. g ( τ, x ) =
Revised: 2 i.e. g ( τ, x ) =

Original: ¯ y ∗ y | x,τ =
Revised: ¯ y ∗ y | x,τ =

Original: E y ∼ p ( y | x,τ ) [ y ] .
Revised: E y ∼ p ( y | x,τ ) [ y ] .

Original: Proof.
Revised: Proof.

Original: Proof is trivial, replace A ( h, τ )( x ) with g ( τ, x ) since h is not used.
Revised: Proof is trivial, replace A ( h, τ )( x ) with g ( τ, x ) since h is not used.

Original: In this case, there is no difference with having an adaptation rule A equipped with another function h and a monolithic meta-learner g .
Revised: In this case, there is no difference with having an adaptation rule A equipped with another function h and a monolithic meta-learner g .

Original: Theorem Q.4.
Revised: Theorem Q.4.

Original: The minimizer to equation 12:
Revised: The minimizer to equation 12:

Original: [Equation]
Revised: [Equation]

Original: (15) where ¯ y ∗ y | x,τ = E y ∼ p ( y | x,τ ) [ y ] and l is the squared loss l (ˆ y, y ) =
Revised: (15) where ¯ y ∗ y | x,τ = E y ∼ p ( y | x,τ ) [ y ] and l is the squared loss l (ˆ y, y ) =

Original: (ˆ y − y ) 2 .
Revised: (ˆ y − y ) 2 .

Original: Proof.
Revised: Proof.

Original: WLOG, consider the minimizer of equation 12 in the discrete case.
Revised: WLOG, consider the minimizer of equation 12 in the discrete case.

Original: In particular, we have:
Revised: In particular, we have:

Original: [Equation]
Revised: [Equation]

Original: Note we can minimize the above point-wise w.r.t.
Revised: Note we can minimize the above point-wise w.r.t.

Original: x only (and not also w.r.t.
Revised: x only (and not also w.r.t.

Original: τ as we did in proof
Revised: τ as we did in proof

Original: Q.2).
Revised: Q.2).

Original: Thus, we have want:
Revised: Thus, we have want:

Original: [Equation]
Revised: [Equation]

Original: at this point it is interesting to observe the disadvantage of supervised learning methods with fixed functions without dependence on the task is that they are forced to consider all task τ at once.
Revised: at this point it is interesting to observe the disadvantage of supervised learning methods with fixed functions without dependence on the task is that they are forced to consider all task τ at once.

Original: We proceed to take derivatives as in proof Q.2 but with this objective:
Revised: We proceed to take derivatives as in proof Q.2 but with this objective:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We proceed by noticing that E τ ∼ p ( τ ) [ p ( x | τ )]
Revised: We proceed by noticing that E τ ∼ p ( τ ) [ p ( x | τ )]

Original: = p ( x ) , thus:
Revised: = p ( x ) , thus:

Original: [Equation]
Revised: [Equation]

Original: as required by the rightmost RHS of equation 15.
Revised: as required by the rightmost RHS of equation 15.

Original: Theorem Q.5.
Revised: Theorem Q.5.

Original: The minimizer in equation 15 reduces to an expectation only over w.r.t.
Revised: The minimizer in equation 15 reduces to an expectation only over w.r.t.

Original: p ( τ ) of ¯ y ∗ y | x,τ under benchmarks that are balanced.
Revised: p ( τ ) of ¯ y ∗ y | x,τ under benchmarks that are balanced.

Original: Formally
Revised: Formally

Original: [Equation]
Revised: [Equation]

Original: where ¯ y ∗ y | x,τ =
Revised: where ¯ y ∗ y | x,τ =

Original: E y ∼ p ( y | x,τ ) [ y ] and under assumption A1: p ( x | τ ) is a constant, i.e. p ( x | τ ) =
Revised: E y ∼ p ( y | x,τ ) [ y ] and under assumption A1: p ( x | τ ) is a constant, i.e. p ( x | τ ) =

Original: k XT ∈ R , ∀ x ∈ X, ∀ τ ∈ T and l is the squared loss l (ˆ y, y )
Revised: k XT ∈ R , ∀ x ∈ X, ∀ τ ∈ T and l is the squared loss l (ˆ y, y )

Original: =
Revised: =

Original: (ˆ y − y ) 2 .
Revised: (ˆ y − y ) 2 .

Original: Proof.
Revised: Proof.

Original: Recall equation 15:
Revised: Recall equation 15:

Original: [Equation]
Revised: [Equation]

Original: due to Bayes’s rule we have p ( τ | x ) =
Revised: due to Bayes’s rule we have p ( τ | x ) =

Original: p ( τ ) p ( x | τ ) p ( x ) and equation 15 can be re-written as follows:
Revised: p ( τ ) p ( x | τ ) p ( x ) and equation 15 can be re-written as follows:

Original: [Equation]
Revised: [Equation]

Original: under assumption A1 we have that p ( x | τ ) does not depend on as a function of x or τ .
Revised: under assumption A1 we have that p ( x | τ ) does not depend on as a function of x or τ .

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: Thus we have:
Revised: Thus we have:

Original: [Equation]
Revised: [Equation]

Original: as required.
Revised: as required.

Original: Remark Q.6 .
Revised: Remark Q.6 .

Original: Note that assumption A1 holds for the common MiniImagenet few-shot learning data set, where p ( x | τ ) = 1600 .
Revised: Note that assumption A1 holds for the common MiniImagenet few-shot learning data set, where p ( x | τ ) = 1600 .

Original: Remark Q.7 .
Revised: Remark Q.7 .

Original: In addition, because all classes are equally likely (e.g. p ( class ) = 164 for the meta-train set)
Revised: In addition, because all classes are equally likely (e.g. p ( class ) = 164 for the meta-train set)

Original: we have p ( τ ) is the same constant independent of the task τ .
Revised: we have p ( τ ) is the same constant independent of the task τ .

Original: Proof in the appendix, lemma Q.8.
Revised: Proof in the appendix, lemma Q.8.

Original: Theorem Q.8.
Revised: Theorem Q.8.

Original: If the tasks are equally likely, then equation 17 becomes an average over conditional predictions over all tasks.
Revised: If the tasks are equally likely, then equation 17 becomes an average over conditional predictions over all tasks.

Original: Formally, if p ( τ ) = 1 T then equation 17 becomes:
Revised: Formally, if p ( τ ) = 1 T then equation 17 becomes:

Original: [Equation]
Revised: [Equation]

Original: under the squared loss l (ˆ y, y )
Revised: under the squared loss l (ˆ y, y )

Original: =
Revised: =

Original: (ˆ y − y ) 2 .
Revised: (ˆ y − y ) 2 .

Original: Proof.
Revised: Proof.

Original: Since f ( x ) =
Revised: Since f ( x ) =

Original: E τ ∼ p ( τ ) (cid:104)
Revised: E τ ∼ p ( τ ) (cid:104)

Original: ¯ y ∗ y | x,τ (cid:105) then, plugging p ( τ ) =
Revised: ¯ y ∗ y | x,τ (cid:105) then, plugging p ( τ ) =

Original: 1 T completes proof.
Revised: 1 T completes proof.

Original: Remark Q.9 .
Revised: Remark Q.9 .

Original: It is interesting to note that without adaptation or dependence on the task τ being solved, the supervised learned meta-learner is suboptimal compared to the meta-learned solution.
Revised: It is interesting to note that without adaptation or dependence on the task τ being solved, the supervised learned meta-learner is suboptimal compared to the meta-learned solution.

Original: The proof is simple, and it follows because the meta-learned decision rule was chosen to minimize each term individually, but the supervised learned decision is not of that form.
Revised: The proof is simple, and it follows because the meta-learned decision rule was chosen to minimize each term individually, but the supervised learned decision is not of that form.

Original: Proof in appendix
Revised: Proof in appendix

Original: Q.11.
Revised: Q.11.

Original: Unfortunately, note that this does not necessarily apply to previous work [Tian et al., 2020].
Revised: Unfortunately, note that this does not necessarily apply to previous work [Tian et al., 2020].

Original: Remark Q.10 .
Revised: Remark Q.10 .

Original: Note that remark Q.9 does not apply to work [Tian et al., 2020] because that work does depend on a task τ during meta-test time by adapting the final layer even if the representation is fixed.
Revised: Note that remark Q.9 does not apply to work [Tian et al., 2020] because that work does depend on a task τ during meta-test time by adapting the final layer even if the representation is fixed.

Original: Remark Q.11 .
Revised: Remark Q.11 .

Original: The supervised learning decision rule is suboptimal compared to the meta-learned decision rule.
Revised: The supervised learning decision rule is suboptimal compared to the meta-learned decision rule.

Original: Q.3 The supervised Learning Solution is equivalent to the Meta-Learning solution when there is low task diversity
Revised: Q.3 The supervised Learning Solution is equivalent to the Meta-Learning solution when there is low task diversity

Original: Sketch argument: The main idea is that because all tasks are very similar (task diversity is low) – it essentially means that τ is not truly an input to the adaptation rule or monolithic meta-learner).
Revised: Sketch argument: The main idea is that because all tasks are very similar (task diversity is low) – it essentially means that τ is not truly an input to the adaptation rule or monolithic meta-learner).

Original: Equivalently, the problem is essentially a single task problem, so the task is implicitly an input to any method used.
Revised: Equivalently, the problem is essentially a single task problem, so the task is implicitly an input to any method used.

Original: Therefore, since the task conditioning does not exist, then the optimization problem is the same for the meta-learned solution and when there is a fixed supervised learning feature extractor.
Revised: Therefore, since the task conditioning does not exist, then the optimization problem is the same for the meta-learned solution and when there is a fixed supervised learning feature extractor.

Original: Theorem Q.12.
Revised: Theorem Q.12.

Original: Assume τ 1 = τ 2 for any tasks in T and the data sets are balanced (i.e. same number of images x for each task).
Revised: Assume τ 1 = τ 2 for any tasks in T and the data sets are balanced (i.e. same number of images x for each task).

Original: Then we have the meta-learned solution is the same as the supervised learning solution with shared embeddings: f sl ( x )
Revised: Then we have the meta-learned solution is the same as the supervised learning solution with shared embeddings: f sl ( x )

Original: =
Revised: =

Original: A ( f ml , τ )( x ) .
Revised: A ( f ml , τ )( x ) .

Original: Proof.
Revised: Proof.

Original: Consider the optimization problem, for supervised learning:
Revised: Consider the optimization problem, for supervised learning:

Original: [Equation]
Revised: [Equation]

Original: If every pair of tasks is equal, it means their distributions are equal p ( x, y | τ )
Revised: If every pair of tasks is equal, it means their distributions are equal p ( x, y | τ )

Original: =
Revised: =

Original: p ( x, y ) (meaning τ can be ignored).
Revised: p ( x, y ) (meaning τ can be ignored).

Original: Thus, the solution to the supervised learning problem is: f sl ( x ) =
Revised: Thus, the solution to the supervised learning problem is: f sl ( x ) =

Original: E τ E p ( x,y )
Revised: E τ E p ( x,y )

Original: [ y ] =
Revised: [ y ] =

Original: E p ( x,y )
Revised: E p ( x,y )

Original: [ y ] =
Revised: [ y ] =

Original: y ∗| x .
Revised: y ∗| x .

Original: Now for the meta-learning problem we have: A ( f ml , τ )( x ) =
Revised: Now for the meta-learning problem we have: A ( f ml , τ )( x ) =

Original: y ∗| x,τ =
Revised: y ∗| x,τ =

Original: E y ∼ p ( y | x,τ ) [ y ] but due to every pair of tasks being equal means p ( x, y | τ ) =
Revised: E y ∼ p ( y | x,τ ) [ y ] but due to every pair of tasks being equal means p ( x, y | τ ) =

Original: p ( x, y ) (i.e. all task share the same distributions)
Revised: p ( x, y ) (i.e. all task share the same distributions)

Original: we have: A ( f ml , τ )( x ) =
Revised: we have: A ( f ml , τ )( x ) =

Original: E y ∼ p ( y | x,τ ) [ y ] =
Revised: E y ∼ p ( y | x,τ ) [ y ] =

Original: E y ∼ p
Revised: E y ∼ p

Original: ( y | x [ y ] =
Revised: ( y | x [ y ] =

Original: y ∗| x which is the same as the solution as in f sl .
Revised: y ∗| x which is the same as the solution as in f sl .

Original: Thus f sl ( x ) =
Revised: Thus f sl ( x ) =

Original: A ( f ml , τ )( x ) .
Revised: A ( f ml , τ )( x ) .

Original: Remark Q.13 .
Revised: Remark Q.13 .

Original: Proofs were presented in the discrete case clearly, but it is trivial to expand them to the continuous case – e.g., using integrals instead of summations.
Revised: Proofs were presented in the discrete case clearly, but it is trivial to expand them to the continuous case – e.g., using integrals instead of summations.