P UTTING T HEORY TO W ORK : F ROM L EARNING B OUNDS TO M ETA -L EARNING A LGORITHMS
Anonymous authors
Paper under double-blind review
A BSTRACT
Most of existing deep learning models rely on excessive amounts of labeled training data in order to achieve state-of-the-art results, even though these data can be hard or costly to get in practice.
One attractive alternative is to learn with little supervision, commonly referred to as few-shot learning (FSL), and, in particular, meta-learning that learns to learn with few data from related tasks.
Despite the practical success of meta-learning, many of its algorithmic solutions proposed in the literature are based on sound intuitions, but lack a solid theoretical analysis of the expected performance on the test task.
In this paper, we review the recent advances in meta-learning theory and show how they can be used in practice both to better understand the behavior of popular meta-learning algorithms and to improve their generalization capacity.
This latter is achieved by integrating the theoretical assumptions ensuring efﬁcient meta-learning in the form of regularization terms into several popular meta-learning algorithms for which we provide a large study of their behavior on classic few-shot classiﬁcation benchmarks.
To the best of our knowledge, this is the ﬁrst contribution that puts the most recent learning bounds of meta-learning theory into practice for the popular task of few-shot classiﬁcation.
I NTRODUCTION
Since the very seeding of the machine learning ﬁeld, its algorithmic advances were inevitably followed or preceded by the accompanying theoretical analyses establishing the conditions required for the corresponding algorithms to learn well.
Such a synergy between theory and practice is reﬂected in numerous concepts and learning strategies that took their origins in the statistical learning theory: for instance, the famous regularized risk minimization approach is directly related to the minimization of the complexity of the hypothesis space, as suggested by the generalization bounds established for supervised learning (Vapnik, 1992), while most of the adversarial algorithms in transfer learning ( e.g. , DANN from (Ganin & Lempitsky, 2015)) follow the theoretical insights provided by the seminal theory of its domain (Ben-David et al., 2010).
Even though many machine learning methods now enjoy a solid theoretical justiﬁcation, some more recent advances in the ﬁeld are still in their preliminary state which requires the hypotheses put forward by the theoretical studies to be implemented and veriﬁed in practice.
One such notable example is the emerging ﬁeld of meta-learning , also called learning to learn (LTL), where the goal is to produce a model on data coming from a set of (meta-train) source tasks to use it as a starting point for learning successfully a new previously unseen (meta-test) target task with little supervision.
This kind of approach comes in particularly handy when training deep learning models as their performance crucially depends on the amount of training data that can be difﬁcult and/or expensive to get in some applications.
Several theoretical studies (Baxter, 2000; Pentina & Lampert, 2014;
Maurer et al., 2016; Amit & Meir, 2018;
Yin et al., 2020) 1 provided probabilistic meta-learning bounds that require the amount of data in the meta-train source task and the number of meta-train tasks to tend to inﬁnity for efﬁcient meta-learning.
While capturing the underlying general intuition, these bounds do not suggest that all the source data is useful in such learning setup due to the additive relationship between the two terms mentioned above.
To tackle this drawback, two very recent studies (Du et al., 2020; Tripuraneni et al., 2020) aimed at ﬁnding deterministic assumptions that lead to faster learning rates allowing meta-learning algorithms to beneﬁt from all the source data.
Contrary to probabilistic bounds that have been used to derive novel learning strategies for meta-learning algorithms (Amit & Meir, 2018; Yin et al., 2020), there was no attempt to verify the validity of the assumptions leading to the fastest known learning rates in practice or to enforce them through an appropriate optimization procedure.
In this paper, we bridge the meta-learning theory with practice by harvesting the theoretical results from Tripuraneni et al.
(2020) and Du et al.
(2020), and by showing how they can be implemented algorithmically and integrated, when needed, to popular existing meta-learning algorithms used for few-shot classiﬁcation (FSC).
This latter task consists in classifying new data having seen only few training examples, and represents one of the most prominent examples where meta-learning has shown to be highly efﬁcient.
More precisely, our contributions are three-fold:
We identify two common assumptions from the theoretical works on meta-learning and show how they can be veriﬁed and forced via a novel regularization scheme.
We investigate whether these assumptions are satisﬁed for popular meta-learning algorithms and observe that some of them naturally satisfy them, while others do not.
With the proposed regularization strategy, we show that enforcing the assumptions to be valid in practice leads to better generalization of the considered algorithms.
The rest of the paper is organized as follows.
After presenting preliminary knowledge on the metalearning problem in Section 2, we detail the existing meta-learning theoretical results with their corresponding assumptions and show how they can be enforced via a novel regularization technique in Section 3.
Then, we provide an experimental evaluation of several state-of-the-art FSL methods in Section 4 and highlight the different advantages brought by the proposed regularization technique in practice.
Finally, we conclude and outline the future research perspectives in Section 5.
P RELIMINARY K NOWLEDGE
We start by formally deﬁning the meta-learning problem following the model described in Du et al.
To this end, we assume having access to T source tasks characterized by their respective data generating distributions { µ t } Tt =1 supported over the joint input-output space
X
× Y with X ⊆ R d and Y ⊆ R .
We further assume that these distributions are observed only through ﬁnite size samples of size n 1 grouped into matrices X t = ( x t, 1 , . . .
, x t,n 1 ) ∈ R n 1 × d and vectors of outputs y t =
( y t, 1 , . . .
, y t,n 1 ) ∈ R n 1 , ∀ t ∈
[[ T ]] := { 1 , . .
, T } .
Given this set of tasks, our goal is to learn a shared representation φ belonging to a certain class of functions Φ := { φ | φ : X → V , V
⊆ R k } and linear predictors w t ∈ R k , ∀ t ∈
[[ T ]] grouped in a matrix W ∈ R T × k .
More formally, this is done by solving the following optimization problem:
[Equation]
where (cid:96) :
Y × Y → R + is a loss function.
Once such a representation is learned, we want to apply it to a new previously unseen target task observed through a pair ( X T +1 ∈ R n 2 × d , y T +1 ∈ R n 2 ) containing n 2 samples generated by the distribution
µ T +1 .
We expect that a linear classiﬁer w learned on top of the obtained representation leads to a low true risk over the whole distribution µ T
+1 .
More precisely, we ﬁrst use ˆ φ to solve the following problem:
[Equation]
Then, we deﬁne the true target risk of the learned linear classiﬁer ˆ w T +1 as:
[Equation]
and want it to be as low and as close as possible to the ideal true risk L ( φ ∗ , w ∗ T +1 ) where w ∗ T +1 and φ ∗ satisfy: ∀ t ∈
[Equation] T + 1]] and [Equation] x , y ) ∼
[Equation]
Equivalently, most of the works found in the literature analyze the excess risk deﬁned as
ER (ˆ φ, ˆ w T +1 ) :
= L (ˆ φ, ˆ w T +1 ) − L ( φ ∗ , w ∗ T +1 ) and seek to upper-bound it in terms of quantities involved in the learning process. {+Remark 
+}
We note that many popular meta-learning algorithms used to solve the few-shot classiﬁcation task do not follow exactly the approach described above.
However, we believe that the exact way of how this is done algorithmically (with or without the support set, with or without learning episodes) does not change the statistical learning challenge of it which is to learn a model that can provably generalize with little supervision.
Traditional statistical learning theory tells us that the generalization in this case will be poor (not enough target data and impossible to rely on data coming from different probability distributions), while the theoretical works we built upon suggest that source data may contribute in improving the generalization of the learned model alongside the target data if the assumptions described below are satisﬁed.
F ROM T HEORY TO P RACTICE
In this section, we highlight the main theoretical contributions that provably ensure the success of meta-learning in improving the performance on the previously unseen meta-test task with the increasing number of meta-train tasks and the amount of data available for them.
We then concentrate our attention on the most recent theoretical advances leading to the fastest learning rates and show how the assumptions used to obtain them can be forced in practice through a novel regularization strategy.
3. W HEN DOES M ETA - LEARNING P ROVABLY W ORK ?
One critical requirement for meta-learning to succeed in few-shot classiﬁcation task is that a representation learned on meta-train data should be useful for learning a good predictor on the meta-test data set.
Such a requirement is reﬂected by the fact that the excess target risk is bounded by a quantity that involves the number of samples in both meta-train and meta-test samples and the number of available meta-train tasks.
To this end, ﬁrst studies in the context of meta-learning relied on probabilistic assumption (Baxter, 2000; Pentina & Lampert, 2014; Maurer et al., 2016; Amit & Meir, 2018;
Yin et al., 2020) stating that meta-train and meta-test tasks distributions are all sampled i.i.d.
from the same random distribution.
Intuitively, this means that source and target tasks are independent, which does not reﬂect real-world applications of few-shot learning where the former are often different draws (without replacement) from the same dataset.
Under this unrealistic assumption, the above-mentioned works obtained the bounds having the following form:
[Equation]
Such a guarantee implies that even with the increasing number of source data, one would still have to increase the number of tasks as well, in order to draw the second term to 0 .
A natural improvement to this bound was then proposed by Du et al.
(2020) and Tripuraneni et al.
(2020) that obtained the bounds on the excess risk behaving as
[Equation]
and [Equation] respectively, where k (cid:28) d is the dimensionality of the learned representation and ˜ O ( · ) hides logarithmic factors.
Both these results show that all the source and target samples are useful in minimizing the excess risk.
Thus, in the FSL regime where target data is scarce, all source data helps to learn well.
From a set of assumptions made by the authors in both of these works 2 , we note the following two: Satisﬁed assumptions
Violated assumptions 𝐰 " 𝐰𝐰 $ 𝑅 & ≈ 𝜎 *+, 𝜎 *./
Source tasks 𝐖= [𝐰 " ,𝐰 4 ,𝐰 $ ] 𝟏 / || 𝐰 "
|| 𝟏 / ||
𝐰 " || 𝟏 / || 𝐰 4 || 𝟏 / || 𝐰 4 ||
𝟏/||𝐰 $ || 𝟏/||𝐰
$ ||
Target tasks 𝑅 " ≫ 1 𝜎 &’( 𝜎 &*+
Source tasks
Target tasks 𝐖 =[𝐰 0 ,𝐰 2 ,𝐰 3 ] 𝐰𝐰𝐰𝟏 / ||
𝐰 0 || 𝟏 / ||
{+𝐰 
+}
|| 𝟏 / ||
𝐰 2 || 𝟏 / ||
𝐰 2 || 𝟏 / ||
𝐰 3 || 𝟏 / ||
𝐰 3 || Assumption 1. The matrix of optimal predictors W ∗ should cover all the directions in R k evenly.
More formally, this can be stated as
[Equation]
where σ i ( · ) denotes the i th singular value of W ∗ .
As pointed out by the authors, such an assumption can be seen as a certain measure of diversity between the source tasks that are expected to be complementary to each other in order to provide a meaningful representation for a previously unseen target task.
Assumption 2.
The norm of the optimal predictors w ∗ should not increase with the number of tasks seen during meta-training 3 .
This assumption roughly boils down to saying that the classiﬁcation margin of linear predictors should remain constant thus avoiding over- or under-specialization to the seen tasks.
While being highly insightful, the authors did not provide any experimental evidence suggesting that verifying these assumptions in practice helps to learn more efﬁciently in the considered learning setting.
To bridge this gap, we propose a general regularization scheme that allows to enforce these assumptions when learning the matrix of predictors in several popular meta-learning algorithms.
3. P UTTING T HEORY TO W ORK
Below, we propose a regularization strategy based on the theoretical insights derived from the advances in the meta-learning theory described above.
The intuition behind these regularization terms is illustrated in Figure 1.
Ensuring assumption 1.
As many other theoretical results in the statistical learning literature, the assumption given in Eq.
3 is stated for the true optimal matrix of the linear predictors W ∗ which remains unknown in practice.
However, one can assume that the meta-learning process leads to a consistent estimation of W ∗ and expect from the output matrix W to be close to the latter and thus, to satisfy the same assumptions too.
To this end, we propose to compute the singular values of the matrix of predictors W during the meta-training stage and follow its evolution during the learning episodes.
In practice, this can be done by performing the Singular Value Decomposition (SVD) on the matrix W ∈ R T × k with a computational cost of O ( Tk 2 ) ﬂoating-point operations (ﬂop).
However, as T is typically quite large, we propose a more computationally efﬁcient solution that is to take into account only the last batch of N predictors (with N (cid:28) T ) grouped in the matrix
W N ∈ R N × k that capture the latest dynamics in the learning process.
Furthermore, we note that
[Equation]
[Equation] ]] , implying that we can calculate the SVD of W N W (cid:62) N (or W (cid:62) N W N for k ≤ N ) and retrieve the singular values from it afterwards.
With this computational strategy in hand, we now want to verify whether the optimal linear predictors w t are spread out to cover all directions in the embedding space by tracking the evolution of the ratio of singular values during the training process deﬁned as
[Equation]
For the sake of conciseness, we use R σ instead of R σ ( W N ) thereafter.
According to the theory, we expect R σ to decrease gradually during the training thus improving the generalization capacity of the learned predictors and preparing them for the target task.
When we want to enforce such a behavior in practice, we propose to use R σ as a regularization term in the training loss of popular meta-learning algorithms.
Alternatively, as the smallest singular value σ N ( W N ) can be arbitrarily close to 0 and lead to numerical errors, we propose a more convenient replacement of R σ given by the entropy of the vector of singular values deﬁned as follows:
[Equation]
where σ ( W N ) is the vector of eigenvalues of W N and softmax( σ ( W N ))
i is the i th output of the softmax function.
Analogously to R σ , we write H σ instead of H σ ( W N ) from now on.
Since the distribution with the highest entropy is the uniform distribution, adding R σ or − H σ as a regularization term leads to a better coverage of R k by ensuring a nearly identical importance regardless of the direction.
Ensuring assumption 2.
In addition to the full coverage of the embedding space by the linear predictors, the meta-learning theory assumes that the norm of the linear predictors does not increase with the number of tasks seen during meta-training, i.e. , (cid:107) w (cid:107)
= O (1) or, equivalently, (cid:107) W (cid:107) 2 F =
O ( T ) .
If this assumption does not hold in practice, we propose to regularize the norm of linear predictors during training or directly normalize the obtained linear predictors ¯ w = w (cid:107) w (cid:107) 2 .
The ﬁnal meta-training loss with the theory-inspired regularization terms is given as:
[Equation]
and depending on the considered algorithm, we can replace R σ by − H σ and/or replace w t by ¯ w t instead of regularizing with (cid:107) W N (cid:107) 2 F .
To the best of our knowledge, such regularization terms based on insights from the advances in metalearning theory have never been used in the literature before.
We also further use the basic quantities involved in the proposed regularization terms as indicators of whether a given meta-learning algorithm naturally satisﬁes the assumptions ensuring an efﬁcient meta-learning in practice or not.
3. R ELATED WORK
Below, we discuss several related studies aiming at improving the general understanding of metalearning, and mention other regularization terms speciﬁcally designed for meta-learning.
Understanding meta-learning As mentioned in the introduction, a comprehensive theory for metalearning is still lacking and is limited to the contributions discussed in Section 3.1.
However, several recent works aimed to shed light on phenomena commonly observed in meta-learning by evaluating different intuitive heuristics.
For instance, Raghu et al.
(2020) investigated whether the popular gradient-based M AML algorithm achieves state-of-the-art performance due to rapid learning with signiﬁcant changes in the representations when deployed on target task, or due to feature reuse where the learned representation remains almost intact.
They establish that the latter factor is dominant and propose a new variation of M AML that freezes all but task-speciﬁc layers of the neural network when learning new tasks.
In another study provided by Goldblum et al.
(2020), the authors explain the success of meta-learning approaches by their capability to either cluster classes more tightly in feature space (task-speciﬁc adaptation approach), or to search for meta-parameters that lie close in weight space to many task-speciﬁc minima (full ﬁne-tuning approach).
Finally, the effect of the number of shots on the classiﬁcation accuracy was studied theoretically and illustrated empirically in
Cao et al.
(2020) for the popular metric-based P ROTO N ET algorithm.
Our paper is complementary to all other works mentioned above as it investigates a new aspect of meta-learning that has never been studied before, while following a sound theory.
Also, we provide a more complete experimental evaluation as the three different approaches of meta-learning (based on gradient, metric or transfer learning), separately presented in Raghu et al.
(2020), Cao et al.
(2020) and Goldblum et al.
(2020), are now compared together.
Other regularization strategies Regularization is a common tool to reduce model complexity during learning for better generalization, and the variations of its two most famous instances given by weight decay (Krogh & Hertz, 1992) and dropout (Srivastava et al., 2014) are commonly used as a basis in meta-learning literature as well.
In general, regularization in meta-learning is applied to the weights of the whole neural network (Balaji et al., 2018;
Yin et al., 2020), the predictions (Jamal & Qi, 2019; Goldblum et al., 2020) or is introduced via a prior hypothesis biased regularized empirical risk minimization (Pentina & Lampert, 2014; Kuzborskij & Orabona, 2017; Denevi et al., 2018a;b; 2019).
Our proposal is different from all the approaches mentioned above for the following reasons.
First, we do not regularize the whole weight matrix learned by the neural network but the linear predictors of its last layer contrary to what was done in the methods of the ﬁrst group, and, more speciﬁcally, the famous weight decay approach (Krogh & Hertz, 1992).
The purpose of the regularization in our case is also completely different: weight decay is used to improve generalization though sparsity in order to avoid overﬁtting, while our goal is to keep the classiﬁcation margin unchanged during the training to avoid over-/under-specialization to some source tasks.
Similarly, spectral normalization proposed by Miyato et al.
(2018) to satisfy the Lipschitz constraint in GANs through dividing W values by σ max ( W ) does not affect the ratio between σ max ( W ) and σ min ( W )and serves a completely different purpose.
Second, we regularize the singular values (through their entropy or their ratio) of the matrix of linear predictors obtained in the last batch of tasks instead of the predictions, as done by the methods of the second group ( e.g. , using the theoretic-information quantities in Jamal & Qi (2019) and Yin et al.
Finally, the works of the last group are related to the online setting with convex loss functions only, and, similarly to the algorithms from the second group, do not speciﬁcally target the spectral properties of the learned predictors.
Last, but not least, our proposal is built upon the most recent advances in the meta-learning ﬁeld leading to faster learning rates contrary to previous works on the subject.
Having introduced how to verify the assumptions of interest and our new regularization terms, we set out to implement them for popular meta-learning algorithms on the commonly considered FSL problem.
P RACTICAL R ESULTS
In this section, we use extensive experimental evaluations to answer the following two questions:
Q1)
Do popular meta-learning methods naturally satisfy the learning bounds assumptions?
Q2) Does ensuring these assumptions help to (meta-)learn more efﬁciently?
To this end, we ﬁrst run the original implementations of popular meta-learning methods to see what their natural behavior is.
Then, we observe the impact of forcing them to closely follow the theoretical setup. [Figure]
R σ converges during training on both datasets for B ASELINE ++ (similarly to P ROTO N ET ) whereas it diverges for B ASELINE on tieredImageNet.
With our regularization, (cid:107) W N (cid:107) F and R σ are constant during training, which is in accordance with theory.
4. E XPERIMENTAL SETUP
Datasets & Baselines For our evaluation, we focus on the few-shot image classiﬁcation problem on three benchmark datasets, namely: 1) Omniglot (Lake et al., 2015) consisting of 1 , 623 classes with 20 images/class of size 28 × 28 ; 2) miniImageNet (Ravi & Larochelle, 2017) consisting of 100 classes with 600 images of size 84 × 84 per class and 3) tieredImageNet (Ren et al., 2018) consisting of 779 , 165 images divided into 608 classes.
For each dataset, we follow the commonly adopted experimental protocol used in Finn et al.
(2017) and Chen et al.
and use a four-layer convolution backbone (Conv-4) with 64 ﬁlters as done by Chen et al.
On Omniglot, we perform 20-way classiﬁcation with 1 shot and 5 shots, while on miniImageNet
and tieredImageNet we perform 5-way classiﬁcation with 1 shot and 5 shots.
Finally, we evaluate four FSL methods: two popular meta-learning strategies, namely, M AML (Finn et al., 2017), a gradient-based method, and Prototypical Networks (P ROTO N ET )
(Snell et al., 2017), a metric-based approach; two popular transfer learning baselines, termed as B ASELINE and B ASELINE ++ (Ravi & Larochelle, 2017;
Gidaris & Komodakis, 2018; Chen et al., 2019).
Even though these baselines are trained with the standard supervised learning framework, such a training can also be seen as learning a single task in the LTL framework.
Implementation details Enforcing Assumptions 1 and 2 for M AML is straightforward as it closely follows the LTL framework of episodic training.
For each task, the model learns a batch of linear predictors and we can directly take them as W N to compute its SVD.
Since the linear predictors are the weights of our model and change slowly, regularizing the norm (cid:107) W N (cid:107) F and the ratio of singular values R σ does not cause instabilities during training.
Meanwhile, metric-based methods do not use linear predictors but compute a similarity between features.
In the case of P ROTO N ET , the similarity is computed with respect to class prototypes ( i.e. the mean features of the images of each class).
Since they act as linear predictors, a ﬁrst idea would be to regularize the norm and ratio of singular values of the prototypes.
Unfortunately, this latter strategy hinders the convergence of the network and leads to numerical instabilities.
Most likely because prototypes are computed from image features which suffer from rapid changes across batches.
Consequently, we regularize the entropy of singular values H σ instead of the ratio R σ to avoid instabilities during training to ensure Assumption 1 and we normalize the prototypes to ensure
Assumption 2 by replacing w t with ¯ w t in Eq. 4.
For transfer learning methods B ASELINE and B ASELINE ++, the last layer of the network is discarded then new linear predictors are learned during meta-testing.
Thus, we only regularize the norm (cid:107) W N (cid:107) F of the new linear predictors learned during the ﬁnetuning phase of meta-testing.
Similarly to M AML , we compute the ratio of singular values R σ with the last layer of the network during training and ﬁne-tuning phase.
Remark 2 While we choose well-established meta-learning algorithms for our comparison, we also note that the proposed regularization can be integrated similarly into their more recent variations (Park & Oliva, 2019; Lee et al., 2019) establishing current state-of-the-art on the considered benchmarks.
Finally, using models that do not rely on linear predictors is also possible but might be more difﬁcult as it would require upstream work to understand which part of the model acts as predictors (as done for P ROTO N ET in our case) and how to compute and track the desired quantities.
4. I NSIGHTS
Q1 – Verifying the assumptions According to theory, (cid:107) W N (cid:107) F and R σ should remain constant or converge toward a constant value when monitoring the last N tasks.
From Fig. 2(a), we can see that for M AML (Fig. 2(a) top), both (cid:107) W N (cid:107) F and R σ increase with the number of tasks seen during training, whereas P ROTO N ET (Fig. 2(a) bottom) naturally learns the prototypes with a good coverage of the embedding space, and minimizes their norm.
This behavior is rather peculiar as neither of the two methods speciﬁcally controls the theoretical quantities of interest, and still, P ROTO N ET manages to do it implicitly.
As for the transfer learning baselines (Fig. 2(b) top and bottom), we expect them to learn features that cover the embedding space with R σ rapidly converging towards a constant value.
As can be seen in Fig. 2(b), similarly to P ROTO N ET , B ASELINE ++ naturally learns linear predictors that cover the embedding space.
As for B ASELINE , it learns a good coverage for
Omniglot dataset, but fails to do so for the more complicated tieredImageNet dataset.
The observed behavior of these different methods leads to a conclusion that some meta-learning algorithms are inherently more explorative of the embedding space.
Q2 –
Ensuring the assumptions Armed with our regularization terms, we now aim to force the considered algorithms to verify the assumptions when it is not naturally done.
In particular, for
M AML we regularize both (cid:107) W N (cid:107) F and R σ in order to keep them constant throughout the training.
Similarly, we regularize R σ during the training of B ASELINE and B ASELINE ++, and both (cid:107) W N (cid:107) F and R σ during the ﬁnetuning phase of meta-testing.
For P ROTO N ET , we enforce a normalization of the prototypes.
According to our results for Q1, regularizing the singular values of the prototypes through the entropy H σ is not necessary.
Based on the obtained results, we can make the following conclusions.
First, from Fig. 2(a) (left, middle) and Fig. 2(b) (left), we note that for all methods considered, our proposed methodology used to enforce the theoretical assumptions works as expected, and leads to a desired behavior during the learning process.
This means that the differences in terms of results presented in Tableare explained fully by this particular addition to the optimized objective function.
Second, from the we [Table] shape of the accuracy curves provided in Fig.
2(a) (right) and the accuracy gaps when enforcing the assumptions given in Table 1, we can see that respecting the assumptions leads to several signiﬁcant improvements related to different aspects of learning.
On the one hand, we observe that the ﬁnal validation accuracy improves signiﬁcantly in all benchmarks for meta-learning methods and in most of experiments for B ASELINE (except for Omniglot, where B ASELINE already learns to regularize its linear predictors).
In accordance with the theory, we attribute the improvements to the fact that we fully utilize the training data which leads to a tighter bound on the excess target risk and, consequently, to a better generalization performance.
On the other hand, we also note that our regularization reduces the sample complexity of learning the target task, as indicated by the faster increase of the validation accuracy from the very beginning of the meta-training.
Roughly speaking, less meta-training data is necessary to achieve a performance comparable to that obtained without the proposed regularization using more tasks.
Finally, we note that B ASELINE ++ and P ROTO N ET methods naturally satisfy some assumptions: both learn diverse linear predictors by design, while B ASELINE ++ also normalizes the weights of its linear predictors.
Thus, these methods do not beneﬁt from additional regularization 6 .
C ONCLUSION
In this paper, we studied the validity of the theoretical assumptions made in recent papers applied to popular meta-learning algorithms and proposed practical ways of enforcing them.
On the one hand, we showed that depending on the problem and algorithm, some models can naturally fulﬁll the theoretical conditions during training.
Some algorithms offer a better covering of the embedding space than others.
On the other hand, when the conditions are not veriﬁed, learning with our proposed regularization terms allows to learn faster and improve the generalization capabilities of meta-learning methods.
The theoretical framework studied in this paper explains the observed performance gain.
Notice that no speciﬁc hyperparameter tuning was performed as we rather aim at showing the effect of ensuring learning bounds assumptions than comparing performance of the methods.
Absolute accuracy results are detailed in the Supplementary materials.
While this paper proposes an initial approach to bridging the gap between theory and practice in meta-learning, some questions remain open on the inner workings of these algorithms.
In particular, being able to take better advantage of the particularities of the training tasks during meta-training could help improve the effectiveness of these approaches.
Self-supervised meta-learning and multiple target tasks prediction are also important future perspectives for the application of meta-learning.