R OBUST M ETA - LEARNING WITH N OISE VIA E IGEN -R EPTILE
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent years have seen a surge of interest in meta-learning techniques for tackling the few-shot learning (FSL) problem.
However, the meta-learner’s initial model is prone to meta-overﬁt, as there are only a few available samples with sampling noise.
Besides, when handling the data sampled with label noise for FSL, meta-learner could be extremely sensitive to label noise.
To address these two challenges that
FSL with sampling and label noise.
In particular, we ﬁrst cast the meta-overﬁtting problem (overﬁtting on sampling and label noise) as a gradient noise problem since few available samples cause meta-learner to overﬁt on existing examples (clean or corrupted) of an individual task at every gradient step.
We present Eigen-Reptile (ER) that updates the meta-parameters with the main direction of historical taskspeciﬁc parameters to alleviate gradient noise.
Speciﬁcally, the main direction is computed by a special mechanism for the parameter’s large size.
Furthermore, to obtain a more accurate main direction for Eigen-Reptile in the presence of label noise, we propose Introspective Self-paced Learning (ISPL) that constructs a plurality of prior models to determine which sample should be abandoned.
We have proved the effectiveness of Eigen-Reptile and ISPL, respectively, theoretically and experimentally.
Moreover, our experiments on different tasks demonstrate that the proposed methods outperform or achieve highly competitive performance compared with the state-of-the-art methods with or without noisy labels.
I NTRODUCTION
Meta-learning, also known as learning to learn, is the key for few-shot learning (FSL) (Vinyals et al., 2016;
Wang et al., 2019a).
One of the meta-learning methods is the gradient-based method, which usually optimizes meta-parameters as initialization that can fast adapt to new tasks with few samples.
However, fewer samples mean a higher risk of meta-overﬁtting, as the ubiquitous sampling noise in mini-batch cannot be ignored.
Moreover, existing gradient-based meta-learning methods are fragile with few samples.
For instance, a popular recent method, Reptile (Nichol et al., 2018), updates the meta-parameters towards the inner loop direction, which is from the current initialization to the last task-speciﬁc parameters.
Nevertheless, as shown by the bold line of Reptile in Figure 1, with the gradient update at the last step, the update direction of meta-parameters has a signiﬁcant disturbance, as sampling noise leads the meta-parameters to overﬁt on the few trained samples at gradient steps.
Many prior works have proposed different solutions for the meta-overﬁtting problem, such as using dropout (Bertinetto et al., 2018; Lee et al., 2020), and modifying the loss function (Jamal & Qi, 2019) etc., which stay at the model level.
This paper casts the meta-overﬁtting problem as a gradient noise problem that from sampling noise while performing gradient update (Wu et al., 2019).
Neelakantan et al.
(2015) .etc
have proved that adding additional gradient noise can improve the generalization of neural networks with large samples.
However, it can be seen from the model complexity penalty that the generalization of the neural network will increase when the number of samples is larger.
To a certain extent adding gradient noise is equivalent to increasing the sample size.
As for FSL, there are only a few samples of each task.
In that case, the model will not only remembers the contents that need to be identiﬁed but also overﬁts on the noise (Zhang et al., 2016).
High-quality manual labeling data is often time-consuming and expensive.
Low-cost approaches to collect low-quality annotated data, such as from search engines, will introduce label noise.
Moreover, training meta-learner requires a large number of tasks, so that it is not easy to guarantee the quality of data.
Conceptually, the initialization learned by existing meta-learning algorithms can severely
[Figure]
Parameters generated by the noisy labels
[Figure]
eigenvector
[Figure]
Eigen
[Figure]
Reptile
Label noise
Gradient
[Figure]
Alleviate gradient noise
Update with main direction degrade in the presence of noisy labels.
Intuitively, as shown in FSL with noisy labels of Figure 1, noisy labels cause a large random disturbance in the update direction.
It means that label noise (Frénay & Verleysen, 2013) leads the meta-learner to overﬁt on wrong samples, which can be seen as further aggravating the inﬂuence of gradient noise.
Furthermore, conventional algorithms about learning with noisy labels require much data for each class (Hendrycks et al., 2018;
Patrini et al., 2017).
Therefore, these algorithms cannot be applied to noisy FSL problem, since few available samples per class.
So it is crucial to propose a method to address the problem of noisy FSL.
In this paper, we propose Eigen-Reptile (ER).
In particular, as shown in Figure 1, Eigen-Reptile updates the meta-parameters with the main direction of task-speciﬁc parameters that can effectively alleviate gradient noise.
Due to the large scale of neural network parameters, it is unrealistic to compute historical parameters’ eigenvectors.
We introduce the process of fast computing the main direction into FSL, which computes the eigenvectors of the inner loop step scale matrix instead of the parameter scale matrix.
Furthermore, we propose Introspective Self-paced Learning (ISPL), which constructs multiple prior models with randomly sampling.
Then prior models will discard high loss samples from the dataset.
We combine Eigen-Reptile with ISPL to address the noisy FSL problem, as ISPL can improve the main direction computed with noisy labels.
Experimental results show that Eigen-Reptile signiﬁcantly outperforms the baseline model by 5 .
35% and 3 .
66% on corrupted Mini-ImageNet of 5-way 1-shot and clean Mini-ImageNet of 5-way 5shot, respectively.
Moreover, the proposed algorithms outperform or are highly competitive with state-of-the-art methods on few-shot classiﬁcation tasks.
The main contributions of this paper can be summarized as follows:
• We cast the meta-overﬁtting issue (overﬁtting on sampling and label noise) as a gradient noise issue under the meta-learning framework.
• We propose Eigen-Reptile that can alleviate gradient noise effectively.
Besides, we propose ISPL, which improves the performance of Eigen-Reptile in the presence of noisy labels.
• The proposed methods outperform or achieve highly competitive performance compared with the state-of-the-art methods on few-shot classiﬁcation tasks.
R ELATED W ORK
There are three main types of meta-learning approaches: metric-based meta-learning approaches (Ravi & Larochelle, 2016; Hochreiter et al., 2001; Andrychowicz et al., 2016; Liu et al., 2018;
Santoro et al., 2016), model-based meta-learning approaches (Vinyals et al., 2016;
Koch et al., 2015;
Mordatch, 2018; Sung et al., 2018; Snell et al., 2017; Oreshkin et al., 2018; Shyam et al., 2017) and gradient-based meta-learning approaches (Finn et al., 2017; Nichol et al., 2018; Jamal & Qi, 2019;
Zintgraf et al., 2018; Li et al., 2017; Rajeswaran et al., 2019; Finn et al., 2018).
In this paper, we focus on gradient-based meta-learning approaches which can be viewed as the bi-level loop.
The goal of the outer loop is to update the meta-parameters on a variety of tasks, while task-speciﬁc parameters are learned through only a small amount of data in the inner loop.
In addition, some algorithms achieve state-of-the-art results by additionally training a 64-way classiﬁcation task on meta-training set (Yang et al., 2020; Hu et al., 2020), we do not compare these algorithms for fairness.
Meta-Learning with overﬁtting.
Due to too few samples, meta-learner inevitably tends to overﬁt in FSL.
Zintgraf et al.
(2018) introduces additional context parameters to the model’s parameters, which can prevent meta-overﬁtting.
Furthermore, Bertinetto et al.
ﬁnd that regularization such as dropout can alleviate meta-overﬁtting in their prior work; Jamal & Qi (2019) propose a novel paradigm of Task-Agnostic Meta-Learning (TAML), which uses entropy or other approaches to minimize the inequality of initial losses beyond the classiﬁcation tasks to improve the generalizability of meta-learner.
All these methods stay at the model level.
However, we solve the meta-overﬁtting problem from the gradient aspect.
We propose Eigen-Reptile, which updates the meta-parameters by the main direction of task-speciﬁc parameters to alleviate meta-learner overﬁt on noise.
Learning with noisy labels.
Learning with noisy labels has been a long-standing problem.
There are many approaches to solve it, such as studying the denoise loss function (Hendrycks et al., 2018;
Patrini et al., 2017; Jindal et al., 2016; Patrini et al., 2017; Wang et al., 2019b; Arazo et al., 2019), relabeling (Lin et al., 2014), and so on.
Nevertheless, most of these methods require much data for each class.
Gao et al.
(2019) proposes a model for noisy few-shot relation classiﬁcation but without good transferability.
For noisy FSL, a gradient-based meta-learner is trained to optimize an initialization on various tasks with noisy labels.
As there are few samples of each class, the traditional algorithms for noisy labels cannot be applied.
When the existing gradient-based meta-learning algorithms, such as Reptile, update meta-parameters, they focus on the samples that generate the last gradient step.
And these samples may be corrupted, which makes the parameters learned by meta-learner susceptible to noisy labels.
For noisy FSL, we propose ISPL based on the idea of
Self-paced Learning (SPL) (Kumar et al., 2010; Khan et al., 2011; Basu & Christensen, 2013; Tang et al., 2012) to learn more accurate main direction for Eigen-Reptile.
ISPL constructs prior models to decide which sample should be discarded when train task-speciﬁc models, and this process can be regarded as the introspective process of meta-learner.
In contrast, the model with SPL learns the samples gradually from easy to complex, and the model itself decides the order.
P RELIMINARIES
Gradient-based meta-learning aims to learn a set of initialization parameters φ that can be adapted to new tasks after a few iterations.
The dataset D is usually divided into the meta-training set
D meta − train and meta-testing set D meta − test , which are used to optimize meta-parameters and evaluate its generalization, respectively.
For meta-training, we have tasks {T i } Bi =1 drawn from task distribution p ( T ) , each task has its own train set D train and test set D test , and the tasks in metatesting D meta − test are deﬁned in the same way.
Note that there are only a small number of samples for each task in FSL.
Speciﬁcally, the N-way K-shot classiﬁcation task refers to K examples for each of the N classes.
Generally, the number of shots used in meta-training should match the one used at meta test-time to obtain the best performance (Cao et al., 2019).
In this paper, we will increase the sample size appropriately to get the main direction of individual tasks during meta-training.
To minimize the test loss L ( D test , (cid:101) φ ) of an individual task, meta-parameters need to be updated n times to get good task-speciﬁc parameters (cid:101) φ .
That is minimizing
[Equation] [Equation]
where U n represents n inner loop steps through gradient descent or Adam (Kingma & Ba, 2014) on batches from D train , q (cid:16) ˆ y
= y
| x, φ , (cid:101) φ (cid:17) is the predictive distribution.
When considering updating the meta-parameters in the outer loop, different algorithms have different rules.
In the case of Reptile, after n inner loop steps, the meta-parameters can be updated as: φ ←− φ + β ( (cid:101) φ − φ ) , where β is a scalar stepsize hyperparameter that controls the update rate of meta-parameters.
E IGEN -R EPTILE FOR CLEAN AND CORRUPTED DATA
The proposed Eigen-Reptile alleviates gradient noise to alleviate meta-learner overﬁtting on sampling and label noise.
Furthermore, ISPL improves the performance of Eigen-Reptile in noisy FSL.
4. T HE E IGEN -R EPTILE A LGORITHM
To alleviate gradient noise to improve the generalizability of meta-learner, we propose Eigen-Reptile, which updates d meta-parameters with the main direction of task-speciﬁc parameters.
We train the task-speciﬁc model with n inner loop steps that start from the meta-parameters φ with few examples.
Let i-th column W : ,i ∈ R
d
× 1 of parameter matrix W ∈ R
d × n be the parameters after i-th gradient update, e.g., W : ,i
= U
i ( φ ) .
And treat W : ,i as a d -dimensional parameter point w i in the parameter space.
e ∈ R
d
× 1 is a unit vector that represents the main direction of n parameter points in W .
Intuitively, projecting all parameter points onto e should retain the most information.
We represent the parameter points by a straight line of the form w = w + l e , which shows that the straight line passes through the mean point w and the signed distance of a point w from w is l .
Then we get the loss function J ( l 1 , l 2 , · · · , l n , e )
=
(cid:80) n i =1 (cid:107) w + l i e − w i
(cid:107) 2 .
And determine the signed distance l of each point by partially differentiating J with respect to l i , we get l i =
e (cid:62) ( w i − w ) .
Plugging in this expression for l i in J , we get
J
[Equation]
= −
[Equation]
where S = (cid:80) ni
=1 ( w i − w )( w i − w ) (cid:62) is a scatter matrix.
According to Eq.2, minimizing J is equivalent to maximizing: e (cid:62) Se .
Note that e needs to be roughly consistent with the gradient update direction V in the process of learning task-speciﬁc parameters.
Use Lagrange multiplier method as
[Equation]
, where
[Equation]
We get the objective function
[Equation]
where λ (cid:54) = 0 , µ ≥
[Equation]
4) then partially differentiating g in Eq.4 with respect to µ, e, λ, η ,
[Equation]
According to Eq.5, if η = 0 , then V and e are orthogonal, which obviously does not meet our expectations.
So we get η (cid:54) = 0 , and µ = 0 .
Then Se = λ e .We can see e is the eigenvector of S corresponding to the largest eigenvalue λ , as we need the main direction.
It should be noted that even if Eq.3 is not directly related to the e , in ER, Eq.3 must be retained because it determines the update direction of the outer-loop.
Otherwise, the algorithm will not converge.
A concerned question about Se = λ
e is that the scatter matrix
S ∈ R
d × d grows quadratically with the number of parameters d .
As the high dimensionality of parameters typically used in neural networks, computing eigenvalue and eigenvector of S could come at a prohibitive cost (the worst-case complexity is O (cid:0) d 3 (cid:1) ).
Centralize W by subtracting the mean w , and scatter matrix S = W W (cid:62) .
To avoid calculating the eigenvector of S directly, we focus on W (cid:62) W .
As W (cid:62) W (cid:98) e = (cid:98) λ (cid:98) e .
Multiply both sides of the equation with W ,
[Equation]
It can be found from Eq.6 that W (cid:62) W ∈ R n × n and W W (cid:62) ∈
R
d × d have the same eigenvalue, λ = (cid:98) λ .
Furthermore, we get the eigenvector of W W (cid:62) as e = W (cid:98) e .
The main advantage of Eq.6 is that the intermediate matrix W (cid:62) W now grows quadratically with the inner loop steps.
As we are interested in FSL, n is very small.
It will be much easier to compute the eigenvector (cid:98) e of W (cid:62) W ,
O (cid:0) n 3 (cid:1) , which can be ignored (detailed analysis refer to Appendix B).
Then we get the eigenvector e of W W (cid:62) based on (cid:98) e .
Moreover, we project parameter update vectors w i +1 − w i , i = 1 , 2 , · · · , n −on e to get the corresponding update stepsize ν , so meta-parameters φ can be updated as
[Equation]
where β is a scalar stepsize hyperparameter that controls the update rate of meta-parameters.
The
Eigen-Reptile algorithm is summarized in Algorithm 1 of Appendix A .
To illustrate the validity of
Eigen-Reptile theoretically, we present Theorem 1 as follow:
Theorem 1 Assume that the gradient noise variable x follows Gaussian distribution (Hu et al., 2017;
Jastrz˛ebski et al., 2017; Mandt et al., 2016), x ∼ N (cid:0) 0 , σ 2 (cid:1) .
Furthermore, x and neural network parameter variable are assumed to be uncorrelated.
The observed covariance matrix C equals noiseless covariance matrix C t plus gradient noise covariance matrix C x .
Then, we get
[Equation]
where P t and P are the orthonormal eigenvector matrices of C t and C respectively, Λ t and Λ are the corresponding diagonal eigenvalue matrices, and I is an identity matrix.
It can be seen from
Eq.8 that C and C t has the same eigenvectors.
We defer the proof to the Appendix C.
Theorem 1 shows that eigenvectors are not affected by gradient noise.
So Eigen-Reptile can ﬁnd a more generalizable starting point for new tasks without overﬁtting on noise.
[Figure]
HE
[Figure]
Self-paced learning (SPL) learns the samples from low losses to high losses, which is proven beneﬁcial in achieving a better generalization result (Khan et al., 2011;
Basu & Christensen, 2013; Tang et al., 2012).
Besides, the losses of samples are determined by the model itself.
Nevertheless, in meta-learning setting, meta-learner is trained on various tasks, and initial model may have lower losses for trained classes and higher losses for unseen classes or noisy samples.
For this reason, we cannot train the task-speciﬁc model by the same way as SPL to solve noisy FSL problem.
In this paper, we use multiple prior models to vote on samples and decide which should be abandoned.
As shown in Figure 2, even though the two categories of yellow and green show an excellent distribution that can be well separated, some samples are marked wrong.
To address this noisy label problem, we build three prior models.
Specially, we randomly sample three times, and model 1 is trained with a corrupted label.
Due to different samples learned by prior models, building multiple models to vote on the data will obtain more accurate losses.
Such a learning process is similar to human introspection, and we call it Introspective Self-paced Learning (ISPL).
Moreover, samples with losses above a certain threshold will be discarded.
Furthermore, we imitate SPL to add the hidden variable v = 0 or 1 that is decided by Q prior models before the loss of each sample to control whether the sample should be abandoned.
So we get the task-speciﬁc loss as
L ISPL ( φ , v ) = h (cid:88)
i = v i L ( x i , y i , φ ) , where v i = arg min v v i Q
Q (cid:88) j
=
L j
[Equation]
where h is the number of samples from dataset D train , γ is the sample selection parameter, which gradually decreases, parameter of model j is φ j = U n ( D j , φ ) , D j ∈ D train .
Note that we update the meta-parameters with the model trained on h samples from D train .
The ISPL is summarized in
Algorithm 2 of Appendix A .
Intuitively, it is difﬁcult to say whether discarding high-loss samples containing correct samples and wrong samples will improve the accuracy of eigenvector, so we will use Theorem 2 to prove the effectiveness of ISPL.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
Theorem 2 Let W o be the parameter matrix generated by the corrupted samples.
Compute the eigenvalues and eigenvectors of the expected observed parameter matrix
[Equation]
(10) where C tr is the covariance matrix generated by true samples, λ is the observed largest eigenvalue, λ o is the largest eigenvalue in the corrupted diagonal eigenvalue matrix Λ o .
According to Eq.21, if λ o /λ is smaller, the observed eigenvector e is more accurate.
Assume that the discarded high loss samples have the same contributions ξ to λ and λ o , representing the observed and corrupted main directional variance, respectively.
Note that these two kinds of data have the same effect on the gradient updating of the model, so this assumption is relatively reasonable.
Furthermore, it is easy to ﬁnd that ( λ o − ξ ) / ( λ − ξ ) is smaller than λ
o /λ .
Theorem 2 shows that discard the high loss samples can help improve the accuracy of the observed eigenvector of parameter matrix learned with corrupted labels.
So ISPL can improve the performance of Eigen-Reptile, as it discards high loss samples in D train .
E XPERIMENTAL R ESULTS AND D ISCUSSION
In our experiments, we aim to (1) evaluate the effectiveness of Eigen-Reptile to alleviate gradient noise (sampling and label noise), (2) determine whether Eigen-Reptile can alleviate gradient noise in a realistic problem, (3) evaluate the improvement of ISPL to Eigen-Reptile in the presence of noisy labels, (4) validate theoretical analysis through numerical simulations.
The code and data for the proposed model are provided for research purposes 1 .
5. M ETA - LEARNING WITH N OISE ON R EGRESSION
In this experiment, we evaluate Eigen-Reptile by the 1D sine wave K -shot regression problem (Nichol et al., 2018).
Each task is deﬁned by a sine curve y ( x ) =
Asin ( x + b ) , where the amplitude
A ∼ U ([0 . 1 , 5 . 0]) and phase b ∼ U ([0 , 2 π ]) .
The amplitude A and phase b are varied between tasks.
The goal of each task is to ﬁt a sine curve with the data points sampled from the corresponding y ( x ) .
We calculate loss in (cid:96) 2 using 50 equally-spaced points from the whole interval [ − 5 . 0 , 5 . 0] for each task.
The loss is
[Equation]
[Table]
where (cid:98) y
( x ) is the predicted function that start from the initialization learned by meta-learner.
The K -shot regression task ﬁts a selected sine curve through K points, here K = 10 .
For the regressor, we use a small neural network, which is the same as Nichol et al.
(2018), except that the activation functions are Tanh.
Speciﬁcally, the small network includes an input layer of size 1, followed by two hidden layers of size 64, and then an output layer of size 1.
In this part, we mainly compare Reptile and Eigen-Reptile.
Both meta-learners use the same regressor and are trained for 30000 iterations with inner loop steps 5, batch size 10, and a ﬁxed inner loop learning rate α = 0 . 02 .
[Figure]
We report the results of Reptile and Eigen-Reptile in Figure 3.
It can be seen that the curve ﬁtted by
Eigen-Reptile is closer to the true green curve, which shows that Eigen-Reptile performs better.
According to Jamal & Qi (2019), the initial model that has a larger entropy before adapting to new tasks would better alleviate meta-overﬁtting.
As shown in Figure 3, from 1 to 30000 iterations, Eigen-Reptile is more generalizable than Reptile as the initial blue line of
Eigen-Reptile is closer to a straight line, which shows that the initialization learned by Eigen-Reptile is less affected by gradient noise.
Furthermore, Figureshows that Eigen-Reptile converges faster and gets a lower loss than Reptile.
5. M ETA - LEARNING IN R EALISTIC P ROBLEM
We evaluate our method on a popular few-shot classiﬁcation dataset: Mini-ImageNet (Vinyals et al., 2016).
The Mini-Imagenet dataset contains 100 classes, each with 600 images.
We follow Ravi &
Larochelle (2016) to divide the dataset into three disjoint subsets: meta-training set, meta-validation set, and meta-testing set with 64 classes, 16 classes, and 20 classes, respectively.
And we follow the few-shot learning protocols from prior work (Vinyals et al., 2016), except that the number of the meta-training shot is 15, which is still much smaller than the number of samples required by traditional tasks.
Moreover, we run our algorithm on the dataset for the different number of shots and compare our results to the state-of-the-art results.
What needs to be reminded is that approaches that use deeper, residual networks can achieve higher accuracies (Gidaris & Komodakis, 2018), so for a fair comparison, we only compare algorithms that use convolutional networks as Reptile does.
Speciﬁcally, our model follows Nichol et al.
(2018), which has 4 modules with a 3 × 3 convolutions and 64 ﬁlters, 2 × 2 max-pooling etc..
The images are downsampled to 84 × 84 , and the loss function is the cross-entropy error.
We use the Adam optimizer with β 1 = 0 in the inner loop.
Our model
[Table]
[Table]
Regarding some hyperparameters analysis, defer to Appendix E
[Table]
The results of Eigen-Reptile and other meta-learning approaches are summarized in Table 1.
The proposed Eigen-Reptile (64 ﬁlters) outperforms and achieves highly competitive performance compared with other algorithms for 5-shot and 1-shot classiﬁcation problems, respectively.
More speciﬁcally, for 1-shot, the result of MC (128 ﬁlters) with a higher capacity network is better than that of EigenReptile.
However, as a second-order optimization algorithm, the computational cost of MC will be much higher than Eigen-Reptile.
And obviously, the result of Eigen-Reptile is much better than
Reptile for each task.
Compared with Reptile, Eigen-Reptile uses the main direction to update the meta-parameters to alleviate the meta-overﬁtting caused by gradient noise.
More importantly,
Eigen-Reptile outperforms the state-of-the-art meta-overﬁtting preventing method Meta-dropout (Lee et al., 2019), which is based on regularization.
This result shows that the effectiveness of addressing the meta-overﬁtting problem from the perspective of alleviating gradient noise
[Table]
5. M ETA - LEARNING WITH L ABEL N OISE
We conduct the 5-way 1-shot experiment with noisy labels generated by corrupting the original labels of Mini-Imagenet.
More speciﬁcally, in this section, we only focus on symmetric label noise, as correct labels are ﬂipped to other labels with equal probability, i.e., in the case of symmetric noise of ratio p , a sample retains the correct label with probability 1 − p .
It becomes some other label with probability p/ ( N − 1) .
An example of symmetric noise is shown in Figure 6.
Furthermore, the asymmetric label noise experiment is conducted in Appendix F. Note that we only introduce noise in the train set during meta-training, where the meta-training shot is 30.
Moreover, all meta-learners with 32 ﬁlters are trained for 10000 iterations, with a learning rate of 0.001 in the inner loop.
The sample selection parameter γ = 10 that decreases by 0.6 every 1000 iterations.
The other settings of this experiment are the same as in section 5.2.
As shown in Table 3, with the increase of the ratio p , the performance of Reptile decreases rapidly.
When p = 0 .
5 , the initialization point learned by Reptile can hardly meet the requirements of quickly adapting to new tasks with few samples.
On the other hand, Eigen-Reptile is less affected by noisy labels than Reptile, especially when the noise ratio is high, i.e., p = 0 .
The experimental results also verify the effectiveness of ISPL, as Eigen-Reptile+ISPL achieves better results than Eigen-Reptile when p (cid:54) = 0 .
It also can be seen that ISPL plays a more signiﬁcant role when p is higher.
However, when p = 0 , ISPL harms Eigen-Reptile, as ISPL only discards correct samples.
In addition, ISPL does not signiﬁcantly improve Reptile, especially when p = 0 .
This is because too many high-loss samples are removed, causing Reptile to fail to converge quickly with the same number of iterations.
These experimental results show that Eigen-Reptile and ISPL can effectively alleviate the gradient noise problem caused by noisy labels, thereby alleviating the meta-overﬁtting on corrupted samples.
C ONCLUSION
In this paper, we cast the meta-overﬁtting problem (overﬁtting on sampling and label noise) as a gradient noise problem.
Then, we propose a gradient-based meta-learning algorithm Eigen-Reptile.
It updates the meta-parameters through the main direction, which has been proven by theory and experiments that it can alleviate the gradient noise effectively.
Furthermore, to get closer to real-world situations, we introduce noisy labels into the meta-training dataset, and the proposed ISPL constructs prior models to select samples for Eigen-Reptile to get more accurate main direction.
A PSEUDO-CODE
Require: Distribution over tasks P ( T ) , outer step size β 1: Initialize meta-parameters φ 2: while not converged do 3:
W = [ ] , ν =4:
Sample batch of tasks {T i } Bi =1 ∼ P ( T ) 5: for each task T i do 6: φ i =
φ 7:
Sample train set D train of T i 8: for j = 1 , 2 , 3 , ..., n do 9: φ ji = U
j ( D train , φ i ) 10:
W appends w j = flatten ( φ ji ) , w j ∈ R d ×11: end for 12:
Mean centering, W = W − w , w ∈ R d ×13:
Compute eigenvalue matrix (cid:98) Λ and eigenvector matrix (cid:98) P of scatter matrix W (cid:62) W 14:
Eigenvalues λ 1 > λ 2 > · · · > λ n in (cid:98) Λ 15:
Compute eigenvector matrix of W W (cid:62) , P = W (cid:98) P 16:
Let the eigenvector corresponding to λ 1 be a unit vector (cid:107) e 1 i (cid:107) 22 =17: for j = 1 , 2 , 3 , ..., n − 1 do 18: ν
= ν
+ ( W : ,j +1 − W : ,j ) e 1 i 19: end for 20: e 1 i
= λ 1 (cid:80) nm =1 λ m × e 1 i 21:
Calculate the approximate direction of task-speciﬁc gradient update V : 22:
V = 1 (cid:98) n/ 2 (cid:99) (cid:80) (cid:98) n/ 2 (cid:99) i =1 W : ,n − i +1 − W : ,i 23: if e 1 i · V < 0 then 24: e 1 i = − e 1 i 25: end if 26: end for 27:
Average the main directions to get ˜
e
= (1 /B )
(cid:80) Bi
=1 e 1 i 28:
Update meta-parameters φ ←− φ + β × ν/B × ˜ e 29: end while
Require: Dataset D train , initialization φ , batch size b , selection parameter γ , attenuation coefﬁcient µ , the number of prior models Q 1: Initialize network parameters φ ∗ =
φ for a sampled task 2: for j = 1 , 2 , 3 , · · · , Q do 3:
Sample examples D j from D train for training model j , φ j = U m ( D j , φ ∗ ) 4: end for 5: Train task-speciﬁc parameters: 6: for i = 1 , 2 , 3 , · · · , n do 7:
Compute hidden variable vector v : 8: v = arg min v v q (cid:80) bq =1 L q − γ (cid:80) bq =1 v q , where L q = 1 Q (cid:80) Qj =1 L j ( x q , y q , φ j ) 9:
Update task-speciﬁc parameters φ ∗ : 10: φ ∗ =
arg min φ ∗ L ISPL ( φ ∗ , v ) 11: γ = γ − µ 12: end for
B A LGORITHM C OMPLEXITY A NALYSIS
As for Eigen-Reptile, the cost of single gradient descent in the inner-loop is O ( d ) , where d is the number of network parameters.
The cost of the covariance matrix computations is O ( n 2 d ) , where n is the number of inner-loop.
Moreover, the worst-case complexity of computing eigenvalue decomposition is O ( n 3 ) .
Finally, the computational complexity of restoring eigenvector is O ( nd ) .
We set the maximal number of outer-loop to T .
Hence the overall time complexity is O ( T ( n 2 d + n 3 + nd + nd )) .
As in FSL, n is usually less than 10 (for this paper n = 7 ), so the overall time complexity is O ( Td ) .
As for Reptile, the computational complexity is also O ( Td ) , which means that the time complexity of both Reptile and Eigen-Reptile is much lower than the second-order optimization algorithms.
As for spatial complexity, Eigen-Reptile needs to store a d × n matrix and a n × n matrix.
The overall space complexity is O ( d ) , while the spatial complexity of Reptile is O ( d ) , too.
It can be seen that, compared to Reptile, Eigen-Reptile is the same in spatial complexity and time complexity.
Still, its accuracy is much higher than that of Reptile.
C T HEOREM
Gradient update always with gradient noise inserted at every iteration, which caused Reptile, MAML, etc. cannot ﬁnd accurate directions to update meta-parameters.
In this section, we will prove that
Eigen-Reptile can alleviate gradient noise.
Theorem 3 Assume that the gradient noise variable x follows Gaussian distribution (Hu et al., 2017;
Jastrz˛ebski et al., 2017; Mandt et al., 2016), x ∼ N (cid:0) 0 , σ 2 (cid:1) .
Furthermore, x and neural network parameter variable are assumed to be uncorrelated.
The observed covariance matrix C equals noiseless covariance matrix C t plus gradient noise covariance matrix C x .
Then, we get
[Equation]
(12) where P t and P are the orthonormal eigenvector matrices of C t and C respectively, Λ t and Λ are the corresponding diagonal eigenvalue matrices, and I is an identity matrix.
It can be seen from
Eq.12 that C and C t has the same eigenvectors.
Proof C.
In the following proof, we assume that the probability density function of gradient noise variable x follows Gaussian distribution, x ∼ N (cid:0) 0 , σ 2 (cid:1) .
Treat the parameters in the neural network as variables, and the parameters obtained by each gradient update as samples.
Furthermore, gradient noise and neural network parameters are assumed to be uncorrelated.
For observed parameter matrix W ∈ R d × n , there are n samples, let W i, : ∈ R 1 × n be the observed values of the i-th variable W i , and W =
[ W (cid:62) 1 , : , · · · , W (cid:62) i, : , · · · , W (cid:62) d, : ] (cid:62) .
Similarly, we denote the noiseless parameter matrix by W t =
[( W t 1 , : ) (cid:62) , · · · , ( W ti, : ) (cid:62) , · · · , ( W td, : ) (cid:62) ] (cid:62) , and
[Equation]
Where X =
[ X (cid:62) 1 , : , · · · , X (cid:62) i, : , · · · , X (cid:62) d, : ] (cid:62) is the dataset of noise variables.
Then, centralize each variable by
[Equation]
So we get W =
[ W (cid:62) 1 , · · · , W (cid:62) d ] (cid:62) .
Suppose W t is also centralized by the same way and get
W t =
[ W t (cid:62) 1 , · · · , W t (cid:62) d ] (cid:62) .
Then, we have:
[Equation]
Computing the covariance matrix of W :
[Equation]
Since W t and X are uncorrelated, W t X (cid:62) and XW t (cid:62) are approximately zero matrices.
Thus:
[Equation] [Equation]
The component C x ( i, j ) is the correlation between X i and X j which corresponds to the i-th and j-th rows of X .
As the two noise variables are not related to each other, if i (cid:54)
=
j , then C x ( i, j )
= 0 .
So
C x ∈ R d × d is a diagonal matrix with diagonal elements σ 2 .
Decompose C t as:
[Equation]
[Equation]
where P t is the noiseless orthonormal eigenvector matrix and Λ t is the noiseless diagonal eigenvalue matrix, then
[Equation]
where Λ x = σ 2 I , and I is the identity matrix.
Thus,
[Equation]
where Λ = Λ t + Λ x .
It can be seen from Eq.20 that C and C t has the same eigenvector matrix.
In other words, eigenvector is not affected by gradient noise.
D T HEOREM
In this section, we will prove that discarding high loss samples will result in a more accurate main direction in noisy FSL.
Theorem 4 Let W o be the parameter matrix generated by the corrupted samples.
Compute the eigenvalues and eigenvectors of the expected observed parameter matrix
[Equation]
[Equation]
where C tr is the covariance matrix generated by true samples, λ is the observed largest eigenvalue, λ o is the largest eigenvalue in the corrupted diagonal eigenvalue matrix Λ o .
According to Eq.21, if λ o /λ is smaller, the observed eigenvector e is more accurate.
Assume that the discarded high loss samples have the same contributions ξ to λ and λ o , representing the observed and corrupted main directional variance, respectively.
Note that these two kinds of data have the same effect on the gradient updating of the model, so this assumption is relatively reasonable.
Furthermore, it is easy to ﬁnd that ( λ o − ξ ) / ( λ − ξ ) is smaller than λ
o /λ .
Proof D.1 Here, we use w to represent the parameter point obtained after a gradient update.
For convenience, let w be generated by a single sample,
w ∈ R d × 1 .
Then the parameter matrix can be obtained,
[Equation]
[Equation]
where w o represents the parameters generated by the corrupted sample, and w tr represents the parameters generated by the true sample.
Furthermore, there are n parameter points generated by n samples.
Moreover, there are m corrupted parameter points generated by m corrupted samples.
Mean centering W , and show the observed covariance matrix C as
[Equation]
It can be seen from the decomposition of C that the required eigenvector is related to the parameters obtained from the true samples and the parameters obtained from the noisy samples.
For a single parameter point
[Equation]
As we discard all high loss samples that make the model parameters change signiﬁcantly, and the randomly generated noisy labels may cause the gradient to move in any direction, we assume that the variance of corrupted parameter point variables is δ .
Compute the expectations of all variables in the corrupted parameter point
[Equation]
Let the sum of all corrupted 1 n E ( ww (cid:62) ) be Ω o , then
[Equation]

[Equation] [Equation]
And let the sum of all true 1 n ww (cid:62) be C tr .
So the expectation of C can be written as,
[Equation]
Treat eigenvector and eigenvalue as deﬁnite values, we get
[Equation]
) where e is the observed eigenvector, λ is the corresponding eigenvalue.
Divide both sides of the equation by λ .
[Equation]
where λ o is the largest eigenvalue in the corrupted diagonal eigenvalue matrix Λ o , P o is the orthonormal eigenvector matrix of Ω o .
According to Eq.29, if λ o /λ is smaller, e is more accurate.
Discard some samples with the largest losses, which may contain true samples and noisy samples.
Assume that the discarded high loss samples have the same contributions ξ to λ and λ o , as these two kinds of data have the same effect on the gradient updating of the model.
Compare the ratio of eigenvalues before and after discarding, get
[Equation]
[Equation]
Obviously, λ > λ o , and if we don’t discard all samples, then λ > ξ .
So Eq.30 > 0 , which means discarding high loss samples could reduce λ
o /λ .
Therefore, discarding high loss samples can improve the accuracy of eigenvector in the presence of noisy labels.
For further analysis, we assume that any two variables are independently and identically distributed, the expectation of variable a , E ( a )
= (cid:15) .
Thus,
[Equation]
where p is the proportion of noisy labels, np = m .
As can be seen from Eq.31, if p(cid:15) 2 /λ ≈ 0 , then Ω o /λ is a diagonal matrix.
According to proof.
C.1, the observed eigenvector e is unaffected by noisy labels with the corresponding eigenvalue p ( δ 2 + (cid:15) 2 ) λ .
E H YPERPARAMETERS OF E IGEN -R EPTILE
In this section, we follows Lee et al.
; Cao et al.
(2019) to vary the number of inner-loops and the number of corresponding training shots to show the robustness of Eigen-Reptile.
Besides, other hyperparameters are the same as section 5.2.
As shown in Figure 5, after the number of inner-loops i reaches 7, the test accuracy tends to be stable, which shows that changing the number of inner-loops within a certain range has little effect on Eigen-Reptile.
That is, Eigen-Reptile is robust to this hyperparameter.
As for train shot, to make the trained task-speciﬁc parameters as unbiased as possible, we specify train shot roughly satisﬁes (cid:100)
i × batch _ size N (cid:101) + 1 , where N is the number of classes.
So when i = 7 , the number of train shots is 15.
It is important to note that in our experiments, Reptile uses the original implementation hyperparameters, the number of inner-loops is 8, the number of train shots is 15 and the corresponding accuracy is 65.99%.
[Figure]
F T HE A SYMMETRIC L ABEL N OISE E XPERIMENT
Since asymmetric noise is as common as symmetric noise, we focus on asymmetric noise in this section.
As illustrated in asymmetric noise of Figure 6, we randomly ﬂip the labels of one class to the labels of another class without duplication in the meta-training dataset (64 classes).
[Figure]
symmetric noise          0.
[Figure]
We compare our algorithms with Reptile in table 3.
We observe that meta-learning algorithms with asymmetric noise have closer results compared to symmetric noise.
As meta-learner is trained on tasks with the same noise transition matrix, which allows the meta-learner to learn more useful information, the results are higher than that with symmetric noise.
Similar to the symmetric noise results, Eigen-Reptile outperforms Reptile in all tasks, and ISPL plays a more signiﬁcant role when p is higher.
The experimental results also show that when the number of iterations is constant, ISPL does not signiﬁcantly improve or even degrades the results of Reptile.
On the contrary, ISPL can provide Eigen-Reptile with a more accurate main direction in difﬁcult tasks, e.g.,
[Table]
[Table]
G M ETA - LEARNING ON CIFAR-FS
Bertinetto et al.
propose CIFAR-FS (CIFAR100 few-shots), which is randomly sampled from CIFAR-100 (Krizhevsky et al., 2009), containing images of size 32 × 32 .
The settings of
Eigen-Reptile in this experiment are the same as in 5.2.
Moreover, we do not compare algorithms with additional tricks, such as higher way (Bertinetto et al., 2018;
Snell et al., 2017).
It can be seen from Table 4 that on CIFAR-FS, the performance of Eigen-Reptile is still far better than Reptile without any parameter adjustment.
[Table]
Embedded Class Models(Ravichandran et al., 2019)
[Table]
Reptile (64)(Nichol et al., 2018)
[Table]
H N EURAL N ETWORK A RCHITECTURES
This section will show the performance of Eigen-Reptile on Mini-Imagenet when using a larger network as CAML, etc.
Note that we only compare our algorithm with meta-learning algorithms based on the gradient in this section.
As shown in Table 5, when Eigen-Reptile uses a larger convolutional neural network (CNN), higher accuracy can be obtained, which shows that Eigen-Reptile beneﬁts from increasing model expressiveness.
[Table]