E NHANCING M ETA L EARNING VIA M ULTI -O BJECTIVE S OFT I MPROVEMENT F UNCTIONS
A BSTRACT
Meta-learning tries to leverage information from similar learning tasks.
In the commonly-used bilevel optimization formulation, the shared parameter is learned in the outer loop by minimizing the average loss over all tasks.
However, the converged solution may be compromised in that it only focuses on optimizing on a small subset of tasks.
To alleviate this problem, we consider meta-learning as a multi-objective optimization (MOO) problem, in which each task is an objective.
However, existing MOO solvers need to access all the objectives’ gradients in each iteration, and cannot scale to the huge number of tasks in typical meta-learning settings.
To alleviate this problem, we propose a scalable gradient-based solver with the use of mini-batch.
We provide theoretical guarantees on the Pareto optimality or
Pareto stationarity of the converged solution.
Empirical studies on various machine learning settings demonstrate that the proposed method is efficient, and achieves better performance than the baselines, particularly on improving the performance of the poorly-performing tasks and thus alleviating the compromising phenomenon.
I NTRODUCTION
Meta-learning, also known as “learning to learn", aims to enable models to learn more effectively by leveraging information from many similar learning tasks (Hospedales et al., 2020).
In recent years, meta-learning has received much attention for its fast adaptation to new learning scenarios with limited data (Kao et al., 2021; Finn et al., 2017;
Snell et al., 2017; Lee et al., 2019; Nichol et al., 2018; Deleu et al., 2022; Rajeswaran et al., 2019; Vilalta & Drissi, 2002).
It is usually formulated as a bi-level optimization problem (Franceschi et al., 2018; Hong et al., 2020), which finds task-specific parameters in the inner level and minimizes the average loss over tasks in the outer level.
Recently, Wang et al.
(2021) reformulate meta-learning as a multi-task learning problem.
From this perspective, minimizing the average loss in the outer level using (stochastic) gradient descent may not always be desirable.
Specifically, it may suffer from the compromising (or conflicting) phenomenon, in which the converged solution only focuses on minimizing the losses of a small subset of tasks while ignoring the others (Yu et al., 2020; Liu et al., 2021a; Sener & Koltun, 2018).
This compromised solution may thus lead to poor performance.
To alleviate this problem, we propose reformulating meta-learning as a multi-objective optimization (MOO) problem, in which each task is an objective.
The performance of all tasks (objectives) are then considered during optimization (Emmerich & Deutz, 2018).
A popular class of MOO solvers is the gradient-based approach (Liu et al., 2021a; Yu et al., 2020; Sener & Koltun, 2018; Navon et al., 2022; Liu et al., 2021b), with prominent examples such as the multiple-gradient descent algorithm (MGDA) (Désidéri, 2012; Sener & Koltun, 2018), PCGard (Yu et al., 2020), and CAGard (Liu et al., 2021a).
In each iteration, they find a common descent direction among all objective gradients, instead of simply optimizing the average performance over all objectives.
Existing gradient-based MOO methods require using gradients from all the objectives.
However, when formulating meta-learning as a MOO problem with each task being an objective, computing all these gradients in each iteration can become very expensive, as the number of objectives (i.e., tasks) can be huge.
For example, in 5-way 1-shot classification on the miniImageNet data, the total number of meta-training tasks is (cid:0) 645 (cid:1)
≈
7 × 10 6 .
To address this challenge, we propose a scalable MOO solver by using the improvement function (Miettinen & Mäkelä, 1995; Mäkelä et al., 2016; Montonen et al., 2018) with the help of mini-batch.
On the other hand, we show that a trivial extension of existing gradient-based MOO methods with the use of mini-batch does not guarantee Pareto optimality and has poor performance in practice.
Our main contributions are as follows: (i) To alleviate the compromising phenomenon, we reformulate meta-learning as a multi-objective optimization problem in which each task is an objective; (ii) To handle the possibly huge number of tasks, we propose a scalable gradient-based solver.
(iii)
We provide theoretical guarantees on the Pareto optimality or Pareto stationarity of the converged solution.
(iv) Empirical studies on few-shot regression, few-shot classification, and reinforcement learning demonstrate that the proposed method achieves better performance, particularly in improving the performance of the poorly-performing tasks and thus alleviating the compromising phenomenon.
B ACKGROUND
Multi-Objective Optimization (MOO).
In MOO (Marler & Arora, 2004), one aims to minimizem ≥ 2 objectives f 1 ( x ) , . . .
, f m ( x ) :
[Equation]
Definition 2.1.
(Global Pareto optimality) (Miettinen, 2012; Mäkelä et al., 2016)
x ∗ is global Pareto optimal if there does not exist another x such that f τ ( x ∗ ) ≥ f τ ( x ) for all τ ∈ { 1 , . . .
, m } , and f τ ′ ( x ∗ ) > f τ ′ ( x ) for at least one τ ′ ∈ { 1 , . . .
, m } .
The Pareto front (PF) is the set of multi-objective values of all global Pareto-optimal solutions.
Definition 2.2.
(Pareto stationarity) (Miettinen, 2012; Désidéri, 2012)
x ∗ is Pareto-stationary if there exist { u τ } mτ =1 such that ∥ (cid:80) mτ
=1 u τ ∇ x f τ ( x ) ∥
= 0 , u τ ≥ 0 ∀ τ and (cid:80) mτ
=1 u τ = 1 .
Note that global Pareto optimal solutions are also Pareto stationary (Désidéri, 2012).
Analogous to the extension from a stationary point to an ϵ -stationary point (Lin et al., 2020), we extend Pareto stationarity to ϵ -Pareto stationarity.
Obviously, 0 -Pareto stationarity reduces to Pareto stationarity.
Definition 2.3.
( ϵ -Pareto stationarity).
For a given ϵ , x is ϵ -Pareto-stationary iff there exist { u τ } mτ = such that ∥ (cid:80) mτ
=1 u τ ∇ x f τ ( x ) ∥ ≤ ϵ, u τ ≥ 0 ∀ τ and (cid:80) mτ =
u τ = 1 .
Definition 2.4.
(Improvement function) (Montonen et al., 2018)
The improvement function of problem (1) is: H ( x, x ′ )
= max τ =1 ...,m { f τ ( x ) − f τ ( x ′ ) } .
Note that x ∗ satisfying x ∗ =
arg min x H ( x,
x ∗ )
(intuitively, x ∗ cannot be further improved) is Pareto stationary (Montonen et al., 2018).
To find x ∗ , one can perform steepest descent on H :
[Equation]
where x s is the iterate at iteration s , β is the learning rate satisfying H ( x s + βd, x s ) <
H ( x s , x s ) , and λ ′ is a hyper-parameter.
It can be shown that when s → ∞ , x s is Pareto stationary (Montonen et al., 2018).
In this paper, we focus on gradient-based MOO methods, including MGDA (Désidéri, 2012; Sener &
Koltun, 2018), PCGard (Yu et al., 2020), and CAGard (Liu et al., 2021a).
They assign weights to each objective’s gradient and find a common descent direction that decreases the losses of all objectives.
For example, MGDA finds the direction g ∗ ( x )
= (cid:80) mτ
=1 γ ∗ τ ∇ x f τ ( x ) in each iteration, where
[Equation]
Meta-Learning.
Meta-learning aims to achieve good performance with limited data and computation (Hospedales et al., 2020).
Most of them are gradient-based (Nichol et al., 2018; Deleu et al., 2022; Rajeswaran et al., 2019;
Zhou et al., 2019; Shu et al., 2019) or metric-based (Snell et al., 2017;
Lee et al., 2019; Vinyals et al., 2016).
Let T be the set of all m tasks, and w be the shared model parameter.
For a task τ ∈ T , let D τ be its dataset and L τ the corresponding loss function.
It tries to obtain task-specific parameter w τ from the shared w as w ∗ τ ( w ) .
Meta-learning is usually formulated as the following bilevel optimization problem (Ji et al., 2021):
[Equation]
The inner subproblem learns the task-specific parameter w τ for each τ , while the outer subproblem learns w by minimizing the average loss over tasks in T .
As m can be very large, usually a mini-batch B of tasks are uniformly sampled from T , and w is then updated as w s +1 =
w s − β 1 | B | (cid:80) τ ∈ B L τ ( w ∗ τ ( w ))
(Finn et al., 2017).
By taking each L τ ( w ∗ τ ( w )) in (4) as an objective, this can be regarded as a weighted sum in multi-task learning (Wang et al., 2021).
As observed in (Sener & Koltun, 2018; Yu et al., 2020), gradient descent on this weighted sum can suffer from the compromising phenomenon, in which the loss obtained on some task τ
′ can be much larger than the losses on the other tasks.
S OFT I MPROVEMENT M ULTI -O BJECTIVE META -L EARNING (SIMOL)
We take the view of meta-learning as multi-task learning in (Wang et al., 2021)
one step further and consider the meta-learning problem as the following multi-objective optimization (MOO) problem:
[Equation]
[Equation]
in which each task corresponds to an objective.
This considers all the individual tasks instead of simply considering the total loss over all tasks (Liu et al., 2021a).
Recently, Ye et al.
(2021) also use multi-objective learning into meta-learning.
However, their focus is not on addressing the compromising phenomenon and they do not treat each task as an objective.
Instead, besides minimizing the average task loss in (4), they consider adding some other objectives such as robustness to adversarial attacks.
Moreover, MGDA is still used to find the Pareto optimal solution.
However, as in other gradient-based MOO methods (Yu et al., 2020; Liu et al., 2021a), MGDA requires collecting gradients from all m objectives in each iteration (as can be seen from its optimization problem (3)).
This is computationally feasible only when there are a small number of objectives.
2 When each task is treated as an objective, the number of objectives can easily be in the millions (as in performing 5-way 1-shot classification on miniImageNet ).
Another widely adopted MOO based methods are the Chebyshev methods (Miettinen, 2012; Mao et al., 2020; Momma et al., 2022), which leverage the weighted Chebyshev problem to find the Pareto front.
However, these methods also cannot handle a huge number of tasks, as the computational complexity per epoch for these methods is O ( m 2 ) , where m is the number of tasks.
To alleviate this problem, one solution is to use only a mini-batch of objectives in each iteration.
For example, when a subset B of objectives is used, MGDA’s optimization problem in (3) becomes:
[Equation]
However, the descent direction then only considers objectives in B , and the original normalization constraint (cid:80) mτ =1 γ τ = 1 in (3) is also changed to (cid:80)
τ ∈ B γ τ = 1 .
The obtained solution may no longer Pareto optimal.
In the following, we demonstrate this by using a simple toy example with two objectives ( f 1 ( x ) and f 2 ( x ) , where x ∈ R 2 ) from (Liu et al., 2021a; Navon et al., 2022).
3 As can be seen from Figure 1, mini-batch MGDA (with a mini-batch of 1) cannot converge to the Pareto front.
3. S OFT I MPROVEMENT F UNCTION
In this section, we propose a scalable MOO solver with the use of the improvement function (Montonen et al., 2018).
The proposed solver is agnostic to the number of tasks, while still theoretically guaranteeing that the solution is Pareto optimal.
Using Definition 2.4, the improvement function for problem (5) is:
[Equation]
[Figure] Consider the optimization problem
[Equation]
where π is a probability density function on τ .
The following Lemma shows that (6) and (7) are equivalent when π is the Dirac delta distribution concentrated on the task corresponding to the maximum in (6).
All the proofs are in Appendix C.
[Equation]
Using (2) and Lemma [Equation] [Equation] w can be updated as
[Equation]
Taking the first-order approximation
[Equation]
the minimax theorem (Simons, 1995) can be used to swap the min and max operators in (9), as
[Equation]
The following Proposition shows that the inner minimization problem has a closed-form solution.
Proposition 3.2.
[Equation]
[Equation]
d
[Equation]
= −2 λ ′ ∥
E τ ∼ π ∇ w L τ ( w ∗ τ ( w )) | w = w s ∥ 2 , and the optimal d is d ∗ = − 1 λ ′
E τ ∼ π [ ∇ w L τ ( w ∗ τ ( w ) | w = w s ] .
The expectation in Proposition 3.2 requires sampling tasks from π .
An easier alternative is to sample tasks from the uniform distribution U ( · ) over the set T of all tasks, and then weighting each sampled task τ with r ( τ ) ≡ π ( τ ) /U
( τ ) .
Note that
[Equation]
We further parameterize r as a neural network r θ with parameter θ .
Using Proposition 3.2, we can then rewrite (11) as
[Equation]
where the last term (with another hyper-parameter λ ′′ ) is a penalty for enforcing the constraint in (12).
For notational simplicity, we denote the objective in (13) by K ( θ ) .
In principle, θ ∗ can be obtained from (13) by gradient ascent.
However, problem (13) involves an expectation over tasks.
Recall that we have a total of m tasks, and m can be huge.
Hence, using all of them to compute this expectation may not be feasible.
Instead, Let B be a mini-batch of k tasks, and denote the the mini-batched version of the objective in (13) as:
[Equation]
where ˆ λ ′ , ˆ λ ′′ are another set of hyper-parameters (which will be set in Proposition 3.3) corresponding to λ ′ , λ ′′ in (13).
Note that ˜ K T ( θ )
=
K ( θ ) .
Let B be the set of all size- k mini-batches (with k > 1 ).
The following Proposition bounds the difference between K ( θ ) , the original objective in (13), and the version 1 |B| (cid:80) B ∈B ˜ K B ( θ ) based on mini-batches.
Proposition 3.3.
Set
[Equation]
and
[Equation]
We have:
[Equation]
where
C≡
[Equation]
, and
[Equation]
Corollary 3.3.1.
When k ≪ m , (cid:16) K ( θ ) − 1 |B| (cid:80) B ∈B ˜ K B ( θ ) (cid:17) 2 ≤ G 1 k λ ′ + G 2 k λ ′′ .
In the experiments, m ≥ 10 6 , k ≈ 10 2 , and G 1 , G 2 ≤ 10 4 .
When λ ′ ≥ 10 2 , λ ′′ ≤ 10 − 2 , we have|B| (cid:80) B ∈B ˜ K B ( θ ) ∈ [ − 10 3 , − 1] during training, and ( K ( θ ) − 1 |B| (cid:80) B ∈B ˜ K B ( θ )) 2 ≤ 0 .
2 is small.
Thus, Proposition 3.3 shows that K ( θ ) can be decomposed into mini-batches as 1 |B| (cid:80) B ∈B ˜ K B ( θ ) .
This allows us to update θ by SGD over the task mini-batches as:
[Equation]
14) where β ′ is the learning rate.
Similarly, we approximate d ∗ by its mini-batch approximation ˜
d ∗ = − 1 λ ′
| B | [ (cid:80) τ ∈ B ∇ w L τ ( w ∗ τ ( w ))
| w = w s ] and update w as w s [-+
-]+1 =
w s + β ˜ d ∗ .
The whole procedure, which will be called Soft Improvement Multi-Objective Meta Learning (SIMOL), is shown in Algorithm 1.
Step 4 trains the base learner.
In the experiments, we use two popular meta-learning algorithms: MAML (Finn et al., 2017) and prototypical network (PN)
(Snell et al., 2017).
For MAML, the base learner is updated as
[Equation]
For the PN, w ∗ τ ( w ) =
[Equation]
) , where Q τ is the set of query examples for task τ , N C is the number of classes per epoch, f w is the model with parameter w , c k =| S k | (cid:80) ( x i ,y i ) ∈ S k f w ( x i ) , and S k is the set of examples belonging to class k .
Pseudo-codes for SIMOL-based MAML and PN are shown in Algorithms 2 and 3 of Appendix B, respectively. Input: T , batch size k , learning rates β and β ′ for w , ˜ d ∗ = 0 and θ , respectively.
1 for s = 1 , 2 , . . .
, S do Reset ˜ d ∗ = 0 ; for τ = 1 , 2 , . . , k do obtain r θ s ( τ ) ∇ w s L τ ( w ∗ τ ( w s )) for task τ ;
[Equation]
C ONVERGENCE A NALYSIS
Lemma 4.1.
Define
[Equation]
where ∆( θ ) ≡ − 1 λ
E τ ∼
U r θ ( τ ) ⊥
[( L τ ( w ∗ τ ( w ) + d ∗ ) − L τ ( w ∗ τ ( w ))] · ⊥ [ E τ ∼ U r θ ( τ )[ L τ ( w ∗ τ ( w ) + d ∗ ) − L τ ( w ∗ τ ( w ))]] , and ⊥ is the stop gradient operator.
Then, ∇ w R ( θ, w ) | w = w s = − d ∗ , and ∇ θ R ( θ, w s )
=
∇ θ K ( θ ) .
This allows interpreting the updates in (14) and (8) as performing Gradient Descent Ascent (GDA) (Singh et al., 2000) on (16).
Thus, we can leverage game theoretical tools (Lin et al., 2020) in the analysis.
Let ˜ R
( θ, w ; B )
[Equation] ∈ B [ r θ ( τ )] [ L τ ( w ∗ τ ( w )] − [Equation]
L [Equation]
[Equation]
be the mini-batch version of R , and U ( B ) be the uniform distribution over task mini-batches.
The following Theorem shows that Algorithm 1 converges to an ϵ -Pareto stationary point of (5).
Theorem 4.2.
Assume that (i) L τ ( w ∗ τ ( · )) is L-smooth and r θ ( · ) is µ -strongly concave.
(ii) The domain of θ is a convex and bounded set with diameter D > 0 ,
(iii) E B ∼ U ( B ) [ ∇ θ ˜ R
( θ, w ; B ) −∇ θ R ( θ, w )]
= 0 , and E B ∼ U ( B ) ∥∇ θ
˜ R
( θ, w, B )
− ∇ θ R ( θ, w ) ∥ 2 ≤ σ 2 .
Assume the first-order approximation in (10), and take β = Θ (cid:0) 1 /D 2 σ
( L 2 + σ 2 ) (cid:1) , β ′ =
Θ(1 /Lσ 2 ) .
Algorithm 1 converges to an ϵ -Pareto stationary point of (5) with a rate of O (1 /ϵ 8 ) .
If L τ ( w ∗ τ ( w )) is also µ ′ -convex w.r.t.
w and ϵ = 0 , the 0 -Pareto stationary point is also global Pareto optimal.
Assumption (i) is commonly used in the literature (Collins et al., 2020;
Zhou et al., 2021;
Finn et al., 2019); while (ii) and (iii) are from (Lin et al., 2020).
Theorem 4.2 shows that the proposed method can obtain an ϵ -Pareto stationary point (or global Pareto optimal point for convex objectives) regardless of m , the number of tasks/objectives.
Corollary 4.2.1.
Consider the MAML base learner update in (15).
Assume that ∇ w L τ ( w ) is HessianLipschitz continuous, bounded, Lipschitz-continuous, and w is bounded.
Then, Algorithm 1 converges to an ϵ -Pareto stationary point of (5) with a rate of O (1 /ϵ 8 ) .
Corollary 4.2.1 is an application of Theorem 4.2 revealing that SIMOL with MAML can also converge to a Pareto point.
Convergence of the outer loop is slower than the O (1 /ϵ 2 ) rate of standard
MAML (Fallah et al., 2020).
However, standard MAML only guarantees convergence to stationary points of w but not to Pareto-stationary points.
Moreover, as will be seen in Section 5.1, empirically, the proposed method has comparable or even slightly faster convergence speed than MAML and other meta learning baselines.
Besides, most the gradient-based MOO approaches (except CAGrad (Liu et al., 2021a)) do not provide convergence rate analysis; while CAGrad requires that all task gradients are available in each epoch, which is very expensive (as will be demonstrated in Section 5.2).
E XPERIMENTS
In this section, we perform experiments on few-shot regression (Section 5.1), few-shot classification (Section 5.2), and reinforcement learning (Section 5.3).
All experiments are run on a GeForce RTX 2080 Ti GPU and Intel(R)
Xeon(R) CPU E5-2680.
Our implementations are based on the popular open-source meta-learning library Learn2Learn (Arnold et al., 2020).
5. F EW - SHOT R EGRESSION
Setup . We follow the setup in (Finn et al., 2017; Li et al., 2017).
The target function for task τ is y
= a τ sin( x + b τ ) , where a τ and b τ are sampled uniformly from [0 . 1 , 5 . 0] and [0 , π ] , respectively.
We generate 160 , 000 meta-training tasks and 1 , 000 meta-testing tasks.
A multilayer perceptron with 2 fully-connected (FC) layers (each of size 32 ) and ReLU activation is used as meta-learner and re-weighting network.
The re-weighting network uses all mini-batch instances as input.
To ensure that the re-weighting network output is positive, we take the square of its last layer’s output as output.
The backbone meta-learning algorithm is MAML (Finn et al., 2017).
We use Adam (Kingma & Ba, 2014), with an initial learning rate of 0 . 01 , to update the base learners for 5 steps in the inner-loop.
For the outer-loop, we compare (i) minimizing the (single objective) of overall task loss as in MAML; versus performing MOO with (ii) mini-batch MGDA
(Désidéri, 2012; Ye et al., 2021), (iii) mini-batch CAGrad, using the same hyper-parameters as in (Liu et al., 2021a), (iv) mini-batch PCGrad, using the
) , ∇ x ⊥ ( h ( x )) = 0 , where h ( · ) is any differentiable function. same hyper-parameters as in (Yu et al., 2020), (v) the proposed SIMOL, and (vi) updating of (6) with a mini-batch version of the improvement function in (6).
Hyperparameters for MAML, mini-batch
MGDA, and mini-batch CAGrad follow (Finn et al., 2017), while that for the proposed SIMOL are in Appendix D.
The mini-batch size is 16 .
We do not compare with the batch versions of MGDA/CAGrad, as computing all task gradients takes very large memory and time.
The initial learning rate for the outer loop is 0 .
The experiment is repeated three times with different random seeds.
For performance evaluation, we use the mean-squared-error (MSE) over all meta-testing tasks.
We also report the worst- 10% MSE, which is the average MSE for the 10% worst-performing meta-testing tasks.
Results . Table 1 shows the MSE and its 95% confidence interval (computed as in (Finn et al., 2017; Li et al., 2017)).
As can be seen, SIMOL consistently outperforms MAML, and the mini-batch versions of MGDA, CAGrad, PCGrad and improvement function in terms of both the overall and worst-10% MSEs.
Indeed, the mini-batch versions of MGDA, CAGard, PCGrad and improvement function are even worse than the original MAML.
Figure 2 shows the convergence of MSE with the number of training epochs.
As can be seen, SIMOL converges slightly faster than the other baselines.
The ∗ denotes that the improvement over the second-best is statistically significant (at a significance level of 0.1 using the paired t-test) [Table] [Figure]
5. F EW - SHOT I MAGE C LASSIFICATION
Setup . In this section, we perform 5 -way- 1 -shot and 5 -way- 5 -shot classification on the miniImageNet (Ravi & Larochelle, 2016) and tieredImageNet data (Ren et al., 2018).
Following (Finn et al., 2017), we split the miniImageNet dataset into a meta-training set with 64 classes, a meta-validation set with 16 classes, and a meta-testing set with 20 classes.
The total number of meta-training tasks is (cid:0) 645 (cid:1) ≈ 7 . 6 × 10 6 .
Similarly, as in (Zhou et al., 2019), we split the tieredImageNet dataset into a meta-training set with 351 classes, a meta-validation set with 97 classes, and a meta-test set with 160 classes.
The total number of meta-training tasks is (cid:0) 3515 (cid:1) ≈ 4 .
3 × 10 10 .
For both datasets, we randomly select 1 , 000 meta-testing tasks for evaluation.
We use two backbone meta-learning algorithms, MAML and prototypical network (PN)
(Snell et al., 2017), with hyper-parameters following the original papers.
Following (Finn et al., 2017; Li et al with a 3-layer FC as the reweighting network.
The optimizer is Adam.
The learning rates for the meta networks are 0 .
003 for MAML-based methods and 0 .
001 for PN-based methods, respectively.
The learning rate of the reweighting network is 0 .
08 for SIMOL and 0 .
for SIMOL+PN.
The mini-batch size is 32.
More details on the hyperparameters are in Appendix D.
The proposed SIMOL is compared with (i) minimizing the overall task loss as in standard MAML, (ii) mini-batch MGDA, and (iii) mini-batch CAGrad.
We also compare with MAML variants including (iv) Reptile (Nichol & Schulman, 2018), (v) FOMAML (Finn et al., 2017), (vi) MetaMinibatchProx
(Zhou et al., 2019), (vii) TSA-MAML (Zhou et al., 2021), (viii) IMAML (Rajeswaran et al., 2019)) (ix) MTL (Wang et al., 2021), a multi-task learning based maml approach, and the standard prototypical network (Snell et al., 2017).
The evaluation metrics are similar to those in
Section 5.1, but with accuracy instead of MSE.
Experiments are repeated three times with different random seeds.
Results . Tables 2 and 3 show the meta-testing accuraccies and 95% confidence intervals on miniImageNet and tieredImageNet , respectively.
For MAML and its variants, SIMOL consistently outperforms all the other baselines in terms of both the overall and worst-10% accuracies.
The same is also observed on the meta-learning algorithm prototypical network.
This demonstrates that SIMOL is useful for both gradient-based and metric-based meta-learning approaches.
Note that (mini-batch) MGDA and CAGrad do not perform good in terms of both overall and worst10% accuracies, showing that they cannot be straightforwardly extended to the use of mini-batch.
On the other hand, the batch versions of MGDA and CAGrad are computationally impractical.
Tablecompares the per-epoch running time in training stage of standard MAML, SIMOL, and batch MGDA and CAgrad.
Experiment is performed on 5-way-5-shot classification with the MAML algorithm on miniImageNet .
As can be seen, while SIMOL has comparable per-epoch running time as MAML, batch MGDA and CAgrad are much more computationally expensive (around 432 , 000 times slower).
[Table]
5. R EINFORCEMENT L EARNING
In this experiment, experiments are performed on the HalfCheetach-Dir and Walker-Dir environments in Mujoco (Todorov et al., 2012).
In both environments, each task corresponds to a random direction in the XY-plane, and the agent (Walker/ HalfCheetach) learns to run in that direction as far as possible.
The reward is the average velocity minus control costs.
We again use MAML as the meta-learning algorithm, and the base reinforcement learning algorithm is vanilla policy gradient (VPG) (Sutton et al., 1999).
Following (Rothfuss et al., 2018), the policy network has two 64 × 64 FC layers with tanh activation, while the critic is a linear state-value function whose parameters are obtained by minimizing least-square.
The re-weighting network has two 64 × 64 FC layers with ReLU activation.
Following (Zintgraf et al., 2019), we use MAML as the baseline.
Figure 3 shows the convergence of the accumulated reward with the number of training iterations.
As can be seen, SIMOL consistently outperforms MAML.
Moreover, the convergence of MAML is less
[Table]
stable, as also reported by (Rothfuss et al., 2018).
On the other hand
[Table]
the convergence of SIMOL is smoother and more stable
[Table]
[Figure]
SIMOL-VPG and MAML-VPG.
Results are averaged over 3 trials.
5. A BLATION S TUDY
In this experiment, we use the setup in Section 5.1, and vary the number of FC layers in SIMOL’s meta-learner. The number of training epochs is always fixed to 10 , 000 . Table 5 shows the MSE’s on 2-shot regression. as can be seen, the use of 2 FC layers has the best overall MSE and worst- 10%
MSE. the deeper networks may not be sufficiently trained with the fixed number of training epochs, leading to worse performance. [Table] [Table]
C ONCLUSION
In this paper, we propose to avoid the compromising phenomenon in meta-learning by reformulating it as a multi-objective optimization (MOO) problem, in which each task is an objective.
However, current gradient-based MOO solvers cannot scale to a large number of objectives.
With the use of improvement function and mini-batch, we propose a scalable gradient-based solver with theoretical guarantees to Pareto-optimality.
Empirical studies on few-shot regression, few-shot classification, and reinforcement learning demonstrate that the proposed method is efficient, and has good generalization in terms of both overall performance and performance on the poorly-performing tasks.
A CKNOWLEDGEMENT
This research was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (Grant 16200021). A T OY E XAMPLE
The definitions of f 1 and f 2 are: [Equation]
where l [Equation] | [Equation]
[Equation]
, g
[Equation]
( x
[Equation]
(cid:16) ( − x 1 + 7)
[Equation]
1 ∗ ( − x
[Equation]
− 8)
[Equation]
(cid:17) / 10 − 20 , g
[Equation]
( x
[Equation]
(cid:16) ( − x
[Equation]
−
[Equation] [Equation]
+ 0 .
[Equation]
( − x
[Equation]
− 8)
[Equation]
(cid:17)
[Equation]
10 − 20 , c 1 ( x
[Equation]
max( [Equation] [Equation] x
[Equation] [Equation] [Equation] c 2 ( x ) = max(tanh( − 0 .
5 x 2 ) , 0) .
For mini-batch MGDA, and SIMOL the probability to sample task 1 is 2 / 3 and 1 / 3 for task 2.
The learning rates for MGDA and mini-batch MGDA are both 0 .
The learning rate for SIMOL’s meta-learner is 0 .
01 , while that for its re-weighting network is 0 .
Note that the meta-learner and the re-weighting network are represented by learnable vectors.
B P SEUDO -C ODES
Algorithms 2 and 3 show the pseudo-codes for SIMOL-based MAML and PN, respectively. The key differences between SIMOL and MAML/PN are highlighted in blue.
Algorithm 2: for
Input: T , and total epoch S .
B is the batch size, β and β ′ are learning rates for w and θ .
α is the learning rates for inner loop.
1 for epoch s = 1 , 2 , 3 , . . .
, S do Sample tasks 1 , 2 , . . .
B ; for Every task τ do
Receive r θ s ( τ ) ;
Compute adapted parameters with gradient descent w ∗ τ ( w ) =
w s − α ∇ w L τ ( w s ) ; w s +1 =
w s − β s 1 B (cid:80) B τ =1 r θ s ( τ ) ∇ w L τ ( w ∗ τ ( w )) ; θ s +
= θ s + β ′ s ˜ K
( θ, B ) ;
1 for epoch s = 1 , 2 , 3 , . . .
, S do
Sample tasks 1 , 2 , . . .
B ; for Every task τ do for Every class c do
Select the support S cτ and query set Q τ ;
Compute prototype c τ = 1 N C (cid:80) ( x i ,y i )
∈ S cτ f θ ( x i ) ;
Calculate w ∗ τ ( w ) = 1 | Q τ | N C (cid:80) x ∈ Q τ exp( −∥ f w ( x ) − c k ∥ 2 ) (cid:80) k ′ exp( −∥ f w ( x ) − c k ′ ∥ 2 ) ;
Receive r θ s ( τ ) ; w s +1 =
w s − β s 1 B (cid:80) Bτ =1 r θ s ( τ ) ∇ w L τ ( w ∗ τ ( w )) ; θ s +
= θ s + β ′ s ˜ K
( θ, B ) ;
C P ROOFS C. P ROOF OF L EMMA 3.
Proof.
Note that [Equation] where p
[Equation]
0 otherwise .
First, we have
[Equation]
This can be done by setting π ( τ ) =
p ( τ ) .
Next, we show that
[Equation]
This is established since
[Equation]
= 0 , where τ ′ =
[Equation]
(a) is due to the fact that [ L
[Equation]
] , ∀ τ based on the property of τ ′ .
Therefore
[Equation] we have [Equation]
and
[Equation]
Thus
[Equation]
C. P ROOF OF P ROPOSITION 3.
Proof.
Using the first-order Taylor expansion, [Equation]
[Equation]
Taking the derivatives w.r.t. d ,
[Equation]
Setting the above to zero
[Equation]
we have E U (
[Equation]
[Equation] θ[-( τ ) ∇ w L D τ ( w ∗ τ ( w s ))] + d = 
 and we find the optimum:-]
[Equation]
w
[Equation]
] + d =
[Equation]
and
[Equation]
Putting d ∗ back to the objective in (17)
[Equation]
we have:
[Equation]
Next, we have the following two Lemmas.
Lemma C.1.
When L τ ( w ∗ τ ( w s + d ))
≈ L τ ( w ∗ τ ( w s ) + ∇ w L τ ( w ∗ τ ( w s ))
⊤ d , Eq.
(9) is convex w.r.t. d and concave.
w.r.t. r θ ( τ ) .
Proof.
Putting L τ ( w ∗ τ ( w s + d ))
≈ L τ ( w ∗ τ ( w s ) + ∇ w L τ ( w ∗ τ ( w s ))
⊤ d into Eq.
(9), we have:
[Equation]
For ∇ w L τ ( w ∗ τ ( w s ))
⊤ d
+ λ ′ 2 ||
d || 2 , since ∇ w L τ ( w ∗ τ ( w s ))
⊤ d is both convex and [-concave, and λ ′ 
-]concave w.r.t.
d , and λ ′ 2 || d || 2 is convex w.r.t. d . Then, ∇ w L τ ( w ∗ τ ( w s ))
{+⊤ d + λ ′ 
+}
∥ d ∥ 2 is convex w.r.t d .
Thus, the sum of two convex functions ∇ w L τ ( w ∗ τ ( w s ))
⊤ d + λ ′ {+
 ∥ d ∥+} 2 is convex w.r.t. d .
Thus, Eq.
(9) is convex w.r.t. d .
Similarly, since − λ ′′ 2 ( E U ( τ ) [ r θ ( τ )] − 1) 2 is concave w.r.t. r θ ( τ ) .
Therefore, Eq.
(9) is also concave w.r.t. r θ ( τ ) .
Lemma C.2.
When L τ
[Equation]
w ∗ τ
[Equation]
w s + d
[Equation]
L τ
[Equation]
Proof.
The above equation is established by the minimax theorem (Simons, 1995) when Eq.
(9) is convex w.r.t. d and concave w.r.t.
r θ ( τ ) . This holds by using Lemma C.1.
Lemma C.3.
If every B ∈ B is a set consists of a unique selection of k ( m > k > 1 ) tasks out of m (the total number of tasks) tasks without replacement. For any continuous function f ( · ) , we have:
[Equation]
where C 1 =
[Equation]
Proof.
Note that
[Equation]
B ∈B (cid:88) τ ′ [Equation] (cid:88) τ
[Equation]
B
[Equation]
τ
[Equation]
[Equation]
[Equation]
(cid:104)(cid:0)
[Equation]
(cid:1) [Equation] (cid:0) n − 1 B − [Equation]
(cid:1) (cid:88)
B ∈B (cid:88) τ [Equation] B [Equation]
( τ )
[Equation]
 =
[Equation]
(cid:88)
[Equation]
[Equation] f ( τ )] [Equation] (cid:104)(cid:0) n − 1 B − 1 (cid:1)
[Equation]
12 (cid:0) n − 2 B − 2 (cid:1)(cid:105) (cid:0) n − 1 B − 1 (cid:1) m(cid:88)
B ∈B (cid:88) τ ∈ B f ( τ )
 =
C
[Equation]
[ (cid:88)
[Equation]
(cid:34)(cid:88)
[Equation]
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
[ f ( [Equation] [Equation] (a) is due to the multinomial theorem (Bolton, 1968) and 1 ( τ, τ ′ ) =
(cid:26) 2 τ ̸ = τ ′ 1 otherwise .
(b) is due to the fact that every task τ has occurred exactly (cid:0) n − 1 B − 1 (cid:1) times, and every tuple ( τ, τ ′ ) has occurred exactly (cid:0) n − 2 B − 2 (cid:1) times.
Lemma C.4.
If every B ∈ B is a set consists of a unique selection of k ( m > k > 1 ) tasks out of m (the total number of tasks) tasks without replacement. Let g ( τ ) =
[ g 1 ( τ ) , g 2 ( τ ) , . . . ,
g m ( τ )] , where g i ( · ) ’s are continuous functions. We have
[Equation]
[-where C 1 = k 2 ( m − 2 k − 2 ) m 2 , and C 
-](cid:88)
B
[Equation]
(cid:88) τ ∈ B ∥ g ( τ ) ∥ 2 , where C 1 = k 2 ( m − 2 k − 2 ) m 2 {+, and C 
+}
= (cid:104)(cid:0) m − 1 k − 1 (cid:1) − 1 2 (cid:0) m − 2 k − 2 (cid:1)(cid:105) / (cid:0) m − 1 k − 1 (cid:1) m 2 (cid:0)
m − 2 k − 2 (cid:1) .
Proof.
Observe that ∥ g ( τ ) ∥ 2 = (cid:80)
i g i ( τ ) 2 .
The remaining follows from Lemma C.3.
Proof.
(of Proposition 3.3) Using Lemmas C.3 and C.4, we have:
[Equation]
where ˜ d ∗ := 1 | B | (cid:80) B ∈B r θ
[Equation]
τ
[Equation]
∇ w L [Equation]
w [Equation] ) | w = w s . Recall ˆ λ ′ and ˆ λ ′′ are defined in Sec.
Then,
[Equation]
By noting that ˜ K T ( θ ) is exactly K ( θ )
[Equation]
we obtain the first claim.
Regarding the second claim
[Equation]
note that when m is large,
[Equation]
The first approximation is due to the fact that (cid:0) m − 1 k − 1 (cid:1) ≫ 12 (cid:0) m − 2 k − 2 (cid:1) .
The third and forth equations are due to the property of combinatorics ( (cid:0) mk (cid:1) =
mk (cid:0) m − 1 k − 1 (cid:1) ).
Putting 1 k − 1 back into Eq.
(18), we obtain the results.
C. P ROOF OF L EMMA 4.
Proof.
We have:
[Equation]
Also notice that:
[Equation]
Therefore, ∇ w R ( θ, w ) | w = w s = − d ∗ .
C. P ROOF OF T HEOREM 4.
Definition C.5.
(Weak Global Pareto Optimality) (Mäkelä et al., 2016)
A solution x ∗ of problem (5) is weakly global Pareto optimal if there does not exist another x such that f τ ( x ∗ ) ≥ f τ ( x ) for τ ∈ { 1 , . . .
, m } .
Theorem C.6.
(Theorem 5 in (Miettinen & Mäkelä, 1995))
For a multi-objective optimization problem min w [ f 1 ( w ) , f 2 ( w ) , . . .
, f m ( w )] and its corresponding improvement function H ( w, w ∗ ) .
A necessary condition for w ∗ ∈ R n to be weakly global Pareto optimal is that w ∗ =
arg min w H ( w, w ∗ ) .
Moreover, if f i ( w ) is convex ∀ i , then it is a sufficient condition.
We observe the following.
By replacing f τ =
L τ ◦ w ∗ τ ( w ) , ∀ τ , the above theorem can be directly applied to the MAML setting.
2) f in our setting can be a neural network. Thus, the convexity condition for f τ ( w ) can be hard to satisfy.
Here, we give the a more relaxed version of the above theorem:
Theorem C.7.
w ∗ := arg min w H ( w, w ∗ ) is Pareto stationary.
Proof.
Using Theorem 2 in (Miettinen & Mäkelä, 1995), we have [Equation] w [Equation]
)) , where ∂ is reloaded as sub-gradient, and conv is the convex set. Note that 0 ∈ ∂ w H ( w, w ∗ ) due to the fact that the element in the convex set of the union of subgradients is still a sub-gradient, and for w ∈ { w | 0 ∈ ∂ w H ( w, w ∗ ) } , the sub-gradient ∂ w H ( w, w ∗ ) is zero.
Then, we always have (cid:80) i
w i ∂
w ( f i ( w ) − f i ( w ∗ ))
= 0 , (cid:80) i w i = 1 , based by definition of a convex set, where w i ∈ [0 , 1] , ∀ i is a real number.
By simplifying the above term, we have (cid:80) i w i ∂ ( f i ( w ) − f i ( w ∗ ))
= (cid:80) i
w i
∂
w f i ( w ) = 0 .
which is exactly the definition of Pareto stationary point.
We now show convergence of Algorithm 1.
Lemma C.8.
In each epoch s of Algorithm 1, set ˆ λ ′′ = (cid:26)
[Equation]
∞ otherwise , we have:
[Equation]
When equality holds, w s is a stationary point.
Proof.
By Lemma C.2, the solutions of
[Equation]
are the same as those from
[Equation]
Solving θ , the above minimax problem degenerates to min
[Equation]
Note that the above optimization problem is exactly min d H ( w s + d, w s ) + ˆ λ ′
||
d
|| 2 , where d is the steepest descent direction.
Then, due to the property of steepest descent direction, we always have H ( w s + d, w s ) ≤ H ( w s , w s ) , and equality holds iff || d
|| 2 = 0 , where w s a stationary point according to the definition.
Lemma C.9.
Algorithm 1 converges to a Pareto stationary point.
Moreover, if L τ ( w ∗ τ ( w )) is convex, Algorithm 1 converges to a global weak Pareto optimal point.
Proof.
From Lemma C.8, the sequence { H ( w s + d, w s , w s +11: m , w s 1: m ) } s is decreasing.
Also obviously, H ( w s + d, w s , w s +11: m , w s 1: m ) has a lower bound.
Thus, using the monotone convergence theorem,
Algorithm 1 converges.
Using Theorem C.7, it is Pareto stationary.
Together with Theorem C.6, it is global Pareto optimal if L τ ( w ∗ τ ( w )) is convex.
Lemma C.10.
(i)
If L τ ( w ∗ τ ( w + d )) is L-smooth and µ ′ -convex w.r.t. w and d , and r θ ( τ ) is µconcave, r θ ( τ ) and L τ ( w ∗ τ ( w s +1 )) are bounded (i.e., r θ ( τ ) ≤ C max and L τ ( w ∗ τ ( w s +1 )) ≤ C ′ max ), then R ( θ, w ) is L + C max µ ′ convex w.r.t. w and 2 L -smooth w.r.t.
w, θ and C ′ max µ concave w.r.t.
θ . (ii)
If L τ ( w ∗ τ ( w + d )) is L -smooth w.r.t. w and d , and r θ ( τ ) is µ -concave, then
˜ R
( θ, w, B ) is L -smooth w.r.t. w, θ , and C ′ max µ -concave w.r.t.
θ .
Proof.
For claim (i), we first prove that for any differentiable functions f, g , ⊥ [ f ( x )] ⊥ [ g ( x )] is 0-smooth and convex w.r.t. x .
Note that
[Equation]
due to the property of the stop gradient operator.
Thus,
[Equation]
For convexity, by using the property of 0-smoothness, we have
[Equation]
Thus, ⊥ [ f ( x )] ⊥ [ g ( x )] is convex due to the definition of convexity. Therefore, ⊥ [ f ( x )] ⊥ [ g ( x )] is also 0-smooth and convex w.r.t.
x .
A direct application of the above, we obtain ⊥ [ f ( x )] is 0-smooth and convex w.r.t.
x by setting g ( x ) ≡ 1 .
Using the above, we have ⊥ [( L τ ( w ∗ τ ( w ) + d ) − L τ ( w ∗ τ ( w ))] , ⊥ [ E
[Equation]
∼ U r θ ( τ )[ L τ ( w ∗ τ ( w ) + d ∗ ) − L τ ( w ∗ τ ( w ))]] , ⊥ [( L D T ( w [Equation] ( w ) [Equation]
d [Equation] [Equation] w
[Equation]
w ))]] ⊥ E
[Equation]
∼ U r θ ( T )[ L D T ( w ∗ T ( w ) + d ∗ ) − L D T ( w ∗ T ( w ))] are also 0-smooth and convex. Therefore r θ ( T ) ⊥ [( L D T ( w ∗ T ( w ) + d ∗ ) − L D T ( w ∗ T ( w ))]] ⊥
[ E T ∼ U r θ ( T )
[ L D T ( w ∗ T ( w ) + d ∗ ) − L D T ( w ∗ T ( w ))]] is convex and L -smooth (as
L ≥ 0 ).
Also, we have ⊥ ( r θ ( τ ))
[ L τ ( w ∗ τ ( w )]] is C max µ ′ -convex and L -smooth due to our assumption.
Then, E τ ∼ U [ ⊥ ( r θ ( τ ))
[ L τ ( w ∗ τ ( w )]] is C max µ ′ -convex and L -smooth since the addition of convex functions implies E τ ∼ U
[ ⊥ ( r θ ( τ ))
[ L τ ( w ∗ τ ( w )]] is C max µ ′ -convex, and the adding of smooth functions implies E τ ∼ U [ ⊥ ( r θ ( τ )) [ L τ ( w ∗ τ ( w )]] is L -smooth, and taking the average does not affect the results.
Thus, R ( θ, w ) is C max µ ′ -convex and L -smooth.
For r θ , since − λ ′ 2 ∇ θ ( E p [ r θ s ( τ )] − 1) 2 + ∆( θ ) is concave, ∆( θ ) is 2 µC ′ max -concave.
Then ∆( θ ) − λ ′′ 2 ( 1 B (cid:80) τ
[ r θ ( τ )] − 1) 2 is also C ′ max µ -concave.
Thus R ( θ, w ) is C ′ max µ -concave.
[-Assume the first-order approximation in Eq. (11), and take β = Θ (cid:0) 1 /D 2 σ 
 ( L 2 + σ 2 ) (cid:1) , β ′ = Θ(1 /Lσ 2 ) .-]The proof of claim (ii) is similar.
Proof.
(of Theorem 4.2) Recall Lemma C.10 on the properties of convex and smooth for ˜ R .
Combine it with the assumptions in Theorem 4.2, and use Theorem 4.9 in (Lin et al., 2020), we obtain the bound of O ( 1 ϵ 8 ) when B = 1 .
When B > 1 , note that using Lemma A.2 in (Lin et al., 2020), we have:
[Equation]
Therefore, let σ 2 ′ =
σ 2 B and use Theorem 4.9 in (Lin et al., 2020) shows the bound of O ( 1 ϵ 8 ) .
The remaining part that the fixed point w of max θ ˜ R ( θ, w, B ) is global Pareto optimal (resp.
Pareto stationary) can be obtained by using Lemma C.9, which says that the fixed point of Algorithm 1 is global Pareto optimal (resp.
Pareto stationary).
Finally, we show that using SIMOL on MAML can guarantee convergence.
Proof.
(of Corollary 4.2.1) First, we show that if ∇ w L τ ( w ) is Hessian-Lipschitz continuous, bounded and Lipschitz-continuous, and ∥ w ∥ is bounded, then w ∗ τ ( w ) is also Hessian-Lipschitz continuous, bounded and Lipschitz-continuous.
Note that
[Equation]
It is easy to see that w ∗ τ ( w ) is also Hessian-Lipschitz continuous, bounded and Lipschitz continuous.
Applying Lemma 3 in (Collins et al., 2020), we obtain that L τ ( w ∗ τ ( w )) is C -smooth, where C is positive.
Then setting C =
L and using Theorem 4.2, we get the desire result.
D H YPER - PARAMETER S ELECTION OF SIMOL
For the few-shot regression experiment (section 5.1), the regularization parameters ˆ λ ′ , ˆ λ ′′ are selected from { 0 . 001 , 0 . 01 , 0 . 1 , 1 } , and learning rate β ′ is selected from { 0 . 01 , 0 . 03 , 0 . 1 , 0 . 3 , 1 } based on the validation set.
For the few-shot classification experiments (section 5.2), we use the ˆ λ ′ , ˆ λ ′′ ) combination selected from few-shot regression, while the learning rate β ′ is selected from { 0 . 0003 , 0 . 0008 , 0 . 01 , 0 . 03 , 0 . 08 , 0 . 1 } for the 1-shot miniImageNet task based on the validation set.
this is then also used in the other few-shot classification experiments.