Implicit Kernel Meta-Learning Using Kernel Integral Forms
Abstract
Meta-learning algorithms have made significant progress in the context of meta-learning for image classification but less attention has been given to the regression setting.
In this paper we propose to learn the probability distribution representing a random feature kernel that we wish to use within kernel ridge regression (KRR).
We introduce two instances of this meta-learning framework, learning a neural network pushforward for a translationinvariant kernel and an affine pushforward for a neural network random feature kernel, both mapping from a Gaussian latent distribution.
We learn the parameters of the pushforward by minimizing a meta-loss associated to the KRR objective.
Since the resulting kernel does not admit an analytical form, we adopt a random feature sampling approach to approximate it.
We call the resulting method
Implicit Kernel Meta-Learning (IKML).
We derive a meta-learning bound for IKML, which shows the role played by the number of tasks T , the task sample size n , and the number of random features M .
In particular the bound implies that M can be the chosen independently of T and only mildly dependent on n .
We introduce one synthetic and two real-world meta-learning regression benchmark datasets.
Experiments on these datasets show that IKML performs best or close to best when compared against competitive meta-learning methods.
INTRODUCTION
A common scenario in the real world is learning from similar tasks in order to transfer this knowledge to new tasks.
In machine learning this setting is called meta-learning or learning-to-learn [Baxter, 2000, Thrun and Pratt, 1998], where we assume that a set of tasks are sampled from a metadistribution on supervised learning problems.
The goal is to design meta-algorithms which from a set of tasks output a learning algorithm.
This algorithm should perform well on average with respect to the tasks sampled from the metadistribution, analogous to having low risk in supervised learning.
Meta-learning operates on top of an inner algorithm, tuning it to perform better on new tasks.
The meta-algorithm acts at an outer level, relying on the inner algorithm to compute a meta-loss and corresponding meta-gradient, based on which a meta-parameter associated to the inner algorithm is updated [see e.g. Franceschi et al., 2018].
For example, in regression settings, a common choice of inner algorithm is ridge regression and the meta-parameter is a representation or embedding shared across the tasks that we wish to meta-learn
[Bertinetto et al., 2018].
There has been considerable interest in meta-learning for few-shot image classification [Finn et al., 2017, Koch et al., 2015, Li et al., 2017, Ren et al., 2018, Rusu et al., 2018,
Snell et al., 2017, Vinyals et al., 2016] but less attention has been given to designing meta-learning algorithms for regression.
Most few-shot regression benchmarks fall under that of interpolating sinusoidals or a variety thereof [Finn et al., 2017, 2018, Oreshkin et al., 2018] which lacks many aspects of real-world regression problems such as being multivariate and noisy.
This highlights the importance of more realistic meta-learning regression datasets and how to design meta-learning algorithms in this setting.
In this paper we aim to close this gap.
Meta-learning algorithms employ a variety of different kinds of base algorithms, ranging from metric based to optimization based, and to black-box ones.
A common theme is to learn a shared representation which lead to faster adaptation of a base learning algorithm to new tasks.
Often the representation is modeled by a neural network.
Indeed, recently [Raghu et al., 2019, Tian et al., 2020] observed that the representation is the most important part of meta-learning algorithms.
In this paper we extend this thinking further, in that we implicitly learn the representation via a kernel function from a large class of kernels defined by a random feature form.
This kernel is in turn implicitly parametrized by a neural network pushforward which is learned by a meta-algorithm.
When using the random feature family of translation-invariant kernels this has two main advantages: since kernel algorithms can be expressed in terms of inner products of features which are simple to compute we don’t have to work with this high-dimensional feature space directly.
We show that modeling the kernel directly leads to improved performance in the meta-learning regression case.
A second advantage is that translation invariant kernels might be used as “plug-in” representations.
We also experiment with using a neural network random feature representation, effectively combining ensembling with with random features.
Contributions
The principal contribution of this paper is a method for meta-learning regression together with a bound on the excess risk which highlights how problem-specific quantities impact the number of random features needed to generalize.
In particular, our method can be used to learn within a family of translation invariant kernels that is wellsuited when using kernel ridge regression as the class of base learning algorithms.
According to Bochner’s Theorem [see e.g. Rahimi et al., 2007], these kernels are parameterized by a distribution in the frequency space.
In line with [Li et al., 2019], we parametrize this distribution as a neural network pushforward.
The weights of the network are learned from a sequence of datasets within a meta-learning setting.
Although we focus on distributions in the context of Bochner’s theorem, our framework extends directly to radial kernels using Schoenberg’s theorem [Schoenberg, 1938].
Additionally we experiment with using a neural network random feature kernel, an extension of R2D2 [Bertinetto et al., 2018], and show competitive performance.
Finally, we introduce three novel meta-learning regression benchmark datasets, one synthetic and two real-world and show that our algorithm ranks at the top or close to competing meta-learning regression algorithms.
We believe these results, including the theoretical guarantees together with the flexibility and ease of our method, make it a competitive candidate to be used as a plug-in meta-learning algorithm in general contexts.
Related Work Learning-to-learn or meta-learning can be traced back to at least [Schmidhuber, 1987] with one seminal work being [Baxter, 2000].
Well-developed theory exists in the batch case [Maurer, 2005, Maurer et al., 2016] and lately similar results have been developed in the online setting [Balcan et al., 2019, Denevi et al., 2019].
Recent advances in the image few-shot classification setting [Fei-Fei et al., 2006, Lake et al., 2011] starting with the work of [Finn et al., 2017, Snell et al., 2017, Vinyals et al., 2016] has lead to renewed interest in meta-learning, notably from the deep learning community by formulating it as an optimization problem [Ravi and Larochelle, 2017].
While classification has received a lot of interest, regression has been given less attention.
Some examples are given by [Patacchiola et al., 2020, Titsias et al., 2020, Tossou et al., 2019] who apply gaussian processes [Williams and Rasmussen, 2006] together with deep kernel learning [Wilson et al., 2016] to regression.
From the ridge regression point of view; Kong et al.
[2020] investigate theoretically the metamixed linear regression setting while Nguyen et al.
[2021] applied kernel ridge regression (KRR) to meta-learn dataset compression.
Our work can be traced directly to ideas from [Li et al., 2019, Sinha and Duchi, 2016, Zhen et al., 2020] to leverage the characterization provided by Bochner’s theorem for kernel learning [Cristianini et al., 2006, Ong et al., 2005].
In [Sinha and Duchi, 2016] they fine-tune a convex combination of sampled kernels in a supervised learning setting using kernel target alignment [Cristianini et al., 2006].
We also mention the work [Zhen et al., 2020] which apply variational inference to optimize a latent variable model for few-shot learning, and [Li et al., 2019] where they learn an implicit kernel using a pushforward in the case that the learning objective is linear in the kernel evaluations.
Organization Sec.
2 introduces the meta-learning setting.
We describe our proposed method in Sec. 3, analyze it in
Sec. 4 and benchmark it in Sec.
We discuss our findings in Sec. 6.
META-LEARNING PROBLEM
In this section we introduce the main elements of the metalearning setting and introduce the notion of stochastic metalearning algorithm.
To this end, we first recall the standard notion of supervised learning problem.
Supervised Learning Given an input X and output Y set, a supervised learning problem is characterized by a data generating distribution µ ∈ P ( X × Y ) on the joint space
X × Y and a loss function ℓ : Y × Y
→ R measuring prediction errors.
The goal of a supervised learning problem is to find a map f : X → Y minimizing the risk
[Equation]
In practice, the data generating distribution is unknown and only a finite number n tr of examples D tr = ( x i , y i ) n tr i = independently sampled from µ are available (denoted here by D tr ∼ µ n tr ).
A learning algorithm is a function mapping datasets into candidate solutions to (1).
Learning Algorithm Let Y X
= { f : X → Y} be the space of all functions from X to Y and D the space of all datasets of any size on X × Y .
Then, a learning algorithm (referred to as inner algorithm in meta-learning) is a function A : D → Y X , mapping datasets D ∈ D to functions f : X → Y .
Typically, learning algorithms are parametrized as A ( · ) =
A ( θ, · ) , by a set of so-called hyperparameters (here referred to as meta-parameters ) θ ∈ Θ , that allow to adapt the algorithm to the specific problem.
Typical examples of hyperparameters include the regularizer in Tikhonov regularization or the number of iterations of an early-stopping procedure.
Ideally, we aim to find the best meta-parameter for a given task, namely the θ minimizing the expected risk R ( A ( θ, D )) .
We do not have access to µ but we can sample a validation set D val ∼ µ n val and consider the empirical risk
[Equation]
as a suitable proxy.
Since D val and D tr are sampled independently, ˆ R ( A ( θ, D tr ) , D val ) is an unbiased estimator of R µ ( A ( θ, D tr )) .
Given a train and validation set
D =
( D tr , D val ) , the process of minimizing the meta-loss
[Equation]
with respect to θ is known as cross-validation .
Meta-Learning The meta-learning paradigm lifts the notion of cross-validation to the level of multiple tasks: assuming that we have access to many supervised learning problems (or tasks) sharing some form of similarity, metalearning aims to find a single set of meta-parameters θ that works well across all tasks.
More formally, we assume that the tasks are sampled from a meta-distribution ρ .
From each µ ∈ P ( X × Y ) sampled from ρ , we then sample a pair of datasets D =
( D tr , D val ) ∼ µ n with n = n tr + n val
(even though in the following we assume n tr and n val to be fixed for simplicity, our discussion can be extended to more general settings).
Then, meta-learning is formulated as the problem of finding the meta-parameters θ ∈ Θ minimizing the transfer risk [Denevi et al., 2018]
[Equation]
If L ( · , D ) is (sub)differentiable, we can adopt standard stochastic first order method (e.g. SGD or Adam [Kingma and Ba, 2015]) to approximate the optimal meta-parameters.
This consists in iteratively sampling a task µ t ∼ ρ and a train-val split D t ∼ µ nt at each time step t = 1 , . .
, T .
Then, update the meta-parameters, e.g. via the SGD rule θ t +
=
θ t − η ∇ θ L ( θ, D t ) .
We refer to Alg. 1 for a concrete example in the setting discussed in this work.
Meta Representation Learning In practice, the above approach might pose computational challenges since, by the chain rule, differentiating L requires computing ∇ θ A ( θ, D ) .
Depending on the inner algorithm A , its gradient with respect to the meta-parameters θ might be hard to compute or not even exist.
In the literature, a wide range of metalearning strategies have been proposed, considering different choices of inner algorithm A and meta-parameters θ .
For example, [Bertinetto et al., 2018] considered the case that
A performs ridge regression (see Sec. 3.1) and θ parameterizes the weights of a feature map ϕ θ : X → R d (e.g. a neural network).
Leveraging the closed-form solution of the ridge regression estimator, this allows us to efficiently compute the gradient ∇ θ A ( θ, D ) .
In settings where A is minimizing the empirical risk but with a loss function that does not admit a closed form, we can adopt a bi-level optimization perspective [Franceschi et al., 2018].
This amounts to interpret A as returning the T -th iteration of an iterative optimization algorithm.
This allows to access ∇ θ A ( θ, D ) by recursively differentiating along the iterates.
This approach is related to the well-known MAML algorithm
[Finn et al., 2017], which proposed to perform fine-tuning of a shared starting network f θ : X → Y with weights θ , that is adapted by A ( θ, D ) =
f θ ′ to each new task by performing a step of gradient descent θ ′ =
θ − η ∇ θ ˆ R ( ϕ θ , D ) , to fit the training data.
In the following we introduce the family of inner algorithms (and corresponding parameters) proposed in this work to tackle the meta-learning problem.
IMPLICIT KERNEL META-LEARNING
We now introduce the propose meta-learning strategy.
While most previous work focused on learning a shared data representation or feature map [Bertinetto et al., 2018, Finn et al., 2017, Franceschi et al., 2018] across tasks, here we propose the dual approach of learning a shared kernel function.
3.1 REPRODUCING KERNELS AND FEATURE MAPS
Reproducing kernels are a well-established tool in machine learning, at the root of most non-parametric algorithms [Schölkopf and Smola, 2002].
They consist of positive definite functions K : X × X
→ R that may be interpreted as a similarity between data points.
A fundamental result dating back to Moore and Aronszajn [see e.g. Aronszajn, 1950, Cucker and Smale, 2002, Schölkopf and Smola, 2002, and references therein] establishes that a kernel is into one-to-one correspondence with a (possibly infinite dimensional)
Hilbert space H K of real-valued functions on
X , such that for every x ∈ X and f ∈ H K , the function
K ( x, · ) ∈ H K and ⟨ f, K ( x, · ) ⟩ K = f ( x ) , where ⟨· , ·⟩ K denotes the inner product in H K .
A kernel is in duality with the notion of feature map: given a mapping ϕ : X → H into a Hilbert space H with inner product ⟨· , ·⟩ such that
K ϕ ( x, x ′ ) ≡ ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ is a reproducing kernel.
The converse is also true, namely for any kernel K there exists a Hilbert space H and feature map ϕ K : X → H such that K ( x, x ′ ) =
⟨ ϕ K ( x ) , ϕ K ( x ′ ) ⟩
[Aronszajn, 1950]; when X is compact we can choose H = ℓ 2 , the space of square summable sequences.
A key practical advantage of kernels is that they allow to learn functions parametrized as f ( x ) = ⟨ f, ϕ K ( x ) ⟩ even when H K is infinite dimensional (namely the feature vector ϕ K ( x ) has infinitely many entries).
As a concrete example we recall the case of kernel ridge regression [see e.g Caponnetto and De Vito, 2007,
Steinwart and Christmann, 2008] which we will use also as plug-in inner algorithm for the proposed meta-learning approach in this work.
Kernel Ridge Regression Kernel ridge regression (KRR) performs Tikhonov regularization using the least-square loss function over the space of hypotheses associated to a reproducing kernel [see e.g. Schölkopf and Smola, 2002].
More precisely, assume Y ⊂ R .
Given a dataset D tr = ( x i , y i ) , and a kernel function K : X × X → R , KRR is the algorithm
[Equation]
with λ > 0 a regularization parameter.
Thanks to the reproducing property of the kernel, (5) can be solved in closed form.
We have that for any x ∈ X , that
[Equation]
with α =( G + λnI ) − 1 y , where G =( K ( x i , x j )) n tr i,j =1 is the n tr × n tr kernel (Gram) matrix, I the d × d identity matrix, and with some abuse of notation we let y =
( y 1 , . . .
, y n tr )
⊤ ∈ R n tr be the vector of output examples.
Notice that we highlighted the dependency of KRR with respect to the kernel K .
This suggests that in meta-learning settings one might be interested in learning the kernel as a meta-parameter.
3.2 LEARNING TRANSLATION INVARIANT KERNELS
The definition of positive definite function underlying the notion of reproducing kernel is very general.
Therefore, to formulate the problem of meta-learning a kernel, we need first to identify a suitable family.
In [Rudi and Rosasco, 2017] they introduce a "recipe" for random feature kernels defined by a random feature map φ : X × Ω → R o and a distribution τ so that any kernel in this family has the form
[Equation]
[Equation]
Given the focus of this work towards regression settings, we first consider the class of translation invariant kernels, which are particularly suited to deal with such settings and are interpretable ( see e.g. Fig.
2 in the appendix).
Let
X = R d .
A kernel K is called translation invariant if
K ( x, x ′ )
=
g ( x − x ′ ) for some function g : R d → R ; a wellknown example is the Gaussian K
[Equation]
with σ > 0 .
A famous theorem by Bochner [see e.g. Rahimi et al., 2007, Rudin, 1962, Sriperumbudur and Szabo, 2015], adapted here to real-valued kernels, establishes that any properly re-scaled continuous bounded translation invariant function K : R d × R d → R is a kernel if and only if there exists a probability measure τ ∈ P ( R d ) such that
[Equation]
which can be written in the form of (7) by expanding the cosine using the trigonometric identity cos( x − y )
=
cos( x ) cos( y )
+ sin( x ) sin( y ) .
We call any kernel that can be written in the form of (7) a Bochner kernel .
Eq.
(8) implies that we can represent the class of translation invariant kernels as T =
{ K τ | τ ∈ P ( R d ) } .
Thus we can translate the problem of learning a kernel to that of learning a probability distribution.
This perspective is in line with the implicit kernel learning approach devised in [Li et al., 2019] for generative modeling and single task settings.
The second type of kernel is inspired by the success of using neural network to extract features and is given by letting φ ( x, ω ) be a neural network with ω the weights and τ a distribution over ω .
Pushforward Models To learn the underlying distribution τ we consider a parametrization in terms of a pushforward model.
More formally, let N be the unit Gaussian distribution over a latent space Z and let ψ θ : Z → R d be a vectorvalued function parameterized by a vector θ ∈ Θ (e.g. a neural network with weights θ ).
We denote by τ θ = ψ θ # N the probability distribution such that, the process of sampling ω ∼ τ θ is equivalent to first sampling z ∼ N and then taking ψ θ ( z )
=
ω .
1 This is the strategy adopted to model the generator distribution in generative adversarial networks (GAN) settings and in the implicit kernel learning approach of [Li et al., 2019].
Several alternatives for the latent distribution N are possible (e.g. uniform).
Under the notation above, we adopt as inner algorithm,
[Equation]
namely KRR trained with a translational invariant kernel
K τ θ meta-parametrized by the pushforward map τ θ .
Below we give an example where a parametrization of τ θ yields an analytic form for the corresponding K τ θ ; see the appendix for a derivation.
Example 1 (Affine Pushforward Maps) .
Let θ =
( Q, b ) with Q ∈ R d × d and b ∈ R d and consider the affine pushforward map ψ ( Q,b ) ( s ) =
Qs + b .
In these settings, the kernel
K τ ( Q,b ) can be expressed analytically as
[Equation]
The example above identifies a relevant family of kernels that are particularly amenable for meta-learning.
Thanks to the analytic form of affine pushforward kernels, we can easily compute meta-gradients and thus directly minimize the transfer risk E ( θ ) .
On the other hand, if we consider more expressive maps ψ θ , we will hardly be able to obtain
K τ θ in analytic form.
Still, this may be well worth the effort: while for large training sets the difference between (10) and a more sophisticate kernel may be less severe since any universal kernel is optimal [Caponnetto and De Vito, 2007], in the few-shot learning setting (where we have small training sets) the inductive bias plays an important role and being able to modify the kernel in a flexible way is key.
3.3 STOCHASTIC META-LEARNING
The discussion above highlighted that except for a few special cases (see e.g. Example 1), given a distribution τ
θ
it is not possible to compute the kernel K τ θ (and its gradient with respect to θ ) analytically.
In principle, this might prevent us from applying meta-learning algorithms of the form in (9).
To circumvent this issue, we consider a strategy based on random features [Rahimi et al., 2007, Rudi and
Rosasco, 2017].
Rather than evaluating K τ θ , we sample a set S =
( s j ) Mj =
1 from N and then approximate the ideal
Bochner kernel by the random features kernel
K
[Equation]
where ˆ τ θS = 1 M (cid:80) Mj
=1 δ ψ θ ( s j ) is an empirical distribution associated to τ θ and δ ω denotes a Dirac’s delta centered in ω ∈ R
d which we call frequency .
Thanks to the characterization of K τ θ as an expectation in (8), we have that
[Equation]
namely K ˆ τ θS is an unbiased estimator of K τ θ .
It is possible to prove also non-asymptotic results bounding the distance beetween the two kernels in sup norm [Rahimi et al., 2007].
Stochastic Meta-Learning We now introduce a stochastic variant to the meta-learning approach from Sec. 2, by defining the meta-loss associated to a set of random features
L ( θ, S, D )
=
ˆ R ( A KRR ( K τ θS , D tr ) , D val ) ,
[Equation] [Equation]
and the corresponding transfer risk
[Equation]
which we will also denote E ( θ, S ) when wanting to highlight the dependence on S explicitly.
In this work we propose to address the stochastic metalearning problem
[Equation]
Alg. 1 provides the pseudocode for a (meta) stochastic gradient descent algorithm applied to this problem.
At each iteration t = 1 , . . .
, T , we sample a new task µ t and datasets
Algorithm
Input: meta-distribution ρ , step-sizes ( γ t ) ∞
t =1 , number of random features M , initial meta-parameters θ 0 , total number of iterations T .
For t = 1 , . . .
, T
Sample a task/dataset D =
( D tr , D val ) from ρ
Sample M random features S from N
Form K τ θtS and compute L ( S, θ t , D ) as in (13)
Get ∇ θ L ( θ t , S, D )
=
A UTO G RAD ( L ( · , S, D ) , θ t )
Update θ t +1 ← θ t − γ t ∇ L ( θ t , S, D )
Return θ T
D t =
( D tr t , D val t ) ∼ µ nt and a set of random features
S t ∼ N M and then perform a gradient descent step in the direction of ∇ θ L ( θ t , S t , D t ) .
Note that the gradient can be computed by means of automatic differentiation (A UTO G RAD )
[see e.g. Baydin et al., 2018].
Many other strategies to perform this optimization step are available, such as Adam [Kingma and Ba, 2015].
While using a large number of random feature may seem expensive, both training and prediction time is linear in M , see the section on computational complexity in the appendix.
We refer to this method as Implicit Kernel Meta-Learning (IKML) .
GENERALIZATION BOUND
We now study the generalization ability of the proposed meta-learning method.
In particular, our goal is to study the effect of the number of random features on the performance of the meta algorithm.
To present our observations we focus for simplicity on the case that the meta loss uses the task dataset for both training and validation, that is we use the empirial risk
[Equation]
which is the empirical error of KRR with kernel (11) on the dataset D instead of (13).
For a collection of datasets ( D t )
Tt =1 and a sample S =
( s j ) Mj =
1 from N , define the multitask empirical risk
[Equation]
We aim to bound the excess transfer risk
[Equation]
where θ ∗ ∈ Θ is such
that E ( θ ∗ ) =
min θ E ( θ ) and ˆ θ is the minimizer of the multitask empirical risk, which we call the multitask empirical risk minimizer (MERM) which in practice we approximate by the solution returned by Alg. 1.
Theorem 1.
Assume that Z = X
× Y ⊆
R d × [0 , 1] , ρ is a meta-distribution on Z , the loss ℓ ( y, ˆ y ) =
( y − ˆ y ) 2 and kernel family K
=
{ K τ θ | θ ∈ Θ } is a family of Bochner kernels parameterized by some latent distribution N with support on R l and a family of measurable functions { ψ θ :
R l → R d | θ ∈ Θ } .
For any n, M, T ∈ N let the training task datasets D 1 , . . .
, D T be given by iteratively sampling a task µ t ∼ ρ and D t ∼ µ nt and S ∼ N M , the family of inner algorithms being KRR with kernels K τ θ ∈ K and fixed regularization parameter λ > 0 and ˆ θ being the MERM over the task datasets and random features.
Then, for δ ∈ (0 , 1) , with probability at least 1 − δ over the datasets and random features
[Equation]
where
R n,M
[Equation]
the random variables
ϵ i,j,t being i.i.d
Rademacher and D ∼ ˆ ρ means first sampling µ ∼ ρ and then D ∼ µ n , and
G
[Equation]
Proof Sketch.
We discuss the key elements of the proof and present the full details in the appendix.
We write E M (ˆ θ ) −
E ( θ ∗ ) =
E S
[ E M (ˆ θ, S ) − E ( θ ∗ )] and decompose the term inside the expectation as
[Equation]
where ˆ E ( θ, S ) and ˆ E T ( θ, S ) are the average empirical error and the multitask empirical error, for the meta-parameter θ and random features S ; – see the secion on the bound in the appendix.
Bounding the terms (A) and (E) leads to (19) while bounding the terms (B) and (D) leads to (20).
The term (C) is the optimization error and is negative if we can minimize the empirical risk objective.
Finally the term (F) is bounded using [Tropp, 2019, Theorem 2.1] and auxiliary results presented in the appendix.
We now comment on the implications of the above theorem.
The first term in the r.h.s.
of (19) contains the unnormalized Rademacher complexity R n,M,T of the set θ ∈ Θ } ⊆ R n × M × T .
This is a measure of the capacity of the RKHS’s
we consider as part of using the kernel family K and quantifies the kernel families ability to fit random noise.
While this quantity requires a case by case analysis it is often of order √ T .
Since in meta-learning the number of tasks is very large this term is negligible in many practical scenarios.
For example following the reasoning in [Oneto et al., 2020] we obtain that
R n,M,T
= O ( √ nMT ) .
The number of random features should then be chosen so that the quantity (21) is smaller than (20).
G ∗ n represents the size of best RKHS needed to explain the data averaged over the possible datasets sampled from the environment.
In some sense it represents the degrees of freedom of the best model θ ∗ given the metadistribution.
A direct computation gives the condition
[Equation]
Since G ∗ n ∈ [1 , n ] , we conclude that the number of random features needed by the algorithm in order to be competitive with meta-learning without random feature approximation is independent of the number of tasks and only mildly dependent on n .
For example, assuming λ = 1 / √ n we obtain that M = Ω( n 32 log n ) or M = Ω( n 52 log n ) when G ∗ n =or G ∗ n = n , respectively.
The case that G ∗ n = O ( n ) requiring more random features corresponds to a low rank Gram matrix, meaning that the tasks are strongly related.
This is however worth the effort since in this case the optimal risk
E ( θ ∗ ) we compare to will be very small, because the optimal low rank kernel makes learning very easy.
Finally we note that λ being in the denominator of all terms is an artifact due to comparing to the best KRR algorithm θ ∗ instead of the quantity E ∗ =
E µ ∼ ρ R µ ( f µ ) where f µ = E [ y |· ] is the optimal predictor for the distribution µ [see Denevi et al., 2019, for a discussion].
EXPERIMENTAL RESULTS
We evaluate the performance of the proposed meta-learning strategy on both synthetic and real experiments against several baselines.
We make all datasets and code available as a
Github repository.
5.1 SYNTHETIC MULTIVARIATE REGRESSION
For IKML to be effective in realistic meta-learning regression scenarios it is important that it can approximate nontrivial functions defined on R d where d ≫ 1 .
To investigate this we create a synthetic high-dimensional meta-learning regression setting where each task is sampled from an RKHS
H with a “complicated” kernel K o .
In particular, we choose
K o to be the kernel given by Bochner’s theorem and a pushforward of a 3-layers Multi-Layer Perceptron (MLP) with
32 hidden units per layer, ReLU activation functions and a 16-dimensional latent Gaussian distribution.
The network was initialized with weights given by the PyTorch [Paszke et al., 2019] default initialization scaled by 100.
Since this kernel lacks an analytic form, we sample 10000 frequencies and use the random features kernel from (11) in its place.
The tasks are generated from a distribution on f ∈ H and a marginal distribution on inputs fixed across all tasks.
For each task we sample n = n tr + n val = 50 + 50 inputs ( x i ) ni
=1 , a function f and create the task ( x i , f ( x i ))
ni
=1 , for more details see Sec.
5 in the appendix.
We compare the following meta-learning algorithms:
IKML.
Alg.
1 parameterizing the pushforward ψ θ for the measure τ θ with a three-layer MLP with hidden dimension set to 32 and the dimension of the latent space Z
= R 16 .
The number of random features is set to M = 10 4 .
Gaussian MKL meta-KRR (GMKL).
Multiple Kernel Learning (MKL) with KRR as inner algorithm.
The metaalgorithm consists in learning the weights of a kernel K = (cid:80) kj =
λ j K j that is a convex combinations of Gaussian kernels K j ( x, x ′ )
=
exp( − 12 σ 2 j ∥ x − x ′ ∥ 2 ) with lengthscale σ j taken from an log-equidistant grid from 10 − 3 to 10 3 .
The meta-learning algorithms learns the weights λ parameterized in terms of the vector z ∈ R k as λ j =
[Equation]
MAML [Finn et al., 2017].
Optimizing through inner gradient descent with MLP to learn a good initalization in the outer loop.
We use a three-layer MLP with 32 hidden units and ReLU activation functions.
R2D2 [Bertinetto et al., 2018].
Ridge regression as inner algorithm, learning a shared feature map in the outer loop.
We use a three-layer MLP with 32 hidden units and ReLU activation functions.
Oracle.
Running a separate instance of KRR on each task, with the same kernel K o used to generate the tasks, and finding λ by cross validation on the test set.
5.2 REAL-WORLD DATA EXPERIMENTS
We evaluate the proposed approach on two new real world meta-learning regression datasets adapted to the metalearning setting from the UCI repository [Dua and Graff, 2017].
Apart from IKML and Gaussian MKL meta-KRR, we used the following algorithms in our experiments: LS Biased Regularization [Denevi et al., 2019] (LSBR).
Running linear ridge regression with biased regularization λ
∥
f − θ ∥in the inner algorithm, learning the bias θ in the outer loop.
ANP [Kim et al., 2019].
Learns to map datasets to stochastic processes over functions using neural networks to do metalearning.
Predictor is the conditional mean of the stochastic process.
Gaussian Oracle KRR (GO).
Gaussian KRR addressing each task as a separate learning problem but cross-validating the kernel bandwidth and regularization parameters σ 2 and λ on the average validation error directly on the meta-test set.
We chose the baselines from landmark papers in the fewshot learning (MAML, R2D2, ANP, LSBR) and multiplekernel learning (GMKL, GO) literature applicable to regression.
We think these are natural baselines to compare against.
For both meta-learning datasets, we run the algorithms above in an online fashion where we use a meta-batch of 4 tasks per iteration sampled from the meta-train set.
For
IKML we fix the number of random features to 20000 which is on the order of Ω( n 5 / 2 log( n )) if we would have pooled the train and validation set of 25 datapoints to one train set of size 50 .
Note however that further experiments show that in practice we can get away with as little as 2500 random features while mainting performance.
Every 250 steps we sample 1000 tasks from the meta-validation set and evaluated the average meta-loss for each algorithm and save the model parameters.
After training we sample 3000 tasks from the meta-test set.
For the meta-test evaluation, for all algorithms, we use the meta-parameters with the lowest meta-validation error and get the test performance for all algorithms.
We measure performance in terms of the root mean square error (RMSE).
This procedure was run 5 times over different random seeds in order to get learning curves and results on the meta-test set.
Below we describe the datasets and comment on the empirical evidence.
Air
Quality
The Beijing Air Quality dataset [Zhang et al., 2017] is a time-series dataset measuring air-quality and meterological factors at 12 air-quality monitoring sites.
The meterological data for each site is matched with the closest of available weather stations.
The data was collected hourly and from the period March 1st, 2013 to February 28th, 2017.
Further details in Sec.
4.1 in the appendix.
We generate a task of train and validation size n tr , n val by randomly picking a station and picking a contiguous subsequence of size n = n tr + n val at random from the split.
We append the feature “ t ” which is the local order of data points and then randomly assign n tr of the n points to the train set and the rest to the validation set.
This can be seen as a reconstruction problem: given data from sensor of which some have failed, we want to infer the output given an input at some points in time.
We choose to use n tr = n val = 25 .
After experimenting we use the following configuration of the algorithms; For Gaussian MKL meta-KRR we use
Gaussian kernels with lengthscale sampled geometrically from 1 to 10 12 and learn the coefficients and regularisation parameter using the same parameterization as in the synthetic experiment with Adam and a meta-learning rate of 0.001.
For LS Biased Regularization we learn the bias and regularisation parameter using Adam with meta-learning rate 0.01.
We parameterized MAML with a 2-layer MLP with 64 hidden dimensions and with inner learning rate 10 −and one adaptation step, learning the initialization using
Adam with meta-learning rate of 0.001.
We found that using a very small inner learning rate and few steps was important to get MAML to converge.
For R2D2, IKML and ANP we cross-validated to find the best set of hyparparameters, see
Sec. 5 and Tab. 1 in the appendix for more information.
For Gaussian meta-KRR we learn the lengthscale and regularisation parameter using Adam with a meta-learning rate of 0.001.
We benchmark a neural network IKML, called
IKML-MLP in where we use a 4-layer MLP with 64 hidden units, 8 output features and 500 random features trained using Adam with learning rate of 3 · 10 − 4 , see Sec.
5 in the appendix.
From Tab. 1 we can see that IKML performs best with R2D and IKML-MLP close seconds.
Gas Sensor The Gas Sensor Modulation dataset [Burgués et al., 2018] is a collection of multivariate timeseries collected in a controlled environment using MOX sensors for
CO detection sampled at 3.5 Hz.
Each task corresponds to a subsampled time-series from an experiment.
As noted in [Burgués et al., 2018] the regression tasks are hard due to being heteroscedastic, non-normal and non-linear as a function of time but with tasks sharing a lot of structure, making it suitable as a meta-learning regression dataset.
Further details in Sec.
4.2 in the appendix.
We benchmark the algorithms for n tr = n val = 20 .
After experimenting we use the following configuration of the algorithms; For Gaussian MKL meta-KRR we use 20 Gaussian kernels with lengthscale chosen geometrically from 1 to 10 8 and learn the coefficients and regularisation parameter using the same parameterization as in the synthetic experiment with Adam and a meta-learning rate of 0.001.
For LS
Biased Regularization we learn the bias and regularisation parameter using Adam with meta-learning rate 0.01.
We parameterized MAML with a 4-layer MLP with
[Table] [Table]
hidden units with inner learning rate 10 − 4 and one adaptation step, learning the initialization using Adam with metalearning rate of 10 − 4 .
For R2D2, IKML and ANP we crossvalidated to find the best set of hyparparameters, see Sec.and Tab. 1 in the appendix for more information.
For Gaussian meta-KRR we learn the lengthscale and regularization parameter using Adam with a meta-learning rate of 0.001.
IKML-MLP is as for Gas Sensor, but with 2 layers andrandom features.
As can be seen from the table, IKML and MAML gets a low meta-test error after R2D2 and IKML-MLP as can be seen in Tab. 1.
Additional Metrics For the algorithms R2D2, IKML and
ANP, with the same setting and training strategy as outlined for Air Quality and Gas Sensor datasets, we evaluate them on two additional metrics: mean average error (MAE) and symmetric mean absolute scaled error (SMAPE)
[Chicco et al., 2021].
From Tab.
2 we see that IKML performs the best on all metrics on the Air Quality dataset with ANP performing poorly.
For Gas Sensor R2D2 performs best
[Table]
except for SMAPE where ANP performs much better.
CONCLUSION AND FUTURE WORK
We introduced a framework for implicit kernel metalearning (IKML) in context of translation-invariant and deep random kernel families.
Our approach focuses on problems where data does not present a clear input structure (in contrast e.g. to image classification settings) and using a plug-in translation invariant kernel might be a safer strategy.
Our approach leverages the characterization of random feature kernels, in particular the translation invariant kernels granted by Bochner’s theorem and ideas from the random features literature to learn it in practice.
We derive a novel bound on the excess transfer risk shedding light on how to choose the number of random features.
To validate our method we introduced two real-world meta-learning regression datasets.
IKML achieve best or close-to-best performance on all of the datasets against state-of-the-art methods designed for few-shot image classification.
We hypothesize that when the data does not have enough structure (e.g. in most regression settings), learning a deep representation – as done by stateof-the-art methods such as MAML or R2D2 – may be less effective.
We leave further investigation of this question to future work.
We close by mentioning three relevant directions for future research: i) Conditional meta-learning Is it possible to extend the framework to conditional meta-learning?
One way would be to use KTA similar to [Sinha and Duchi, 2016] and adjusting the initial starting kernel similar to MAML; ii)
Theoretical guarantees Can we show that IKML converges to a stationary point for benign settings?
This would require understanding the bias-variance decomposition of the gradient; iii) Alternative Kernel Classes Can we extend IKML to other kernel families?
An example is dot-product kernels [Kar and Karnick, 2012].
ACKNOWLEDGEMENTS
The authors would like to thank comments by the anonymous reviewers which helped improve the paper.
John Isak
Texas Falk is supported through the Department of Computer Science at UCL, and this work was partially carried out while he was a visiting doctoral student at Istituto Italiano di
Tecnologia, Genoa.
Carlo Ciliberto acknowledges the support of the Royal Society (grant SPREM RGS\R1\201149) and
Amazon.com Inc. (Amazon Research Award – ARA).