Uncertainty-Aware Meta-Learning for Multimodal Task Distributions
Abstract
Meta-learning is a popular approach for learning new tasks with limited data (i.e., few-shot learning ) by leveraging the commonalities among different tasks.
However, meta-learned models can perform poorly when context data is limited, or when data is drawn from an out-of-distribution (OoD) task.
Especially in safety-critical settings, this necessitates an uncertainty-aware approach to meta-learning.
In this work, we present U N L I M I TD ( uncertainty-aware meta-learning for multimodal task distributions ), a novel method for meta-learning that (1) makes probabilistic predictions on in-distribution tasks efficiently, (2) is capable of detecting OoD context data at test time, and (3) performs on heterogeneous, multimodal task distributions.
To achieve this goal, we take a probabilistic perspective and train a parametric, tuneable distribution over tasks on the meta-dataset.
We construct this distribution by performing Bayesian inference on a linearized neural network, leveraging Gaussian process theory.
We demonstrate that
U N L I M I TD’s predictions compare favorably to, and outperform in most cases, the standard baselines, especially in the low-data regime.
Furthermore, we show that U N L I M I TD is effective in detecting data from OoD tasks.
Finally, we confirm that both of these findings continue to hold in the multimodal task-distribution setting.
Introduction
Meta-learning has emerged as a popular approach to enable models to perform well on new tasks using limited data.
It involves first a meta-training process, when the model learns valuable features from a set of tasks.
Then, at test time, using only few context data from a new, unseen task, the model (1) adapts to this new task and then (2) infers by making predictions on new, unseen query inputs from the same task.
A popular baseline for meta-learning, which has attracted a large amount of attention, is Model-Agnostic Meta-Learning (MAML)
[Finn et al., 2017], in which the adaptation process consists of fine-tuning the parameters of the model via gradient descent.
However, meta-learning methods can often struggle in several ways.
First, accurate prediction becomes challenging when context data is limited.
As these predictions can be untrustworthy, this necessitates the development of meta-learning methods that can express uncertainty during adaptation [Yoon et al., 2018, Harrison et al., 2018].
In addition, meta-learning models may not successfully adapt to “unusual” tasks, i.e., when test-time context data is drawn from an out-of-distribution (OoD) task not well represented in the training dataset [Jeong and Kim, 2020, Iwata and Kumagai, 2022].
Finally, special care has to be taken when learning tasks that have a large degree of heterogeneity.
An important example is the case of tasks with a multimodal distribution, i.e., when there are no common features shared across all the tasks, but the tasks can be broken down into subsets (modes) in a way that the ones from the same subset share common features [Vuorio et al., 2019].
Our contributions.
This paper presents U N L I M I TD, a novel meta-learning method that leverages probabilistic tools to address the aforementioned issues.
Specifically, U N L I M I TD models the true distribution of tasks with a learnable distribution constructed over a linearized neural network and uses analytic Bayesian inference to perform uncertainty-aware adaption.
We further extend the framework to the multimodal task-distribution setting.
Problem statement
A task T i consists of a function f i from which data is drawn.
At test time, the prediction steps are broken down into (1) adaptation , that is identifying f i using K context datapoints ( X i , Y i ) from the task, and (2) inference , that is making predictions for f i on the query inputs X i ∗ .
We take a probabilistic, functional perspective and represent a cluster by p ( f ) , a theoretical distribution over the function space that describes the probability of a task belonging to the cluster.
In practice, however, we are not given p ( f ) , but only a meta-training dataset D that we assume is sampled from p ( f ) :
D = { ( (cid:102)
X i , (cid:101) Y i ) } Ni =1 , where N is the number of training tasks, and ( (cid:102)
X i , (cid:101) Y i ) ∼
T i is the entire pool of data from which we can draw subsets of context data.
Consequently, in the meta-training phase, we aim to optimize ˜ p ξ ( f ) to capture properties of p ( f ) , using D .
Our Approach: U N L I M I TD
In our approach, we choose ˜ p ξ ( f ) to be the GP distribution over functions that arises from a Gaussian prior on the weights of the linearization of a neural network.
Recent works [Jacot et al., 2018, Neal, 1996, Azizan et al., 2021, Lee et al., 2017] have demonstrated that the linearized deep neural network, wherein the Jacobian of the network operates as the feature map, can well approximate the training behavior of wide nonlinear deep neural networks.
Appendix A.1 contains detail about the GP and linearized neural networks, on which the proposed framework is based.
Consider a particular task T i and a batch of K context data ( X i ∈ R N x × K , Y i ∈ R N y × K ) .
The resulting GP prior predictive distribution after evaluating on the context inputs, is Y |
X i ∼
N ( µ Y | X i , Σ Y | X i ) , where
[Equation]
where J ( · , · ) : R P × R N x × K → R N y K × P is the vectorized Jacobian of the network with respect to the parameters on the given K inputs.
In this setup, the parameters ξ of ˜ p ξ ( f ) that we wish to optimize are the linearization point θ 0 ∈ R P , and the parameters of the prior over the weights ( µ , Σ ) .
Given this Gaussian prior, it is straightforward to compute the joint NLL of the context labels
Y i ,
[Equation]
The NLL (a) serves as a loss function quantifying the quality of ξ during training and (b) serves as an uncertainty signal at test time to evaluate whether context data ( X i , Y i ) is OoD. Given this model, adaptation is tractable as we can condition this GP on the context data analytically.
3.1 Tractably parameterizing the prior covariance over the weights
When working with deep neural networks, the number of weights P can surpass 10 6 .
The memory complexity of U N L I M I TD quickly increases with P due to the dense prior covariance matrix over the weights Σ ∈ R P × P .
Learning a low-dimensional representation of the covariance.
To relieve the burden, this paper proposes a low-rank representation of Σ , allowing for a learnable weight-space prior covariance that can encode correlations.
Specifically, we consider a covariance of the form Σ = Q ⊤ diag ( s 2 ) Q , where Q is a fixed projection matrix on an s -dimensional subspace of R P .
In this case, the parameters that are learned are ξ =
( θ 0 , µ , s ) .
We define S :=
diag ( s 2 ) .
The computation of the covariance of the prior predictive (equation 1) could then be broken down into two steps:
[Equation]
which requires a memory footprint of O ( P ( s + N y K ))
, if we include the storage of the Jacobian.
Because N y K ≪ P in typical deep learning contexts, it suffices that s ≪ P so that it becomes tractable to deal with this new representation of the covariance.
One way to choose the informative projection matrix Q is constructing the projection map by choosing the top s eigenvectors of the Fisher information matrix (FIM) evaluated on the training dataset D .
Recent work has shown that the FIM for deep neural networks tends to have rapid spectral decay [Sharma et al., 2021], which suggests that keeping only a few of the top eigenvectors of the FIM is enough to encode an expressive task-tailored prior.
We named this approach U N L I M I TD-F. The detail about U N L I M I TD-F and other parameterizing methods (U N L I M I TD-I and U N L I M I TD-R) are described in Appendix A.2 and Appendix A.3, respectively.
3.2 Generalizing the structure to a mixture of Gaussians
When learning on multiple clusters of tasks, p ( f ) can become non-unimodal.
Instead of using single
GP, we can capture this multimodality by structuring ˜ p ξ ( f ) as a mixture of GPs.
Building a more general structure.
We assume that at train time, a task T i comes from any cluster with equal probability.
Thus, we construct ˜ p ξ ( f ) as an equal-weighted mixture of α GPs.
For each element of the mixture, the structure is similar to the single cluster case, where the parameters of the cluster’s weight-space prior are given by ( µ j , Σ j ) .
We choose to have both the projection matrix Q and the linearization point θ 0 (and hence, the feature map ϕ ( · ) =
J ( θ 0 , · ) ) shared across the clusters.
This yields improved computational efficiency, as we can compute the projected features once, simultaneously, for all clusters.
This yields the parameters ξ α = ( θ 0 , Q , ( µ 1 , s 1 ) , . .
, ( µ α , s α )) .
This can be viewed as a mixture of linear regression models, with a common feature map but separate, independent prior distributions over the weights for each cluster.
These separate distributions are encoded using the low-dimensional representations S j for each Σ j .
Notice how this is a generalization of the single cluster case, for when
α = 1 , ˜ p ξ ( f ) becomes a Gaussian and ξ α = ξ .
Prediction and likelihood computation.
The NLL of a batch of inputs under this mixture model can be computed as:
[Equation]
where NLL j ( X i , Y i ) is the NLL with respect to each individual Gaussian, as computed in equation 2.
To make exact predictions, we would require conditioning this mixture model.
As this is not directly tractable, we propose to first infer the cluster from which a task comes from, by identifying the
Gaussian G j 0 that yields the highest likelihood for the context data (cid:0)
X i , Y i (cid:1) .
Then, we can adapt by conditioning G j 0 with the context data and finally infer by evaluating the posterior on the query X i ∗ .
3.3 Meta-training the Parametric Task Distribution
The key to our meta-learning approach is to estimate the quality of ˜ p ξ ( f ) via the NLL of context data from training tasks, and use its gradients to update the parameters of the distribution ξ .
Optimizing this loss over tasks in the dataset draws ˜ p ξ ( f ) closer to the empirical distribution present in the dataset, and hence towards the true distribution p ( f ) .
The training details can be found in Appendix [-A.
-]A.
Computing the likelihood.
In the algorithms, the function G AUSS NLL( Y i ; m , K ) stands for NLL of Y i under the Gaussian N ( m, K ) (see equation 2).
In the mixture case, we instead use M IXT NLL, which wraps equation 3 and calls G AUSS NLL for the individual NLL computations (see discussion in Section 3.2).
In this case, µ becomes { µ j } j = α j =1 and s becomes { s j } j = α j =1 when applicable.
Finding the FIM-based projections.
The FIM-based projection matrix aims to identify the elements of ϕ =
J ( θ 0 , X ) that are most relevant for the problem (see Section 3.1 and Appendix A.3).
However, this feature map evolves during training, because it is θ 0 -dependent.
How do we ensure that the directions we choose for Q remain relevant during training?
We leverage results from Fort et al. [2020], stating that the NTK (the kernel associated with the Jacobian feature map, see Section A.1.2) changes significantly at the beginning of training and that its evolution slows down as training goes on.
This suggests that as a heuristic, we can compute the FIM-based directions after partial training, as they are unlikely to deviate much after the initial training.
For this reason, U N L I M I TD-F (Algorithm 1) first calls U N L I M I TD-I (Algorithm 3) before computing the FIM-based Q .
Then the usual training takes place with the learning of s in addition to θ 0 and µ .  Related work
Bayesian inference with linearized DNNs.
Bayesian inference with neural networks is often intractable because the posterior predictive has rarely a closed-form expression.
Whereas U N L I M I TD linearizes the network to allow for practical Bayesian inference, existing work has used other approximations to tractably express the posterior.
For example, it has been shown that in the infinitewidth approximation, the posterior predictive of a Bayesian neural network behaves like a GP [Neal, 1996, Lee et al., 2017].
This analysis can in some cases yield a good approximation to the Bayesian posterior of a DNN [Garriga-Alonso et al., 2018]. It is also common to use Laplace’s method to approximate the posterior predictive by a Gaussian distribution and allow practical use of the Bayesian framework for neural networks. This approximation relies in particular on the computation of the Hessian of the network: this is in general intractable, and most approaches use the so-called
Gauss-Newton approximation of the Hessian instead [Ritter et al., 2018].
Recently, it has been shown that the Laplace method using the Gauss-Newton approximation is equivalent to working with a certain linearized version of the network and its resulting posterior GP [Immer et al., 2021].
Meta-learning.
MAML is a meta-learning algorithm that uses as adaptation a few steps of gradient descent [Finn et al., 2017].
It has the benefit of being model-agnostic (it can be used on any model for which we can compute gradients w.r.t.
the weights), whereas U N L I M I TD requires the model to be a differentiable regressor.
MAML has been further generalized to probabilistic meta-learning models such as PLATIPUS or BaMAML [Yoon et al., 2018, Finn et al., 2018], where the simple gradient descent step is augmented to perform approximate Bayesian inference.
These approaches, like ours, learn (during meta-training) and make use of (at test-time) a prior distribution on the weights.
In contrast, however, U N L I M I TD uses exact Bayesian inference at test-time.
MAML has also been improved for multimodal meta-learning via MMAML [Vuorio et al., 2019, Abdollahzadeh et al., 2021].
Similarly to our method, they add a step to identify the cluster from which the task comes from [Vuorio et al., 2019].
OoD detection in meta-learning has been studied by Jeong and Kim [2020], who build upon MAML to perform OoD detection in the classification setting, to identify unseen classes during training.
Iwata and Kumagai [2022] also implemented OoD detection for classification, by learning a Gaussian mixture model on a latent space.
U N L I M I TD extends these ideas to the regression task, aiming to identify when test data is drawn from an unfamiliar function.
ALPaCA is a Bayesian meta-learning algorithm for neural networks, where only the last layer is
Bayesian [Harrison et al., 2018].
Such framework yields an exact linear regression that uses as feature map the activations right before the last layer.
Our work is a generalization of ALPaCA, in the sense that U N L I M I TD restricted to the last layer matches ALPaCA’s approach.
More on this link between the methods is discussed in Appendix A.5.
Results and Discussion
We wish to evaluate following key aspects of U N L I M I TD.
(1) At test time, how do the probabilistic predictions compare to baselines?
(2) How well does the detection of context data from OoD tasks perform?
We construct a multimodal task distribution, regression problems inspired from Vuorio et al.
[2019], with training data consisting of sinusoids as well as lines with varying slopes.
OoD tasks are quadratic functions.
We evaluate the performance of the proposed approach and compared −−  4 x − 0 .0 . 0 2 .y MSE=0.4963( K =1)
TargetContextdataPrediction+/-1.96 σ −−  4 x − 0 .0 .
0 2 .y
MSE=0.0062( K =5)
TargetContextdataPrediction+/-1.96 σ −−  4 x − 0 .0 .
0 2 .y
MSE=0.0012( K =10)
TargetContextdataPrediction+/-1.
[Figure]
−−  4 x − 0 .0 .
0 2 .y
MSE=0.5476( K =1)
TargetContextdataPrediction+/-1.96 σ −−  4 x − 0 .
[Figure]
[Figure]
The standard deviation is from the posterior predictive distribution.
Note how the uncertainty levels are coherent with the actual prediction error.
Also, note how uncertainty decreases when there is more context data.
Notice how U N L I M I TD-F recovers the shape of the sine even with a low number of context inputs.
Finally, note how U N L I M I TD-F is able to reconstruct the sine even when trained on fewer tasks (bottom).
More comprehensive plots available in Figure 5.
[Figure]
[Figure]
it to the U N L I M I TD with single GP structure as well as baselines including MAML and MMAML.
Details on the problems is in Appendix A.7, while the experiment details are in Appendix A.8 and
A.9.
Further discussion about our models on the unimodal setting is described in Appendix A.6.1.
Both the OoD detection and the prediction performances are better with the mixture structure than with the single GP structure (Figure 2), indicating that the mixture model is a useful structure for ˜ p ξ ( f ) .
This is reflected in the quality of the learned priors (see Appendix A.6.2 for more results).
Further note how our U N L I M I TD have efficient OoD detection and outperforms MAML; it achieves much better generalization when decreasing the number of context samples K (Figure 2).
This demonstrates the strength of our probabilistic approach for multimodal meta-learning: even if the probabilistic assumptions are not optimal, the predictions are still accurate and can beat baselines.
The performance and trade-off among different variants (i.e., U N L I M I TD-I, U N L I M I TD-R, and
U N L I M I TD-F) of our proposed method is further discussed in Appendix [-A.6.
-]A.6.
A Appendix A.1 Background A.1.1 Bayesian linear regression and Gaussian Processes
Efficient Bayesian meta-learning requires a tractable inference process at test time.
In general, this is only possible analytically in a few cases.
One of them is the Bayesian linear regression with Gaussian noise and a Gaussian prior on the weights.
Viewing it from a nonparametric, functional approach, this model is equivalent to a Gaussian process (GP)
[Rasmussen and Williams, 2005].
Let X =
( x 1 , . .
, x K ) ∈
R N x × K be a batch of K N x -dimensional inputs and let y =
( y 1 , . . .
, y K ) ∈ R N y K be a vectorized batch of N y -dimensional outputs.
In the Bayesian linear regression model, these quantities are related according to y = ϕ ( X )
⊤ ˆ θ + ε ∈ R N y K where ˆ θ ∈ R P are the weights of the model, and the inputs are mapped via ϕ : R N x × K → R P × N y K .
Notice how this is a generalization of the usual one-dimensional linear regression ( N y = 1 ).
If we assume a Gaussian prior on the weights ˆ θ ∼ N ( µ , Σ ) and a Gaussian noise ε ∼ N ( 0 , Σ ε ) with Σ ε = σ 2 ε I , then the model describes a multivariate Gaussian distribution on y for any X .
Equivalently, this means that this model describes a GP distribution over functions, with mean and covariance function (or kernel)
[Equation]
This GP enables tractable computation of the likelihood of any batch of data ( X , Y ) given this distribution over functions.
The structure of this distribution is governed by the feature map ϕ and the prior over the weights, specified by µ and Σ .
This distribution can also easily be conditioned to perform inference.
Given a batch of data ( X , Y ) , the posterior predictive distribution is also a GP, with an updated mean and covariance function
[Equation]
Here, µ post ( X ∗ ) represents our model’s adapted predictions for the test data, which we can compare to Y ∗ to evaluate the quality of our predictions, for example, via mean squared error (assuming that test data is clean, following Rasmussen and Williams [2005]).
The diagonal of cov post ( X ∗ , X ∗ ) can be interpreted as a per-input level of confidence that captures the ambiguity in making predictions with only a limited amount of context data.
A.1.2 The linearization of a neural network yields an expressive linear regression model
As discussed, the choice of feature map ϕ plays an important role in specifying a linear regression model.
In the deep learning context, recent work has demonstrated that the linear model obtained when linearizing a deep neural network with respect to its weights at initialization, wherein the
Jacobian of the network operates as the feature map, can well approximate the training behavior of wide nonlinear deep neural networks [Jacot et al., 2018, Neal, 1996, Azizan et al., 2021, Lee et al., 2017].
Let f be a neural network f : ( θ , x t ) (cid:55)→ y t , where θ ∈ R P are the parameters of the model, x ∈ R N x is an input and y ∈ R N y an output.
The linearized network (w.r.t.
the parameters) around θ 0 is
[Equation]
where J θ ( f )( · , · ) : R P × R N x → R N y × P is the Jacobian of the network (w.r.t.
the parameters).
In the case where the model accepts a batch of K inputs X = ( x 1 , . . .
, x K ) and returns Y = ( y 1 , . . .
, y K ) , we generalize f into g :
R P × R N x × K → R N y × K , with Y =
g
( θ , X ) .
Consequently, we generalize the linearization:
[Equation]
where J ( · , · ) : R P × R N x × K → R N y K × P is a shorthand for J θ ( g )( · , · ) .
Note that we have implicitly vectorized the outputs, and throughout the work, we will interchange the matrices R N y × K and the vectorized matrices R N y K .
This linearization can be viewed as the N y K -dimensional linear regression
[Equation]
(6) where the feature map ϕ θ 0 ( · ) : R N x × K → R P × N y K is the transposed Jacobian J ( θ 0 , · ) ⊤ .
The parameters of this linear regression ˆ θ =
( θ − θ 0 ) are the correction to the parameters chosen as the linearization point.
Equivalently, this can be seen as a kernel regression with the kernel k θ
( X 1 , X 2 ) =
J ( θ 0 , X 1 ) J ( θ 0 , X 2 ) ⊤ , which is commonly referred to as the Neural Tangent
Kernel (NTK) of the network.
Note that the NTK depends on the linearization point θ 0 .
Building on these ideas, Maddox et al.
[2021] show that the NTK obtained via linearizing a DNN after it has been trained on a task yields a GP that is well-suited for adaptation and fine-tuning to new, similar tasks.
Furthermore, they show that networks trained on similar tasks tend to have similar Jacobians, suggesting that neural network linearization can yield an effective model for multi-task contexts such as meta-learning.
In this work, we leverage these insights to construct our parametric functional distribution ˜ p ξ ( f ) via linearizing a neural network model.
Parameterizing the
prior
covariance over the weights
Imposing a unit covariance.
One simple way to tackle this issue would be to remove Σ from the learnable parameters ξ , i.e., fixing it to be the identity Σ
= I P .
In this case, ξ = ( θ 0 , µ ) .
This computational benefit comes at the cost of model expressivity, as we lose a degree of freedom in how we can optimize our learned prior distribution ˜ p ξ ( f ) .
In particular, we are unable to choose a prior over the weights of our model that captures correlations between elements of the feature map.
A trade-off between feature-map expressiveness and learning a rich prior over the weights.
Note that even if a low-dimensional representation of Σ enriches the prior distribution over the weights, it also restrains the expressiveness of the feature map in the kernel by projecting the P -dimensional features J ( θ 0 , X ) on a subspace of size s ≪ P via Q .
This presents a trade-off: we can use the full feature map, but limit the weight-space prior covariance to be the identity matrix by keeping Σ =
I (case U N L I M I TD-I).
Alternatively, we could learn a low-rank representation of Σ by randomly choosing s orthogonal directions in R P , with the risk that they could limit the expressiveness of the feature map if the directions are not relevant to the problem that is considered (case U N L I M I TD-R).
As a compromise between these two cases, we can choose the projection matrix more intelligently and project to the most impactful subspace of the full feature map — in this way, we can reap the benefits of a tuneable prior covariance while minimizing the useful features that the projection drops.
To select this subspace, we construct this projection map by choosing the top s eigenvectors of the
Fisher information matrix (FIM) evaluated on the training dataset D (case U N L I M I TD-F).
Recent work has shown that the FIM for deep neural networks tends to have rapid spectral decay [Sharma et al., 2021], which suggests that keeping only a few of the top eigenvectors of the FIM is enough to encode an expressive task-tailored prior.
A.3 A tractable way of finding the perturbation directions in weight space that impact the most the predictions of an entire dataset
Deep neural networks have a large number of parameters, making the feature map ϕ θ 0 highdimensional.
However, recent work has shown that only a small subspace of the weight space is impactful.
For example, to perform continual learning, Farajtabar et al.
[2020] leverage the fact that it is sufficient to update the parameters orthogonally to a few directions only to avoid catastrophic forgetting.
Sagun et al.
[2017] have shown that the Hessian of a deep neural network can be summarized in a few number of directions, due to rapid spectral decay.
This encourages finding a method to extract these meaningful directions of the weight space.
A.3.1 Link with the Fisher Information Matrix
We define these main directions as the ones that have the most impact on the predictions of a whole dataset.
To find them, we first find a way to quantify the influence of an infinitesimal weight perturbation.
Using the second-order approximation of that quantity, we then describe in the deeplearning context a tractable way to find the directions.
Setting We take the same setting as Section A.1.2, and we describe a method to quantify the influence of a parameter perturbation ˜ θ on the predictions of a dataset of tasks D .
To do so, we leverage a probabilistic interpretation of the model: we assume a Gaussian pdf over the observations for given inputs and parameters p θ ( Y | X ) ∼ N ( g ( θ , X ) , Σ ε ) , where the covariance of the noise Σ ε is diagonal Σ ε = σ 2 ε I (just as in Section A.1.1).
Perturbation of the prediction of a batch of inputs.
Before quantifying the influence of the perturbation on the predictions of the whole task dataset, we do it for the prediction of a batch of inputs g ( θ 0 , X ) .
We borrow the method from Sharma et al.
[2021]: we quantify the influence of a parameter perturbation by computing the Kullback-Leibler divergence between p θ
( Y | X ) and p θ 0 +˜
θ ( Y | X ) .
The expansion is:
[Equation]
where F ( θ 0 , X ) :
=
J ( θ 0 , X ) ⊤ Σ − 1 ε J ( θ 0 , X ) =
σ − 2 ε J ( θ 0 , X ) ⊤ J ( θ 0 , X ) ∈ R P
× P is the empirical Fisher Information Matrix (FIM) of the batch of inputs X , computed on the parameters θ 0 .
Generalization: perturbation of the prediction of a dataset.
Now we can define the influence of a parameter perturbation on the whole training dataset D , by generalizing the previous definition:
[Equation]
Using equation 7, this quantity verifies:
[Equation]
which gives a natural definition for the FIM of the whole dataset by analogy with equation 7:
[Equation]
The expansion in equation 8 shows that the FIM of the dataset is a second-order approximation describing the influence of a parameter perturbation over the entire dataset.
In particular, the eigenvectors of F ( θ 0 , D ) with the highest eigenvalues are the directions that impact the most the predictions.
A.3.2 Computing the top eigenvectors of the Fisher Information Matrix of the dataset in a deep learning context
Naively computing the top eigenspace of F ( θ 0 , D ) requires processing a P × P matrix, which is intractable in the deep-learning context (where P can surpass 10 6 ).
Instead, we decide to use the method used by Sharma et al.
[2021], which leverages a low-rank-approximation-based technique (namely matrix sketching) developed by Tropp et al.
Sketching the FIM of the dataset The key idea behind this technique is to build two small random sketches of the FIM, ( Y , W ) :
= S ( F ( θ 0 , D )) , that with high probability contain enough information to reconstruct the top s eigenvectors of F ( θ 0 , D ) .
The linearity of S simplifies the sketching process by breaking down the computation into individual sketches:
[Equation]
In particular, the sketch ( Y , W ) can be updated in-place and does not require to store all the individual sketches.
Given a sketch budget k + l (Sharma et al.
[2021] recommends choosing k :
= 2 s + 1 and l := 4 s + 3 ) and two random normal matrices Ω ∈ R k × P and Ψ ∈ R l × P , the random individual sketches ( W i , Y i ) are defined as:
[Equation]
where J i :=
J ( θ 0 , (cid:102) X i ) .
Sketch-based computation of the top eigenspace Once the sketches are computed, the function
F IXED R ANK S YM A PPROX by Tropp et al.
[2017] computes the first eigenvectors and eigenvalues of the FIM of the dataset.
Overall, the memory footprint to find the sketches and the top eigenspace is O
( P ( s + N y M )) , where s is the number of queried eigenvectors and M is the size of (cid:102) X i .
As long as s ≪ P , this computation is tractable, given that N y M ≪ P is usual deep learning contexts.
Algorithm 2 summarizes the process that yields the FIM-based projections via sketching.
We drop the scaling coefficient σ − 2 ε as it doesn’t affect the computation, given that we only want orthogonal eigenvectors and eigenvalues.[-Algorithm 
-]
Require: s (desired size of the subspace)
1: k ← 2 s + 1 2: l ← 4 s + 3 3: Draw Ω ∈ R k × P , Ψ ∈ R l × P , two random normal matrices 4: Initialize Y =
∈ R P × k , W = 0 ∈ R l × P 5: for all training task T i do : 6: J
i ← J ( θ 0 , (cid:102) X i ) 7: Y ← Y + 1 /N
( Ω J ⊤ i ) J i )
⊤ 8: W ← W + 1 /N ( Ψ J ⊤ i )
J i
A.4 Meta-training the Parametric Task Distribution
The key to our meta-learning approach is to estimate the quality of ˜ p ξ ( f ) via the NLL of context data from training tasks, and use its gradients to update the parameters of the distribution ξ .
Optimizing this loss over tasks in the dataset draws ˜ p ξ ( f ) closer to the empirical distribution present in the dataset, and hence towards the true distribution p ( f ) .
We present three versions of U N L I M I TD, depending on the choice of structure of the prior covariance over the weights (see Section 3.1 and Appendix A.2 for more details).
U N L
I M I TD-I (Algorithm 3) is the meta-training with the fixed identity prior covariance.
U N L
I M I TD-R and U N L
I M I TD-F (Algorithm 4) learn a low-dimensional representation of that prior covariance, either with random projections or with FIM-based projections.
1: Initialize do : 3: ε ) ▷ J
= J ( θ 0 , X i ) 6:
Update θ
[Table]
θ 0 ∪ µ (cid:80) i NLL i
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
do
[Table]
Sample 9: for all T
Y
) do : 10:
NLL i ←
[Table]
AUSS i ; , JQ ⊤ diag ( 2 ) QJ ⊤ + Σ ε ) ▷ J
= J ( θ 0 , X i ) 11:
[Table] [Table]
[Table] [Table]
Computing the likelihood.
In the algorithms, the function G AUSS NLL( Y i ; m , K ) stands for NLL of Y i under the Gaussian N ( m, K ) (see equation 2).
In the mixture case, we instead use M IXT NLL, which wraps equation 3 and calls G AUSS NLL for the individual NLL computations (see discussion in Section 3.2).
In this case, µ becomes { µ j } j = α j =1 and s becomes { s j } j = α j =1 when applicable.
Finding the FIM-based projections.
The FIM-based projection matrix aims to identify the elements of ϕ =
J ( θ 0 , X ) that are most relevant for the problem (see Section 3.1 and Appendix A.3).
However, this feature map evolves during training, because it is θ 0 -dependent.
How do we ensure that the directions we choose for Q remain relevant during training?
We leverage results from Fort et al. [2020], stating that the NTK (the kernel associated with the Jacobian feature map, see Section A.1.2) changes significantly at the beginning of training and that its evolution slows down as training goes on.
This suggests that as a heuristic, we can compute the FIM-based directions after partial training, as they are unlikely to deviate much after the initial training.
For this reason, U N L I M I TD-F (Algorithm 4) first calls U N L I M I TD-I (Algorithm 3) before computing the FIM-based Q that yields intermediate parameters θ 0 and µ .
Then the usual training takes place with the learning of s in addition to θ 0 and µ .
A. U N L I M I TD as a generalization of ALPaCA
Restraining the linearization to the last layer Remember the linear regression of equation 6, that we obtained by linearizing the network with all its layers.
Let’s separate the parameters of the network θ into two: the parameters of all the layers but the last one λ , and the parameters of the last layer ρ : θ = λ ∪ ρ .
We assume that the last layer is dense with biases: we will note ρ w the weight matrix and ρ b the biases of this last layer.
N ψ will stand as the dimension of the activations right before the last layer: in particular, ρ w ∈ R N y × N ψ and ρ b ∈ R N y .
P ′ will stand as the size of ρ : in our case,
P ′ =
N ψ × N y + N y .
We implicitly vectorize ρ , such that:
[Equation]
We now restrain the linear regression to the last layer as follows:
[Equation]
where:
• ψ λ ( · ) : R N x × K → R N ψ × K stands for the function that maps the inputs and the activations right before the last layer; • J ′ ( · , · ) : R P ′
×
R N ψ × K → R N y K × P ′ stands for the jacobian of the last layer with respect to the parameters ρ .
We can write the jacobian in closed-form due to the linearity of the last layer:
[Equation]
Note that J ′ ( ρ 0 , ψ λ ( x )) does not depend on the linearization point ρ 0 (could be expected, given the linearity of the last layer).
• ρ − ρ 0 ∈ R P
′ is the correction to the last parameters.
We will note the correction to the weights as ˆ ρ w := ρ w − ρ w 0 and the correction to the bias as ˆ ρ b :=
ρ b − ρ b 0 .
Using Kronecker’s product identities, the linear regression in equation 9 can be rewritten:
[Equation]
Also, as a side note, another way of getting this linear regression (equation 10) is to rewrite the initial linearization of Section A.1.2, but with respect to ρ only:
[Equation]
Doing it this way has the benefit to show that the linearization is exact when restricted to the last layer.
For non-linear neural networks, the linearization is always an approximation.
Adapting U N L I M I TD to the last layer Just like what we did in the general case, we use the GP theory to make Bayesian inference on this linear regression.
The new parameters of the GP ˜ p ξ ( f ) are now ξ = ( λ , µ , Σ ) , where µ and Σ are the parameters of the Gaussian prior over ˆ ρ .
Note how ρhave disappeared from ξ : contrary to the general case, the linearization point is not optimized, as it does not impact the computation.
Also note that λ has replaced ρ 0 , as it parameterizes the feature map.
Comparison with ALPaCA
In the ALPaCA setting, the linear regression is:
[Equation]
Note that ˆ ρ w still plays the same role in the linear regression, but is not any correction anymore.
Substracting 11 - 10 yields:
[Equation]
U N L I M I TD restricted to the last layer is closely related to ALPaCA, in that they both perform a linear regression with the same kernel: the only difference lies in the additional bias term that is not considered in ALPaCA.
Thus, we can think of U N L I M I TD as a generalization of ALPaCA to all the layers of the network.
A.6 Additional results A.6.1 Additional results (single-cluster case)
Unimodal meta-learning: The meta-learned prior accurately fits the tasks.
First, we investigate the performance of U N L I M I TD on a unimodal task distribution consisting of sinusoids of varying amplitude and phase, using the single GP structure for ˜ p ξ ( f ) .
We evaluate the performance of the proposed approaches and MAML. We also compare the results between training with an infinite amount of available sine tasks (infinite task dataset), and with a finite amount of available tasks (finite task dataset). Examples of predictions at the test time are available in Figure 5, along with confidence levels. In both OoD detection and quality of predictions, U N L I M I TD-R, and U N L I M I TD-F perform better than U N L I M I TD-I (Figure 3), and this is reflected in the quality of the learned prior ˜ p ξ ( f ) in each case (see Appendix A.6.1).
With respect to the trade-off mentioned in Section 3.1, we find that
−−  4 x −− 2 −    y
In-distOoD1OoD
[Figure]
10 K 0 . [Figure]
AU C - R
O
C
s c
o r e
[Figure] F(inﬁnite) [Figure] F(ﬁnite)
[Figure] inﬁnite [Figure] (ﬁnite
[Figure]
10 K 0 .0 .0 .0 .0 .0 .
M S E
UNLIMTD
[Figure] F(inﬁnite)UNLIMTD [Figure] F(ﬁnite)UNLIMTD-R(inﬁnite)UNLIMTD-R(ﬁnite
[Figure]
UNLIMTD-I(inﬁnite)UNLIMTD-I(ﬁnite)
[Figure]
(inﬁnite
[Figure] ﬁnite [Figure] [Figure]
U N L I M I TD-R, U N L I M I TD-F) −−  4 x 2610 k ( 0 , x ) UNLIMTD-I(inﬁnite)UNLIMTD-I(ﬁnite) (b) Covariance cov ( f (0) , f ( x ))
(U N L I M I TD-I) −−  4 x −−   k ( 0 , x ) UNLIMTD-F(inﬁnite)UNLIMTD-F(ﬁnite)UNLIMTD-R(inﬁnite)UNLIMTD-R(ﬁnite) (c) Covariance cov ( f (0) , f ( x ))
(U N L I M I TD-R, U N L I M I TD-F) for small networks, a rich prior over the weights matters more than the full expressiveness of the feature map, making both U N L I M I TD-R, and U N L I M I TD-F appealing.
However, after running further experiments on a deep-learning image-domain problem, this conclusion does not hold for deep networks (see Appendix A.6.3), where keeping an expressive feature map is important (U N L I M I TD-I and U N L I M I TD-F are appealing in that case).
Thus, U N L I M I TD-F is the variant that we retain, for it allows similar or better performances than the other variants in all situations.
Comparison between variants In both OoD detection and quality of predictions, U N L I M I TDR, and U N L I M I TD-F perform better than U N L I M I TD- I (Figure 3), and this is reflected in the quality of the learned prior ˜ p ξ ( f ) in each case (see Appendix A.6.1).
With respect to the trade-off mentioned in Section 3.1, we find that for small networks, a rich prior over the weights matters more than the full expressiveness of the feature map, making both U N L I M I TD-R, and U N L I M I TD-F appealing.
However, after running further experiments on a deep-learning image-domain problem, this conclusion does not hold for deep networks (see Appendix A.6.3), where keeping an expressive feature map is important (U N L I M I TD-I and U N L I M I TD-F are appealing in that case).
Thus,
U N L I M I TD-F is the variant that we retain, for it allows similar or better performances than the other variants in all situations.
Quality of the priors To qualitatively analyze the prior after meta-training on the sines, we plot the mean functions and the covariance functions of the resulting GP, that is ˜ p ξ ( f ) (Figure 4).
All the trainings yield a similar mean for ˜ p ξ ( f ) (that is a cosine with amplitude 1.5 and offset 1) (Figure 4a), which is close to the theoretical mean of p ( f ) (a cosine with amplitude 2.5 and offset 1).
The covariance function of U N L
I M I TD-R and U N L I M I TD-F (Figure 4c) resembles what we would expect for p ( f ) (e.g. periodicity, negative correlation between 0 and π , etc.), but it is not the case for
U N L
I M I TD-I (Figure 4b).
This empirical analysis confirms the quantitative comparison in terms of
OoD detection and prediction performance carried in Section 5.
Examples of predictions Figure 5 summarizes the predictions of the model meta-trained on sines, for a varying number of context inputs K , breaking down all the different cases of training.
A.6.2 Additional results (multi-cluster case)
Quality of the priors Figure 6 and Figure 7 show the mean and covariance functions of the GP (when training with a single GP), and the mean and covariance functions of the GPs composing the mixture(when training with a mixture of GPs).
We note that in the case of the mixture, both of the
Gaussians composing the mixture have correctly captured the common features shared by each of the clusters (respectively the linear and the sine cluster).
For instance, the mean of the line cluster is the zero function, which matches Figure 6b, and the correlation between x = 1 and the other inputs is correctly rendered for linear tasks (Figure 7b).
When learning with a single GP, the learnt mean and covariance do not match any of the two clusters.
For example, the mean has an intermediate offset between the offset of the sine cluster and the line cluster.
This empirical analysis comforts the conclusions of Section 5: the mixture model yields better results than the single GP case.
A.6. U N L I M I TD yields effective predictions on large-scale vision problems
We consider a regression-vision meta-learning problem from Gao et al.
[2022], Shapenet1D, aiming to predict object orientations in space.
In this problem, each task consists of a different object of which we want to predict the orientation.
For each task, the context data consists of some images of the same object, but with different orientations; the query inputs are other images of the same object, with unknown orientations.
Details on the problems and the datasets can be found in appendix A.7.
We train a deep learning model on Shapenet1D, and we compare the performances on the test set between U N L I M I TD-I, U N L
I M I TD-R and U N L I M I TD-F (Figure 8).
More training and test details can be found in the Appendix A.10.
Both U N L I M I TD-I and U N L I M I TD-F yield better performances than MAML, achieving low angle errors.
U N L I M I TD-R however gives poor results, worse than that of MAML.
In terms of the trade-off of Section 3.1, our conclusion from small networks does not scale up to deep models.
Here, the loss of valuable features is perceptible (that is what happens with U N L I M I TD-R, when the randomness of the directions may drop such features), and not learning a rich prior over the weights is not burdensome (U N L I M I TD-I).
However, U N L I M I TD-F plays a role of compromise, by learning the prior covariance while keeping the few important features of the jacobian, as it gives comparable results to U N L I M I TD-I.
A.7 Details on the regression problems A.7.1 Simple regression problems
Our simple regression problems are inspired by Vuorio et al.
[2019]’s work.
They consists of three clusters of different types of tasks, and varying offset.
The first cluster consists of sines with constant frequency and offset, but with a varying amplitude and phase:
[Equation]
The second cluster consists of lines with a varying slope and no offset:
[Equation]
The last cluster consists of quadratic functions, with a varying quadratic coefficient and phase, and with a constant offset:
[Equation]
In all these clusters, we add an artificial Gaussian noise on the context observations N (0 , 0 . 05) .
The query datapoints remain noiseless, to remain coherent with the assumption from Section A.1. borrowed from Rasmussen and Williams [2005].
A.7.2 Vision problem
We consider the meta-learning vision, regression problem recently created by Gao et al.
[2022] (namely Shapenet1D), that consists in estimating objects orientation from images.
The objects have a vertical angular degree of freedom, and Figure 9 shows an example of such objects in different positions.
We use the same training and test datasets as Gao et al.
[2022], with no kind of augmentation (in particular, no artificial noise on the ground-truth angles).
In particular, we use the same intra-category (IC) evaluation dataset (that is, objects from the same categories as the objects used for training) and cross-category (CC) evaluation dataset (that is, objects from different categories as the ones used for training).
A.8 Training and test details for the single cluster case A.8.1 Training details of U N L I M I TD (single cluster case)
The model is a neural network with 2 hidden layers, 40 neurons each, with a ReLU non-linearity.
In our single-cluster experiment, the cluster is the sine cluster (see Appendix A.7).
In the case where there are an infinite number of available sine tasks during training ( ie N = + ∞ ), the training is performed with n = 24 tasks per epoch, and at each epoch the context inputs are randomly drawn from [ − 5 , 5] (which means that M = + ∞ ).
In the case where we restrict the available tasks to a finite number, we randomly choose N = 10 tasks and M = 50 context datapoints per task before training (they are shared among all the “finite” trainings) and perform the trainings with n = 6 tasks per epoch.
For all the trainings, the number of context inputs seen during training is K = 10 .
For all trainings, we train U N L I M I TD on 60 , 000 epochs.
When dealing with the U N L I M I TD-F case, we allow half of the epochs ( 30 , 000 ) to the training before finding the intermediate θ 0 and µ (see Algorithm 4), and the other half after .
In the U N L I M I TD-F case with infinite available tasks, a finite number of tasks is needed to compute the FIM: thus we build an artificial finite dataset of N = 100 and M = P (arbitrary, but chosen so that J ( θ 0 , (cid:102)
X i )
J ( θ 0 , (cid:102)
X i )
⊤ gets a chance to be full-rank ie contain as much information as possible), that is used only for that computation.
In the U N L I M I TD-F and the U N L I M I TD-R cases, the subspace size is s = 10 .
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0 .
The noise σ ε = 0 .
05 , equal to the noise added to the context data.
We compute the NLL using Rasmussen and
Williams [2005]’s implementation.
A.8.2 Training details of MAML (single cluster case)
We train MAML baselines to compare our results with that of MAML.
A large part of these hyperparameters is directly inspired from Finn et al.
[2017]’s work.
The model is a neural network with 2 hidden layers, 40 neurons each, with a ReLU non-linearity.
In our single-cluster experiment, the cluster is the sine cluster (see Appendix A.7).
In the case where there are an infinite number of available sine tasks during training ( ie N = + ∞ ), the training is performed with n = 24 tasks per epoch, and at each epoch the context and query inputs are randomly drawn from [ − 5 , 5] .
In the case where we restrict the available tasks to a finite number, we randomly choose N = 10 tasks and M = 50 datapoints per task (used for both context and query batches) before training (they are shared among all the “finite” trainings) and perform the trainings with n = 6 tasks per epoch.
For all trainings, the number of context datapoints is K = 10 , and the number of query datapoints is
L = 10 .
We meta-train for 70 , 000 epochs.
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0 .
The inner learning-rate is kept constant, at 0 .
The number of inner updates is 5 during training, and 10 at test time.
A.8.3 Test details (single-cluster case)
For the OoD detection evaluation, we plot the AUC as a function of the number of the context inputs
K .
The AUC is computed using the NLL of the context inputs wrt to ˜ p ξ ( f ) (our uncertainty metric).
The true-positives are the OoD tasks (lines and quadratic tasks) flagged as such; the false-positives are the in-distribution tasks (sines) flagged as OoD.
For the predictions, we plot the average and ci95 of the MSE on 1,000 tasks (100 queried inputs each).
In the U N L I M I TD-R case, we also compute the average and ci95 on 5 different random projections trainings.
A.9 Training and test details for the multi cluster case A.9.1 Training details of U N L I M I TD (multi-cluster case)
The model is a neural network with 2 hidden layers, 40 neurons each, with a ReLU non-linearity.
In our multi-cluster experiment, α = 2 : the clusters consist of the sine cluster and the linear cluster (see
Appendix A.7).
For all the trainings, the training is performed with n = 24 tasks per epoch (with an infinite number of available sine tasks and linear tasks during training ie N = + ∞ ), and at each epoch the context inputs are randomly drawn from [ − 5 , 5] (which means that M = + ∞ ).
In accordance with our equal probability assumption from Section 3.2, at each epoch n/ 2 = 12 tasks come from the sine cluster and n/ 2 = 12 tasks come from the linear cluster.
For all trainings, the number of context inputs seen during training is K = 10 .
For all trainings, we train U N L I M I TD algorithm on 60 , 000 epochs: because we deal with the U NL I M I TD-F case, we allow half of the epochs ( 30 , 000 ) to the training before finding the intermediate θ 0 and { µ j } j = α j =
(see Algorithm 4), and the other half after .
In the U N L I M I TD-F case, a finite number of tasks is needed to compute the FIM: thus we build an artificial finite dataset of N = 100 and M = P (arbitrary, but chosen so that J ( θ 0 , (cid:102)
X i )
J ( θ 0 , (cid:102)
X i ) ⊤ gets a chance to be full-rank ie contain as much information as possible), that is used only for that computation.
For all trainings, the subspace size is s = 10 .
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0 .
The noise σ ε = 0 .
05 , equal to the noise added to the context data.
We compute the NLL using Rasmussen and
Williams [2005]’s implementation.
When training with M IXT , we make sure to initialize s 1 and s 2 randomly with N ( 0 , 0 .
5 I ) , so that the meta-learning can effectively differentiate the two clusters.
Also, in the M IXT case, the mean µ is unique before computing the FIM.
Once we have found the projection directions, we initialize ( µ 1 , µ 2 ) with the intermediate µ , thus yielding two Gaussians.
A.9.2 Training details of MAML (multi-cluster case)
We reimplement and train a MAML baseline to compare our results with that of MAML.
A large part of these hyperparameters is directly inspired from Finn et al.
[2017]’s work.
The model is a neural network with 2 hidden layers, 40 neurons each, with a ReLU non-linearity.
In our multi-cluster experiment, α = 2 : the clusters consist of the sine cluster and the linear cluster (see
Appendix A.7).
The training is performed with n = 24 tasks per epoch (with an infinite number of available sine and linear tasks during training ie N = + ∞ ) and at each epoch the context and query inputs are randomly drawn from [ − 5 , 5] .
In accordance with our equal probability assumption from Section 3.2, at each epoch n/ 2 = 12 tasks come from the sine cluster and n/ 2 = 12 tasks come from the linear cluster.
For all trainings, the number of context datapoints is K = 10 , and the number of query datapoints is
L = 10 .
We meta-train for 70 , 000 epochs.
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0 .
The inner learning-rate is kept constant, at 0 .
The number of inner updates is 5 during training, and 10 at test time.
A.9.3 Training details of MMAML (multi-cluster case)
We train a MMAML by using Vuorio et al.
[2019]’s code, using the best setting mentionned in their paper (FiLM).
We adapt their code to train it on our sine and linear clusters: we add an offset to their sine cluster (via their parameter bias ), change the phase from sin( x − φ ) to sin( x + φ ) and specify via the arguments the slope range ( [ − 1 , 1] ) and
the y-intercept range ( [0 , 0] , because it does not vary in our case) of the line tasks.
We also change the number of context inputs that we set to
K = 10 , to remain coherent with the rest of the trainings.
The rest of the hyperparameters are kept identical to the command specified in the repository of MMAML.
Finally, at test time we make the query ground-truths noiseless, to remain coherent with the rest of the test conditions.
A.9.4 Test details (multi-cluster case)
For the OoD detection evaluation, we plot the AUC as a function of the number of the context inputs
K .
The AUC is computed using the NLL of the context inputs wrt to ˜ p ξ ( f ) (our uncertainty metric).
The true-positives are the OoD tasks (quadratic tasks) flagged as such; the false-positives are the in-distribution tasks (lines and sines) flagged as OoD.
For the predictions, we plot the average and ci95 of the MSE on 1,000 tasks (100 queried inputs each): half of them are sines and half of them are lines.
A.10 Training and test details for the vision problem
The model is a deep neural network, identical to the one used by Gao et al.
[2022] except for the last layer: instead of doing a one-dimensional regression (where the output stands for an angle prediction), we perform a two-dimensional regression (where the output stands for a cosine and sine prediction).
This choice is motivated by the fact that the Gaussian noise assumed in Section A.1.1 cannot capture the complexity of an angle error (e.g., predicting 361 ◦ should yield a low error when compared to the ground-truth angle 0 ◦ ), but better renders the MSE that can be applied on cosine and sine.
A.10.1 Training details of U N L I M I TD (vision case)
For all the trainings, there are n = 10 tasks per epoch and K = 15 context inputs per task.
When dealing with the U N L I M I TD-F case, we allow half of the epochs ( 5 , 000 ) to the training before finding the intermediate θ 0 and µ (see Algorithm 4), and the other half after.
In the U N L I M I TD-F and U N L I M I TD-R cases, the subspace size is s = 100 .
For all the trainings, the meta-optimizer is Adam with an initial learning rate of 0 .
The noise is σ ε = 0 .
We compute the NLL using Rasmussen and Williams [2005]’s implementation.
A.10.2 Training details of MAML (vision case)
We train a MAML baseline to compare our results with that of MAML.
A large part of these hyperparameters is directly inspired from Gao et al.
The training is performed with n = 10 tasks per epoch.
The number of context datapoints is K = 15 , and the number of query datapoints is L = 10 .
We meta-train for 50 , 000 epochs.
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0 . 0005 .
The inner learning-rate is kept constant, at 0 .
The number of inner updates is 5 during training, and 20 a test time.
A.10.3 Test details (vision problem)
At test time, we wrap our model with the arctan function to convert the predictions angle predictions.
Then, we compare the angle predictions with the ground-truth angles.
To do so, we use the following error from Gao et al.
[2022] to compare two angles:
[Equation]
where E β ± ,β ∗ = | y ± 360 − y ∗ | and E β,β ∗ = | y − y ∗ | .
When plotting the performance (Figure 8), we plot the average and ci95 on 100 tasks (15 queried inputs each).
For U N L I M I TD-R, the average and ci95 are computed on 5 different random projection trainings.
−−  4 x −
[Table] [Table]
[Table] [Table]
[Table] [Table]
y
[Table]
TargetContextdataPrediction+/
[Table]
−−
[Table]
x −
[Table] [Table]
[Table] [Table]
[Table] [Table]
y
[Table] [Table]
−
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Figure] [Figure]
[Table]
[Table]
[Table]
[Table]
[Figure]
−−  4 x − 1 . 0 − 0 .0 .
0 2 .3 .m
e a n
UNLIMTD-F(singleGP)
[Figure]
−−  4 x − 1 . 0 − 0 .0 .
0 2 .3 .m
e a n
UNLIMTD-F(mixture):gaussian
[Figure]
−−  4 x − 1 . 0 − 0 .0 .
0 2 .3 .m
e a n
UNLIMTD-F(mixture):gaussian
[Figure]
−−  4 x −−   k ( 0 , x )
UNLIMTD-F(singleGP)
−−  4 x
[Figure]
−   k (
[Figure]
x )
UNLIMTD-F(
[Figure] [Figure]
[Figure]
function cov ( f (1)
[Figure]
f ( x )) of the first GP of the mixture (U N L I M I TD-F with
[Figure]
[Figure] [Figure]
k (
[Figure]
(c)
Covariance function cov ( f (0) , f ( x )) of the second
GP of the mixture (U N L
I M
I TD- F with a mixture of GPs
[Figure]
Note how the covariance of the single GP is not accurate for any of the two clusters.
[Figure] [Figure]