N EURAL V ARIATIONAL D ROPOUT P ROCESSES
A BSTRACT
Learning to infer the conditional posterior model is a key step for robust metalearning.
This paper presents a new Bayesian meta-learning approach called Neural Variational Dropout Processes (NVDPs).
NVDPs model the conditional posterior distribution based on a task-speciﬁc dropout; a low-rank product of Bernoulli experts meta-model is utilized for a memory-efﬁcient mapping of dropout rates from a few observed contexts.
It allows for a quick reconﬁguration of a globally learned and shared neural network for new tasks in multi-task few-shot learning.
In addition, NVDPs utilize a novel prior conditioned on the whole task data to optimize the conditional dropout posterior in the amortized variational inference.
Surprisingly, this enables the robust approximation of task-speciﬁc dropout rates that can deal with a wide range of functional ambiguities and uncertainties.
We compared the proposed method with other meta-learning approaches in the few-shot learning tasks such as 1D stochastic regression, image inpainting, and classiﬁcation.
The results show the excellent performance of NVDPs.
I NTRODUCTION
In traditional machine learning, a large amount of labeled data is required to train deep models (LeCun et al., 2015).
In practice, however, there are many cases where it is impossible to collect sufﬁcient data for a given task.
On the other hand, humans can quickly understand and solve a new task even from few examples (Schmidhuber, 1987; Andrychowicz et al., 2016).
This distinguishing characteristic of humans is referred to as the meta-learning ability, which enables them to accumulate the past learning experiences into general knowledge and to utilize it for efﬁcient learning in the future.
Incorporating the meta-learning capability into artiﬁcial machines has long been an active research topic of machine learning (Naik & Mammone, 1992; Vinyals et al., 2016;
Snell et al., 2017;
Ravi & Larochelle, 2017;
Finn et al., 2017; Lee et al., 2019c; Hospedales et al., 2021).
Recently, Bayesian meta-learning methods have been attracting considerable interest due to incorporating the uncertainty quantiﬁcation of Bayesian frameworks into the efﬁcient model adaptation of meta-learning approaches.
An earlier study (Grant et al., 2018) extended the optimization-based deterministic approach model-agnostic meta-learning (MAML)
(Finn et al., 2017) to a hierarchical
Bayesian frameworks (Daum´e III, 2009).
Later, the optimization-based variational inference (Yoon et al., 2018; Finn et al., 2018; Ravi & Beatson, 2019; Lee et al., 2019a; Nguyen et al., 2020) and the model-based Bayesian meta-learning (Gordon et al., 2018; Garnelo et al., 2018a;b; Iakovleva et al., 2020) methods were presented.
These methods achieved outstanding outcomes in various few-shotregression and classiﬁcation tasks.
In fact, the adaptation of deep models that can estimate uncertainties is a key study for reliable machine learning systems in real-world applications with high judgmental risks such as medical AI or autonomous driving systems.
Inspired by recent studies, we propose a new model-based Bayesian meta-learning approach, neural variational dropout processes (NVDPs).
The main contribution of this work is to design a new type of Neural Network (NN) based conditional posterior model that can bypass the under-ﬁtting and posterior collapsing of the existing approaches (Gordon et al., 2018; Garnelo et al., 2018a;b; Iakovleva et al., 2020).
NVDPs extend the simple yet effective posterior modeling of Variational Dropout (VD) (Kingma et al., 2015; Gal & Ghahramani, 2016; Molchanov et al., 2017;
Hron et al
Liu et al., 2019) in the context of meta-learning.
A novel low-rank product of Bernoulli experts meta-model is utilized in the dropout posterior to learn task-speciﬁc dropout rates conditioned on a few learning examples.
In this way, a full conditional posterior model over the NN’s parameters can be efﬁciently obtained.
In addition, we also propose a new type of prior to optimize the conditional dropout posterior in variational inference, which supports the robust training of conditional posteriors.
Although it is developed in conjunction with NVDPs, the formulation allows its adoption to other recent methods, and we show the effectiveness.
We have evaluated NVDPs compared with other methods on various few-shot learning tasks and datasets.
The experiments show that NVDPs can circumvent the posterior collapsing and under-ﬁtting and achieve outstanding performances in terms of log-likelihood, active learning efﬁciency, prediction accuracy, and generalization.
B AYESIAN M ETA -L EARNING 2. A MORTIZED V ARIATIONAL I NFERENCE F RAMEWORK IN T HE M ULTI -T ASK D ATA
A goal in meta-learning is to construct models that can quickly solve new tasks from small amounts of observed data.
To achieve this, it is important to learn general (or task-invariant) structures from multiple tasks that can be utilized for efﬁcient model adaptation when necessary.
Bayesian metalearning (Gordon et al., 2018; Garnelo et al., 2018b; Ravi & Beatson, 2019) formulates this objective of meta-learning as an amortized variational inference (VI) of the posterior distribution in multi-task environments.
Suppose a collection of T related tasks is given, and each t -th task has the training data D t containing N i.i.d.
observed tuples ( x t , y t ) =
( x ti , y ti ) Ni
=1 .
Then, the evidence lower-bound (ELBO) over the log-likelihood of the multi-task dataset can be derived as:
[Equation]
Here, the p ( y t | x t , φ t ) is the likelihood (or NN model) on t -th training data and φ t is t -th task-speciﬁc variable ( i.e . a latent representation or weights of NN) and q ( φ t |D t ; θ ) is a tractable amortized variational posterior model utilized for the approximation of the true unknown posterior distribution of φ t for each given t -th task (i.e p ( φ t |D t ) )
(Kingma & Welling, 2013; Gordon et al., 2018; Garnelo et al., 2018b; Ravi & Beatson, 2019; Iakovleva et al., 2020).
The parameter θ represents the common structures that can be efﬁciently learned across multiple different tasks.
The prespeciﬁed prior distribution p ( φ t ) in the KL term provides a stochastic regularization that can help to capture the task-conditional uncertainty and prevent the collapsing of q ( φ t |D t ; θ ) .
In fact, the maximization of the the evidence lower bound (ELBO), right side of equation 1, with respect to the conditional variational posterior model is equivalent to the minimization of (cid:80)
Tt =1 KL ( q ( φ t |D t ; θ ) || p ( φ t |D t )) .
Essentially, the goal in the amortized VI of Bayesian meta-learning is to learn the inference process of the conditional posterior distribution via the variational model q ( φ t |D t ; θ ) and the shared general structure θ across multiple tasks since these task-invariant knowledge can later be utilized for the efﬁcient adaption of the NN function on new unseen tasks.
The approximation of the conditional posterior also enables the ensemble model approach and uncertainty quantiﬁcation.
2. T HE E XISTING C ONDITIONAL P OSTERIOR AND P RIOR M ODELINGS
Many of the recent Bayesian meta-learning approaches can be roughly categorized into the optimization-based (Grant et al., 2018; Yoon et al., 2018;
Finn et al., 2018; Ravi & Beatson, 2019;
Lee et al., 2019a; Nguyen et al., 2020) or the model-based posterior approximation approaches (Gordon et al., 2018; Garnelo et al., 2018a;b; Kim et al., 2019; Iakovleva et al., 2020).
In the optimizationbased approaches, the task-speciﬁc variable φ t can be seen as the adapted NN weights.
For example, the deterministic model-agnostic meta-learning (MAML)
(Finn et al., 2017) can be considered as learning a Dirac delta
variational posterior modeling (i.e., q ( φ t |D tC ; θ )
≈ δ ( φ t − SGD j ( D tC , θ )) ) where the goal is to learn the shared global initialization NN’s parameters θ such that a few j steps of SGD updates on the small subset 2 D tC of the t -th dataset D t provides a good approximation of the task-speciﬁc weights φ t .
Many optimization-based Bayesian meta-learning approaches such as LLAMA (Grant et al., 2018), PLATIPIS (Finn et al., 2018), BMAML (Yoon et al., 2018), and
ABML (Ravi & Beatson, 2019) have incorporated Gaussian types of posterior and prior models into the deterministic adaptation framework of MAML in order to improve the robustness of models.
The optimization-based approaches are applicable to various types of few-shot learning tasks, but the adaptation cost at test time is computationally expensive due to the inversion of Hessian or kernel matrix; this may restrict their usage in environments with low computational resources.
On the other hand, the model-based Bayesian meta-learning approaches such as VERSA (Gordon et al., 2018) or Neural Processes (Garnelo et al., 2018a;b) allow an instant estimation of the Bayesian predictive distribution at test time via NN-based conditional posterior modeling.
VERSA employs an additional NN-based meta-model g θ ( · ) to directly predict the Gaussian posterior of agent NN’s task-speciﬁc weight φ t from the small context set ( i.e. , q ( φ t |D tC ; θ ) =
N ( φ t | ( µ, σ ) =
g θ
( D tC )) ).
In this case, the shared structure θ represents the meta-model’s parameters.
However, the direct approximation of agent NN weights in VERSA could limit their scalability due to the large dimensionality of the weights; they only consider the task-speciﬁc weights for the single softmax output layer.
In addition, VERSA did not utilize any task-speciﬁc prior p ( φ t ) , so the conditional posterior could turn into a deterministic one while training with the Monte Carlo approximation (Iakovleva et al., 2020).
Instead of the NN’s weights, the Neural Processes (NPs) model the conditional posterior with a latent representation ( i.e. , q ( z t |D t ; θ ) =
N ( z t | ( µ, σ ) =
g θ
( D t )) )
(Garnelo et al., 2018b).
Regarding the task-speciﬁc latent representation as a second input to the likelihood p ( y t | x t , z t ) (similar to Kingma & Welling (2013) or Edwards & Storkey (2017)) provides an efﬁcient way to transfer the conditional information (Garnelo et al., 2018b).
However, the following studies indicate that
NPs often suffer from under-ﬁtting (Kim et al., 2019) and posterior-collapsing (Grover et al., 2019;
Le et al., 2018).
We conjecture that these behaviors are partially inherent to its original variational formulation; the conditional posteriors of NPs depend on the whole task data while training, which did not empirically show good generalization performance at test.
In addition, it is well known that the highly ﬂexible NN-based likelihood can neglect the latent representation in variational inference (Hoffman & Johnson, 2016;
Sønderby et al., 2016; Kingma et al., 2016; Chen et al., 2017; Yeung et al., 2017;
Alemi et al., 2018; Lucas et al., 2019; Dieng et al., 2018); we also demonstrate the posterior collapsing of the NPs in our experiments.
N EURAL V ARIATIONAL D ROPOUT P ROCESSES
This section introduces a new model-based Bayesian meta-learning approach, called Neural Variational Dropout Processes (NVDPs).
Unlike the existing methods such as NPs or VERSA employing conditional latent representation or direct modeling of NN’s weights, NVDPs extend the posterior modeling of the variational dropout in the context of meta-learning.
We also introduce a new type of task-speciﬁc prior for optimizing the conditional dropout posterior in variational inference.
3. A C ONDITIONAL D ROPOUT P OSTERIOR
Variational dropout (VD) (Kingma et al., 2015; Gal & Ghahramani, 2016; Molchanov et al., 2017;
Hron et al., 2018; Liu et al., 2019) is a set of approaches that models the variational posterior distribution based on the dropout regularization technique.
The dropout regularization randomly turns off some of the Neural Network (NN) parameters during training by multiplying the discrete Bernoulli random noises to the parameters (Srivastava et al., 2014; Hinton et al., 2012).
This technique was originally popularized as an efﬁcient way to prevent NN model’s over-ﬁtting.
Later, the fast dropout (Wang & Manning, 2013; Wan et al., 2013) reported that multiplying the continuous noises sampled from Gaussian distributions works similarly to the conventional Bernoulli dropout (Srivastava et al., 2014) due to the central limit theorem.
The Gaussian approximation of the Bernoulli dropout is convenient since it provides a fully factorized (tractable) posterior model over the NN’s parameters (Wang & Manning, 2013; Wan et al., 2013).
In addition, VD approaches often utilize sparse priors for regularization (Kingma et al., 2015; Molchanov et al., 2017; Hron et al., 2018; 2017; Liu et al., 2019).
This posterior and prior distribution modeling in VD enables the learning of each independent dropout rates over the NN parameters as variational parameters.
This distinguishes the VD approaches from the conventional dropouts that use a single ﬁxed rate over all parameters.
However, the dropout posterior in the conventional VD becomes ﬁxed once trained and it is unable to express the conditional posterior according to multiple tasks.
[Figure]
( + ( ) (cid:1) + ( ) (cid:1) + ( ) , $ - .$ i=1…
S t=1…
T
Meta NN
KL
Variationalprior
Conditional posterior
[Figure] [Figure]
$ !
$ - .$
i
[Figure]
1…N / .$
[Figure]
Conditional dropout posterior.
We propose a new amortized variational posterior model that can be efﬁciently adapted for each given task.
Suppose we train a fully connected NN of L layers, then a conditional dropout posterior 3 based on the task-speciﬁc dropout rates P t over the K × D dimensional deterministic parameters θ of each l -th layer of the NN can be given as follows:
[Equation]
The parameter 4 θ k,d is shared across different tasks, representing the common task-invariant structure.
Then the task-speciﬁc NN parameter φ t are fully described by the mean and variance of each independent Gaussian distribution via θ k,d and P k,d .
Note that the variational posterior is explicitly conditioned on the subset of the t -th training dataset D tC
= { x ti , y ti } Si =1 ( ⊆ D t ) known as the t -th context set.
The key idea of conditional posterior modeling in NVDPs is to employ a NNbased meta-model to predict the task-speciﬁc dropout rate P tk,d from the small context set D tC .
The meta-model learning to approximate P tk,d for each given task is deﬁned as:
[Equation]
The set representation r t is deﬁned as the mean of features obtained from each data in t -th context set D tC ( i.e. , r t = (cid:80)
Si =1 h ω ( x ti , y ti ) /S , where h ω is a feature extracting NN parameterized by ω ), summarizing order invariant set information (Edwards & Storkey, 2017; Lee et al., 2019b).
The g ψ ( · ) is Meta NN model parameterized by ψ to predict a set of logit vectors (i.e., a ∈ R K , b ∈ R D and c ∈ R ).
Then, the sigmoid function with a learnable temperature parameter τ (i.e., s τ ( · )
= 1 / (1 + exp( − ( · ) /τ ) ) is applied to them to get the low-rank components of task-speciﬁc dropout rates P tk,d : the row-wise s ( a k ) , column-wise s ( b d ) , and layer-wise dropout rate s ( c ) .
In other words, the task-speciﬁc dropout rate P tk,d is obtained by multiplying low-rank components of the conditionally approximated dropout rates from the meta-model g ψ ( D tC ) (see Figure1 (a)).
The product of n Bernoulli random variables is also a Bernoulli variable (Leemis, 2020).
By exploiting this, we interpret the approximation of the task-speciﬁc dropout rates in terms of the low-rank product of Bernoulli experts.
Unlike VERSA whose meta-model’s complexity is O ( LKD ) to model the full NN weight posterior directly, the complexity of meta-model in NVDPs is O
( L ( K + D + 1 )); the meta-model’s role is only to predict the low-rank components of the task-speciﬁc dropout rates.
With the shared parameter θ , this can greatly reduce the complexity of the posterior distribution approximation of the high-dimensional task-speciﬁc weights with only a few learning examples.
The product model tends to give sharp probability boundaries, which is often used for modeling the high dimensional data space (Hinton, 2002).
3. T HE T ASK -S PECIFIC V ARIATIONAL P RIOR
To optimize the conditional dropout posterior in equation 2, a speciﬁcation of the prior distribution p ( φ t ) is necessary to get a tractable derivation of the KL regularization term of the ELBO in equation 1.
The question is how we can deﬁne the effective task-speciﬁc prior.
In fact, one important requirement in the choice of the prior distribution in the conventional VD framework (Kingma et al., 2015) is that the analytical derivation of the KL term ( i.e., KL ( q ( φ t |D tC ; θ ) || p ( φ t )) in equation 1) should not depend on the deterministic NN parameter θ .
This allows the constant optimization of the ELBO w.r.t all the independent variational parameters ( i.e., θ l,k,d and P l,k,d for all l = 1 . . .
L , k = 1 . . .
K and d = 1 . . .
D ).
One way of modeling the prior is to employ the log-uniform prior p (log( | φ | )
∝ c as in the conventional VD (Kingma et al., 2015).
However, a recently known limitation is that the log-uniform prior is an improper prior ( e.g. , the KL divergence between the posterior and the log-uniform prior is inﬁnite).
This could yield a degeneration of the dropout posterior model to a deterministic one (Molchanov et al., 2017; Hron et al., 2018; 2017; Liu et al., 2019).
Besides, the conventional prior used in VD approaches does not support the task-dependent regularization.
Variational prior.
We introduce a new task-speciﬁc prior modeling approach to optimize the proposed conditional posterior; it is approximated with the variational prior deﬁned by the same dropout posterior model in equation 2 except that the model is conditioned on the whole task data (i.e. p ( φ t )
≈ q ( φ t |D t ) ).
The KL divergence between the conditional dropout posterior (with the only context set) and the variational prior (with the whole task data) can be derived as:
[Equation]
where both P k,d and ˆP k,d are the dropout rates predicted from the meta-model but with different conditional set information: P k,d is obtained from the small context set D tC , while ˆP k,d is from the whole task set D t .
Interestingly, the analytical derivation of the KL is independent of the shared parameter θ , thus this satisﬁes the necessary condition to be used as a prior in the VI optimization of the dropout posterior.
Figure 1(b) depicts the variational prior is conditioned on the whole dataset.
The shared posterior model for the task-speciﬁc prior introduced here was inspired by recent
Bayesian meta-learning approaches (Garnelo et al., 2018b; Kim et al., 2019; Iakovleva et al., 2020), but some critical differences are that: 1) we have developed the variational prior to regularize the task-speciﬁc dropout rates in the optimization of the conditional dropout posterior, 2) the denominator and the numerator of the KL divergence term in equation 4 are reversed compared with the existing approaches.
In fact, some other recent studies of the amortized VI inference (Tomczak &
Welling, 2018; Takahashi et al., 2019) analytically derived that the optimal prior maximizing VI objective is the variational posterior aggregated on the whole dataset: p ∗ ( φ )
=
(cid:82) D q ( φ |D ) p ( D ) .
Thus, we hypothesized the conditional posterior model that depends on the whole dataset should be used to approximate the optimal prior p ∗ ( φ t )
≈ q ( φ t |D t ) since the variational model conditioned on the aggregated set representation with much larger context (e.g., D t ⊇ D tC ) is likely to be much closer to the optimal task-speciﬁc prior than the model conditioned only on a subset.
The experiments in
Section 5 demonstrate that the variational prior approach provides a robust regularization for the conditional dropout posterior and the similar formulation is also applicable to the latent variable based conditional posterior models (Garnelo et al., 2018b; Kim et al., 2019).
3. S TOCHASTIC V ARIATIONAL I NFERENCE
With the conditional dropout posterior deﬁned in equation 2 and the KL regularization term in equation 4, we can now fully describe the ELBO objective of NVDPs for the multi-task dataset as:
[Equation]
The goal is to maximize the ELBO equation 5 w.r.t.
the variational parameters ( i.e. θ , ψ , ω , and τ ) of the posterior q ( φ t |D tC ; θ ) deﬁned in equation 2.
The optimization of these parameters is done by using the stochastic gradient variational Bayes (SGVB) (Kingma & Welling, 2013; Kingma et al., 2014).
The basic trick in the SGVB is to parameterize the random weights φ t ∼ q ( φ t |D t ) using a deterministic differentiable transformation φ t = f ( (cid:15), D tC ) with a non-parametric i.i.d.
noise (cid:15) ∼ p ( (cid:15) ) .
Then, an unbiased differentiable minibatch-based Monte Carlo estimator, ˆ L ( θ, ψ, ω, τ ) , of the ELBO of NVDPs can be deﬁned as:
[Equation]
T (cid:48) and M are the size of randomly sampled mini-batch of tasks and data points, respectively, per each epoch ( i.e., { x ti , y ti } Mi =1 =
D t ∼ {D t } T (cid:48) t =1 ∼ D ).
D tC is the subset set of D t discussed in section 3.1 and 3.2.
The transformation of the task-speciﬁc weights is given as φ tk,d = f ( (cid:15) k,d , D tC ) =
(1 − P tk,d ) θ k,d + (cid:113) P tk,d (1 − P tk,d ) θ k,d (cid:15) k,d with (cid:15) k,d ∼ N (0 , 1) from the equation 2.
The intermediate weight φ tk,d is now differentiable with respect to θ k,d and P tk,d .
Also, P tk,d (and ˆP tk,d ) is deterministically computed by the meta-model parameterized by ψ , ω , and τ as in equation 3.
Thus, the estimator ˆ L ( θ, ψ, ω, τ ) in equation 6 is differentiable with respect to all the variational parameters and can be optimized via the SGD algorithm.
In practice, a local reparameterization trick 5 is further utilized to reduce the variance of gradient estimators on training (Kingma et al., 2015).
R ELATED W ORKS
Gaussian Processes(GPs)
(Rasmussen & Williams, 2005) are classical Bayesian learning approaches closely related to NVDPs; GPs allow an analytical derivation of the posterior predictive distribution given observed context points based on Gaussian priors and kernels.
However, classical GPs require an additional optimization procedure to identify the suitable kernel for each task, and the time complexity is quadratic to the number of contexts.
Recently introduced model-based Bayesian meta-learning methods (Garnelo et al., 2018b;a; Kim et al., 2019; Gordon et al., 2018;
Iakovleva et al., 2020) offer NN-based conditional posterior models that can be efﬁciently learned from data and later provide an instant posterior predictive estimation at test time.
One important aspect of these model-based methods have in common is the utilization of permutation invariant set representation as an input to the meta-model (Edwards & Storkey, 2017; Lee et al., 2019b; Bloem-Reddy & Teh, 2020; Teh & Lecture, 2019), which makes a conditional posterior invariant to the order of the observed context data.
This property, known as exchangeability , is a necessary condition to deﬁne a stochastic process according to Kolmogorov’s Extension Theorem (Garnelo et al., 2018b; Kim et al., 2019; Øksendal, 2003).
NVDPs also use the set representation to approximate the task-speciﬁc dropout rates in the conditional dropout posterior.
Another related work to NVDPs is Meta Dropout (Lee et al., 2019a).
They also proposed a novel input-dependent dropout approach; Gaussian-based stochastic perturbation is applied to each preactivation feature of NN functions.
Their approach, however, is constructed using the optimization-based meta-learning methods (Finn et al., 2017; Li et al., 2017) without directly considering the set representation.
NVDPs extend the dropout posterior of Variational Dropout (VD) (Kingma et al., 2015; Wang & Manning, 2013; Wan et al., 2013;
Liu et al., 2019; Molchanov et al., 2017) in the context of model-based meta-learning employing set representation and introduce a new concept of variational prior that can be universally applied to other Bayesian learning approaches (Garnelo et al., 2018b; Kim et al., 2019).
Gal & Ghahramani (2016) also discussed a theoretical connection between VD and GPs.
E XPERIMENTS
Metrics in regression.
In the evaluation of the conditional NN models’ adaptation, the newly observed task data D ∗ is split into the input-output pairs of the context set D ∗ C = { x i , y i } Si =1 and the target set D ∗ T = { x i , y i } Ni =
( D ∗ C (cid:54)⊆ D ∗ T in evaluation (Garnelo et al., 2018b; Gordon et al., 2018)).
The log-likelihood (LL),
[Equation]
E q ( φ
[Equation]
D
[Equation]
C )
[log p ( y i | x i , φ ∗ )] , measures the performance of the NN model over the whole dataset D ∗ conditioned on the context set .
The reconstructive log-likelihood (RLL), 1 S (cid:80) i ∈D ∗ C E q ( φ ∗ |D ∗ C )
[log p ( y i | x i , φ ∗ )] , measures how well the model reconstructs the data points in the context set .
A low RLL is a sign of under-ﬁtting.
The predictive log-likelihood (PLL), 1 N (cid:80) i ∈D ∗ T E q ( φ ∗ |D ∗ C )
[log p ( y i | x i , φ ∗ )] , measures the prediction on the data points not in the context set (i.e. the target set).
A low PLL is a sign of over-ﬁtting.
NP
NP+CNP
NVDP NP+VP  22 - -  2 -   -  2 -   -   -   -   -
F i x e d v a r i a n c e
L e a r n e d v a r i a n c e
[Figure]
[Table]
NP NP
[Equation]
VP
[Table]
[Equation]
[Table]
GP Samples.
The 1D regression task is to predict unknown functions given some observed context points; each function (or data point) is generated from a Gaussian Process (GP) with random kernel, then the standard data split procedure is performed (i.e. S ∼ U (3 , 97) and N ∼ U
[ S + 1 , 100) ) at train time.
For the baseline models, Neural Process model (NP) and NP with additional deterministic representation (NP+CNP) described in (Garnelo et al., 2018b;a; Kim et al., 2019) is compared.
We also adopted the variational prior (VP) into the representation based posterior of NP and its variants (NP+VP and NP+CNP+VP).
For all models (including our NVDP), we used the same ﬁxed variance and learned variance likelihood architecture depicted in (Kim et al., 2019): the agent NN withhidden layers of 128 units with LeLU activation (Nair & Hinton, 2010) and an output layer of 1 unit for mean (or an additional 1 unit for variance).
The dimensions of the set representation r t were ﬁxed to 128 .
The meta NNs in the conditional dropout posterior in NVDP has 4 hidden layers of 128 units with LeakyReLU and an output layer of 257 units (i.e. K + D + 1 ) for each layer of the agent model.
All models were trained with Adam optimizer (Kingma & Ba, 2015) with learning rate 5eand 16 task-batches for 0 .
5 million iterations.
On validation, 50000 random tasks (or functions) were sampled from the GP function generator, and the split data of S ∼ U (3 , 97) and N = 400 − S were used to compute the log-likelihood (LL) and other evaluation metrics.
Table 1 summarizes the validation results of 1D regression with the GP dataset.
NVDP achieves the best LL scores compared with all other baselines in both the ﬁxed and learned variance likelihood model settings.
The NVDPs record high RLL on the observed data points and excellent PLL scores on the unseen function space in the new task; this indicates that the proposed conditional dropout posterior approach can simultaneously reduce the under-ﬁtting and over-ﬁtting of the agent model compared with the other baselines.
When VP is applied to NP or NP+CNP, the PLL scores tend to increase by meaningful margins in all cases.
This supports that the proposed variational prior (VP) approach can also reduce the over-ﬁtting of latent representation-based conditional posterior.
Figure 2 visualizes the few-shot 1D function regression results in both model settings.
In the ﬁxed variance setting, the functions sampled from NPs show high variability but cannot ﬁt the context data well.
NP+CNP can ﬁt the context data well but loses epistemic uncertainty a lot due to the collapsing of the conditional posterior.
On the other hand, the functions from NVDPs show a different behavior; they capture the function’s variability well while also ﬁtting the observed context points better.
NVDPs also approximate the mean and variance of unknown function well in learned variance setting.
[Figure]
S a m p l e 2 S a m p l e 1 C o n t e x t
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
to
[Table]
points.
NVDPs can use its estimation to quickly improve LLs, while other models are learning slowly.
Active learning with regression.
To further compare the uncertainty modeling accuracy, we performed an additional active learning experiment on the GP dataset described above.
The goal in active learning is to improve the log-likelihood of models with a minimal number of context points.
To this end, each model chooses additional data points sequentially; the points with maximal variance across the sampled regressors were selected at each step in our experiment.
The initial data point is randomly sampled within the input domain, followed byadditional points that are selected according to the variance estimates.
As seen in Figure 3, NVDPs outperform the others due to their accurate variance estimation, especially with a small number of additional points, and show steady improvement with less under-ﬁtting.
Image completion tasks.
The image completion tasks are performed to validate the performance of the models in the more complex function spaces (Garnelo et al., 2018b;a; Kim et al., 2019).
Here, we treat the image samples from MNIST (LeCun et al., 1998) and CelebA (Liu et al., 2015) as unknown functions.
The task is to predict a mapping from normalized 2D pixel coordinates x i ( ∈ [0 , 1] 2 ) to pixel intensities y i ( ∈ R 1 for greyscale, ∈ R 3 for RGB) given some context points.
We used the same learned variance baselines implemented in the GP data regression task (except the 2D input and 3D output for the agent NN and r t = 1024 were used for CelebA).
At each iteration, the images in the training set are split into S context and N target points (e.g., S ∼ U (3 , 197) , n ∼ U [ N + 1 , 200) at train and S ∼ U (3 , 197) , N = 784 − S at validation).
Adam optimizer with a learning rate 4e-4 and 16 task batches with 300 epochs were used for training.
The validation is performed on the separated validation set.
To see the generalization performance on a completely new dataset, we also tested the models trained on MNIST to the Omniglot validation set.
Table 2 summarizes the validation results of image completion tasks.
NVDPs achieve the outperforming LLs compared with all other baselines.
Interestingly, NVDPs (trained on the MNIST
[Table]
achieve the best results on the Omniglot dataset.
Figure 4 shows the image reconstruction results with a varying number of random context pixels.
NP generated various image samples when the number of contexts was small (e.g., m ≤ 30 or half), but those samples could not approximate the true unknown images well compared with the other models.
The NP+CNP achieves crisp reconstruction results compared with NP and also shows a good generalization performance on the
Omniglot dataset, but the sampled images (or functions) from NP+CNP had almost no variability due to its posterior collapsing behavior.
On the other hand, the samples from NVDPs exhibit comparable reconstruction results while also showing a reasonable amount of variability.
In addition,
NVDPs also present an outstanding generalization performance on the unseen Omniglot dataset.
This implies that NVDPs not only ﬁt better in the complex function regression but can also capture more general knowledge that can be applied to new unseen tasks.
Few-shot classiﬁcation tasks.
NVDPs can also be successfully applied for few-shot classiﬁcation tasks; we have tested NVDPs on standard benchmark datasets such as Omniglot (Lake et al., 2015) and MiniImagenet (Ravi & Larochelle, 2017) with other baselines: VERSA (Gordon et al., 2018),
CNP (Garnelo et al., 2018a), Matching Nets (Vinyals et al., 2016), Prototypical Nets (Snell et al., 2017), MAML (Finn et al., 2017),
Meta-SGD (Li et al., 2017) and Meta-dropout (Lee et al., 2019a).
For the classiﬁer, we used one-layer NNs with hidden units of 512 .
For the meta-model, we used two-layer NNs with hidden units of 256 similar to VERSA’s conditional posterior model.
For an image input, the NN classiﬁer outputs one logit value per each class; class-speciﬁc dropout rates for the NN classiﬁer are computed with the image features r t ∈ R 256 aggregated by the same class in the few-shot context examples.
The same deep (CONV5) feature extractor architecture is used as in VERSA (Gordon et al., 2018).
For each class, 1 or 5 few-shot context samples (i.e., labeled images) are given.
Among 5 or 20 classes, only the logit value related to the true label is maximized.
We use the same batch sizes, learning rate, and epoch settings depicted in VERSA (Gordon et al., 2018).
Table 3 summarizes the results.
NVDPs achieve higher predictive accuracy than the model-based meta-learning approaches CNP and VERSA and is comparable with the stateof-the-art optimization-based meta-learning approaches such as MAML or Meta-Dropout on the
Omniglot dataset.
NVDPs also record a good classiﬁcation accuracy in the MiniImageNet dataset.
C ONCLUSION
This study presents a new model-based Bayesian meta-learning approach, Neural Variational
Dropout Processes (NVDPs).
A novel conditional dropout posterior is induced from a meta-model that predicts the task-speciﬁc dropout rates of each NN parameter conditioned on the observed context.
This paper also introduces a new type of variational prior for optimizing the conditional posterior in the amortized variational inference.
We have evaluated the proposed method compared with the existing approaches in various few-shot learning tasks, including 1D regression, image inpainting, and classiﬁcation tasks.
The experimental results demonstrate that NVDPs simultaneously improved the model’s adaptation to the context data, functional variability, and generalization to new tasks.
The proposed variational prior could also improve the variability of the representation-based posterior model (Garnelo et al., 2018b).
Adopting the advanced set representations of (Lee et al., 2019b; Kim et al., 2019; Volpp et al., 2020) or investigating more complicated architectures (Chen et al., 2019; Gordon et al., 2020; Foong et al., 2020) to NVDPs are interesting future works.
A CKNOWLEDGMENTS
This work was supported by Center for Applied Research in Artiﬁcial Intelligence(CARAI) grant funded by Defense Acquisition Program Administration(DAPA) and Agency for Defense Development(ADD)
(UD190031RD) and Basic Science Research Program through National Research Foundation of Korea (NRF) funded by the Korea government (MSIT) (NRF-2020R1A2B5B03095585).
Gunhee Kim is the corresponding author.
Insu Jeon also thanks Jaesik Yoon, Minui Hong and
Kangmo Kim for helpful comments.
A D ERIVATION OF THE ELBO OF NVDP S
This section describes a detailed derivation of the evidence lower-bound (ELBO) of NVDPs in equation 5.
Given the context data D tC
= ( x tC , y tC ) and target data D t = ( x t , y t ) , the KL divergence between the true unknown posterior distribution over parameter p ( φ t | x t , y t ) and the (conditional) variational posterior q ( φ t |D tC ) is given by:
[Equation]
The step (7) is due to Bayes rule of p ( φ t | x
[Equation]
, y
[Equation]
=
[Equation]
where φ t is often assumed to be independent of x t .
By reordering (8), we get
[Equation]
The lower-bound in (9) is due to the non-negativity of the omitted term (cid:80) T t =1 D KL ( q ( φ t |D tC )
|| p ( φ t |D t ))
In the lower-bound of (9), the choice of p ( φ t ) is often difﬁcult since the biased prior can lead to over-ﬁtting or under-ﬁtting of model.
However, some recent studies of the amortized VI inference (Tomczak & Welling, 2018; Takahashi et al., 2019) analytically discussed that the optimal prior in the amortized variational inference is the aggregated conditional posterior model on whole dataset: p ∗ ( φ ) = (cid:82) D q ( φ |D ) p ( D ) .
Usually the aggregated posterior cannot be calculated in a closed form due to the expensive computation cost of the integral.
However, the aggregation on whole dataset in the model-based conditional posterior is constructed based on the set representation.
This motivates us to deﬁne the conditional posterior given the whole task dataset as an empirical approximation of the optimal prior (i.e., p ∗ ( φ t )
≈ q ( φ t |D t ) ).
We call this a variational prior .
Thus, the approximation of p t ( φ ) in (9) with the variational prior yields the approximated lower bound of (10).
B D ERIVATION OF THE KL D IVERGENCE IN THE ELBO OF NVDP S
One important requirement in the choice of the prior is that the analytical derivation of the KL divergence term in equation (10) should not depend on the deterministic NN parameter θ (Kingma et al., 2015; Molchanov et al., 2017;
Hron et al., 2018; 2017; Liu et al., 2019).
This allows the constant optimization of the ELBO w.r.t all the independent variational parameters ( i.e . θ l,k,d and
P l,k,d for all l = 1 . . .
L , k = 1 . . .
K , and d = 1 . . .
D ).
In fact, the conditional posterior deﬁned in (2) is a Gaussian distribution, thus KL ( q ( φ t |D tC ; θ ) ||
q ( φ t |D t ; θ )) is analytically deﬁned as:
[Equation]
where both P and ˆP are the dropout rates predicted from the meta-model via the equation (3) but with different conditional set information: P is obtained from the small context set D tC , while ˆP is from the whole task set D t .
(11) is derived using the analytical formulation of KL divergence between two Gaussian distributions.
(13) is equivalent to the KL term deﬁned in (4) of the manuscript (except that the constant term 1/2 is omitted for brevity).
Interestingly, the analytical derivation of the KL is independent of the shared parameter θ , thus this satisﬁes the necessary condition to be used as a prior in the VI optimization of the dropout posterior.
The KL divergence in (13) intuitively means that the dropout rates predicted from a small context set should be close to the dropout rates predicted from a much larger context set while training.
The experiments validated that this surprisingly works well to induce a robust conditional functional uncertainty.
However, one practical issue while training the dropout rate with the KL term (13) is that the dropout rate can converge to zeros during the early training period due to the larger gradients from the KL than from the likelihood.
In practice, we adopt the dropout rate clipping technique often used in other Variational Dropout approaches (Kingma et al., 2015; Hron et al., 2018; Liu et al., 2019; Molchanov et al., 2017; Gal & Ghahramani, 2016; Liu et al., 2019).
We use the dropout rate range of (0.01, 0.99) for all experiments.
C A DDITIONAL E XPERIMENTAL D ETAILS ON THE 1D REGRESSION .
Setup.
We explored the 1D function regression on the data generated from the synthetic GPs with varying kernels 6 in the previous work (Garnelo et al., 2018b;a; Kim et al., 2019) that is suitable for measuring the uncertainty adaptation.
For the baseline models, Neural Process model (NP) and NP with additional deterministic representation (NP+CNP) described in (Garnelo et al., 2018b;a; Kim et al., 2019) is compared.
We also adopted the variational prior (VP) into the representation-based posterior of NP and its variants (NP+VP and NP+CNP+VP).
For all models (including our NVDP), we used the same ﬁxed variance and learned variance likelihood architecture depicted in (Kim et al., 2019): the agent NN with 4 hidden layers of 128 units with LeLU activation (Nair & Hinton, 2010) and an output layer of 1 unit for mean (or an additional 1 unit for variance).
The dimensions of the set representation r t were ﬁxed to 128 .
The meta NNs in the conditional dropout posterior in
NVDPs have 4 hidden layers of 128 units with LeakyReLU and an output layer of 257 units (i.e.
K + D + 1 ) for each layer of the agent model.
All models were trained with Adam optimizer (Kingma & Ba, 2015) with learning rate 5e-4 for 0 .
5 million iterations.
We draw 16 functions (a batch) from
GPs at each iteration.
Speciﬁcally, at every training step, we draw 16 functions f ( · ) from GP prior with the squared-exponential kernels, k
( x, x (cid:48) ) =
σ 2 f exp( − ( x − x (cid:48) ) 2 / 2 l 2 ) , generated with lengthscale l ∼ U (0 . 1 , 0 . 6) and function noise level σ f ∼ U (0 . 1 , 1 . 0) .
Then, x values was uniformly sampled from [ − 2 , 2] , and corresponding y value was determined by the randomly drawn function (i.e. y = f
( x ) , f ∼ GP ).
And the task data points are split into a disjoint sample of m contexts and n targets as m ∼ U (3 , 97) and n ∼ U [ m + 1 , 100) , respectively.
In the test or validation, the numbers of contexts and targets were chosen as m ∼ U (3 , 97) and n = 400 − m , respectively.
50000 functions were sampled from GPs to compute the log-likelihood (LL) and other scores for validation.
The architecture of NVDP model is as follows:
[Table]
( µ, σ ) (
[Table]
σ = 0 .
1 + 0 .
9 · softplus(logstd))
[Table]
where g ( l ) ( r ) is the meta NN for the l -th layer of the decoder
[Table]
and K ( l ) × D ( l ) is the number of parameters in the l -th layer
[Table]
D A DDITIONAL E XPERIMENTAL D ETAILS ON THE IMAGE COMPLETION TASK .
Setup.
The image completion tasks are performed to validate the performance of the models in more complex function spaces (Garnelo et al., 2018b;a;
Kim et al., 2019)
Here, we treat the image samples from MNIST (LeCun et al., 1998) and CelebA (Liu et al., 2015) as unknown functions.
The task is to predict a mapping from normalized 2D pixel coordinates x i ( ∈ [0 , 1] 2 ) to pixel intensities y i ( ∈ R 1 for greyscale, ∈ R 3 for RGB) given some context points.
For 2D regression experiments, we used the same learned variance baselines implemented in the GP data regression task, except the input and output of the decoder are changed according to dataset, e.g., x i ∈ [0 , 1] 2 , and y i ∈
Rfor MNIST (or ∈ R 3 and r t = 1024 for CelebA).
At each iteration, the images in the training set are split into S context and N target points (e.g., S ∼ U (3 , 197) , n ∼ U [ N + 1 , 200) at train and S ∼ U (3 , 197) , N = 784 − S at validation).
Adam optimizer with a learning rate 4e-4 and 16 batches with 300 epochs were used for training.
The validation is performed on the separated validation images set.
To see the generalization performance on a completely new dataset, we also tested the models trained on MNIST to the Omniglot validation set.
E A DDITIONAL EXPERIMENTS ON 1D FUNCTION REGRESSION ON A TOY TRIGONOMETRY DATASET
Setup.
To further investigate how the proposed meta-model utilizes the common structures of the NN parameters, we trained a small NVDP model (of the size of 13-12-12-2) on a mixture of scaled trigonometric functions: x is sampled in a range of [ − π, π ] , and y is determined by y
= a ∗ f (2 ∗ x − b ∗ π ) where f is one of three functions sine, cosine, and tanh with the probability of one third, and a ∼ U (1 . 5 , 2) and b ∼ U ( − 0 .
We used the training procedures in the next section except the the simple model architecture and learning rate 5 e − 4 .
For the small NVDP model on 1D function regression with Trigonometry dataset, the following architecture was employed:
[Equation]
Deterministic)
[Equation]
Encoder
[Equation] [Equation]
Sample Sample
[Figure]
[Equation]
Sample Sample
[Equation]
where g ( l ( r ) is the meta NN for the l -th layer of the decoder, and K ( l ) × D ( l ) is the number of parameters in the l -th layer.
[Table] [Table]
S
[Table]
l e
[Table]
m p l e
C o n t e x t
NP
NP+VP
NP+CNP
NP+CNP+VP
NVDP
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
[Table]
MNIST
Results.
Figure 11 displays the trained NVDP on the trigonometry function dataset.
The NVDP could capture the probability of dropout for each parameter θ of the agent, successfully predicting the task-speciﬁc trigonometric functions conditioned on the small ( S = 4 ) context set.
This shows that the task-speciﬁc dropout rates can transform a single conventional NN agent to express multiple functions.
It is interesting to see that the contexts from sine and cosine functions yield similar dropout rates for the second layer.
On the other hand, the contexts from tanh function results in different dropout structures for all layers.
[Figure]
half
[Figure]
S a m p l e 2 S a m p l e
C o n t e x t
NP
NP+VP
NP+CNP
NP+CNP+VP
NVDP
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
[Figure]
MNIST (train) to Omniglot (test)
CelebA
[Figure]
S a m p l e 2 S a m p l e
C o n t e x t
NP
NP+VP
NP+CNP
NP+CNP+VP
NVDP
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
[Figure]
[Figure]
NP
NP+VP
NP+CNP
NP+CNP+VP
NVDP
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
S a m p l e 2 S a m p l e S a m p l e 2 S a m p l e
[Figure] [Figure]