E MPIRICAL B AYES T RANSDUCTIVE M ETA -L EARNING WITH S YNTHETIC G RADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging unlabeled information in the query set to learn a more powerful meta-model.
To develop our framework we revisit the empirical Bayes formulation for multi-task learning.
The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior of each task.
We derive a novel amortized variational inference that couples all the variational posteriors into a meta-model, which consists of a synthetic gradient network and an initialization network.
The combination of local KL divergences and synthetic gradient network allows for backpropagating information from unlabeled data, thereby enabling transduction.
Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classiﬁcation signiﬁcantly outperform previous state-of-the-art methods.
I NTRODUCTION
While supervised learning of deep neural networks can achieve or even surpass human-level performance (He et al., 2015; Devlin et al., 2018), they can hardly extrapolate the learned knowledge beyond the domain where the supervision is provided.
The problem of solving rapidly a new task after learning several other similar tasks is called meta-learning (Schmidhuber, 1987; Bengio et al., 1991; Thrun & Pratt, 1998); typically, the data is presented in a two-level hierarchy such that each data point at the higher level is itself a dataset associated with a task, and the goal is to learn a meta-model that generalizes across tasks.
In this paper, we focus on few-shot learning (Vinyals et al., 2016), an instance of meta-learning problems, where a task t , in meta-testing , consists of an unlabeled set x t :=
{ x t,i } ni =1 and a labeled set (aka support set )
d lt := { ( x lt,i , y lt,i ) } n l i =1 , and the goal is to predict the corresponding labels, namely y t
= { y t,i } ni =1 , for the unlabeled set.
In meta-training , y t is provided as ground truth.
The set d t :=
( x t , y t ) is sometimes referred to as query set .
A particular important distinction to make is whether each task is solved in a transductive or inductive manner.
The inductive setting is what was originally proposed by Vinyals et al.
(2016): we use d lt to train a model and test it on x t (one example at a time).
Transduction, however, has the advantage of being able to see all points in x t before making predictions.
We argue that the transductive setting is more relevant to the problem since, as in semi-supervised learning, an inductive learner can always be built from a transductive one (Chapelle et al., 2006).
In fact, Nichol et al.
(2018) notice that most of the existing meta-learning methods follow the transductive setting unintentionally since they use x t implicitly via the batch normalization (Ioffe & Szegedy, 2015).
Due to the hierarchical structure of the data, it is natural to formulate meta-learning as an instance of hierarchical Bayes (HB) (Good, 1980; Berger, 1985), or alternatively, empirical Bayes (EB) (Robbins, 1985; Kucukelbir & Blei, 2014).
The difference is that the latter restricts the learning of meta-parameters to point estimates.
In this paper, we focus on the EB model, since it largely simpliﬁes the training and testing without losing the strength of the HB formulation.
The idea of using HB or EB for meta-learning is not new: Amit & Meir (2018) derive an objective similar to that of HB using PAC-Bayesian analysis; Grant et al.
(2018) show that MAML (Finn et al., 2017) can be understood as a EB method; Ravi & Beatson (2018) consider a HB extension to MAML and compute posteriors via amortized variational inference.
However, unlike our proposal, these
[Figure]
methods do not take advantage of the unlabeled set.
Roughly speaking, they construct the variational posterior as a function of the labeled set d lt without taking advantage of the unlabeled set x t .
The situation is similar in gradient based meta-learning methods (Finn et al., 2017; Ravi & Larochelle, 2016;
Li et al., 2017; Nichol et al., 2018; Flennerhag et al., 2019) and many other meta-learning methods (Vinyals et al., 2016; Snell et al., 2017; Gidaris & Komodakis, 2018), where the mechanisms used to generate the task-speciﬁc parameters rely on groundtruth labels, thus, there is no place for the unlabeled set to contribute.
We argue that this is a suboptimal choice, which may lead to overﬁtting when the labeled set is small and hinder the possibility of zero-shot learning (when the labeled set is not provided).
An exception is Liu et al.
They reuse the label propagation algorithm
(Zhu et al., 2003) for transductive inference within each task and show that transduction is useful for boosting the performance.
In this paper, we propose to use synthetic gradient (Jaderberg et al., 2017) to enable transductivity, such that the variational posterior is implemented as a function of the labeled set d lt and the unlabeled set x t .
The synthetic gradient is produced by a neural network and learned to be a surrogate of the true gradient.
The optimization process is similar to the inner gradient descent in MAML, but it iterates on the unlabeled set x t rather than on labeled set d lt , since it does not rely on y t to compute the true gradient.
The labeled set for an unseen task is now optional, which is only used to generate the initialization in our case.
In summary, our main contributions are the following:
In section 2 and section 3, we develop a novel empirical Bayes formulation with transduction for meta-learning.
To perform amortized variational inference, we propose a parameterization for the variational posterior based on synthetic gradient descent, which incoporates the contextual information from all the inputs of the query set.
In section 4, we show in theory that a transductive variational posterior yields better generalization performance.
Besides, we show that the proposed empirical Bayes formulation is equivalent to the information bottleneck principle considered by Achille & Soatto (2017).
We thus call our method synthetic information bottleneck (SIB).
In section 6, we verify our proposal empirically.
Our experimental results demonstrate that our method signiﬁcantly outperforms the state-of-the-art meta-learning methods on standard few-shot classiﬁcation benchmarks.
M ETA - LEARNING WITH TRANSDUCTIVE INFERENCE
The goal of meta-learning is to train a meta-model on a collection of tasks, such that it works well on another disjoint collection of tasks.
Suppose that we are given a collection of N tasks for training.
The associated data is denoted by D :=
{ d t = ( x t , y t ) }
Nt =1 .
In the case of few-shot learning, we are given in addition a support set d lt for each task.
In this section, we revisit the classical empirical
Bayes model for meta-learning.
Then, we propose to use a transductive scheme in the variational inference by constructing the variational posterior as a function of x t .
2. E MPIRICAL B AYES MODEL
Due to the hierarchical structure among data, it is natural to consider a hierarchical Bayes model for the marginal likelihood
[Equation]
The generative process is illustrated in Figure 1 (left, in solid arrows): ﬁrst, a meta-parameter ψ is sampled from the hyper-prior p ( ψ ) ; then, for each task, a task-speciﬁc parameter w t is sampled from the prior p ( w t
| ψ ) ; ﬁnally, the dataset is drawn from the likelihood p ( d t | w t ) 1 .
In particular, since different tasks may require different losses, we assume the log-likelihood takes a general form:
[Equation]
where (cid:96) t denotes the task-speciﬁc loss , e.g., the cross entropy loss.
The ﬁrst argument in (cid:96) t is the prediction, denoted by ˆ y t,i , for the i -th example, which takes as input the feature representation f
( x t,i ) and the task-speciﬁc weight w t .
Rather than following a fully Bayesian approach, we leave some random variables to be estimated by a frequentist approach, e.g., f is a part of the likelihood model for which we use a point estimate.
As such, the posterior inference about these variables will be largely simpliﬁed.
For the same reason, we derive the empirical Bayes (Robbins, 1985; Kucukelbir & Blei, 2014), which interprets ψ in a frequentist way:
[Equation]
The overall model formulation is the same as the ones considered by Amit & Meir (2018); Grant et al.
; Ravi & Beatson (2018).
2. A MORTIZED INFERENCE WITH TRANSDUCTION
Focusing on the empirical Bayes model (3), we derive an evidence lower bound (ELBO) on the log-likelihood by introducing a variational distribution q θ t ( w t ) for each task with parameter θ t :
[Equation]
Maximizing the ELBO in (4) with respect to θ 1 , . .
, θ N and ψ is equivalent to
[Equation]
However, the optimization in (5), as N increases, becomes more and more expensive in terms of the memory footprint and the computational cost.
We therefore wish to bypass this heavy optimization and to take advantage of the fact that individual KL terms indeed share the same structure.
To this end, instead of introducing N different variational distributions, we consider a commonly parameterized family of distributions, which is deﬁned implicitly by a deep neural network φ taking as input x t .
Note that we do not include y t as an input because it is not available during meta-testing.
Replacing each q θ t by q φ ( x t ) , (5) can be written as
[Equation]
[Equation]
which is also known as amortized variational inference in the literature (Kingma & Welling, 2013;
Rezende et al., 2014).
Note that this inference scheme is transductive since for testing each point in x t we will use the entire x t due to the variational posterior q φ ( x t ) .
Alternatively, we can derive an inductive inference scheme by using the support set d l t to construct a variational posterior q φ ( d lt ) , since d lt and x t are disjoint.
As an example, MAML (Finn et al., 2017) is an inductive method, where φ ( d lt ) is realized as θ Kt , the K -th iterate of the stochastic gradient descent
[Equation]
In fact, nothing prevents us to come up with an even better variational posterior q φ ( x t ,d lt ) , shown in dashed arrows in Figure 1 (a), which is again transductive by deﬁnition.
In a nutshell, the meta-model includes f, ψ from empirical Bayes and the amortization φ for inference.
To obtain a closed-form KL term in (6), we restrict ourselves to Gaussian models 2 , such that both q φ ( x t ) and p ψ are Gaussian distributions with diagonal covariance.
V ARIATIONAL INFERENCE WITH SYNTHETIC GRADIENTS
It is however non-trivial to design a network architecture to implement the amortization φ ( x t ) directly since x t is itself a dataset.
The strategy adopted by neural processes (Garnelo et al., 2018) is to aggregate the information from all individual examples via a permutation invariant function.
However, as pointed out by Kim et al.
(2019), such a strategy tends to underﬁt x t because the aggregation does not necessarily attain the most relevant information for identifying the task-speciﬁc parameter.
We instead design a neural network φ ( x t ) to parameterize the optimization process of θ t .
More speciﬁcally, consider a stochastic gradient descent on θ t for optimizing (5) with step size η :
[Equation]
We would like to parameterize this optimization dynamics up to the K -th step via φ ( x t ) , such that q θ Kt is a good approximation of the optimum q θ (cid:63)t .
It consists of parameterizing (a the initialization θ 0 t and (b) the gradient ∇ θ t D KL ( q θ t (cid:107) p f · p ψ ) .
By doing so, θ Kt becomes a function of φ , ψ and x t 3 , we therefore realize q φ ( x t ) as q θ Kt .
For (a), we opt to either let θ 0 t = λ to be a global data-independent initialization as in MAML (Finn et al., 2017) or let θ 0 t = λ
( d lt ) with a few supervisions from the support set, where λ can be implemented by a permutation invariant network described in Gidaris & Komodakis (2018).
In the second case, the features of the support set will be ﬁrst averaged in terms of their labels and then scaled by a learned vector of the same size.
For (b), the fundamental reason that we parameterize the gradient is because we do not have access to y t during the meta-testing phase.
Note that we are able to follow (8) in meta-training to obtain q θ (cid:63)t ( w t )
∝ p f
( d t | w t ) p ψ ( w t ) .
To make a consistent parameterization in both meta-training and meta-testing, we thus discard y t when constructing the variational posterior.
Regarding the true gradient, a key observation is that, under a reparameterization w t = w t ( θ t , (cid:15) ) with (cid:15) ∼ p ( (cid:15) ) , ∇ θ t D KL (cid:16)
[Equation]
(cid:107) p
[Equation]
p
[Equation]
(cid:17)
[Equation]
where all the terms can be computed without y t except for ∂(cid:96) t
∂ ˆ y t,i , thus, we introduce a deep neural network ξ (ˆ y t,i ) to synthesize it.
The idea of synthetic gradients was originally proposed by Jaderberg et al.
(2017) to parallelize the back-propagation.
Here, the purpose of ξ (ˆ y t,i ) is to update θ t regardless of the groundtruth labels, which is slightly different from its original purpose.
Besides, we do not introduce an additional loss to force ξ (ˆ y t,i ) to approximate ∂(cid:96) t
∂ ˆ y t,i since ξ (ˆ y t,i ) will be learned to yield a reasonable θ Kt even without mimicking the true gradient.
To sum up, we have derived a particular implementation of φ ( x t ) by parameterizing the ideal meanﬁeld update, namely (8), on the query set d t , such that the meta-model φ includes an initialization network λ and a synthetic gradient network ξ .
Speciﬁcally, we have φ ( x t )
=
θ
Kt , the K -th iterate of the following update:
[Equation]
The overall algorithm is depicted in Algorithm 1.
A comparison with MAML is shown in Figure 1.
Rather than viewing (9) as an optimization process, it may be more precise to think of it as a part of the computation graph created in the forward-propagation.
As an extension, if we were deciding to estimate the feature network f in a Bayesian manner, we would have to compute the gradient of gradient wrt f in the case of MAML.
This is super costly from a computational point of view and needs technical simpliﬁcations (Nichol et al., 2018).
By introducing a series of synthetic gradient networks in a way similar to Jaderberg et al.
(2017), the computation will be decoupled into computations within each layer, and thus becomes more feasible.
T HEORETICAL ANALYSIS
In this section, we study the generalization ability of the empirical Bayes model.
Empirical Bayes induces a simpler graphical model compared to hierarchical Bayes due to the conditional independence: w t ⊥⊥ D \ d t | d t (see Appendix A for a full discussion).
In this case, rather than viewing w 1 , . . .
, w N as N different random variables, we may view { ( w t , d t ) } Nt =1 as iid samples drawn from q ( w, d ) =
q ( d ) q ( w | d ) , where q ( d ) is the underlying data distribution and q ( w | d ) =
q φ ( d ) ( w ) , the ideal variational posterior.
The data distribution q ( d ) is in fact a marginal distribution computed by q ( d )
=
(cid:82) t q ( t ) q ( d | t ) , where t is the random variable representing the task.
To simplify the analysis, we do not distinguish samples (i.e. d ) drawn from different tasks.
We may call q ( w | d ) pseudo posterior since as the family induced by φ becomes larger it would have to approximate the true posterior.
Correspondingly, we have q ( d | w ) =
q ( w | d ) q ( d ) q ( w ) , the pseudo likelihood , where q ( w ) =
E q ( d ) q ( w | d ) is sometimes referred to as aggregated posterior
(Makhzani et al., 2015;
Tomczak & Welling, 2017).
Before going into details, we ﬁrst introduce a few notations: the entropy is deﬁned as H p ( x ) :=
E p ( x ) [ − log p ( x )] ; the mutual information is given by I p ( x ; y ) :
= D KL (cid:0) p ( x , y ) (cid:107) p ( x ) p
( y ) (cid:1)
; the cross entropy is deﬁned as H p,q ( x ) :=
E p ( x ) [ − log q ( x )] .
To quantify the generalization ability of the meta-model is equivalent to quantify the generalization of the resulting pseudo posterior.
To this end, we ﬁrst identify the empirical risk for a single task as
L ( w, d ) := 1 n (cid:80) ni =1 (cid:96) t
(ˆ y i ( f ( x i ) , w ) , y i ) .
Then, the average empirical risk is the expectation over all possible samples and weights, which depends on the pseudo posterior and the data distribution:
[Equation] [Equation]
The true risk should be independent of any particular pseudo posterior, where we sample the taskspeciﬁc weight from the aggregated posterior:
[Equation]
Finally, the generalization error is deﬁned as
[Equation]
Intuitively, the generalization error measures how much the empirical risk concentrated on the true risk is.
That is, we would like to bound the tail of the distribution over all possible values of the empirical risk.
By taking an expectation of the objective in (6) with respect to the choice of D , we have E
[Equation]
q
[Equation]
w | d )
[Equation]
p ( d | w ) p ( w )) .
We show in Proposition 1 that the quantity on the right hand side directly affects the generalization performance.
Theorem 1.
Given distributions q ( w | d ) , q ( d ) , p ( w ) and p ( d | w ) , if (cid:96) t (ˆ y ( w ) , y ) is σ -subgaussian for all w , the following inequalities hold:
[Equation]
The equality in (13) holds
if q ( w ) = p ( w ) .
The equality in (14) holds
if q ( d ) = p ( d ) .
The proof of Theorem 1 can be found in Appendix B.
The inequality (13) basically says that (6) can be seen a regularized empirical risk minimization in which the regularization term is an upper bound of the mutual information between the weight and the sample.
In general, there is a tradeoff between the generalization error and the empirical risk controlled by the coefﬁcient n 2 σ 2 .
If n is small, then we are in the overﬁtting regime.
This is the case of many existing gradient-based meta-learning methods, where they basically implement the pseudo posterior q ( w | d ) by q φ ( d l ) ( w ) .
Recall that d l is the support set which is fairly small by the deﬁnition of few-shot learning.
Consequently, we expect these methods to have large generalization errors in light of the above analysis.
On the other hand, if we were following the transductive setting, the sample size n would be larger since we implement the pseudo posterior as q φ ( x ) ( w ) or q φ ( x,d l ) ( w ) .
An obvious message from the inequality (14) is that, if we choose appropriate likelihood, prior and pseudo posterior such that p ( w, d ) is aligned with q ( w, d ) , the empirical Bayes model can actually be pretty good, that is, both the generalization error and the empirical risk are close to zero.
If we look at the message more carefully, it also implies that a good likelihood is the key to represent the data well; the inference model is less important as long as the aggregated posterior coincides with the prior.
Besides, (13) reveals a connection between empirical Bayes and information bottleneck (Tishby et al., 2000; Achille & Soatto, 2017).
The right hand side of (13) is exactly the IB objective considered by
Achille & Soatto (2017) with a coefﬁcient equal to 1 .
We ﬁnd this connection interesting, thus, we call our method synthetic information bottleneck (SIB).
E XPERIMENTS : ZERO - SHOT REGRESSION ON SPINNING LINES
To verify the proposed algorithm, we ﬁrst look at a toy multi-task problem, where we have the full control of the data generation.
It is called zero-shot since there is no labeled set or support set attached for each task during meta-testing.
Denote by D train := { d t } Nt =1 the train set, which consists of datasets of size n : d = { ( x i , y i ) } ni =1 .
We construct a dataset d by ﬁrstly sampling iid Gaussian random variables as inputs: x i ∼ N ( µ, σ 2 ) .
Then, we generate the weight for each dataset by calculating the mean of the inputs and shifting with a Gaussian random variable (cid:15) w : w = 1 n (cid:80) i x i + (cid:15) w , (cid:15) w ∼ N ( µ w , σ 2 w ) .
The output for x
i is y i =
w · x i .
We decide ahead of time the hyperparameters µ, σ, µ w , σ w for generating x i and y i .
Recall that a weighted sum of iid Gaussian random variables is still a Gaussian random variable.
Speciﬁcally, if w = (cid:80) i c i x i and x i ∼ N ( µ i , σ 2 i ) , then w ∼ N ( (cid:80)
i c i µ i , (cid:80) i c
2 i σ 2 i ) .
Therefore, we have p ( w ) =
N ( µ + µ w , 1 n σ 2 + σ 2 w ) .
On the other hand, if we are given a dataset d of size n , the only uncertainty about w comes from (cid:15) w , that is, we should consider x i as a constant given d .
Therefore, the posterior p ( w | d ) = N ( 1 n (cid:80) ni
=1 x i + µ w , σ 2 w ) .
Although the data generation is relatively simple, the problem is difﬁcult from a regression point of view.
Note that, for each x , there are potentially multiple corresponding y ’s.
In other words, we are solving a one-to-many mapping, which is not a function by deﬁnition.
We use a simple implementation of SIB to solve this problem.
For a task t , the loss (cid:96) t is the mean squared error, and the variational posterior is realized by
[Equation]
The synthetic gradient network ξ ( y ) is implemented by a three-layer MLP with hidden size 8 .
In addition, we let p ψ ( w ) to be a Gaussian.
Thus, we have three sets of parameters in total: ( λ, ξ, ψ ) .
As a comparsion, we implement a Bayesian neural network (BNN) (Blundell et al., 2015) with diagonal
Gaussian variational posterior as the baseline, which has exactly twice the number of weights as ξ (since it has the mean and the standard deviation).
In the experiment, we sample 240 tasks respectively for both D train and D test .
We learn SIB and BNN on D train for 150 epochs using the ADAM optimizer (Kingma & Ba, 2014), with learning rate set to 10 − 3 and batch size to 8 .
Other hyperparameters are set as follows: n = 32 , K = 3 , µ = 0 , σ = 1 , µ w = 1 , σ w = 0 .
The results are shown in Figure 2.
It is clear that both D KL ( q θ Kt ( w ) (cid:107) p ( w | d )) and D KL ( p ψ ( w ) (cid:107) p ( w )) are close to zero indicating the success of the learning.
In contrast to SIB, the MLP baseline tends to predict the mean value for each x .
The learned λ = 0 .
9771 captures the mean value as well (since initially y = λx ), but the dynamics in equation (15) are able to adaptively reﬁne the weights, overall learned in an end-to-end fashion.
It is interesting to see how θ kt evolves gradually towards the ground truth by varying K (Figure 2 middle).
[Figure]
.−
1 .− 1 .− 0 .0 .0 .1 .1 .2 .y
SIB dynamics
GTinitk=1k=2k= k= − 2 .
− 1 .
5 − 1 . 0 − 0 .
x − 2 .− 1 .− 1 .− 0 .0 .0 .1 .1 .2 .y
Prediction comparison
GTBaselineSIBGTBaselineSIBGTBaselineSIB
E XPERIMENTS : FEW - SHOT CLASSIFICATION
In this section we present our experimental results on MiniImageNet (Vinyals et al., 2016) and
CIFAR-FS (Bertinetto et al., 2018)
6. S ETUP
Datasets We conduct experiments for few-shot classiﬁcation on two datasets.
Each dataset is composed of disjoint training, validation and testing categories.
MiniImageNet is proposed by
Vinyals et al.
(2016), which contains 100 categories, splitted into 64 training classes, 16 validation classes and 20 testing classes, where each category consists of 600 image-label pairs and each image is of size 84 × 84.
CIFAR-FS is proposed by Bertinetto et al.
(2018), which is created by dividing the original CIFAR-100 into 64 training classes, 16 validation classes and 20 testing classes; each image is of size 32 × 32.
Network architectures Following Gidaris & Komodakis (2018); Qiao et al.
; Gidaris et al.
(2019), we implement f by a 4-layer convolutional network (Conv-4-64 or Conv-4-128 4 ) or a
WideResNet (WRN-28-10) (Zagoruyko & Komodakis, 2016).
We pretrain the feature network f ( · ) on the 64 training categories for a stardard 64 -way classiﬁcation.
We reuse the feature averaging network proposed by Gidaris & Komodakis (2018) as our initialization network λ ( · ) , which basically averages the feature vectors of all data points from the same category and then scale each feature dimension differently by a learned coefﬁcient.
For the synthetic gradient network ξ ( · ) , we implement a three-layer MLP with hidden-layer size 8 k .
Finally, for the predictor ˆ y ij ( · , w i ) , we adopt the cosine-similarity based classiﬁer advocated by Chen et al.
and Gidaris & Komodakis (2018).
Evaluation metrics
In few-shot classiﬁcation, a task (aka episode) t consists of a query set d t and a support set d lt .
When we say the task t is k -way- n l -shot we mean that d lt is formed by ﬁrst sampling k categories from a pool of categories; then, for each sampled category, n l examples are drawn and a new label taken from { 0 , . .
, k − 1 } is assigned to these examples.
By default, each query set contains 15 k examples.
The goal of this problem is to predict the labels of the query set, which are provided as ground truth during training.
The evaluation is the average classiﬁcation accuracy on the tasks created with testing categories.
We sample 2000 and 5000 testing tasks respectively for
MiniImageNet and CIFAR-FS following Gidaris et al.
Training details We use ADAM with batch size 8 for 60 epochs, where the learning rate is 10 −and we may optionally dropp the learning rate by a factor 0 .
1 at epoch 3, 10, 25, 50.
During training, we freeze the feature network.
To select the best hyper-parameters, for each dataset, we sampletasks from the validation categories and reuse them at each training epoch.
We use these validation tasks to select the best meta-model and then use it for the ﬁnal evaluation on the tasks sampled from testing categories.
6. C OMPARISON TO STATE - OF - THE - ART META - LEARNING METHODS
In Table 1, we show a comparison between the state-of-the-art approaches and several variants of our method (varying K or f ( · ) ).
First of all, comparing SIB ( K = 3 ) to SIB ( K = 0 ), we observe a clear improvement, which suggests that, by taking a few synthetic gradient steps, we do obtain a better variational posterior as promised.
For 1-shot learning, SIB (when K = 3 or K = 5 ) signiﬁcantly outperforms previous methods on both MiniImageNet and CIFAR-FS.
For 5-shot learning, the results are are also comparable.
It should be noted that the performance boost is consistently observed with different feature networks, which suggests that SIB is a general method for few-shot learning.
However, we also observe a potential limitation of SIB: when the support set is relatively large, e.g., 5-shot, with a good feature network like WRN-28-10, the initialization θ 0 t can already be close to a local minima, then if we have relatively large learning rate for the synthetic gradient descent, the result of θ Kt may yield no improvement over that of θ 0 t .
[Table]
C ONCLUSION
We have presented an empirical Bayesian framework for meta-learning.
To enable an efﬁcient variational inference, we followed the amortized inference paradigm [Table] and proposed to use a transductive scheme for constructing the variational posterior.
To implement the transductive inference, we make use of two neural networks: a synthetic gradient network and an initialization network, which together enables a synthetic gradient descent on the unlabeled data to generate the parameters of the amortized variational posterior dynamically.
We have studied the theoretical properties of the proposed framework and shown that it yields signiﬁcant performance boost on MiniImageNet and
CIFAR-FS for few-shot classiﬁcation
A PPENDIX A C ONDITIONAL INDEPENDENCE IMPLIED BY EB
In the case of the hierarchical Bayes formulation note that the posterior over the parameter w t for a given task, namely, p ( w t | d t ) , is different from its global counterpart, namely, p ( w t |D ) .
Indeed, p ( w t |D ) is coupled by
[Equation]
[Equation]
Hence the global posterior takes a form of
[Equation]
On the other hand, the local posterior is given by
[Equation]
which is not equal to p ( w t |D ) apparently.
On the other hand, in the empirical Bayes approach, learning is only coupled via the estimation of ψ by the type-II likelihood, because for a ﬁxed choice of ψ , we have
[Equation]
which makes the global and local posteriors coincide with each other:
[Equation]
B P ROOF OF T HEOREM
Before going into details, we ﬁrst introduce a few notations, deﬁnitions
[Equation]
and a technical lemma.
• The entropy is deﬁned as H p ( x ) :
= E p ( x ) [ − log p ( x )] ; • The mutual information is given by I p ( x ; y ) :
= D KL (cid:0) p ( x , y ) (cid:107) p ( x ) p
( y ) (cid:1) .
• Very often, we decompose the mutual information as I p ( x ; y )
=
H p ( x ) − H p ( x | y ) with H p ( x | y ) :=
E p ( x , y ) [ − log p ( x | y )] the conditional entropy .
• The cross entropy is deﬁned as H p,q ( x ) :=
E p ( x ) [ − log q ( x )] .
Similarly, we have the cross conditional entropy H p,q ( x | y ) :
= E p ( x , y ) [ − log q ( x | y )] and the cross mutual information I p,q ( x ; y ) =
H p,q ( x ) − H p,q ( x | y ) .
Deﬁnition 1.
A random variable x is subgaussian if there exists a positive number σ such that E [exp( λ ( x − E x ))]
≤ exp( σ 2 λ 2 / 2) for all λ ∈ R .
Lemma 1 (Xu & Raginsky (2017)) .
If g( x , y ) is σ -subgaussian, then
[Equation]
Now, we are ready to prove Theorem 1.
Proof.
The expected KL divergence can be rewritten as follows:
[Equation]
Note that I q,p ( d ; w ) =
I q,p ( w ; d ) and D KL (cid:0) q ( w ) (cid:13)(cid:13) p ( w ) (cid:1) =
iff q ( w ) =
E q ( d ) q
[Equation]
|
[Equation]
=
p ( w ) .
Similarly, we have
[Equation] [Equation]
by removing the term H q,p ( d ) .
Let us rewrite (21) as
[Equation]
which attains the right hand side of the second inequality.
Recall that L ( w, d ) :
= 1 n (cid:80) ni =1 (cid:96) t (ˆ y i
( f ( x i ) , w ) , y i ) .
If (cid:96) t (ˆ y i ( f ( x i ) , w ) , y i ) is σ -subgaussian for all w , then L ( w, d ) is σ/ √ n -subgaussian due to the iid assumption on d .
Thus, by Lemma 1, we have (cid:12)(cid:12) gen ( q ( w | d ) , q ( d ))
(cid:12)(cid:12) ≤ (cid:114)
σn I q ( w ; d ) .
On the other hand, H q,p ( d | w ) = ˆ R ( q ( w | d ) , q ( d )) .
Combining both, we have n
[Equation]
as desired.
Recall that information bottleneck (IB) involves an optimization
[Equation]
Thus, we have established a connection between local empirical Bayes and information bottleneck.
The idea of IB is to view w as a compressed representation of d from a rate-distortion perspective :
• I q ( w ; d ) is known as rate , which is a regularization term discouraging memorization; • − I q,p ( w ; d ) =
H q,p ( d | w ) − H q,p ( d ) is naturally a measure of distortion since H q,p ( d | w ) is equal to the expected negative log-likelihood and H q,p ( d ) is a constant wrt q ( w | d ) .
Thus, solving (6) amounts to minimizing an upper bound of the IB objective if we generalize the posterior as follows:
[Equation]
C I MPORTANCE OF SYNTHETIC GRADIENTS
To further verify the effectiveness of the synthetic gradient descent, we implement an inductive version of SIB inspired by MAML, where the initialization θ 0 t is generated exactly the same way as
SIB using λ ( d lt ) , but it then follows the iterations in (7) as in MAML rather than follows the iterations in (9) as in standard SIB.
We conduct an experiment on CIFAR-FS using Conv-4-64 feature network.
The results are shown in
Table 2.
It can be seen that there is no improvement over SIB ( K = 0 ) suggesting that the inductive approach is insufﬁcient.
[Table]
Training on
[Table]
shot
Training on
[Table]
Training on 5-shot
Testing on
Testing on Testing on
[Table]
shot
[Table]
shot
[Table]