B AYESIAN F EW -S HOT C LASSIFICATION WITH
O NE - VS -E ACH
P ´ OLYA -G AMMA
A UGMENTED
G AUSSIAN
P
ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot classiﬁcation (FSC), the task of adapting a classiﬁer to unseen classes given a small labeled dataset, is an important step on the path toward human-like machine learning.
Bayesian methods are well-suited to tackling the fundamental issue of overﬁtting in the few-shot scenario because they allow practitioners to specify prior beliefs and update those beliefs in light of observed data.
Contemporary approaches to Bayesian few-shot classiﬁcation maintain a posterior distribution over model parameters, which is slow and requires storage that scales with model size.
Instead, we propose a Gaussian process classiﬁer based on a novel combination of P´olya-gamma augmentation and the one-vs-each softmax approximation (Titsias, 2016) that allows us to efﬁciently marginalize over functions rather than model parameters.
We demonstrate improved accuracy and uncertainty quantiﬁcation on both standard few-shot classiﬁcation benchmarks and few-shot domain transfer tasks.
I NTRODUCTION
Few-shot classiﬁcation (FSC) is a rapidly growing area of machine learning that seeks to build classiﬁers able to adapt to novel classes given only a few labeled examples.
It is an important step towards machine learning systems that can successfully handle challenging situations such as personalization, rare classes, and time-varying distribution shift.
The shortage of labeled data in
FSC leads to uncertainty over the parameters of the model, known as model uncertainty or epistemic uncertainty .
If model uncertainty is not handled properly in the few-shot setting, there is a signiﬁcant risk of overﬁtting.
In addition, FSC is increasingly being used for risk-averse applications such as medical diagnosis (Prabhu, 2019) and human-computer interfaces (Wang et al., 2019) where it is important for a few-shot classiﬁer to know when it is uncertain.
Bayesian methods maintain a distribution over model parameters and thus provide a natural framework for capturing this inherent model uncertainty.
In a Bayesian approach, a prior distribution is ﬁrst placed over the parameters of a model.
After data is observed, the posterior distribution over parameters is computed using Bayesian inference.
This elegant treatment of model uncertainty has led to a surge of interest in Bayesian approaches to FSC that infer a posterior distribution over the weights of a neural network (Finn et al., 2018; Yoon et al., 2018; Ravi & Beatson, 2019).
Although conceptually appealing, there are several practical obstacles to applying Bayesian inference directly to the weights of a neural network.
Bayesian neural networks (BNNs) are expensive from both a computational and memory perspective.
Moreover, specifying meaningful priors in parameter space is known to be difﬁcult due to the complex relationship between weights and network outputs (Sun et al., 2019).
Gaussian processes (GPs) instead maintain a distribution over functions rather than model parameters.
The prior is directly speciﬁed by a mean and covariance function, which may be parameterized by deep neural networks.
When used with Gaussian likelihoods, GPs admit closed form expressions for the posterior and predictive distributions.
They exchange the computational drawbacks of BNNs for cubic scaling with the number of examples.
In FSC, where the number of examples is small, this is often an acceptable trade-off.
When applying GPs to classiﬁcation with a softmax likelihood, the non-conjugacy of the GP prior renders posterior inference intractable.
Many approximate inference methods have been proposed to circumvent this, including variational inference and expectation propagation.
In this paper we investigate a particularly promising class of approaches that augments the GP model with a set of auxiliary random variables, such that when the auxiliary variables are marginalized out the original model is recovered (Albert & Chib, 1993; Girolami & Rogers, 2006; Linderman et al., 2015).
Such augmentation-based approaches typically admit efﬁcient Gibbs-sampling procedures for generating posterior samples which when combined with Fisher’s identity (Douc et al., 2014) can be used to optimize the parameters of the mean and covariance functions.
In particular, augmentation with P´olya-gamma random variables (Polson et al., 2013) makes inference tractable in logistic models.
Naively, this is useful for handling binary classiﬁcation, but in this paper we show how to extend this augmentation to classiﬁcation with multiple classes by using the one-vs-each softmax approximation (Titsias, 2016), which can be expressed as a product of logistic sigmoids.
In this work, we make several contributions:
• We propose a novel GP classiﬁcation method that combines the one-vs-each softmax approximation (Titsias, 2016) with P´olya-gamma augmentation for tractable inference.
• We demonstrate competitive classiﬁcation accuracy of our method on standard FSC benchmarks and challenging domain transfer settings.
• We propose several new benchmarks for uncertainty quantiﬁcation in FSC, including calibration, robustness to input noise, and out-of-episode detection.
• We demonstrate improved uncertainty quantiﬁcation of our method on the proposed benchmarks relative to standard few-shot baselines.
R ELATED W ORK
Our work is related to both GP classiﬁcation methods for handling non-conjugacy of classiﬁcation likelihoods and Bayesian approaches to FSC.
We summarize the most relevant work here.
2. GP C LASSIFICATION
Non-augmentation approaches.
There are several classes of approaches for applying Gaussian processes to classiﬁcation.
The most straightforward method, known as least squares classiﬁcation (Rifkin & Klautau, 2004; Williams & Rasmussen, 2006) treats class labels as real-valued observations with a Gaussian likelihood.
Laplace’s approximation constructs a Gaussian approximate posterior centered at the posterior mode.
Variational approaches (Titsias, 2009) maximize a lower bound on the log marginal likelihood.
In expectation propagation (Minka, 2001; Kim & Ghahramani, 2006), local Gaussian approximations to the likelihood are ﬁtted iteratively until convergence.
Augmentation approaches.
Augmentation-based approaches introduce auxiliary random variables such that the original model is recovered when marginalized out.
Girolami & Rogers (2006) propose a Gaussian augmentation for multinomial probit regression.
Linderman et al.
(2015) utilize P´olyagamma augmentation (Polson et al., 2013) and a stick-breaking construction to decompose a multinomial distribution into a product of binomials.
Galy-Fajou et al.
(2019) proposes a logistic-softmax likelihood for classiﬁcation and uses Gamma and Poisson augmentation in addition to P´olya-gamma augmentation in order to perform inference.
2. F EW - SHOT C LASSIFICATION
Meta-learning.
A common approach to FSC is meta-learning, which seek to learn how to update neural network parameters.
The Meta-learner LSTM (Ravi & Larochelle, 2017) learns a meta-level
LSTM to recurrently output a new set of parameters for the base learner.
MAML (Finn et al., 2017) learns deep neural networks to perform well on the task-speciﬁc loss after one or a few steps of gradient descent on the support set by directly backpropagating through the gradient descent procedure itself.
LEO (Rusu et al., 2018) performs meta-learning in a learned low-dimensional latent space from which the parameters of a classiﬁer are generated.
Metric learning.
Metric learning approaches learn distances such that input examples can be meaningfully compared.
Siamese Networks (Koch et al., 2015) learn a shared embedding network along with a distance layer for computing the probability that two examples belong to the same class.
Matching Networks (Vinyals et al., 2016) uses a nonparametric classiﬁcation in the form of attention over nearby examples, which can be interpreted as a form of soft k -nearest neighbors in the embedding space.
Prototypical Networks (Snell et al., 2017) make predictions based on distances to nearest class centroids.
Relation Networks (Sung et al., 2018) instead learn a more complex neural network distance function on top of the embedding layer.
Bayesian Few-shot Classiﬁcation.
More recently, Bayesian FSC approaches that attempt to infer a posterior over task-specﬁc parameters have appeared.
Grant et al.
(2018) reinterprets MAML as an approximate empirical Bayes algorithm and propose LLAMA, which optimizes the Laplace approximation to the marginal likelihood.
Bayesian MAML (Yoon et al., 2018) instead uses Stein
Variational Gradient Descent (SVGD) (Liu & Wang, 2016) to approximate the posterior distribution over model parameters.
VERSA (Gordon et al., 2019) uses amortized inference networks to obtain an approximate posterior distribution over task-speciﬁc parameters.
ABML (Ravi & Beatson, 2019) uses a few steps of Bayes by Backprop (Blundell et al., 2015) on the support set to produce an approximate posterior over network parameters.
GPs for Few-shot Learning.
There have been relatively few works applying GPs to few-shot learning.
Tossou et al.
(2019) consider Gaussian processes in the context of few-shot regression with
Gaussian likelihoods.
GPNet (Patacchiola et al., 2019) use Gaussian processes with least squares classiﬁcation to perform few-shot classiﬁcation and learn covariance functions parameterized by deep neural networks.
More recently, Titsias et al.
(2020) applies GPs to meta-learning with the variational information bottleneck.
B ACKGROUND
In this section we review P ´ olya-gamma augmentation for binary classiﬁcation and the one-vs-each approximation before introducing our method in Section 4.
3. P ´ OLYA -G AMMA A UGMENTATION
Suppose we have a vector of logits ψ ∈ R N with corresponding binary labels y ∈ { 0 , 1 } N .
The logistic likelihood is
[Equation]
where σ ( · ) is the logistic sigmoid function.
Let the prior over ψ be Gaussian: p ( ψ ) =
N ( ψ | µ , Σ ) .
In Bayesian inference, we are interested in the posterior p ( ψ | y )
∝ p ( y | ψ ) p ( ψ ) but the form of (1) does not admit analytic computation due to non-conjugacy.
The main idea of P´olya-gamma augmentation is to introduce auxiliary random variables ω to the likelihood such that the original model is recovered when ω is marginalized out: p ( y | ψ )
= (cid:82) p ( ω ) p ( y | ψ , ω ) d ω .
Conditioned on ω ∼ PG ( b, c ) , the likelihood is proportional to a diagonal Gaussian (see Section A for a full derivation):
[Equation]
where κ i =
y i − 1 / 2 and Ω = diag ( ω ) .
The conditional distribution over ψ given y and ω is now tractable:
[Equation]
where ˜ Σ
=
( Σ − 1 + Ω ) − 1 .
The conditional distribution of ω given ψ and y can also be easily computed:
[Equation]
where the last expression follows from the exponential tilting property of P´olya-gamma random variables.
This suggest a Gibbs sampling procedure in which iterates ω ( t ) ∼ p ( ω | y , ψ ( t − 1) ) and ψ ( t ) ∼ p ( ψ | X , y , ω ( t ) ) are drawn sequentially until the Markov chain reaches its stationary distribution, which is the joint posterior p ( ψ , ω | y ) .
Fortunately, efﬁcient samplers for the P´olya-gamma distribution have been developed (Windle et al., 2014) to facilitate this.
3. O NE - VS -E ACH A PPROXIMATION TO S OFTMAX
The one-vs-each (OVE) approximation (Titsias, 2016) was formulated as a lower bound to the softmax likelihood in order to handle classiﬁcation over a large number of output classes, where computation of the normalizing constant is prohibitive.
We use the OVE approximation not to deal with extreme classiﬁcation, but rather due to its compatibility with P´olya-gamma augmentation, as we shall soon see.
The one-vs-each approximation can be derived by ﬁrst rewriting the softmax likelihood as follows:
[Equation]
where f (cid:44) ( f 1 , . .
, f C ) (cid:62) are the logits.
Since (cid:81) i (1 + α i ) ≥ (1 + (cid:80) i α i ) for α i ≥ 0 , the softmax likelihood (5) can be bounded as follows:
[Equation]
which is the OVE lower bound.
This expression avoids the normalizing constant and factorizes into a product of pairwise sigmoids.
O NE - VS -E ACH P ´ OLYA -G AMMA GP S
We now introduce our method for GP-based Bayesian few-shot classiﬁcation, which utilizes a novel combination of P ´ olya-gamma augmentation and the one-vs-each (OVE) approximation.
4. OVE AS A L IKELIHOOD F UNCTION
Suppose we have access to examples X ∈ R N × D with corresponding one-hot labels Y ∈ { 0 , 1 } N × C , where C is the number of classes.
We consider the logits jointly as a single vector f (cid:44) ( f 11 , . . .
, f 1 N , f 21 , . . . , f 2 N , . . .
, f C 1 , . . .
, f CN ) (cid:62) and place an independent GP prior on the logits for each class: f c ( x ) ∼ GP ( m ( x ) , k
( x , x (cid:48) )) .
Therefore we have p ( f | X ) =
N ( f | µ , K ) , where µ ci = m ( x i ) and K is block diagonal with K cij =
k ( x i , x j ) for each block K c .
The P ´ olya-gamma integral identity used to derive (2) does not have a multi-class analogue and thus a direct application of the augmentation scheme to the softmax likelihood is nontrivial.
Instead, we propose to directly replace the softmax with an OVE-based likelihood function, which is the same as
[Equation] [Equation]
We use this likelihood not to handle extreme classiﬁcation as Titsias (2016), but instead due to its close relationship with the softmax likelihood while maintaining tractable inference with P´olyagamma augmentation.
Compared to other choices of likelihoods used by previous approaches, there are several reasons to prefer OVE.
Compared to the Gaussian augmentation approach of Girolami & Rogers (2006), P´olya-gamma augmentation has the beneﬁt of fast mixing and moreover a single value of ω can capture much of the marginal distribution over function values 1 .
The stick-breaking construction of Linderman et al.
(2015) induces a dependence on the ordering of classes, which leads to undesirable asymmetry.
Finally, the logistic-softmax likelihood of Galy-Fajou et al.
(2019) requires three augmentations and careful learning of the mean function to avoid a priori underconﬁdence (see Section J for more details).
4. P OSTERIOR I NFERENCE VIA G IBBS S AMPLING
Deﬁne the matrix A (cid:44) OVE-MATRIX ( Y ) to be a CN × CN sparse block matrix with C row partitions and C column partitions.
Each block A cc (cid:48) is a diagonal N × N matrix deﬁned as follows:
[Equation]
where Y · c (cid:48) denotes the c (cid:48) th column of Y .
Now the binary logit vector ψ (cid:44)
Af ∈ R CN will have entries equal to f
y
i
i − f ci for each unique combination of c and i, of which there are CN in total.
The OVE likelihood can now be written as p OVE ( Y | ψ )
= 2 N (cid:81) NCj =1 σ ( ψ j ) , where the 2 N term arises from the N cases in which
ψ j = 0 due to comparing the ground truth logit with itself.
Analogous to (2), the likelihood of ψ conditioned on ω is proportional to a diagonal Gaussian:
[Equation]
where κ j = 1 / 2 and Ω = diag ( ω ) .
By exploiting the fact that ψ =
Af , we can express the likelihood in terms of f and write down the conditional posterior as follows:
[Equation]
where ˜ Σ
= ( K − 1 + A (cid:62) ΩA ) − 1 , which is an expression remarkably similar to (3).
Analogous to (4), the conditional distribution over ω given f and the data becomes p ( ω | y , f ) =
PG ( ω | 1 , Af ) .
The primary computational bottleneck of posterior inference lies in sampling f from (10).
Since ˜ Σ is a CN × CN matrix, a naive implementation has complexity O ( C 3 N 3 ) .
By utilizing the matrix inversion lemma and conditional Gaussian sampling techniques (Doucet, 2010), this can be brought down to O ( CN 3 ) (See Appendix E for details).
4. L EARNING C OVARIANCE H YPERPARAMETERS FOR F EW - SHOT C LASSIFICATION
We now describe how we apply OVE P ´ olya-gamma augmented GPs to few-shot classiﬁcation.
We assume the standard episodic few-shot setup in which one observes a labeled support set
S =
( X , Y ) .
Predictions must then be made for a query example ( x ∗ , y ∗ ) .
We consider a zeromean GP prior over the class logits f c ( x ) ∼ GP ( 0 , k θ ( x , x (cid:48) )) , where θ are learnable parameters of our covariance function.
These could include traditional hyperparameters such as lengthscales or the weights of a deep neural network as in deep kernel learning (Wilson et al., 2016).
We consider two objectives for learning hyperparameters of the covariance function: the marginal likelihood (ML) p θ ( Y | X ) and the predictive likelihood (PL) p θ ( y ∗ | x ∗ , X , Y ) .
Marginal likelihood measures the likelihood of the hyperparameters given the observed data and is intuitively appealing from a Bayesian perspective.
On the other hand, many standard FSC methods optimize for predictive likelihood on the query set (Vinyals et al., 2016;
Finn et al., 2017; Snell et al., 2017).
Both objectives marginalize over latent functions, thereby making full use of our Bayesian formulation.
The details of these objectives and how we compute gradients can be found in Section
B. Our learning algorithm for both marginal and predictive likelihood may be found in Section C. Details of computing the posterior predictive distribution p ( y ∗ | x ∗ , X , Y , ω ) may be found in Section D.
Finally, details of our chosen “cosine” kernel may be found in Section G.
E XPERIMENTS
We compare classiﬁcation accuracy and uncertainty quantiﬁcation to representative baselines for several major approaches to FSC: ﬁne-tuning, metric learning, gradient-based meta-learning,
Bayesian neural network and GP-based classiﬁers.
An overview of the baselines we used can be found in Section F.2.
One of our aims is to compare methods based on uncertainty quantiﬁcation.
We therefore developed new benchmark evaluations and tasks: few-shot calibration, robustness, and out-of-episode detection.
In order to empirically compare methods, we could not simply borrow the accuracy results from other papers, but instead needed to train each of these baselines ourselves.
For all baselines except Bayesian MAML, ABML, and Logistic Softmax GP, we ran the code from
(Patacchiola et al., 2019) and veriﬁed that the accuracies matched closely to those reported by (Patacchiola et al., 2019).
Additional experimental details may be found in Section F. Code for our experiments may be found in the supplementary materials.
5. C LASSIFICATION ON F EW - SHOT B ENCHMARKS
As mentioned above, we follow the training and evaluation protocol of Patacchiola et al.
for this section.
We train both 1-shot and 5-shot versions of our model in four different settings:
Caltech-UCSD Birds (CUB)
(Wah et al., 2011), mini-Imagenet with the split proposed by Ravi &
Larochelle (2017), as well as two cross-domain transfer tasks: training on mini-ImageNet and testing on CUB, and from Omniglot (Lake et al., 2011) to EMNIST (Cohen et al., 2017).
We employ the commonly-used Conv4 architecture with 64 channels (Vinyals et al., 2016) for all experiments.
Further experimental details and comparisons across methods can be found in the appendix.
Classiﬁcation results are shown in Table 1 and 2.
We ﬁnd that our proposed P´olya-Gamma OVE GPs yield strong classiﬁcation results.
[Table]
5. U NCERTAINTY Q UANTIFICATION THROUGH C ALIBRATION
We next turn to uncertainty quantiﬁcation, an important concern for few-shot classiﬁers.
When used in safety-critical applications such as medical diagnosis, it is important for a machine learning system to defer when there is not enough evidence to make a decision.
Even in non-critical applications, precise uncertainty quantiﬁcation helps practitioners in the few-shot setting determine when a class has an adequate amount of labeled data or when more labels are required, and can facilitate active learning.
We chose several commonly used metrics for calibration.
Expected calibration error (ECE) (Guo et al., 2017) measures the expected binned difference between conﬁdence and accuracy.
Maximum calibration error (MCE) is similar to ECE but measures maximum difference instead of expected difference.
Brier score (BRI) (Brier, 1950) is a proper scoring rule computed as the squared error between the output probabilities and the one-hot label.
For a recent perspective on metrics for uncertainty evaluation, please refer to Ovadia et al.
The results for representative approaches on 5-shot, 5-way CUB can be found in Figure 1.
Our OVE PG GPs are the best calibrated overall across the metrics.
[Table]
[Figure]
5-shot 5-way tasks on CUB (additional calibration results can be found in the Section H).
Metrics are computed on 3,000 random tasks from the test set.
5. R OBUSTNESS TO I NPUT N OISE
Input examples for novel classes in FSC may have been collected under conditions that do not match those observed at training time.
For example, labeled support images in a medical diagnosis application may come from a different hospital than the training set.
To mimic a simpliﬁed version of this scenario, we investigate robustness to input noise.
We used the Imagecorruptions package (Michaelis et al., 2019) to apply Gaussian noise, impulse noise, and defocus blur to both the support set and query sets of episodes at test time and evaluated both accuracy and calibration.
We used corruption severity of 5 (severe) and evaluated across 1,000 randomly generated tasks on the three datasets involving natural images.
The results are shown in Figure 2.
We ﬁnd that in general
Bayesian approaches tend to be robust due to their ability to marginalize over hypotheses consistent with the support labels.
Our approach is one of the top performing methods across all settings.
5. O UT - OF -E PISODE D ETECTION
Finally, we measure performance on out-of-episode detection, another application in which uncertainty quantiﬁcation is important.
In this experiment, we used 5-way, 5-shot support sets at test time but incorporated out-of-episode examples into the query set.
Each episode had 150 query examples: 15 from each of 5 randomly chosen in-episode classes and 15 from each of 5 randomly chosen outof-episode classes.
We then computed the AUROC of binary outlier detection using the negative of
[Figure]
the maximum logit as the score.
Intuitively, if none of the support classes assign a high logit to the example
[Figure]
it can be classiﬁed as an outlier.
The results are shown in Figure 3.
Our approach generally performs the best across the datasets.
[Figure]
D ISCUSSION
In our experiments
[Figure]
we observed that the ﬁne-tuning approaches are strong baselines in terms of accuracy (Baseline ++ in particular), but do not produce reliable estimates of uncertainty.
Methods relying on Gaussian likelihoods (RelationNet and GPNet) tend to also exhibit poor uncertainty quantiﬁcation.
We hypothesize this is due to the ill-suited nature of applying Gaussian likelihoods to the fundamentally discrete task of classiﬁcation.
Optimizing for predictive cross-entropy generally improves classiﬁcation accuracy and to some extent can remedy calibration issues of marginal likelihood-based methods.
The OVE likelihood is better suited to classiﬁcation than the Logistic
Softmax likelihood, as can be seen by comparing the accuracy and calibration results of the ML versions of these models.
Overall, our proposed OVE PG GP demonstrates strong performance across a wide range of scenarios.
C ONCLUSION
In this work, we have proposed a Bayesian few-shot classiﬁcation approach based on Gaussian processes.
Our method replaces the ordinary softmax likelihood with a one-vs-each likelihood and applies P´olya-Gamma augmentation to perform inference.
This allows us to model class logits directly as function values and efﬁciently marginalize over uncertainty in each few-shot episode.
Modeling functions directly enables our approach to avoid the dependence on model size that posterior inference in weight-space based models inherently have.
Our approach compares favorably to baseline FSC methods under a variety of dataset and shot conﬁgurations, including dataset transfer.
We also demonstrate strong uncertainty quantiﬁcation, robustness to input noise, and out-of-episode detection.
We believe that Bayesian modeling in genereal and GPs in particular are powerful tools for handling uncertainty and hope that our work will lead to broader adoption of efﬁcient Bayesian inference in the few-shot scenario.
A D ERIVATION OF P ´ OLYA -G AMMA A UGMENTED L OGISTIC L IKELIHOOD
In this section, we show the derivation for the augmented logistic likelihood presented in Section 3.1.
First, recall the logistic likelihood:
[Equation]
where σ ( · ) is the logistic sigmoid function.
We have a Gaussian prior p ( ψ ) =
N ( ψ | µ , Σ ) and introduce P´olya-gamma auxiliary random variables ω to the likelihood such that the original model is recovered when ω is marginalized out: p ( y | ψ ) =
(cid:82) p ( ω ) p ( y | ψ , ω ) d ω .
The P´olya-gamma distribution ω ∼ PG ( b, c ) can be written as an inﬁnite convolution of gamma distributions:
[Equation]
The following integral identity holds for b > 0 :
[Equation]
(13) where κ = a − b/ 2 and ω ∼ PG ( b, 0) .
Speciﬁcally, when a = y and b = 1 , we recover an individual term of the logistic likelihood (11):
[Equation]
where κ = y − 1 / 2 and ω ∼ PG (1 , 0) .
Conditioned on ω , the batch likelihood is proportional to a diagonal Gaussian:
[Equation]
15) where κ i =
y i − 1 / 2 and Ω = diag ( ω ) .
The conditional distribution over ψ given y and ω is now tractable:
[Equation]
16) where ˜ Σ
= ( Σ − 1 + Ω ) − 1 .
B M ARGINAL L IKELIHOOD AND P REDICTIVE L IKELIHOOD O BJECTIVES
Marginal Likelihood (ML).
The log marginal likelihood can be written as follows:
[Equation]
(17) where p θ ( Y | X , ω ) = (cid:82) p ( Y | f , ω ) p θ ( f | X ) d f . The gradient of the log marginal likelihood can be estimated by posterior samples ω ∼ p θ ( ω | X , Y ) .
In practice, we use a stochastic training objective based on samples of ω from Gibbs chains.
We use Fisher’s identity (Douc et al., 2014) to derive the following gradient estimator: [Equation]
[Equation]
where ω (1) , . .
ω ( M ) are samples from the posterior Gibbs chain.
As suggested by Patacchiola et al.
(2019), who applied GPs to FSC via least-squares classiﬁcation, we merge the support and query sets during learning to take full advantage of the available data within each episode.
Predictive Likelihood (PL).
The expected log predictive likelihood for a query example x ∗ is: [Equation]
[Equation]
We use an approximate gradient estimator again based on posterior samples of ω :
[Equation]
We note that this is not an unbiased estimator of the gradient, but ﬁnd it works well in practice.
C L EARNING A LGORITHM
Our learning algorithm for both marginal and predictive likelihood is summarized in Algorithm 1.
Input: Objective L ∈ {L ML , L PL } , Task distribution T , number of parallel Gibbs chains M , number of steps T , learning rate η .
Initialize hyperparameters θ randomly.
repeat
Sample S =
( X , Y ) , Q =
( X ∗ , Y ∗ ) ∼ T
if L = L ML then
X ← X ∪ X ∗ , Y ← Y ∪ Y ∗ end if
A ← OVE-MATRIX ( Y ) for m = 1 to M do ω ( m )∼ PG (1 , 0) , f ( m )∼ p θ ( f | X ) for t = 1 to T do ψ ( m ) t ← Af ( m ) t −ω ( m ) t ∼ PG (1 , ψ ( m ) t )
f ( m ) t ∼ p θ ( f | X , Y , ω ( m ) t ) end for end for if L =
L ML then θ
[Equation]
else θ ←
[Equation]
end if until convergence
D P OSTERIOR P REDICTIVE D ISTRIBUTION
The posterior predictive distribution for a query example x ∗ conditioned on ω is: [Equation]
[Equation]
where f ∗ are the query example’s logits.
The predictive distribution over f ∗ can be obtained by noting that ψ and the query logits are jointly Gaussian: [Equation]
[Equation]
where K ∗ is the NC × C block diagonal matrix with blocks K θ ( X , x ∗ ) and K ∗∗ is the C × C diagonal matrix with diagonal entries
k θ ( x ∗ , x ∗ ) .
The predictive distribution becomes:
[Equation]
, where
[Equation]
With p ( f ∗ | x ∗ , X , Y , ω ) in hand, the integral in equation (21) can easily be computed numerically for each class c by forming the corresponding OVE linear transformation matrix A c and then performing 1D Gaussian-Hermite quadrature on each dimension of N ( ψ c ∗ | A c µ ∗ , A c Σ ∗ A c (cid:62) ) . E E FFICIENT C ONDITIONAL S AMPLING OF f The Gibbs conditional distribution over f is given by: [Equation]
[Equation]
where ˜ Σ
= ( K − 1 + A (cid:62) ΩA ) − 1 . Naively sampling from this distribution requires O ( C 3 N 3 ) computation since ˜ Σ is a CN × CN matrix. Here we describe a method for sampling from this distribution that requires O ( CN 3 ) computation instead. First, we note that equation (24) can be interpreted as the conditional distribution p ( f | z
= Ω − 1 κ ) resulting from the following marginal distribution p ( f ) and conditional p ( z | f ) :
[Equation]
where we have made implicit the dependence on X , Y , and ω for brevity of notation.
Equivalently, the distribution over f and z can be represented by the partitioned Gaussian
[Equation]
The conditional distribution p ( f | z ) is given as:
[Equation]
where ˜ Σ
= ( K − 1 + A (cid:62) ΩA ) − 1 .
Note that p ( f | z =
Ω − 1 κ ) recovers our desired Gibbs conditional distribution from equation (24).
An efﬁcient approach to conditional Gaussian sampling is due to Hoffman & Ribak (1991) and described in greater clarity by Doucet (2010).
The procedure is as follows:
Sample f 0 ∼ p ( f ) and z 0 ∼ p ( z | f ) .
Return ¯ f = f 0 + KA (cid:62) ( AKA (cid:62) + Ω − 1 ) − 1 ( Ω − 1 κ − z 0 ) as the sample from p ( f | z ) .
K is block diagonal and thus sampling
from
p ( f requires
O
CN
time.
Af can be computed in
O ( CN ) time, since each entry is the difference between f y i i and
f ci for some i and c .
Overall, step 1 requires O (
CN
3 ) time.
We now show how to compute ¯ f from step 2 in O ( CN 3 ) time.
We ﬁrst expand ( AKA (cid:62) + Ω − 1 ) − 1 :
[Equation]
We substitute into the expression for ¯ f :
[Equation]
where we have deﬁned v (cid:44) A (cid:62) Ω ( Ω − 1 κ − z 0 ) .
Now let d (cid:44) ( d 11 , . . .
, d 1 N , d 21 , . . .
, d 2 N , . . .
, d C 1 , . . .
, d CN ) (cid:62) , where d ci =
Y ic (cid:80) c (cid:48)
ω c (cid:48)
i .
Deﬁne Y †
to be the CN × N matrix produced by vertically stacking diag ( Y · c ) , and let W † be the CN × N matrix produced by vertically stacking diag (( ω c 1 , . . .
, ω cN ) (cid:62) ) .
A (cid:62) ΩA may then be written as follows:
[Equation]
(cid:62) , where
[Equation]
[Equation]
[Equation]
[Equation]
Substituting (33) into (32):
[Equation]
Now we expand ( K − 1 + D − SPS (cid:62) ) − 1 :
[Equation]
where E =
( K − 1 + D ) − 1 =
K ( K + D − 1 ) − 1 D − 1 is a block-diagonal matrix that can be computed in O ( CN 3 ) time, since D is diagonal and K is block diagonal.
Now, substituting (38) back into (37),
[Equation]
Note that ( S (cid:62) ES − P − 1 ) − 1 is a 2 N × 2 N matrix and thus can be inverted in O ( N 3 ) time.
The overall complexity is therefore O ( CN 3 ) .
Here we provide more details about our experimental setup for our classiﬁcation experiments, which are based on the protocol of (Patacchiola et al., 2019).
F. D ATASETS
We used the four dataset scenarios described below.
The ﬁrst three are the same used by Chen et al.
and the ﬁnal was proposed by Patacchiola et al.
• CUB.
Caltech-UCSD Birds (CUB)
(Wah et al., 2011) consists of 200 classes and 11, images.
A split of 100 training, 50 validation, and 50 test classes was used (Hilliard et al., 2018; Chen et al., 2019).
• mini-Imagenet.
The mini-Imagenet dataset (Vinyals et al., 2016) consists of 100 classes with 600 images per class.
We used the split proposed by Ravi & Larochelle (2017), which has 64 classes for training, 16 for validation, and 20 for test.
• mini-Imagenet → CUB.
This cross-domain transfer scenario takes the training split of mini-Imagenet and the validation & test splits of CUB.
• Omniglot → EMNIST.
We use the same setup as proposed by Patacchiola et al.
Omniglot (Lake et al., 2011) consists of 1,623 classes, each with 20 examples, and is augmented by rotations of 90 degrees to create 6,492 classes, of which 4,114 are used for training.
The EMNIST dataset (Cohen et al., 2017), consisting of 62 classes, is split into 31 training and 31 test classes.
F. B ASELINES
We compare to a variety of baselines, explained here in more detail.
• Feature Transfer (Chen et al., 2019) involves ﬁrst training an off-line classiﬁer on the training classes and then training a new classiﬁcation layer on the episode.
• Baseline ++ (Chen et al., 2019) is similar to Feature Transfer except it uses a cosine distance module prior to the softmax during ﬁne-tuning.
• Matching Networks (Vinyals et al., 2016) can be viewed as a soft form of k -nearest neighbors that computes attention and sums over the support examples to form a predictive distribution over classes.
• Prototypical Networks
(Snell et al., 2017) computes class means (prototypes) and forms a predictive distribution based on Euclidean distance to the prototypes.
It can be viewed as a Gaussian classiﬁer operating in an embedding space.
• MAML (Finn et al., 2017) performs one or a few steps of gradient descent on the support set and then makes predictions on the query set, backpropagating through the gradient descent procedure.
For this baseline, we simply quote the classiﬁcation accuracy reported by (Patacchiola et al., 2019).
• RelationNet
(Sung et al., 2018) rather than using a predeﬁned distance metric as in Matching Networks or Prototypical Networks instead learns a deep distance metric as the output of a neural network that accepts as input the latent representation of both examples.
It is trained to minimize squared error of output predictions.
• GPNet
(Patacchiola et al., 2019) relies on least squares classiﬁcation to maintain tractability of Gaussian process posterior inference.
This is concurrent work to ours and so we compare to their results with a linear kernel (the latest version at the time the bulk of our experiments were performed).
This work has since been renamed to GPShot.
• Bayesian MAML (Yoon et al., 2018) relies on Stein Variational Gradient Descent (SVGD)
(Liu & Wang, 2016) to get an approximate posterior distribution in weight-space.
We compare to both the non-chaser version, which optimizes cross-entropy of query predictions, and the chaser version, which optimizes mean squared error between the approximate posterior on the support set and the approximate posterior on the merged support & query set.
The non-chaser version is therefore related to predictive likelihood methods and the chaser version is more analogous to the marginal likelihood methods.
For the non-chaser version, we used 20 particles and 1 step of adaptation at both train and test time.
For the chaser version, we also used 20 particles.
At train time, the chaser took 1 step and the leaderadditional step.
At test time, we used 5 steps of adaptation.
Due to the slow performance of this method, we followed the advice of Yoon et al.
and only performed adaptation on the ﬁnal layer of weights, which may help explain the drop in performance relative to MAML.
The authors only released Tensorﬂow code for regression, so we reimplemented this baseline in PyTorch.
Amortized Bayesian Meta-Learning (ABML) (Ravi & Beatson, 2019) performs a few steps of Bayes-by-backprop (Blundell et al., 2015) in order to infer a fully-factorized approximate posterior over the weights.
The authors did not release code and so we implemented our own version of ABML in PyTorch.
We found the weighting on the inner and outer KL divergences to be important for achieving good performance.
We took the negative log likelihood to be mean cross entropy and used an inner KL weight of 0.01 and an outer KL weight of 0.001, which values were arrived upon by doing a small amount of hyperparameter tuning on the Omniglot → EMNIST dataset.
We used α = 1 . 0 and β = 0 .
01 for the Gamma prior over the weights.
We only applied ABML to the weights of the network; the biases were learned as point estimates.
We used 4 steps of adaptation and took 5 samples when computing expectations (using any more than this did not ﬁt into GPU memory).
We used the local reparameterization trick (Kingma et al., 2015) and ﬂipout (Wen et al., 2018) when computing expectations in order to reduce variance.
In order to match the architecture used by Ravi & Beatson (2019), we trained this baseline with 32 ﬁlters throughout the classiﬁcation network.
• Logistic Softmax GP (Galy-Fajou et al., 2019) is the multi-class Gaussian process classiﬁcation method that relies on the logistic softmax likelihood.
Galy-Fajou et al.
(2019) did not consider few-shot, but we use the same objectives described in Section 3.3 of the main paper to adapt this method to FSC.
In addition, we used the cosine kernel (see Section G for a description) that we found to work best with our OVE PG GPs.
For this method, we found it important to learn a constant mean function (rather than a zero mean) in order to improve calibration.
Please refer to Section J for a possible explanation why this is necessary.
F. T RAINING D ETAILS
All methods employed the commonly-used Conv4 architecture (Vinyals et al., 2016)
(see Tablefor a detailed speciﬁcation), except ABML which used 32 ﬁlters throughout.
All of our experiments used the Adam (Kingma & Ba, 2014) optimizer with learning rate 10 − 3 .
During training, all models used epochs consisting of 100 randomly sampled episodes.
A single gradient descent step on the encoder network and relevant hyperparameters is made per episode.
All 1-shot models are trained for 600 epochs and 5-shot models are trained for 400 epochs.
Each episode contained 5 classes (5-way) and 16 query examples.
At test time, 15 query examples are used for each episode.
Early stopping was performed by monitoring accuracy on the validation set.
The validation set was not used for retraining.
We train both marginal likelihood and predictive likelihood versions of our models.
For P´olyagamma sampling we use the PyP´olyaGamma package 2 .
During training, we use a single step of
Gibbs ( T =1).
For evaluation, we run until T = 50 .
In both training and evaluation, we use M =parallel Gibbs chains to reduce variance.
G E FFECT OF K ERNEL C HOICE ON C LASSIFICATION A CCURACY
In this section, we examine the effect of kernel choice on classiﬁcation accuracy for our proposed
One-vs-Each P´olya gamma [Table]
Cosine Kernel.
In the main paper, we showed results for the following kernel, which we refer to as the “cosine” kernel due to its resemblance to cosine similarity:
[Equation]
where g θ ( · ) is a deep neural network that outputs a ﬁxed-dimensional encoded representation of the input and α is the scalar log output scale.
Both θ and α are considered hyperparameters and learned simultaneously as shown in Algorithm 1.
We found that this kernel works well for a range of datasets and shot settings.
We note that the use of cosine similarity is reminiscent of the approach taken by
Baseline ++ method of (Chen et al., 2019), which computes the softmax over cosine similarity to class weights.
Here we consider three additional kernels: linear, RBF, and normalized RBF.
Linear Kernel.
The linear kernel is deﬁned as follows:
[Equation]
(41) where D is the output dimensionality of g θ ( x ) .
We apply this dimensionality scaling because the dot product between g θ ( x ) and g θ ( x (cid:48) ) may be large depending on D .
RBF Kernel.
The RBF (also known as squared exponential) kernel can be deﬁned as follows:
[Equation]
where (cid:96) is the log lengthscale parameter (as with α , we learn the (cid:96) alongside θ ).
Normalized RBF Kernel.
Finally, we consider a normalized RBF kernel similar in spirit to the cosine kernel:
[Equation]
The results of our P ´ olya-gamma OVE GPs with different kernels can be found in Tables 4 and 5.
In general, we ﬁnd that the cosine kernel works best overall.
We hypothesize that this is because they help the embedding network g θ ( · ) to learn linearly separable representations.
In contrast, the RBFbased kernels yields nonlinear decision boundaries with respect to the embedded representation and may not provide the embedding network with as strong of a learning signal.
Further study of the beneﬁts and drawbacks of linear vs. nonlinear kernels is an interesting area of future work.
Cosine is overall the best, followed closely by linear.
RBF kernels perform worse, except for the Omniglot → EMNIST dataset
Evaluation is performed on 5 randomly generated sets episodes
Standard deviation of the mean accuracy is also shown = Marginal Likelihood PL =
Predictive Likelihood.
[Table]
[Table]
H A DDITIONAL C ALIBRATION R ESULTS
In Figure [Table] we include calibration results for mini-Imagenet and Omniglot → EMNIST.
They follow similar trends to the results presented in Section 5.2.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
I Q UANTITATIVE R OBUSTNESS TO I NPUT N OISE R ESULTS
In this section we include quantitative results for the robustness to input noise results presented in
Figure 2.
Results for Gaussian noise are shown in Table 6, impulse noise in Table 7, and defocus blur in Table 8.
[Table] [Table]
+
[Table]
J C OMPARISON OF L IKELIHOODS
In this section we seek to better understand the behaviors of the softmax, OVE, logistic softmax, and Gaussian likelihoods for classiﬁcation.
For convenience, we summarize the forms of these likelihoods below.
• Softmax.
p ( y = c | f ) = exp( f c ) (cid:80) c (cid:48) exp( f c (cid:48) )
• OVE.
p ( y = c | f ) =
(cid:89) c (cid:48) (cid:54) =
c σ ( f c − f c (cid:48) )
• Logistic Softmax.
p ( y = c | f ) =
σ ( f c ) (cid:80) c (cid:48)
σ ( f c (cid:48) )
[Table]
• Gaussian.
p ( y = c | f ) =
(cid:89)
[Equation]
We sampled logits from f ∼ N (0 , 1) and plotted a histogram and kernel density estimate of the maximum output probability max c p ( y = c | f ) for each of the likelihoods, where C = 5 .
Note that because we are interested in predictions here that all output probabilities are normalized to sum to 1 when computing conﬁdences.
The results are shown in Figure 5.
Logistic softmax is a priori underconﬁdent: it puts little probability mass on conﬁdence above 0.4.
This may be due to the use of the sigmoid function which squashes large values of f .
Gaussian likelihood a priori is overconﬁdent in that it puts a large amount of probability mass on conﬁdent outputs.
OVE is slightly closer to softmax compared to Gaussian, yet still overconﬁdent.
Note that this is not a complete explanation, because GP hyperparameters such as the prior mean or Gaussian likelihood variance may be able to compensate for these imperfections to some degree.
Indeed, we found it helpful to learn a constant mean for the logistic softmax likelihood, as mentioned in Section F.2.
[Figure]
K L IKELIHOOD V ISUALIZATION
In order to visualize the various likelihoods under consideration, we consider a trivial classiﬁcation task with a single observed example.
We assume that there are three classes ( C = 3 ) and the single example belongs to the ﬁrst class ( y = 1 ).
We place the following prior on f = ( f 1 , f 2 , f 3 ) (cid:62) : p
[Equation]
[Equation]
In other words, the prior for f 1 and f 2 is a standard normal and f 3 is clamped at zero (for ease of visualization).
We consider here the softmax, Gaussian, logistic softmax (LSM), and one-vs-each (OVE) likelihoods as deﬁned in Table 9.
[Table]
[Equation]
[Table]
[Equation]
One
[Table]
vs
[Table]
Each (OVE)
[Equation]
The posterior for a given likelihood can be computed as:
[Equation]
The likelihoods as deﬁned in Table 9 and corresponding posteriors are plotted in Figure 6 and Figure 7, respectively.
L 2D I RIS E XPERIMENTS
We also conducted experiments on a 2D version of the Iris dataset (Fisher, 1936), which contains 150 examples across 3 classes.
The ﬁrst two features of the dataset were retained (sepal length and width).
We used a zero-mean GP prior and an RBF kernel k ( x , x (cid:48) )
=
exp (cid:18) −2 d ( x , x (cid:48) (cid:19), where d ( · , · ) is Euclidean distance.
We considered class-balanced training set sizes with 1, 2, 3, 4, 5, 10, 15, 20, 25, and 30 examples per class.
For each training set size, we performed GP inference onrandomly generated train/test splits and compared the predictions across Gaussian, logistic softmax, and one-vs-each likelihoods.
Predictions at a test point x ∗ were made by applying the normalized likelihood to the posterior predictive mean ¯ f ∗ .
The predictive probabilities for each likelihood is shown in Figure 8 for a randomly generated train/test split with 30 examples per class.
Test predictive accuracy, Brier score, expected calibration error, and evidence lower bound (ELBO) results across various training set sizes are shown in Figure 9.
The ELBO is computed by treating each likelihood’s posterior q ( f | X , Y ) as an approximation to the softmax posterior p ( f | X , Y ) .
[Equation]
[Figure]
Even though direct computation of the softmax posterior p ( f | X , y ) is intractable, computing the
ELBO is tractable.
A larger ELBO indicates a lower the KL divergence to the softmax posterior.
One-vs-Each performs well for accuracy, Brier score, and ELBO across the training set sizes.
Gaussian performs best on expected calibration error through 15 examples per class, beyond which onevs-each is better.
[Figure]
[Figure]
[Figure]