B AYESIAN F EW -S HOT C LASSIFICATION WITH O NE - VS -E ACH P ´ OLYA -G AMMA A UGMENTED G AUSSIAN P ROCESSES
A BSTRACT
Few-shot classiﬁcation (FSC), the task of adapting a classiﬁer to unseen classes given a small labeled dataset, is an important step on the path toward human-like machine learning.
Bayesian methods are well-suited to tackling the fundamental issue of overﬁtting in the few-shot scenario because they allow practitioners to specify prior beliefs and update those beliefs in light of observed data.
Contemporary approaches to Bayesian few-shot classiﬁcation maintain a posterior distribution over model parameters, which is slow and requires storage that scales with model size.
Instead, we propose a Gaussian process classiﬁer based on a novel combination of P ´ olya-Gamma augmentation and the one-vs-each softmax approximation (Titsias, 2016) that allows us to efﬁciently marginalize over functions rather than model parameters.
We demonstrate improved accuracy and uncertainty quantiﬁcation on both standard few-shot classiﬁcation benchmarks and few-shot domain transfer tasks.
I NTRODUCTION
Few-shot classiﬁcation (FSC) is a rapidly growing area of machine learning that seeks to build classiﬁers able to adapt to novel classes given only a few labeled examples.
It is an important step towards machine learning systems that can successfully handle challenging situations such as personalization, rare classes, and time-varying distribution shift.
The shortage of labeled data in
FSC leads to uncertainty over the parameters of the model, known as model uncertainty or epistemic uncertainty .
If model uncertainty is not handled properly in the few-shot setting, there is a signiﬁcant risk of overﬁtting.
In addition, FSC is increasingly being used for risk-averse applications such as medical diagnosis (Prabhu, 2019) and human-computer interfaces (Wang et al., 2019) where it is important for a few-shot classiﬁer to know when it is uncertain.
Bayesian methods maintain a distribution over model parameters and thus provide a natural framework for capturing this inherent model uncertainty.
In a Bayesian approach, a prior distribution is ﬁrst placed over the parameters of a model.
After data is observed, the posterior distribution over parameters is computed using Bayesian inference.
This elegant treatment of model uncertainty has led to a surge of interest in Bayesian approaches to FSC that infer a posterior distribution over the weights of a neural network (Finn et al., 2018; Yoon et al., 2018; Ravi & Beatson, 2019).
Although conceptually appealing, there are several practical obstacles to applying Bayesian inference directly to the weights of a neural network.
Bayesian neural networks (BNNs) are expensive from both a computational and memory perspective.
Moreover, specifying meaningful priors in parameter space is known to be difﬁcult due to the complex relationship between weights and network outputs (Sun et al., 2019).
Gaussian processes (GPs) instead maintain a distribution over functions rather than model parameters.
The prior is directly speciﬁed by a mean and covariance function, which may be parameterized by deep neural networks.
When used with Gaussian likelihoods, GPs admit closed form expressions for the posterior and predictive distributions.
They exchange the computational drawbacks of BNNs for cubic scaling with the number of examples.
In FSC, where the number of examples is small, this is often an acceptable trade-off.
When applying GPs to classiﬁcation with a softmax likelihood, the non-conjugacy of the GP prior renders posterior inference intractable.
Many approximate inference methods have been proposed to circumvent this, including variational inference and expectation propagation.
In this paper we investigate a particularly promising class of approaches that augment the GP model with a set of auxiliary random variables, such that when they are marginalized out the original model is recovered (Albert & Chib, 1993; Girolami & Rogers, 2006; Linderman et al., 2015).
Such augmentation-based approaches typically admit efﬁcient Gibbs sampling procedures for generating posterior samples which when combined with Fisher’s identity (Douc et al., 2014) can be used to optimize the parameters of the mean and covariance functions.
In particular, augmentation with P´olya-Gamma random variables (Polson et al., 2013) makes inference tractable in logistic models.
Naively, this is useful for handling binary classiﬁcation, but in this paper we show how to extend P´olya-Gamma augmentation to multiple classes by using the one-vs-each softmax approximation (Titsias, 2016), which can be expressed as a product of logistic sigmoids.
We further show that the one-vs-each approximation can be interpreted as a composite likelihood (Lindsay, 1988; Varin et al., 2011), a connection which to our knowledge has not been made in the literature.
In this work, we make several contributions:
• We show how the one-vs-each softmax approximation (Titsias, 2016) can be interpreted as a composite likelihood consisting of pairwise conditional terms.
• We propose a novel GP classiﬁcation method that combines the one-vs-each softmax approximation with P ´ olya-Gamma augmentation for tractable inference.
• We demonstrate competitive classiﬁcation accuracy of our method on standard FSC benchmarks and challenging domain transfer settings.
• We propose several new benchmarks for uncertainty quantiﬁcation in FSC, including calibration, robustness to input noise, and out-of-episode detection.
• We demonstrate improved uncertainty quantiﬁcation of our method on the proposed benchmarks relative to standard few-shot baselines.
R ELATED W ORK
Our work is related to both GP methods for handling non-conjugate classiﬁcation likelihoods and
Bayesian approaches to few-shot classiﬁcation.
We summarize relevant work here.
2. GP C LASSIFICATION
Non-augmentation approaches.
There are several classes of approaches for applying Gaussian processes to classiﬁcation.
The most straightforward method, known as least squares classiﬁcation (Rifkin & Klautau, 2004), treats class labels as real-valued observations and performs inference with a Gaussian likelihood.
The Laplace approximation (Williams & Barber, 1998) constructs a Gaussian approximate posterior centered at the posterior mode.
Variational approaches (Titsias, 2009; Matthews et al., 2016) maximize a lower bound on the log marginal likelihood.
In expectation propagation (Minka, 2001; Kim & Ghahramani, 2006; Hernandez-Lobato & Hernandez-Lobato, 2016), local Gaussian approximations to the likelihood are ﬁtted iteratively to minimize KL divergence from the true posterior.
Augmentation approaches.
Augmentation-based approaches introduce auxiliary random variables such that the original model is recovered when marginalized out.
Girolami & Rogers (2006) propose a Gaussian augmentation for multinomial probit regression.
Linderman et al.
(2015) utilize P´olya-Gamma augmentation (Polson et al., 2013) and a stick-breaking construction to decompose a multinomial distribution into a product of binomials.
Galy-Fajou et al.
(2020) propose a logistic-softmax likelihood for classiﬁcation and uses Gamma and Poisson augmentation in addition to P´olya-Gamma augmentation in order to perform inference.
2. F EW - SHOT C LASSIFICATION
Meta-learning.
A common approach to FSC is meta-learning, which seeks to learn a strategy to update neural network parameters when faced with a novel learning task.
The Meta-learner
LSTM (Ravi & Larochelle, 2017) learns a meta-level LSTM to recurrently output a new set of parameters for a base learner.
MAML (Finn et al., 2017) learns initializations of deep neural networks that perform well on task-speciﬁc losses after one or a few steps of gradient descent by backpropagating through the gradient descent procedure itself.
LEO (Rusu et al., 2019) performs meta-learning in a learned low-dimensional latent space from which the parameters of a classiﬁer are generated.
Metric learning.
Metric learning approaches learn distances such that input examples can be meaningfully compared.
Siamese Networks (Koch, 2015) learn a shared embedding network along with a distance layer for computing the probability that two examples belong to the same class.
Matching Networks (Vinyals et al., 2016) uses a nonparametric classiﬁcation in the form of attention over nearby examples, which can be interpreted as a form of soft k -nearest neighbors in the embedding space.
Prototypical Networks (Snell et al., 2017) make predictions based on distances to nearest class centroids.
Relation Networks (Sung et al., 2018) instead learn a more complex neural network distance function on top of the embedding layer.
Bayesian Few-shot Classiﬁcation.
More recently, Bayesian FSC approaches that attempt to infer a posterior over task-speciﬁc parameters have appeared.
Grant et al.
(2018) reinterpret MAML as an approximate empirical Bayes algorithm and propose LLAMA, which optimizes the Laplace approximation to the marginal likelihood.
Bayesian MAML (Yoon et al., 2018) instead uses Stein
Variational Gradient Descent (SVGD) (Liu & Wang, 2016) to approximate the posterior distribution over model parameters.
VERSA (Gordon et al., 2019) uses amortized inference networks to obtain an approximate posterior distribution over task-speciﬁc parameters.
ABML (Ravi & Beatson, 2019) uses a few steps of Bayes by Backprop (Blundell et al., 2015) on the support set to produce an approximate posterior over network parameters.
CNAPs (Requeima et al., 2019) modulate taskspeciﬁc Feature-wise Linear Modulation (FiLM)
(Perez et al., 2018) layer parameters as the output of an adaptation network that takes the support set as input.
GPs for Few-shot Learning.
There have been relatively few works applying GPs to few-shot learning.
Tossou et al.
(2020) consider Gaussian processes in the context of few-shot regression with Gaussian likelihoods.
Deep Kernel Transfer (DKT)
(Patacchiola et al., 2020) uses Gaussian processes with least squares classiﬁcation to perform few-shot classiﬁcation and learns covariance functions parameterized by deep neural networks.
More recently, Titsias et al.
(2020) applies GPs to meta-learning by maximizing the mutual information between the query set and a latent representation of the support set.
B ACKGROUND
In this section we ﬁrst review P´olya-Gamma augmentation for binary classiﬁcation and the one-vseach approximation before we introduce our method in Section 4.
3. P ´ OLYA -G AMMA A UGMENTATION
The P´olya-Gamma augmentation scheme was originally introduced to address Bayesian inference in logistic models (Polson et al., 2013).
Suppose we have a vector of logits ψ ∈ R N with corresponding binary labels y ∈ { 0 , 1 } N .
The logistic likelihood is
[Equation]
where σ ( · ) is the logistic sigmoid function.
Let the prior over ψ be Gaussian: p ( ψ ) =
N ( ψ | µ , Σ ) .
In Bayesian inference, we are interested in the posterior p ( ψ | y )
∝ p ( y | ψ ) p ( ψ ) but the form of (1) does not admit analytic computation of the posterior due to non-conjugacy.
The main idea of
P´olya-Gamma augmentation is to introduce auxiliary random variables ω to the likelihood such that the original model is recovered when ω is marginalized out: p ( y | ψ )
= (cid:82) p ( ω ) p ( y | ψ , ω ) d ω .
Conditioned on ω ∼ PG (1 , 0) , the batch likelihood is proportional to a diagonal Gaussian (see
Section A for a full derivation):
[Equation]
where κ i =
y i − 1 / 2 and Ω = diag ( ω ) .
The conditional distribution over ψ given y and ω is now tractable:
[Equation]
where ˜ Σ
=
( Σ − 1 + Ω ) − 1 .
The conditional distribution of ω given ψ and y can also be easily computed:
[Equation]
where the last expression follows from the exponential tilting property of P´olya-Gamma random variables.
This suggest a Gibbs sampling procedure in which iterates ω ( t ) ∼ p ( ω | y , ψ ( t − 1) ) and ψ ( t ) ∼ p ( ψ | X , y , ω ( t ) ) are drawn sequentially until the Markov chain reaches its stationary distribution, which is the joint posterior p ( ψ , ω | y ) .
Fortunately, efﬁcient samplers for the P´olya-Gamma distribution have been developed (Windle et al., 2014) to facilitate this.
3. O NE - VS -E ACH A PPROXIMATION TO S OFTMAX
The one-vs-each (OVE) approximation (Titsias, 2016) was formulated as a lower bound to the softmax likelihood in order to handle classiﬁcation over a large number of output classes, where computation of the normalizing constant is prohibitive.
We employ the OVE approximation not to deal with extreme classiﬁcation, but rather due to its compatibility with P ´ olya-Gamma augmentation, as we shall soon see.
The one-vs-each approximation can be derived by ﬁrst rewriting the softmax likelihood as follows:
[Equation]
where f (cid:44) ( f 1 , . .
, f C ) (cid:62) are the logits.
Since in general (cid:81) k
(1 + α k ) ≥ (1 + (cid:80) k
α k ) for α k ≥ 0 , the softmax likelihood (5) can be bounded as follows:
[Equation]
which is the OVE lower bound.
This expression avoids the normalizing constant and factorizes into a product of pairwise sigmoids, which is amenable to P ´ olya-Gamma augmentation for tractable inference.
O NE - VS -E ACH P ´ OLYA -G AMMA GP S
In this section, we ﬁrst show how the one-vs-each (OVE) approximation can be interpreted as a pairwise composite likelihood.
We then we introduce our method for GP-based Bayesian few-shot classiﬁcation, which brings together OVE and P´olya-Gamma augmentation in a novel combination.
4. OVE AS A C OMPOSITE L IKELIHOOD
Titsias (2016) showed that the OVE approximation shares the same global optimum as the softmax maximum likelihood, suggesting a close relationship between the two.
We show here that in fact
OVE can be interpreted as a pairwise composite likelihood version of the softmax.
Composite likelihoods (Lindsay, 1988; Varin et al., 2011) are a type of approximate likelihood often employed when the exact likelihood is intractable or otherwise difﬁcult to compute.
Given a collection of marginal or conditional events { E 1 , . . .
, E K } and parameters f , a composite likelihood is deﬁned as:
[Equation]
where L k ( f | y ) ∝ p ( y ∈
E k | f ) and w k ≥ 0 are arbitrary weights.
In order to make the connection to OVE, it will be useful to let the one-hot encoding of the label y be denoted as y ∈ { 0 , 1 } C .
Deﬁne a set of C ( C − 1) / 2 pairwise conditional events E ij , one each for all pairs of classes i (cid:54) =
j , indicating the event that the model’s output matches the target label for classes i and j conditioned on all the other classes:
[Equation]
where ¬ ij denotes the set of classes not equal to either i or j .
This expression resembles the pseudolikelihood (Besag, 1975), but instead of a single conditional event per output site, the expression in (8) considers all pairs of sites.
Stoehr & Friel (2015) explored similar composite likelihood generalizations of the pseudolikelihood in the context of random ﬁelds.
Now suppose that y c
= 1 for some class c / ∈ { i, j } .
Then p ( y i , y j | y ¬ ij , f ) = 1 due to the one-hot constraint.
Otherwise either y i = 1 or y j = 1 .
In this case, assume without loss of generality that y i =
and y j = 0 and thus
[Equation]
The composite likelihood deﬁned in this way with unit component weights is therefore
[Equation]
Alternatively, we may simply write L OVE ( f | y = i ) =
[Equation]
= i σ ( f i − f j ) , which is identical to the
OVE bound (6).
4. GP C LASSIFICATION WITH THE OVE L IKELIHOOD
We now turn our attention to GP classiﬁcation.
Suppose we have access to examples X ∈ R N × D with corresponding one-hot labels Y ∈ { 0 , 1 } N × C , where C is the number of classes.
We consider the logits jointly as a single vector
[Equation]
and
[Table] [Table]
GP ( m ( x ) , k ( x , x (cid:48) )) .
Therefore we have p ( f | X ) =
N ( f | µ , K ) , where µ ci = m ( x i ) and K is block diagonal with K cij =
k ( x i , x j ) for each block K c .
The P ´ olya-Gamma integral identity used to derive (2) does not have a multi-class analogue and thus a direct application of the augmentation scheme to the softmax likelihood is nontrivial.
Instead, we propose to directly replace the softmax with the OVE-based composite likelihood function from (10) with unit weights.
The posterior over f when using OVE as the likelihood function can be expressed as:
[Equation]
to which P´olya-Gamma augmentation can be applied as we show in the next section.
Our motivation for using a composite likelihood therefore differs from the traditional motivation, which is to avoid the use of a likelihood function which is intractable to evaluate .
Instead, we employ a composite likelihood because it makes posterior inference tractable when coupled with P´olya-Gamma augmentation.
Prior work on Bayesian inference with composite likelihoods has shown that the composite posterior is consistent under fairly general conditions for correctly speciﬁed models (Miller, 2019) but can produce overly concentrated posteriors (Pauli et al., 2011; Ribatet et al., 2012) since each component likelihood event is treated as independent when in reality there may be signiﬁcant dependencies.
Nevertheless, we show in Section 5 that in practice our method exhibits competitive accuracy and strong calibration relative to baseline few-shot learning algorithms.
We leave further theoretical analysis of the OVE composite posterior and its properties for future work.
Compared to choices of likelihoods used by previous approaches, there are several reasons to prefer
OVE.
Relative to the Gaussian augmentation approach of Girolami & Rogers (2006), P´olya-Gamma augmentation has the beneﬁt of fast mixing and the ability of a single value of ω to capture much of the marginal distribution over function values 1 .
The stick-breaking construction of Linderman et al.
(2015) induces a dependence on the ordering of classes, which leads to undesirable asymmetry.
Finally, the logistic-softmax likelihood of Galy-Fajou et al.
(2020) requires three augmentations and careful learning of the mean function to avoid a priori underconﬁdence (see Section F.1 for more details).
4. P OSTERIOR I NFERENCE VIA G IBBS S AMPLING
We now describe how we perform tractable posterior inference in our model with Gibbs sampling.
Deﬁne the matrix A (cid:44) OVE-MATRIX ( Y ) to be a CN × CN sparse block matrix with C row partitions and C column partitions.
Each block A cc (cid:48) is a diagonal N × N matrix deﬁned as follows:
[Equation]
(13) where Y · c (cid:48) denotes the c (cid:48)
th column of Y .
Now the binary logit vector ψ (cid:44)
Af ∈ R CN will have entries equal to f
y
i
i − f ci for each unique combination of c and i, of which there are CN in total.
The OVE composite likelihood can now be written as L ( ψ | Y ) =
2 N (cid:81) NCj =1 σ ( ψ j ) , where the 2 N term arises from the N cases in which ψ j = 0 due to comparing the ground truth logit with itself.
Analogous to (2), the likelihood of ψ conditioned on ω and Y is proportional to a diagonal Gaussian:
[Equation] [Equation]
where κ j = 1 / 2 and Ω = diag ( ω ) .
By exploiting the fact that ψ =
Af , we can express the likelihood in terms of f and write down the conditional composite posterior as follows:
[Equation]
where ˜ Σ
= ( K − 1 + A (cid:62) ΩA ) − 1 , which is an expression remarkably similar to (3).
Analogous to (4), the conditional distribution over ω given f and the data becomes p ( ω | y , f ) =
PG ( ω | 1 , Af ) .
The primary computational bottleneck of posterior inference lies in sampling f from (15).
Since ˜ Σ is a CN × CN matrix, a naive implementation has complexity O ( C 3 N 3 ) .
By utilizing the matrix inversion lemma and Gaussian sampling techniques summarized in (Doucet, 2010), this can be brought down to O ( CN 3 ) .
Details may be found in Section B.
4. L EARNING C OVARIANCE H YPERPARAMETERS FOR F EW - SHOT C LASSIFICATION
We now describe how we apply OVE P ´ olya-Gamma augmented GPs to few-shot classiﬁcation.
We assume the standard episodic few-shot setup in which one observes a labeled support set S =
( X , Y ) .
Predictions must then be made for a query example ( x ∗ , y ∗ ) .
We consider a zero-mean
GP prior over the class logits f c ( x ) ∼ GP ( 0 , k θ ( x , x (cid:48) )) , where θ are learnable parameters of our covariance function.
These could include traditional hyperparameters such as lengthscales or the weights of a deep neural network as in deep kernel learning (Wilson et al., 2016).
We consider two objectives for learning hyperparameters of the covariance function: the marginal likelihood (ML) and the predictive likelihood (PL).
Marginal likelihood measures the likelihood of the hyperparameters given the observed data and is intuitively appealing from a Bayesian perspective.
On the other hand, many standard FSC methods optimize for predictive likelihood on the query set (Vinyals et al., 2016;
Finn et al., 2017; Snell et al., 2017).
Both objectives marginalize over latent functions, thereby making full use of our Bayesian formulation.
The details of these objectives and how we compute gradients can be found in Section C.
Our learning algorithm for both marginal and predictive likelihood may be found in Section D. Details of computing the posterior predictive distribution p ( y ∗ | x ∗ , X , Y , ω ) may be found in Section E.
Finally, details of our chosen “cosine” kernel may be found in Section H.
E XPERIMENTS
In this section, we present our results on few-shot classiﬁcation both in terms of accuracy and uncertainty quantiﬁcation.
Additional results comparing the one-vs-each composite likelihood to the softmax, logistic softmax, and Gaussian likelihoods may be found in Section F.
One of our aims is to compare methods based on uncertainty quantiﬁcation.
We therefore developed new benchmark evaluations and tasks: few-shot calibration, robustness, and out-of-episode detection.
In order to empirically compare methods, we could not simply borrow the accuracy results from other papers, but instead needed to train each of these baselines ourselves.
For all baselines except Bayesian MAML, ABML, and Logistic Softmax GP, we ran the code from (Patacchiola et al., 2020) and veriﬁed that the accuracies matched closely to their reported results.
We have made
PyTorch code for our experiments publicly available 2 .
5. F EW - SHOT C LASSIFICATION
For our few-shot classiﬁcation experiments, we follow the training and evaluation protocol of Patacchiola et al.
We train both 1-shot and 5-shot versions of our model in four different settings:
Caltech-UCSD Birds (CUB)
(Wah et al., 2011), mini-Imagenet with the split proposed by Ravi &
Larochelle (2017), as well as two cross-domain transfer tasks.
The ﬁrst transfer task entails training on mini-ImageNet and testing on CUB, and the second measures transfer from Omniglot (Lake et al., 2011) to EMNIST (Cohen et al., 2017).
Experimental details and an overview of the baselines we used can be found in Section G. Classiﬁcation results are shown in Table 1 and 2.
We ﬁnd that our proposed P ´ olya-Gamma OVE GPs yield strong classiﬁcation results, outperforming the baselines in ﬁve of the eight scenarios.
[Table]
[Table]
U NCERTAINTY Q UANTIFICATION THROUGH C ALIBRATION
We next turn to uncertainty quantiﬁcation, an important concern for few-shot classiﬁers.
When used in safety-critical applications such as medical diagnosis, it is important for a machine learning system to defer when there is not enough evidence to make a decision.
Even in non-critical applications, precise uncertainty quantiﬁcation helps practitioners in the few-shot setting determine when a class has an adequate amount of labeled data or when more labels are required
[Table]
and can facilitate active learning.
[Table]
We chose several commonly used metrics for calibration.
Expected calibration error (ECE) (Guo et al., 2017) measures the expected binned difference between conﬁdence and accuracy.
Maximum calibration error (MCE) is similar to ECE but measures maximum difference instead of expected difference.
Brier score (BRI) (Brier, 1950) is a proper scoring rule computed as the squared error between the output probabilities and the one-hot label.
For a recent perspective on metrics for uncertainty evaluation, please refer to Ovadia et al.
The results for representative approaches on 5-shot, 5-way CUB can be found in Figure 1.
Our OVE PG GPs are the best calibrated overall across the metrics.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
5. R OBUSTNESS TO I NPUT N OISE
Input examples for novel classes in FSC may have been collected under conditions that do not match those observed at training time.
For example, labeled support images in a medical diagnosis application may come from a different hospital than the training set.
To mimic a simpliﬁed version of this scenario, we investigate robustness to input noise.
We used the Imagecorruptions package (Michaelis et al., 2019) to apply Gaussian noise, impulse noise
[Figure]
and defocus blur to both the support set and query sets of episodes at test time and evaluated both accuracy and calibration.
We used corruption severity of 5 (severe) and evaluated across 1,000 randomly generated tasks on the three datasets involving natural images.
The robustness results for Gaussian noise are shown in Figure 2.
Full quantitative results tables for each noise type may be found in Section J. We ﬁnd that in general
Bayesian approaches tend to be robust due to their ability to marginalize over hypotheses consistent with the support labels.
Our approach is one of the top performing methods across all settings.
[Figure]
5. O UT - OF -E PISODE D ETECTION
Finally
[Figure]
we measure performance on out-of-episode detection, another application in which uncertainty quantiﬁcation is important.
In this experiment, we used 5-way, 5-shot support sets at test time but incorporated out-of-episode examples into the query set.
Each episode had 150 query examples: 15 from each of 5 randomly chosen in-episode classes and 15 from each of 5 randomly chosen outof-episode classes.
We then computed the AUROC of binary outlier detection using the negative of the maximum logit as the score.
Intuitively, if none of the support classes assign a high logit to the example, it can be classiﬁed as an outlier.
The results are shown in Figure 3.
Our approach generally performs the best across the datasets.
[Figure]
C ONCLUSION
In this work, we have proposed a Bayesian few-shot classiﬁcation approach based on Gaussian processes.
Our method replaces the ordinary softmax likelihood with a one-vs-each pairwise composite likelihood and applies P´olya-Gamma augmentation to perform inference.
This allows us to model class logits directly as function values and efﬁciently marginalize over uncertainty in each few-shot episode.
Modeling functions directly enables our approach to avoid the dependence on model size that posterior inference in weight-space based models inherently have.
Our approach compares favorably to baseline FSC methods under a variety of dataset and shot conﬁgurations, including dataset transfer.
We also demonstrate strong uncertainty quantiﬁcation, robustness to input noise, and out-of-episode detection.
We believe that Bayesian modeling is a powerful tool for handling uncertainty and hope that our work will lead to broader adoption of efﬁcient Bayesian inference in the few-shot scenario.
A CKNOWLEDGMENTS
We would like to thank Ryan Adams, Ethan Fetaya, Mike Mozer, Eleni Triantaﬁllou, Kuan-Chieh
Wang and Max Welling for helpful discussions.
JS also thanks SK T-Brain for supporting him on an internship that led to precursors of some ideas in this paper.
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through
CIFAR and companies sponsoring the Vector Institute ( https://www.vectorinstitute.
ai/partners ).
This project is supported by NSERC and the Intelligence Advanced Research
Projects Activity (IARPA) via Department of Interior/Interior Business Center (DoI/IBC) contract number D16PC00003.
The U.S. Government is authorized to reproduce and distribute reprints for
Governmental purposes notwithstanding any copyright annotation thereon.
Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies or endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U
S. Government.
A D ERIVATION OF P ´ OLYA -G AMMA A UGMENTED L OGISTIC L IKELIHOOD
In this section, we show the derivation for the augmented logistic likelihood presented in Section 3.1.
First, recall the logistic likelihood:
[Equation]
where σ ( · ) is the logistic sigmoid function.
We have a Gaussian prior p ( ψ ) =
N ( ψ | µ , Σ ) and introduce P´olya-Gamma auxiliary random variables ω to the likelihood such that the original model is recovered when ω is marginalized out: p ( y | ψ ) =
(cid:82) p ( ω ) p ( y | ψ , ω ) d ω .
The P´olya-Gamma distribution ω ∼ PG ( b, c ) can be written as an inﬁnite convolution of Gamma distributions:
[Equation]
The following integral identity holds for b > 0 :
[Equation]
where κ = a − b/ 2 and ω ∼ PG ( b, 0) .
Speciﬁcally, when a = y and b = 1 , we recover an individual term of the logistic likelihood (16):
[Equation] [Equation]
where κ = y − 1 / 2 and ω ∼ PG (1 , 0) .
Conditioned on ω , the batch likelihood is proportional to a diagonal Gaussian:
[Equation]
where κ i =
y i − 1 / 2 and Ω = diag ( ω ) .
The conditional distribution over ψ given y and ω is now tractable:
[Equation]
where ˜ Σ
=
( Σ − 1 + Ω ) − 1 .
B E FFICIENT G IBBS S AMPLING
The Gibbs conditional distribution over f is given by:
[Equation]
where ˜ Σ
= ( K − 1 + A (cid:62) ΩA ) − 1 .
Naively sampling from this distribution requires O ( C 3 N 3 ) computation since ˜ Σ is a CN × CN matrix.
Here we describe a method for sampling from this distribution that requires O ( CN 3 ) computation instead.
First, we note that (22) can be interpreted as the conditional distribution p ( f | z
=
Ω − 1 κ ) resulting from the following marginal distribution p ( f ) and conditional p ( z | f ) :
[Equation]
where we have made implicit the dependence on X , Y , and ω for brevity of notation.
Equivalently, the distribution over f and z can be represented by the partitioned Gaussian
[Equation]
The conditional distribution p ( f | z ) is given as:
[Equation] [Equation]
where ˜ Σ
= ( K − 1 + A (cid:62) ΩA ) − 1 .
Note that p ( f | z =
Ω − 1 κ ) recovers our desired Gibbs conditional distribution from (22).
An efﬁcient approach to conditional Gaussian sampling is due to Hoffman & Ribak (1991) and described in greater clarity by Doucet (2010).
The procedure is as follows:
Sample f 0 ∼ p ( f ) and z 0 ∼ p ( z | f ) .
Return ¯ f = f 0 + KA (cid:62) (
[Equation]
(cid:62) + Ω − 1 ) − 1 ( Ω − 1 κ − z 0 ) as the sample from p ( f | z ) .
K is block diagonal and thus sampling from p ( f ) requires O ( CN 3 ) time.
Af can be computed in
O ( CN ) time, since each entry is the difference between f y i i and
f ci for some i and c .
Overall, step 1 requires O ( CN 3 ) time.
We now show how to compute ¯ f from step 2 in O ( CN 3 ) time.
We ﬁrst expand ( AKA (cid:62) + Ω − 1 ) − 1 :
[Equation] [Equation]
We substitute into the expression for ¯ f :
[Equation]
where we have deﬁned v (cid:44) A (cid:62) Ω ( Ω − 1 κ − z 0 ) .
Now let d (cid:44) ( d 11 , . . .
, d 1 N , d 21 , . . .
, d 2 N , . . .
, d C 1 , . . .
, d CN ) (cid:62) , where d ci =
Y ic (cid:80) c (cid:48)
ω c (cid:48)
i .
Deﬁne Y †
to be the CN × N matrix produced by vertically stacking diag ( Y · c ) , and let W † be the CN × N matrix produced by vertically stacking diag (( ω c 1 , . . .
, ω cN ) (cid:62) ) .
A (cid:62) ΩA may then be written as follows:
[Equation]
, where
[Equation] [Equation]
[Equation]
Substituting (31) into (30):
[Equation]
Now we expand ( K − 1 + D − SPS (cid:62) ) − 1 :
[Equation]
ES
[Equation]
(36) where E =
( K − 1 + D ) − 1 =
K ( K + D − 1 ) − 1 D − 1 is a block-diagonal matrix that can be computed in O ( CN 3 ) time, since D is diagonal and K is block diagonal.
Now, substituting (36) back into (35),
[Equation]
Note that ( S (cid:62) ES − P − 1 ) − 1 is a 2 N × 2 N matrix and thus can be inverted in O ( N 3 ) time.
The overall complexity is therefore O ( CN 3 ) .
C M ARGINAL L IKELIHOOD AND P REDICTIVE L IKELIHOOD O BJECTIVES
Marginal Likelihood (ML).
The log marginal likelihood can be written as follows:
[Equation]
The gradient of the log marginal likelihood can be estimated by posterior samples ω ∼ p θ ( ω | X , Y ) .
In practice, we use a stochastic training objective based on samples of ω from Gibbs chains.
We use
Fisher’s identity (Douc et al., 2014) to derive the following gradient estimator:
[Equation]
) where ω (1) , . .
ω ( M ) are samples from the posterior Gibbs chain.
As suggested by Patacchiola et al.
(2020), who applied GPs to FSC via least-squares classiﬁcation, we merge the support and query sets during learning to take full advantage of the available data within each episode.
Predictive Likelihood (PL).
The log predictive likelihood for a query example x ∗ is:
[Equation]
We use an approximate gradient estimator again based on posterior samples of ω :
[Equation]
We note that this is not an unbiased estimator of the gradient, but ﬁnd it works well in practice.
D L EARNING A LGORITHM
Our learning algorithm for both marginal and predictive likelihood is summarized in Algorithm 1.
Algorithm 1 One-vs-Each P ´ olya-Gamma GP Learning
Input: Objective L ∈ { L ML , L PL } , Task distribution T , number of parallel Gibbs chains M , number of steps T , learning rate η .
Initialize hyperparameters θ randomly.
repeat
Sample S =
( X , Y ) , Q =
( X ∗ , Y ∗ ) ∼ T
if L = L ML then
X ← X ∪ X ∗ , Y ← Y ∪ Y ∗ end if
A ← OVE-MATRIX ( Y ) for m = 1 to M do ω ( m )∼ PG (1 , 0) , f ( m )∼ p θ ( f | X ) for t = 1 to T do ψ ( m ) t ← Af ( m ) t −ω ( m ) t ∼ PG
[Equation]
, ψ ( m ) t ) f
( m ) t ∼ p θ ( f | X , Y , ω ( m ) t ) end for end for if L =
L ML then θ
[Equation]
else
[Equation]
←
[Equation]
end if until convergence
E P OSTERIOR P REDICTIVE D ISTRIBUTION
The posterior predictive distribution for a query example x ∗ conditioned on ω is:
[Equation] [Equation]
where f ∗ are the query example’s logits.
The predictive distribution over f ∗ can be obtained by noting that ψ and the query logits are jointly Gaussian:
[Equation]
where K ∗ is the NC × C block diagonal matrix with blocks K θ ( X , x ∗ ) and K ∗∗ is the C × C diagonal matrix with diagonal entries
k θ ( x ∗ , x ∗ ) .
The predictive distribution becomes:
[Equation]
, Σ
[Equation]
) , where
[Equation]
With p ( f ∗ | x ∗ , X , Y , ω ) in hand, the integral in (42) can easily be computed numerically for each class c by forming the corresponding OVE linear transformation matrix A c and then performing 1D
Gaussian-Hermite quadrature on each dimension of N ( ψ c ∗ | A c µ ∗ , A c Σ ∗ A c (cid:62) ) .
F D ETAILED C OMPARISON OF L IKELIHOODS
In this section we seek to better understand the behaviors of the softmax, OVE, logistic softmax, and Gaussian likelihoods for classiﬁcation.
For convenience, we summarize the forms of these likelihoods in Table 3.
[Table]
[Equation]
Logistic
[Equation]
-vs-Each (OVE)
[Equation]
F. H ISTOGRAM OF C ONFIDENCES
We sampled logits from f c ∼ N (0 , 1) and plotted a histogram and kernel density estimate of the maximum output probability max c p ( y = c | f ) for each of the likelihoods shown in Table 3, where
C = 5 .
The results are shown in Figure 4.
Logistic softmax is a priori underconﬁdent: it puts little probability mass on conﬁdence above 0.4.
This may be due to the use of the sigmoid function which squashes large values of f .
Gaussian likelihood and OVE are a priori overconﬁdent in that they put a large amount of probability mass on conﬁdent outputs.
Note that this is not a complete explanation, because GP hyperparameters such as the prior mean or Gaussian likelihood variance may be able to compensate for these imperfections to some degree.
Indeed, we found it helpful to learn a constant mean for the logistic softmax likelihood, as mentioned in Section G.2.
F. L IKELIHOOD V ISUALIZATION
In order to visualize the various likelihoods under consideration, we consider a trivial classiﬁcation task with a single observed example.
We assume that there are three classes ( C = 3 ) and the single example belongs to the ﬁrst class ( y = 1 ).
We place the following prior on f = ( f 1 , f 2 , f 3 ) (cid:62) :
[Equation]
[Equation]
0 10 0(cid:35)(cid:33) .
[Figure]
In other words the prior for f 1 and f 2 is a standard normal and f 3 is clamped at zero (for ease of visualization).
The likelihoods are plotted in Figure 5 and the corresponding posteriors are plotted in Figure 6.
[Figure]
[Figure]
F. 2D I RIS E XPERIMENTS
We also conducted experiments on a 2D version of the Iris dataset (Fisher, 1936), which contains 150 examples across 3 classes.
The ﬁrst two features of the dataset were retained (sepal length and width).
We used a zero-mean GP prior and an RBF kernel k ( x , x (cid:48) )
=
exp (cid:18) −2 d ( x , x (cid:48) )
(cid:19) , where d ( · , · ) is Euclidean distance.
We considered training set sizes with 1, 2, 3, 4, 5, 10, 15, 20, 25, and 30 examples per class.
For each training set size, we performed GP inference on 200 randomly generated train/test splits and compared the predictions across Gaussian, logistic softmax, and onevs-each likelihoods.
Predictions at a test point x ∗ were made by applying the (normalized) likelihood to the posterior predictive mean ¯ f ∗ .
The predictive probabilities for each likelihood is shown in Figure 7 for a randomly generated train/test split with 30 examples per class.
Test predictive accuracy, Brier score, expected calibration error, and evidence lower bound (ELBO) results across various training set sizes are shown in Figure 8.
The ELBO is computed by treating each likelihood’s posterior q ( f | X , Y ) as an approximation to the softmax posterior p ( f | X , Y ) .
[Equation]
Even though direct computation of the softmax posterior p ( f | X , y ) is intractable, computing the
ELBO is tractable.
A larger ELBO indicates a lower KL divergence to the softmax posterior.
One-vs-Each performs well for accuracy, Brier score, and ELBO across the training set sizes.
Gaussian performs best on expected calibration error through 15 examples per class, beyond which onevs-each is better.
[Figure] [Figure]
G F EW - SHOT E XPERIMENTAL D ETAILS
Here we provide more details about our experimental setup for our few-shot classiﬁcation experiments, which are based on the protocol of (Patacchiola et al., 2020).
G. D ATASETS
We used the four dataset scenarios described below.
The ﬁrst three are the same used by Chen et al.
and the ﬁnal was proposed by Patacchiola et al.
• CUB.
Caltech-UCSD Birds (CUB)
(Wah et al., 2011) consists of 200 classes and 11, images.
A split of 100 training, 50 validation, and 50 test classes was used (Hilliard et al., 2018; Chen et al., 2019).
• mini-Imagenet.
The mini-Imagenet dataset (Vinyals et al., 2016) consists of 100 classes with 600 images per class.
We used the split proposed by Ravi & Larochelle (2017), which has 64 classes for training, 16 for validation, and 20 for test.
• mini-Imagenet → CUB.
This cross-domain transfer scenario takes the training split of mini-Imagenet and the validation & test splits of CUB.
• Omniglot → EMNIST.
We use the same setup as proposed by Patacchiola et al.
Omniglot (Lake et al., 2011) consists of 1,623 classes, each with 20 examples, and is augmented by rotations of 90 degrees to create 6,492 classes, of which 4,114 are used for training.
The EMNIST dataset (Cohen et al., 2017), consisting of 62 classes, is split into 31 training and 31 test classes.
G. F EW - SHOT C LASSIFICATION B ASELINES
Here we explain the few-shot baselines in greater detail.
• Feature Transfer (Chen et al., 2019) involves ﬁrst training an off-line classiﬁer on the training classes and then training a new classiﬁcation layer on the episode.
• Baseline ++ (Chen et al., 2019) is similar to Feature Transfer except it uses a cosine distance module prior to the softmax during ﬁne-tuning.
• Matching Networks (Vinyals et al., 2016) can be viewed as a soft form of k -nearest neighbors that computes attention and sums over the support examples to form a predictive distribution over classes.
• Prototypical Networks
(Snell et al., 2017) computes class means (prototypes) and forms a predictive distribution based on Euclidean distance to the prototypes.
It can be viewed as a Gaussian classiﬁer operating in an embedding space.
• MAML (Finn et al., 2017) performs one or a few steps of gradient descent on the support set and then makes predictions on the query set, backpropagating through the gradient descent procedure.
For this baseline, we simply quote the classiﬁcation accuracy reported by (Patacchiola et al., 2020).
• RelationNet
(Sung et al., 2018) rather than using a predeﬁned distance metric as in Matching Networks or Prototypical Networks instead learns a deep distance metric as the output of a neural network that accepts as input the latent representation of both examples.
It is trained to minimize squared error of output predictions.
• Deep Kernel Transfer (DKT)
(Patacchiola et al., 2020) relies on least squares classiﬁcation (Rifkin & Klautau, 2004) to maintain tractability of Gaussian process posterior inference.
In DKT, a separate binary classiﬁcation task is formed for each class in one-vs-rest fashion by treating labels in {− 1 , +1 } as continuous targets.
We include the results of DKT with the cosine kernel as implemented by Patacchiola et al.
(2020), which is parameterized slightly differently from the version we used in (47):
[Equation]
• Bayesian MAML (Yoon et al., 2018) relies on Stein Variational Gradient Descent (SVGD)
(Liu & Wang, 2016) to get an approximate posterior distribution in weight-space.
We compare to both the non-chaser version, which optimizes cross-entropy of query predictions, and the chaser version, which optimizes mean squared error between the approximate posterior on the support set and the approximate posterior on the merged support & query set.
The non-chaser version is therefore related to predictive likelihood methods and the chaser version is more analogous to the marginal likelihood methods.
For the non-chaser version, we used 20 particles and 1 step of adaptation at both train and test time.
For the chaser version, we also used 20 particles.
At train time, the chaser took 1 step and the leaderadditional step.
At test time, we used 5 steps of adaptation.
Due to the slow performance of this method, we followed the advice of Yoon et al.
and only performed adaptation on the ﬁnal layer of weights, which may help explain the drop in performance relative to MAML.
The authors released Tensorﬂow code for regression only, so we reimplemented this baseline for classiﬁcation in PyTorch.
Amortized Bayesian Meta-Learning (ABML) (Ravi & Beatson, 2019) performs a few steps of Bayes-by-backprop (Blundell et al., 2015) in order to infer a fully-factorized approximate posterior over the weights.
The authors did not release code and so we implemented our own version of ABML in PyTorch.
We found the weighting on the inner and outer KL divergences to be important for achieving good performance.
We took the negative log likelihood to be mean cross entropy and used an inner KL weight of 0.01 and an outer KL weight of 0.001.
These values were arrived upon by doing a small amount of hyperparameter tuning on the Omniglot → EMNIST dataset.
We used α = 1 . 0 and β = 0 .
01 for the Gamma prior over the weights.
We only applied ABML to the weights of the network; the biases were learned as point estimates.
We used 4 steps of adaptation and took 5 samples when computing expectations (using any more than this did not ﬁt into GPU memory).
We used the local reparameterization trick (Kingma et al., 2015) and ﬂipout (Wen et al., 2018) when computing expectations in order to reduce variance.
In order to match the architecture used by Ravi & Beatson (2019), we trained this baseline with 32 ﬁlters throughout the classiﬁcation network.
We trained each 1-shot ABML model for 800 epochs and each 5-shot ABML model for 600 epochs as the learning had not converged within the epoch limits speciﬁed in Section G.3.
• Logistic Softmax GP (Galy-Fajou et al., 2020) is the multi-class Gaussian process classiﬁcation method that relies on the logistic softmax likelihood.
Galy-Fajou et al.
(2020) did not consider few-shot, but we use the same objectives described in Section 4.4 to adapt this method to FSC.
In addition, we used the cosine kernel (see Section H for a description) that we found to work best with our OVE PG GPs.
For this method, we found it important to learn a constant mean function (rather than a zero mean) in order to improve calibration.
G. T RAINING D ETAILS
All methods employed the commonly-used Conv4 architecture (Vinyals et al., 2016)
(see Tablefor a detailed speciﬁcation), except ABML which used 32 ﬁlters throughout.
All of our experiments used the Adam (Kingma & Ba, 2015) optimizer with learning rate 10 − 3 .
During training, all models used epochs consisting of 100 randomly sampled episodes.
A single gradient descent step on the encoder network and relevant hyperparameters is made per episode.
All 1-shot models are trained for 600 epochs and 5-shot models are trained for 400 epochs, except for ABML which was trained for an extra 200 epochs.
Each episode contained 5 classes (5-way) and 16 query examples.
At test time, 15 query examples are used for each episode.
Early stopping was performed by monitoring accuracy on the validation set.
The validation set was not used for retraining.
We train both marginal likelihood and predictive likelihood versions of our models.
For P´olyaGamma sampling we use the PyP´olyaGamma package 3 .
During training, we use a single step of
Gibbs ( T =1).
For evaluation, we run until T = 50 .
In both training and evaluation, we use M =parallel Gibbs chains to reduce variance.
[Table]
H E FFECT OF K ERNEL C HOICE ON C LASSIFICATION A CCURACY
In this section
[Table]
we examine the effect of kernel choice on classiﬁcation accuracy for our proposed
One-vs-Each P ´ olya-Gamma OVE GPs.
Cosine Kernel.
In the main paper, we showed results for the following kernel, which we refer to as the “cosine” kernel due to its resemblance to cosine similarity:
[Equation]
[Equation]
where g θ ( · ) is a deep neural network that outputs a ﬁxed-dimensional encoded representation of the input and α is the scalar log output scale.
Both θ and α are considered hyperparameters and learned simultaneously as shown in Algorithm 1.
We found that this kernel works well for a range of datasets and shot settings.
We note that the use of cosine similarity is reminiscent of the approach taken by
Baseline ++ method of (Chen et al., 2019), which computes the softmax over cosine similarity to class weights.
Here we consider three additional kernels: linear, RBF, and normalized RBF.
Linear Kernel.
The linear kernel is deﬁned as follows:
[Equation]
(48) where D is the output dimensionality of g θ ( x ) .
We apply this dimensionality scaling because the dot product between g θ ( x ) and g θ ( x (cid:48) ) may be large depending on D .
RBF Kernel.
The RBF (also known as squared exponential) kernel can be deﬁned as follows:
[Equation]
where (cid:96) is the log lengthscale parameter (as with α , we learn the (cid:96) alongside θ ).
Normalized RBF Kernel.
Finally, we consider a normalized RBF kernel similar in spirit to the cosine kernel:
[Equation]
The results of our P´olya-Gamma OVE GPs with different kernels can be found in Tablesand 6.
In general, we ﬁnd that the cosine kernel works best overall, with the exception of
Omniglot → EMNIST, where RBF does best.
[Table] [Table]
I A DDITIONAL C ALIBRATION R ESULTS
In Figure
[Table]
we include calibration results for mini-Imagenet and Omniglot → EMNIST.
They follow similar trends to the results presented in Section 5.
[Table]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
J Q UANTITATIVE R OBUSTNESS TO I NPUT N OISE R ESULTS
In this section we include quantitative results for the robustness to input noise results presented in
Figure 2.
Results for Gaussian noise are shown in Table 7, impulse noise in Table 8, and defocus blur in Table 9.
[Table]
[Table] [Table]