T ASK C ALIBRATION FOR D ISTRIBUTIONAL U NCERTAINTY IN F EW -S HOT C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
As numerous meta-learning algorithms improve performance when solving fewshot classiﬁcation problems for practical applications, accurate prediction of uncertainty, though challenging, has been considered essential.
In this study, we contemplate modeling uncertainty in a few-shot classiﬁcation framework and propose a straightforward method that appropriately predicts task uncertainty.
We suppose that the random sampling of tasks can generate those in which it may be hard for the model to infer the queries from the support examples.
Speciﬁcally, measuring the distributional mismatch between support and query sets via class-wise similarities, we propose novel meta-training that lets the model predict with careful conﬁdence.
Moreover, our method is algorithm-agnostic and readily expanded to include a range of meta-learning models.
Through extensive experiments including dataset shift, we present that our training strategy helps the model avoid being indiscriminately conﬁdent, and thereby, produce calibrated classiﬁcation results without the loss of accuracy.
I NTRODUCTION
With the great success of deep learning over the last decade, there has been growing interest in investigating methods that are more intuitive to mimic human intelligence.
One of the desirable characteristics of human cognition is the ability to learn new information quickly.
Few-shot learning is a problem that requires machines to learn with a few examples, and it is usually solved by metalearning algorithms.
Through training over a number of few-shot tasks, the meta-learning model learns to acclimate rapidly to new data and perform desired tasks with the help of prior across-task knowledge.
Noteworthy approaches include learning metric space (Koch et al., 2015;
Snell et al., 2017; Sung et al., 2018), learning update rule (Ravi & Larochelle, 2017; Andrychowicz et al., 2016), or learning an initialization (Maclaurin et al., 2015; Finn et al., 2017).
Meanwhile, calibration is critical in real-life because the model should correctly inform humans or other models of its degree of uncertainty.
Misplaced conﬁdence or overconﬁdence of a deep network can result in dramatically different outcomes in the decision-making process, such as during autonomous driving (Helldin et al., 2013) or medical diagnoses (Cabitza et al., 2017).
Calibration is even more crucial in few-shot learning, for given a few numbers of data at hand, the model is likely to put wrong conﬁdence into the predictions of the unknown data.
Also, existing calibration methods, while effective in general classiﬁcation, are not readily applicable to the few-shot learning.
Therefore, we present a novel method to measure task-level uncertainty and exploit it to make a well-calibrated model.
When the model is meta-trained to solve the few-shot classiﬁcation, it generates a task by randomly choosing classes and sampling support and query examples of corresponding classes.
However, we suppose in this way a variety of tasks can be generated, and there are some tasks in which it may be hard for the model to infer the queries from the support examples.
Common meta-learning approaches force the model to learn anyway, lacking the discussion about task generation.
In this study, we design an algorithm-agnostic calibration method for a few-shot classiﬁcation model by diminishing the learning signal from those ill-deﬁned tasks.
This novel training lets the model predict with careful conﬁdence, ultimately obtaining better calibration ability.
To summarize, our contributions are as follows:
[Figure]
While Task 1 comprises well-distributed D spt and D qry , Taskholds a signiﬁcant support-query discrepancy.
We limit the feedback ﬂow from Task 2.
• We propose a simple and straightforward method by which a few-shot classiﬁcation model learns the ill-deﬁned tasks in a timid way, so as not to place the wrong conﬁdence.
• We extend our approach from an optimization-based to a metric-based framework, conﬁrming that our method is versatile and can be easily expanded to include a range of metalearning models.
• We demonstrate that we can design an uncertainty-aware model that is suitable for calibration and robustness.
To the best of our knowledge, there has been a work that used weighted meta-update in reinforcement learning literature (Xu et al., 2019) to prioritize and give different weights for each trajectory.
However, our work is the ﬁrst one that quantiﬁed task uncertainty and applied it in the few-shot classiﬁcation framework.
Moreover, we validate our work through a dataset shift experiment, which is set to be more challenging and hardly ever addressed due to the degradation of the estimation ability (Snoek et al., 2019).
P RELIMINARIES 2. P ROBLEM S ETUP
A meta-learning problem is composed of tasks from a task distribution p ( τ ) , and for each metatraining task τ ∼ p ( τ ) there is a training set and test set.
We denote each as a support set, D ( τ ) spt and query set, D ( τ ) qry , respectively.
In typical few-shot classiﬁcation scenarios, each task is comprised of data from N randomly selected classes and k support examples per class, with a total of kN examples, { ( x ( τ ) s , y ( τ ) s ) } kNs =1 :=
D ( τ ) spt .
This is called N -way k -shot problem.
Also, there is a disjoint query set, { ( x ( τ ) q , y ( τ ) q ) } k (cid:48)
N q =1 := D ( τ ) qry .
Meta-training is an iterative process of optimization on D qry after being seen D spt .
This setup resembles learning a novel task with only a few examples of training data.
The adaptation on D spt corresponds to training with the novel task data.
Given a few examples, meta-learner is expected to perform predictions on unseen data, which in our setup is D qry .
That is, the meta-learner evolves by acquiring general information across the tasks by optimizing on D qry .
2. U NCERTAINTY IN F EW -S HOT C LASSIFICATION
A good meta learner should not only perform well on given tasks but should also make reasonable decisions based on aleatoric or epistemic uncertainty (Der Kiureghian & Ditlevsen, 2009).
While aleatory is irreducible uncertainty from data, including class overlap, observation noise, or data ambiguity, episteme arises from the stochasticity of the model itself, and can be reduced by obtaining sufﬁcient data.
For example, aleatory is the case when ‘dog’ images have inherently high noise (e.g., blurred and corrupted), and thus, have the appearance of other class such as ‘cat’ (known-unknown).
Episteme arises when the model is not sufﬁciently trained for it to match the images to ‘dog’.
There is another source, distributional uncertainty, which is ascribed to a mismatch between training and test data (Malinin & Gales, 2018).
In few-shot classiﬁcation, this corresponds to the discrepancy between support and query set, often caused by intra-class variations (Chen et al., 2019).
For example, distributional uncertainty may arise when the ‘dog’ class includes support examples as ‘bulldogs’ while by chance, the query is ‘chihuahua’, resulting in large uncertainty of mapping ‘chihuahua’ to any class (unknown-unknown).
Typical meta-learning algorithms do not consider this mismatch, which we call support-query discrepancy, and na¨ıvely adapt to every observed support example.
Appendix A provides the formalized description of distributional uncertainty and its relation to the few-shot learning.
In this work, we propose a straightforward method of training an uncertainty-aware model (Figure 1), where we measure the distributional mismatch within a task and constrain the feedback from uncertain tasks.
T ASK C ALIBRATION M ETHOD
Despite the importance of distributional uncertainty modeling, the problem setup, which restricts the number of data, renders the model extremely sensitive to the distribution of given data.
Therefore, we focus on task-level distributional uncertainty instead of considering that of individual data and models.
We suggest that this can be implemented by a non-Bayesian approach with an appropriate measure.
We propose task calibration model-agnostic meta-learning (TCMAML), a variant of MAML (Finn et al., 2017), that can estimate task distributional uncertainty during meta-training and utilize it to modulate the optimization of global parameters.
Thereafter, we extend our approach to the metric-based learning model.
3. B RIEF R EVIEW OF MAML
Before introducing our new approach, we provide a brief review of model-agnostic meta-learning (MAML)
(Finn et al., 2017) to promote readers’ understanding.
MAML learns a proper initialization of globally shared θ that is quickly adjusted to each task τ
by a few gradient descent steps.
After being adapted to support examples, MAML updates θ by optimizing on the query set, minimizing task-speciﬁc loss L τ with a task-adapted parameter φ τ .
This is summarized as the bi-level optimization.
[Equation]
The meta-update (2) aggregates gradients over multiple tasks, so that the tasks are learned simultaneously.
If we can measure the task uncertainty, adaptive weighting of task-speciﬁc losses is possible, i.e., L = (cid:80) T τ =1 w τ L τ ( φ τ ) where T is the number of tasks in a meta-batch.
Each weight reﬂects the uncertainty of a speciﬁc task.
If a task τ has a large uncertainty component, the small value of w τ will decrease the contribution of ∇ θ L τ .
3. TCMAML
We suggest that class-wise similarity would be a good metric to estimate task uncertainty.
As the similarity between classes increases, it becomes harder to distinguish one from another.
The similarity is measured in the embedding space mapped by h φ τ , the feature extractor before the classiﬁer.
With the pairs of query inputs and labels ( x ( τ ) q , y ( τ ) q ) , we collect the average features for every class c , h φ τ ,c := 1 k (cid:48) (cid:80) { q : y ( τ ) q = c } h φ τ ( x ( τ ) q ) , where h maintains feature dimensions.
h φ τ contains the information about the support and query data because the average features of the query inputs are parameterized by the task parameter
φ τ , while φ τ is derived from the adaptation on the support data as (1).
Therefore, the relations between support and query data determines h φ τ .
We will go deeper in the next section.
We now deﬁne the cosine similarity matrix C τ , as for all i, j ∈ { 1 , · · · , N } ,
[Equation]
(3) calculates how correlated the two class embeddings are.
The diagonal line of C τ is always 1, and the rest of the elements are between [ − 1 , 1] .
To map the similarity matrix into a scalar, we use
Frobenius norm.
For C ∈ R N × N , the Frobenius norm of the matrix is (cid:107)
In few-shot classiﬁcation, it is important to carefully consider the task generation.
More speciﬁcally, we prefer to sample the tasks considering how well the task is deﬁned and how much it is reliable to train with.
The original meta-loss in (2) assumes equal reliability for uniformly sampled tasks; p ( τ ) = 1 T , ∀ τ .
Let q ( τ ) be our desired task generating distribution.
There is no straightforward way to sample from q ( τ ) , but we can alternatively use importance sampling.
[Equation]
Now if we sample the tasks randomly from p ( τ ) and compute the importance weight q ( τ ) , we can easily get the adaptive empirical risk, which allows training with the prior knowledge of task conﬁguration.
TCMAML estimates q ( τ ) with a softmax function over s τ values.
Consequently, we modify the meta-update rule (2) as follows:
[Equation]
where T is a scaling factor.
Designing the importance weight as above implies large weight on the well-deﬁned tasks, where the measured class-wise similarity is low, and small weight on the ill-deﬁned tasks, where the similarity is high.
While the softmax function is an exponential scaling, linear scaling is also possible; w τ = (1 /s τ ) / (cid:80) T τ =1 (1 /s τ ) .
However, we empirically found that the exponential scaling better performs.
Refer to Appendix B.2 for linear scaling TCMAML.
3. R ELATION WITH D ISTRIBUTIONAL U NCERTAINTY
The meta-update (5) indicates that for an uncertain task, a small weight value is multiplied, so that its contribution to the updates of θ is weak.
In this sense, TCMAML considers task distributional uncertainty by measuring the similarity between the query examples.
If we assume L as a crossentropy loss, φ τ from task adaptation (1) is an approximation of the maximum likelihood estimate with observed support examples.
[Equation]
interval general few-shot tasks
Poorly deﬁned indicates extremely large support-query discrepancy; images from mismatched classes, and Well-deﬁned imitates τ ∗ .
More details in Appendix B.4.
Let φ τ ∗ be the parameter from a task where there is no distributional uncertainty.
This is satisﬁed when D ( τ ∗ ) spt has exactly the same distribution as D ( τ ∗ ) qry .
We aim to maximize p ( D ( τ ∗ ) qry | θ ) in meta-learning, which is attained when θ = φ τ ∗ , from eq.(6) and D ( τ ∗ )
spt ≈ D ( τ ∗ ) qry .
We can assume that to make a clear distinction between different classes the embedded representation vectors should be less correlated.
Therefore in task τ ∗ , the query inputs’ embedded vectors, h φ τ ∗ ,c , are less correlated because φ τ ∗ optimally distinguishes D ( τ ∗ ) qry .
In contrast, if there is a signiﬁcant distributional mismatch between D ( τ ) spt and D ( τ ) qry , φ τ is the suboptimal solution to maximize p ( D ( τ ) qry | θ ) , because the parameters cannot infer D ( τ ) qry from D ( τ ) spt .
In the previous example, the parameters adapted from ”bulldog” images might not separately distinguish ”chihuahua” from another class such as ”cat”.
This implies higher correlation between the embedded vectors h φ τ ,c .
Therefore, we can compare the class-wise cosine similarity, S C , computed using the average features in task τ ∗ and τ .
[Equation]
From (3) and (5), the above inequality leads to smaller weights on the loss function of uncertain tasks.
For a general task τ , a signiﬁcant mismatch between the distribution of D ( τ ) spt and D ( τ ) qry increases the inequality gap.
Figure 2 shows the similarity scores of tasks from different distributions with real-world images.
In well-deﬁned tasks as τ ∗ , the adapted model is good at distinguishing the query representations because they are less correlated.
On the other hand, in the extremely discrepant case where the classes of support images do not correspond to those of query images, between-class similarities are high because the model cannot infer the query set from observing the support set.
General fewshot classiﬁcation tasks lie between these well-deﬁned and poorly-deﬁned ones.
TCMAML is thus trained by imposing a prior regarding whether a given input task has high distributional uncertainty or not.
3. E XTENSION TO THE M ETRIC -B ASED M ODEL
We emphasize that our task calibration approach can be easily extended to a metric-based approach.
Unlike optimization-based models such as MAML, a metric-based model does not require a process of explicit adaptation to the task.
Instead, the model calculates the distances between the query and support data in a non-parametric way.
In this study, we use prototypical network, i.e., ProtoNet (Snell et al., 2017), which is a high-performance metric-based few-shot learning method, and propose and apply our task-calibrated version, TCProtoNet.
Using the distance vector of each query to the support prototypes p
= [ p 1 , . .
, p N ] (cid:62) , we collect the average vectors for each class, d φ τ ,c := 1 k (cid:48) (cid:80) { q : y ( τ ) q = c } d ( x ( τ ) q , p ) , where ( x ( τ ) q , y ( τ ) q ) ∈ D ( τ ) qry and d ( ·
, p ) ∈ R N + hold.
The assembly of a similarity matrix using the average vectors and the weight computation are effected similarly as in section 3.2.
We empirically demonstrate that TCProtoNet performs better calibration than vanilla ProtoNet.
The result implies that in a range of few-shot learning models, our method is versatile and applicable to more recent methods such as task dependent adaptive metric, TADAM (Oreshkin et al., 2018).
R ELATED W ORK
Uncertainty estimation in deep learning has been persistently studied despite the challenges (Gal, 2016; Lakshminarayanan et al., 2017; Guo et al., 2017; Jiang et al., 2018).
Meanwhile, Depeweg et al.
(2017) dealt with the decomposition of uncertainty in Bayesian neural networks to better understand and identify their characteristics.
Our work also investigates different sources of uncertainty, initially being inspired by Malinin & Gales (2018), which builds Dirichlet Prior Network to explicitly model distributional uncertainty.
We extend their argument to a few-shot learning framework without requiring any additional parameters or networks.
In parallel, Sensoy et al.
(2018) parameterized Dirichlet using the evidence vector to regularize predictive distribution, to penalize divergence from the uncertain state.
Few-shot image classiﬁcation is one of the most challenging tasks for coping with high degrees of uncertainty (Fe-Fei et al., 2003).
Estimation and judgment over the uncertainty become challenging when encountered with high-variance data such as a large-scale image set.
To overcome this issue, several recent studies have relied on designing Bayesian neural networks.
These networks use posterior approximation to quantify parameter uncertainty, which is a type of epistemic uncertainty.
Probabilistic MAML (Finn et al., 2018) is a variant of MAML (Finn et al., 2017), which approximates the distribution over model parameters by optimizing the variational lower bound.
Bayesian
MAML (Kim et al., 2018) uses the Stein variational gradient descent for more ﬂexible modeling of posterior.
Ravi & Beatson (2019) and Gordon et al.
both amortize approximate inference.
Neural Processes (Garnelo et al., 2018) are a combination of a stochastic process and neural network that learns the distribution of functions while representing uncertainty, although they are yet to be proven capable of solving large-scale classiﬁcation problems.
The Bayesian models must compute posterior inference, and most approaches use empirical approximation, which usually requires heavy computations.
Conversely, our method uses a non-Bayesian approach, which is computationally advantageous and does not rely on posterior approximation.
The TCMAML running time is almost the same as that for vanilla MAML.
E XPERIMENTS 5. E XPERIMENTAL D ETAILS
We mainly used two different datasets for the experiments: mini -ImageNet and CUB-200-2011.
mini -ImageNet (Vinyals et al., 2016) dataset contains 100 classes and 600 images per class.
We followed the standard protocol which was initially provided by Ravi & Larochelle (2017), splitting (train, validation, test) set to (64, 16, 20) non-overlapped classes.
CUB-200-2011 (Wah et al., 2011) dataset is composed of 200 bird species classes incorporating 11,788 ﬁne-grained images.
According to Hilliard et al.
(2018), it is divided into (100, 50, 50) classes for (train, validation, test) set.
Thanks to their work conducted on fair comparison in the few-shot classiﬁcation setup, most of our implementation details follow Chen et al.
both in 1-shot and 5-shot tasks.
Every accuracy reported is the test accuracy, which was evaluated from 1000 tasks randomly drawn from the test set.
By default, we used a Conv-4 backbone, CNN with four convolutional blocks (Vinyals et al., 2016), to not overestimate the performance by using deeper networks.
Furthermore, MAML and TCMAML were trained with a ﬁrst-order approximation of derivatives for the efﬁciency (Finn et al., 2017).
Dataset Shift.
Additionally, we conducted dataset shift experiments (Chen et al., 2019), in which the train set is the entire mini -ImageNet dataset, and validation and test sets are from CUB-200.
miniImageNet → CUB-200 was implemented with a ResNet-18 backbone (He et al., 2016) for better convergence.
The dataset shift is signiﬁcantly more challenging than a typical few-shot classiﬁcation setting because the model has to predict the examples from completely different distributions to those of the train set.
This is an intriguing situation from a practical perspective since for most real-world problems it cannot be assumed that new tasks would be from among those with which the model is familiar.
However, not much research on uncertainty in few-shot learning has been attempted on shifted datasets.
5. C ALIBRATION R ESULTS
Our main contribution is well-calibrated classiﬁcation.
The model is well-calibrated when it makes its decisions based on the predicted uncertainty.
The correctness of a perfectly calibrated model will match the conﬁdence level (the largest value of the softmax output).
For example, the model is miscalibrated or overconﬁdent when the conﬁdence is 0.8 on average, but only 70 out ofsamples are correct.
The quantiﬁed measures for miscalibration include expected calibration error (ECE) and maximum calibration error (MCE),
[Equation]
where acc( · ) and conf( · ), respectively, are the empirical accuracy and average conﬁdence for each bin.
Here, B m , referred to as bin, is a subset of samples such that the sample’s prediction conﬁdence falls into the m -th partition.
Refer to Guo et al.
(2017) for a detailed derivation.
Table 1 and 2 summarize the error rate and accuracy.
MAML+TS denotes temperature scaling after training MAML.
Temperature scaling (Guo et al., 2017) is one of the calibration methods widely used in general classiﬁcation, which learns a temperature parameter to control the conﬁdence.
It works as post-processing on the validation set.
However, in few-shot learning setup, train, validation, and test set all have non-overlapped classes, and the unseen classes are given in the future tasks.
Existing calibration methods such as temperature scaling ﬁnd the best parameter for the known classes, which is not applicable in few-shot classiﬁcation.
This is why MAML+TS produced somewhat random results; a few results are better than MAML but the rest are severely poor.
In most experiments, TCMAML exhibited a signiﬁcantly decreased error rate than vanilla MAML and ABML, as well as TCProtoNet outperforming ProtoNet.
ABML, amortized Bayesian metalearning (Ravi & Beatson, 2019), is one of the Bayesian methods for few-shot learning and is also based on the MAML algorithm.
Notably, while producing calibrated results, TCMAML and TCProtoNet yielded consistent accuracy without incurring the loss of prediction capability.
On the other hand, ABML produced poor accuracy with high ECE/MCE rate except for the 1-shot mini -ImageNet
[Table] [Table]
case.
Also, TCMAML’s improvement was more signiﬁcant in 1-shot experiments compared to the 5-shot, because 1-shot tasks bring more distributional uncertainty that induces the wrong conﬁdence of a model.
For example, in 5-shot mini -ImageNet, MAML was already well-calibrated and there was no more improvement.
Interestingly, TCProtoNet has a better calibration ability than TCMAML.
MAML has explicit innerlevel optimization, which adapts rapidly to the task, regardless of uncertainty.
ProtoNet, on the other hand, updates the parameters using the distance loss between the support prototypes and query set.
Therefore, ProtoNet actually warps the embedding space towards reducing the support-query discrepancy.
Thus, it can be observed that different learning strategies highly inﬂuence the model’s calibration abilities.
5. T ASK U NCERTAINTY R EFLECTS P REDICTABILITY
One might be curious whether task uncertainty correlates well to the difﬁculty of a given task.
Figure 3 (left) shows that our measure of task uncertainty reﬂects the model’s reliability for making correct predictions.
There is a clear trend of low accuracy when the task has large discrepancy in the support and query distribution.
In other words, distributional uncertainty is strongly related to the classiﬁcation accuracy.
However, the epistemic and aleatoric uncertainty are much more exacting to measure and do not intuitively explain the difﬁculty of a task.
We further studied evaluating only those tasks that were estimated to be well-deﬁned.
In Figure(right), we excluded the outlier tasks of which uncertainty values exceeded the threshold we set.
The baseline accuracy for all 1000 tasks is 76.03%.
From removing approximately half of all tasks (when threshold is 3.5), the test accuracy reached 81.01% (gaining + 4.98%).
[Figure]
M ETA -T RAINING WITH C ORRUPTED T ASKS
Our task calibration method is robust against data corruption.
Meta-training with corrupted data is challenging to the model because it introduces large noise that impedes the accumulation and the transfer of meta-knowledge.
Nevertheless, robustness is crucial in that the model should still learn meta-knowledge, even among those tasks, to be transferred to the target tasks.
To this end, we manipulated the task sampling process in meta-train phase.
Speciﬁcally, with some ﬁxed probability, we corrupted the tasks to include random Gaussian images.
That is, a corrupted task consists of 5 support images from mini -ImageNet train set and 15 query images that have pixel values from N (0 , 1) .
We found out that MAML is already robust against this attack due to the informative feature reuse (Raghu et al., 2019) and overﬁtting to the uncorrupted images; thus, there was no improvement with TCMAML.
On the other hand, metric-based model as ProtoNet is fragile because this type of model learns directly from matching the query examples to the support set.
If the support and query set are heterogeneous, ProtoNet is easily damaged by unwanted updates.
[Table]
When 25% of tasks were corrupted ( p = 0 . 25 ), ProtoNet performance dropped to 61.38%.
Not surprisingly, 50% corruption ( p = 0 . 5 ) led to a further decrease, which is 24.01%p lower than without-corruption training.
TCProtoNet managed to exceed its counterpart, 63.45% and 48.65% in 25% and 50% corruption, respectively.
C ONCLUSION
In this study, we asserted that large-scale few-shot classiﬁcation problems should be examined by carefully estimating uncertainty.
We postulate that the few-shot classiﬁcation contains large discrepancy between the support and query set, although typical meta-learning algorithms do not consider this.
We demonstrated that our suggested class-wise similarity score is related to distributional uncertainty and reﬂects the predictability of a task.
Further, we demonstrated that task calibration can be extended easily to other methods, such as metric-based learning, via TCProtoNet.
The calibration results veriﬁed reliable decision-making without the ability to classify being degraded.
Lastly, we found out the task calibration method acquires robustness to the data corruption during metatraining.
A D ETAILED D ESCRIPTION ABOUT D ISTRIBUTIONAL U NCERTAINTY
Here we use the same few-shot classiﬁcation setup as in the paper.
To address the uncertainty in meta-learning, recent works built Bayesian neural networks to formalize posterior distribution over the task-speciﬁc parameters φ τ , which is an adapted result from D ( τ ) spt and global parameter θ .
The main idea of the Bayesian approach is using the posterior of φ τ as a prior when computing the predictive distribution.
Marginalizing over task-speciﬁc parameters would give good estimates of an output, considering model’s epistemic uncertainty.
[Equation]
Malinin & Gales (2018) extended above equation to explicitly capture distributional uncertainty, via parameterizing a distribution over categorical distribution, p ( µ | x , φ ) where µ stands for the categorical probability; µ = [ µ 1 , · · · , µ N ] (cid:62) =
[ p ( y = 1) , · · · , p ( y = N )] (cid:62) .
Given enough training data we can assume a point estimate for the model parameters, ˆ φ .
Then (9) can be reformulated as:
[Equation]
11) where ˆ φ ∼ p
[Equation]
φ |D spt ,
[Equation]
12) p ( µ | x , ˆ φ ) is a prior for categorical distribution, for example Dirichlet prior,
[Equation]
where α c is a non-negative concentration parameter for class c .
In the case when input has high aleatoric uncertainty, p ( µ | x , ˆ φ ) from (13) must be sharp at the center of the ( N − 1) simplex, being conﬁdent of imposing uncertain predictions.
It is satisﬁed with symmetric Dirichlet where α c s are the same and larger than 1.
Conversely, when there exists high distributional uncertainty, desired Dirichlet prior should be ﬂat , i.e., α = [1 , · · · , 1] (cid:62) .
This is because support and query examples form different distributions, so there is no prior belief of mapping query examples to any prediction, x (cid:55)→ y .
Although Malinin & Gales (2018) and Sensoy et al.
(2018) addressed distributional uncertainty by parameterizing Dirichlet prior over categorical distribution, they trained the model with a strong assumption that the distributional mismatch of data is known in advance.
However, in few-shot learning problem setup, it is unknown whether the uncertain prediction is attributed to the distribution, model, or data itself.
Moreover, every few-shot learning task could potentially involve distributional uncertainty due to the limited number of support examples.
Thus, it is not easy for the model to decide how much of the meta-learned prior experience it should make use to solve a given task.
We suppose it is more crucial to detect the distributional mismatch in few-shot image classiﬁcation, with an assumption that the epistemic uncertainty is relatively lower than the aleatoric or distributional uncertainty.
Since deep networks such as CNN extract highly informative features from the images (Chen et al., 2019; Raghu et al., 2019), uncertainty may not strongly depend on the model’s stochasticity.
While the Bayesian models can explicitly represent parameter uncertainty in simple problems, it cannot estimate distributional mismatch within complex data.
Besides, there is another pitfall.
Although few-shot classiﬁcation constitutes a task with only a small amount of examples, there are in fact large enough data in total and meta-learner is trained under millions of randomly composed tasks.
Enough data typically shrink the model posterior and reduce the episteme .
B E XPERIMENTAL M ATERIALS B. H YPERPARAMETERS
We used a Conv-4 backbone except for dataset shift experiments.
The Conv-4 architecture (Vinyals et al., 2016) stacks 4 blocks that are each comprised of (Convolution + BatchNorm + ReLU + Max-
Pool).
When training, we used data augmentation (RandomResizedCrop, ColorJitter, RandomHorizontalFlip), and each image is resized to the resolution of 84 (224 in the dataset shift).
Meta-training lasts 1600 epochs for 5-shot and 2400 epochs for 1-shot experiments, with 25 episodes per epoch.
One episode samples 4 tasks ( T = 4 ) and each task samples 15 queries per class ( k (cid:48)
= 15 ).
We used Adam optimizer (Kingma & Ba, 2014) with learning rate 5e-4.
For MAML, the learning rate in the inner-loop update is 1e-2 and the number of inner-loop steps is 5.
In addition, computing weights for each task requires a scaling factor, according to (5).
T is a scaling hyperparameter that needs to be tuned, depending on the dataset, type of feature extractor, and the size of a matrix (number of classes), etc.
Note that T → ∞ means equally weighted sum, leading to vanilla MAML.
In our experiments, we searched T from { 1/2, 1, 2 } .
B. L INEAR S CALING TCMAML
In (5), we computed w τ with a softmax over the similarity scores.
While the softmax function is an exponential scaling (ES), normalizing by linear scaling (LS) is also possible.
In this case, we do not need a scaling factor.
[Equation]
Table 4 exhibits the results of linear scailng TCMAML.
However, in most experiments, TCMAML (ES) is comparable with or better than TCMAML (LS).
Depending on the similarity score values, linear scaling may have small weight variations across tasks, and produce similar results as MAML.
[Figure]
[Table]
B. R EIMPLEMENTATION OF ABML
ABML (Ravi & Beatson, 2019) is a Bayesian approach that estimates uncertainty in meta-learning.
We reimplemented the ABML algorithm because 1) they only reported mini -ImageNet 5-way 1shot experiment result and 2) our experimental setup is slightly different from theirs.
Therefore, we implemented ABML based on our setup.
First, we sampled 4 tasks within an episode as we did in the TCMAML experiments.
Second,
Ravi & Beatson (2019) did not use data augmentation, but we used the same dataloaders as other experiments of ours.
This includes the same data augmentation scheme.
Third, we used a ﬁrst-order approximation of derivatives as TCMAML.
For the other setups, such as the number of ensemble networks to train and validate, learning rates, and KL-divergence reweighting factor, we followed the original paper’s default values.
However, we did not implement ABML on dataset shift experiment due to its inefﬁciency of memory and computation.
ABML is a Bayesian method, and this implies that ABML is trained by marginalizing over model parameters.
In implementation, this requires a signiﬁcant computation overhead because it should hold several models and receive their gradients in parallel.
During the metatraining of ABML, 5 ensemble networks work in parallel, and during the meta-testing, 10 ensemble networks are employed.
In Figure 4, TCMAML and ABML show an enormous gap of GPU memory usage, even though they are both MAML-based methods.
We can also ﬁnd that ABML takes a lot of time to compute during the meta-train phase.
Because of ABML’s inefﬁciency, we could not apply it to a ResNet-18 model, which is a backbone for dataset shift experiments.
It exceeds the capacity of a single GPU we own, and also it takes excessive amount of time to be converged.
[Figure]
E XPERIMENTAL S ETUP FOR F IGURE
We now describe how we generated Figure 2 in the paper.
We measured the class-wise similarity scores of 1000 training tasks, where the tasks are sampled from some manipulated distribution.
The dataset we used here is mini -ImageNet, and we followed the 5-way 1-shot MAML experiment protocol.
We describe the way that we manipulated the task sampling distribution.
Standard is a general 1-shot experiment.
For the support set, 1 image is sampled from each class.
For the query set, 15 images are sampled from the corresponding class.
Well-deﬁned tasks are sampled so that there is no distributional uncertainty.
Although it is hard to perfectly get rid of the uncertainty, it is possible to make the distribution of D ( τ ) spt and D ( τ ) qry similar.
To this end, we sample 100 support images per class and corresponding 15 query images.
By changing few-shot tasks into data-rich many-shot tasks, we could simulate on the low uncertainty regime.
Adapting to 100 support images (but we did not change the number of adaptation steps) allows θ to be near optimal to maximize the likelihood of the query set.
Poorly-deﬁned tasks are constructed to make extremely large support-query discrepancy, large distributional uncertainty.
As the Standard setting, a Poorly-deﬁned task samples 1 support image and 15 query images, but they are from different classes.
For instance, the class labels of support images can be { hair slide,carousel,wok,photocopier,jellyfish } , and the class labels of query images can be { ant,vase,dalmatian,school bus,king crab } .
B. T EST A CCURACY FOR T ABLE
In Table 5, we display every test accuracy for CUB-200 and dataset shift experiments.
Note that
ABML poorly performs in CUB-200 5-shot, while it maintained comparable performance in 1-shot.
Also, ProtoNet-based methods outstrip MAML-based methods with large margin in dataset shift experiments.
[Table]
C
Q UALITATIVE E XAMPLE
[Figure]
query set weights 0.
[Figure]
[Figure]
In Figure 5, we depicted an example of weight computation given real images as the input data.
This is a sampled batch of tasks, where the minibatch size is 4, from mini -ImageNet train set.
As it describes, TCMAML gives different weights to each task, regarding the support-query discrepancy.
For example, in Task 1, each query image is easily matched to their corresponding support images.
The computed weight to this well-deﬁned task is 0.2732.
However, Task 4 includes somewhat unclear matching between the support and query images.
In this kind of task, even the adapted model cannot conﬁdently distinguish the query images, resulting in the lower weight, 0.1725.
D R ELIABILITY D IAGRAMS
Reliability diagrams (Niculescu-Mizil & Caruana, 2005) visually present model calibration, plotting empirical accuracy versus average prediction conﬁdence, the largest value of the output distribution.
Figure 6 describes reliability diagrams comparing between MAML, ABML and TCMAML.
We can ﬁnd that TCMAML has its bars closer to the y = x line than MAML and ABML, implying more reliable predictions.
By the way, CUB experiments show slightly different aspects.
MAML is difﬁdent on its predictions, with the empirical accuracy higher than the conﬁdence.
The model which is not sufﬁciently conﬁdent can cause another type of problem.
As opposed to the overconﬁdence, low conﬁdence often occurs when there are ambiguous images with low accuracy.
This situation is vulnerable to adversarial attack or perturbation of images (Chakraborty et al., 2018), readily available to decrease overall accuracy due to the model’s low conﬁdence.
In this case, the model which can accurately tell easy tasks apart is needed.
TCMAML again succeeds in calibration with enough conﬁdence in such tasks.
[Figure]