I NTERVAL B OUND I NTERPOLATION FOR F EW - SHOT L EARNING WITH F EW T ASKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot learning aims to transfer the knowledge acquired from training on a diverse set of tasks to unseen tasks from the same task distribution, with a limited amount of labeled data.
The underlying requirement for effective few-shot generalization is to learn a good representation of the task manifold.
This becomes more difficult when only a limited number of tasks are available for training.
In such a few-task few-shot setting, it is beneficial to explicitly preserve the local neighborhoods from the task manifold and exploit this to generate artificial tasks for training.
To this end, we introduce the notion of interval bounds from the provably robust training literature to few-shot learning.
The interval bounds are used to characterize neighborhoods around the training tasks.
These neighborhoods can then be preserved by minimizing the distance between a task and its respective bounds.
We then use a novel strategy to artificially form new tasks for training by interpolating between the available tasks and their respective interval bounds.
We apply our framework to both model-agnostic meta-learning as well as prototype-based metric-learning paradigms.
The efficacy of our proposed approach is evident from the improved performance on several datasets from diverse domains in comparison to recent methods.
I NTRODUCTION
Few-shot learning problems deal with diverse tasks consisting of subsets of data drawn from the same underlying data manifold along with associated labels. The joint distribution of data and corresponding labels which governs the sampling of such tasks is often called the task distribution (Finn et al., 2017; Yao et al., 2022). Consequently, few-shot learning methods attempt to leverage the knowledge acquired by training on a large pool of such tasks to easily generalize to unseen tasks from the same distribution, using only a few labeled examples.
We hereafter refer to the support of the task distribution as the task manifold which is distinct from but closely-related to the data manifold associated with the data distribution.
Since the unseen tasks are sampled from the same underlying manifold governing the task distribution, we should ideally learn a good representation of the task manifold by preserving the neighborhoods from the high-dimensional manifold in the lower-dimensional feature embedding (Tenenbaum et al., 2000; Roweis & Saul, 2000; Van der Maaten & Hinton, 2008).
However, the labels associated with a task can define any arbitrary partitioning of the data.
Therefore, we can attempt to preserve the neighborhood for a task by simply conserving the neighborhoods for the corresponding subset of the data manifold in the feature embedding learned by the few-shot learner.
This facilitates effective generalization to new tasks using a limited amount of labeled data by only updating the classifier as the learned feature embedding would likely require very little adaptation.
However, existing few-shot learning methods lack an explicit mechanism for achieving this.
Further, real-world few-shot learning scenarios like rare disease detection may not have the large number of training tasks required for effective learning, due to various constraints such as data collection costs, privacy concerns, and/or data availability in newer domains (Yao et al., 2022).
In such scenarios, few-shot learning methods are prone to overfit the training tasks, thus limiting the ability to generalization to unseen tasks.
Therefore, in this work, we develop a method to explicitly constrain the feature embedding in an attempt to preserve neighborhoods from the high-dimensional task manifold and to construct artificial tasks within these neighborhoods in the feature space, to improve the performance when a limited number of training tasks are available.
The proposed approach relies on characterizing the neighborhoods from the high-dimensional task manifold and propagating them through the network with the intent to preserve the task neighborhood in the feature space.
We achieve this by employing the concept of interval bounds from the provably robust training literature (Gowal et al., 2019; Morawiecki et al., 2020), i.e. the axis-aligned bounds for the activations in each layer, obtained using interval arithmetic (Sunaga, 1958).
Concretely, as shown in Figure 1, we first define a small ϵ -neighborhood for each few-shot training task and then use Interval Bound Propagation (IBP; Gowal et al., 2019) to obtain the bounding box around the mapping of the corresponding neighborhood in the feature embedding space.
We then explicitly attempt to preserve the ϵ -neighborhoods by minimizing the distance between a task and its respective interval bounds in addition to optimizing the few-shot classification objective.
We further devise a mechanism to construct the artificial tasks by interpolating between a task and its corresponding IBP bounds.
It is important to notice that this setup is distinct from provably robust training for few-shot learning in that we do not attempt to minimize (or calculate for that matter) the worst-case classification loss. [Figure]
[Figure]
Various methods have been proposed to mitigate the few-task few-shot problem using approaches such as explicit regularization (Jamal & Qi, 2019; Yin et al., 2019), intra-task augmentation (Lee et al., 2020;
Ni et al., 2021; Yao et al., 2021) and inter-task interpolation to construct new artificial tasks (Yao et al., 2022).
While inter-task interpolation has been shown to be the most effective among these existing approaches, it suffers from the limitation that the artificially created tasks may be generated away from the task manifold depending on the curvature of the feature embedding space, as there is no natural way to select pairs of task which are close to each other on the manifold (Figure 2(a)).
The interval bounds obtained using IBP, on the other hand, are likely to be close to the original task embedding as we explicitly minimize the distance between a task and its interval bounds.
Thus, using them for interpolation is likely to keep the generated tasks close to the manifold (Figure 2(b))
In essence, the key contributions made in this article advance the existing literature in the following ways: (1) In Section 4.1, we present for the first time, a novel method to synergize few-shot learning with interval bound propagation (Gowal et al., 2019) so as to explicitly lend the ability to preserve task neighborhoods in the feature embedding space of the few-shot learner.
In Section 4.2, we propose the interval bound–based task interpolation technique which can create new tasks (as opposed to augmenting each individual task (Lee et al., 2020;
Ni et al., 2021; Yao et al., 2021)), by interpolating between a task sampled from the task distribution and its interval bounds.
(3) Unlike existing inter-task interpolation methods that require paired tasks for interpolation (Yao et al., 2022), our framework can generate new tasks from only a single task.
This allows the proposed framework to be seamlessly integrated with existing few-shot learning paradigms.
In Section 5, we empirically demonstrate the effectiveness of our proposed approach on both gradientbased meta-learning and prototype-based metric-learning on few-task real-world datasets from multiple domains, in comparison to the recent prior methods.
Finally, we make concluding remarks and also discuss future scopes of research in Section 6.
R ELATED WORKS
Few-shot learning: The aim of few-shot learning is to generalize to new tasks using only a few examples (Wang et al., 2020) through three major strategies. First, one can augment the tasks at the data level (Hariharan & Girshick, 2017). Second, the hypothesis space can be constrained at the model level (Snell et al., 2017). Third, the hypothesis search strategy at the algorithm level can be improved (Finn et al., 2017).
The problem of few-task learning can be even more difficult when there is a scarcity of training tasks in a few-task scenario.
We take the route of Yao et al.
(2022) to offer a novel task augmentation strategy that can work in conjunction with both algorithm-level meta-learning as well as model-level metric-learning methods.
Provable robust training of neural networks: A way to build robust neural networks is to find a differentiable upper bound on the verifiable violation of specifications.
Such upper bounds can then be directly optimized alongside the original loss (Mirman et al., 2018;
Raghunathan et al., 2018;
Wong et al., 2018). IBP (Gowal et al., 2019) follows this direction by explicitly minimizing the worst-case loss inside the ϵ -neighborhood of an input for an arbitrary network with some architectural constraints. However, in our work, instead of building robust networks, we repurpose IBP to characterize the ϵ -neighborhood to learn better representation such that the generalization to new tasks by a few-shot learner becomes easier. Moreover, the bounds of the ϵ -neighborhood obtained through IBP gives us a direct way to construct new artificial tasks when the number of available tasks is scarce.
Manifold learning: Traditional methods like ISOMAP (Tenenbaum et al., 2000), LLE (Roweis & Saul, 2000), t-SNE (Van der Maaten & Hinton, 2008), etc. aims to represent high-dimensional data in lower-dimensional space while preserving the local neighborhoods through manifold learning (Abukmeil et al., 2021). Recent manifold learning approaches mostly employ generative neural networks such as deep belief network (Lee et al., 2009), variational auto-encoders (Connor et al., 2021; Kumar & Poole, 2020), flow-based approaches (Brehmer & Cranmer, 2020; Caterini et al., 2021), etc. In a similar spirit, we repurpose IBP to define ϵ -neighborhoods for few-shot learning tasks and constrain the learned feature embedding to preserve the said neighborhoods.
Task augmentation: To train on datasets with a limited number of tasks, some works directly impose regularization on the few-shot learner (Jamal & Qi, 2019;
Yin et al., 2019).
Another line of work performs data augmentation on the individual tasks (Lee et al., 2020;
Ni et al., 2021; Yao et al., 2021).
Finally, a third direction is to employ inter-task interpolation to mitigate task scarcity (Yao et al., 2022).
Our approach is similar to the third category in that we directly create new artificial tasks.
But, we also differs from all of the above-mentioned methods in that we neither undertake intra-task augmentation nor inter-task interpolation.
P RELIMINARIES
In a few-shot learning problem, we deal with tasks T i ∼ p ( T ) .
Each task T i is associated with a dataset
D i =
( X i , Y i ) , that we further subdivide into a support set D si = ( X si , Y si ) = { ( x si,r , y si,r ) } N s r =1 and a query set D qi =
( X qi , Y qi ) =
{ ( x qi,r , y qi,r ) } N q r =1 .
Given a learning model f θ , where θ denotes the model parameters, few-shot learning algorithms attempt to learn θ to minimize the loss on the query set D qi for each of the sampled tasks using the data-label pairs from the corresponding support set
D si .
Thereafter, the trained model f θ and the support set D sj for new tasks T j can be used to perform inference on the corresponding query set D qj .
In the following subsection, we discuss gradient-based meta-learning while the prototype-based metric-learning is detailed in Appendix A.
Gradient-based meta-learning:
In gradient-based meta-learning, the aim is to learn initial parameters θ ∗ such that a typically small number of gradient update steps using the data-label pairs in the support set D si results in a model f ϕ i that performs well on the query set of task T i .
During the meta-training stage, first, a base learner is trained on multiple support sets D si , and the performance of the resulting models f ϕ
i is evaluated on the corresponding query sets D qi .
The meta-learner parameters θ are then updated such that the expected loss of the base learner on query sets is minimized.
In the meta-testing stage, the final meta-trained model f θ ∗ is fine-tuned on the support set D sj for the given test task T j to obtain the adapted model f ϕ j that can then be used for inference on the corresponding query set D qj .
Considering Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) as an example, the bi-level optimization of the gradient-based meta-learning is formulated as:
[Equation]
while η 0 denotes the inner-loop learning rate used by the base learner to train on D s i for task T i , and
L is the loss function, which is usually the cross-entropy loss for classification problems:
[Equation]
A key requirement for effective few-shot generalization to new tasks for both gradient-based metalearning and prototype-based metric-learning is to learn a good embedding of the high-dimensional manifold characterizing the task distribution p ( T ) , i.e. the task manifold.
Ideally, the learned embedding should conserve the neighborhoods from the high-dimensional task manifold (Tenenbaum et al., 2000; Roweis & Saul, 2000).
Hence, in the following subsection, we discuss Interval Bound
Propagation (IBP) (Gowal et al., 2019) that can be employed to define a neighborhood around a task.
Interval bound propagation: Let us consider a neural network f θ consisting of a sequence of transformations h l , ( l ∈ { 1 , 2 , · · · , L } ) for each of its L layers.
We start from an initial input z
= x to the network along with lower bound z 0 ( ϵ ) =
x − 1 ϵ and upper bound z 0 ( ϵ ) = x + 1 ϵ for an ϵ -neighborhood around the input x .
In each of the subsequent layers l ∈ { 1 , 2 , · · · , L } of the network, we get an activation z l = h l ( z l − 1 ) .
IBP uses interval arithmetic to obtain the corresponding axis-aligned bounds of the form z l ( ϵ ) ≤ z l ≤ z l ( ϵ ) on the activations for the l -th layer.
Given the specific differentiable transformation h l , interval arithmetic yields corresponding differentiable lower and upper bound transformations z l ( ϵ ) =
h l ( z l − 1 ( ϵ ) , z l − 1 ( ϵ )) , and z l ( ϵ ) =
h l ( z l − 1 ( ϵ ) , z l − 1 ( ϵ )) , as described in Appendix C. This ensures that each of the coordinates z l,c ( ϵ ) and z l,c ( ϵ ) of z l ( ϵ ) and
z l ( ϵ ) respectively, satisfies the conditions:
[Equation]
− 1 ) , (3) where e c is the standard c -th basis vector.
Further extending to multiple layers, such as f θ S having the first S layers of f θ , the individual transformations h l and h l for l ∈ { 1 , 2 , · · · , S } can be composed to obtain the corresponding functions f θ S and f θ S , such that z S ( ϵ ) =
f θ S ( z 0 , ϵ ) , and z S ( ϵ ) =
f θ S ( z 0 , ϵ ) .
P ROPOSED M ETHOD
Our proposed method aims to enable the learner f θ to learn a feature embedding that attempts to preserve the ϵ -neighborhoods in the task manifold.
Therefore, in the following subsection, we describe the notion of an ϵ -neighborhood for a training task
T i using IBP and show how they can be preserved to aid in few-shot learning problems.
4. F EW - SHOT LEARNING WITH INTERVAL BOUNDS
Consider the network f θ = f
θ L − S ◦ f θ S where S ( ≤ L ) is a user-specified layer number that demarcates the boundary between the portion f θ S of the model that focuses on feature representation and the subsequent portion f θ L − S responsible for the classification.
For a given training task T i , the Euclidean distances between the embedding f θ S ( x qi,r ) for the query instances and their respective interval bounds f θ S ( x qi,r , ϵ ) and f θ S ( x qi,r , ϵ ) is a measure of how well the ϵ -neighborhood is preserved in the learned feature embedding:
[Equation]
To ensure that the small ϵ -neighborhoods get mapped to small interval bounds by the feature embedding f θ S , we can minimize the losses L LB and L UB in addition to the classification loss
L CE in (2).
Notice that the losses L LB and L UB are never used for the support instances x si,r .
Attempting to minimize a na¨ıve sum of the three losses can cause some issues.
For example, weighing the classification loss L CE too high essentially reduces the proposed method to vanilla few-shot learning.
On the other hand, assigning very high weights to the interval losses L LB and/or L UB may diminish learnability as the preservation of ϵ -neighborhoods gets precedence over classification performance.
Moreover, such static weighting approaches are not capable of adapting to (and consequently mitigating) situations where one of the losses comes to unduly dominate the others.
Thus, we minimize a convex weighted sum L of the three losses:
[Equation]
where t denotes the current training step and w e ( t ) is the weight for the corresponding loss L e , e ∈ { CE, LB, UB } at the t -th training step, which is dynamically calculated based on a softmax across the current values of the three losses:
[Equation]
The hyperparameter γ controls the relative importance of the losses.
If any of the losses become too large, the dynamic weighing scheme strives to restore balance by assigning very high weightage to the concerned loss, thus prioritizing its minimization over that of the other losses.
The changes in the dynamic weights over training steps for IBP-aided MAML (hereafter called MAML+IBP) using “4-CONV” network (Vinyals et al., 2016) on the miniImageNet dataset (Vinyals et al., 2016) is illustrated in Figure 3.
We can observe that while there is a clear ordering to the magnitude of the weights (and therefore the corresponding losses) throughout the entire training run, the weights are in fact able to adapt to changes in loss values to maintain the status quo among the different losses.
[Table]
Motivating results:
In Table 1, we demonstrate the effect of employing IBP-aided training for MAML using “4-CONV” network.
Apart from vanilla MAML, we consider three other regularized variants of MAML, (1) MAML+SN that applies Spectral Normalization (Miyato et al., 2018) to the feature embedding through the first S layers, (2) MAML+GL that uses the distance between the original query set and its perturbed (by additive Gaussian noise) version as an extra loss, and (3) MAML+ULBL that considers the distance between the upper and lower interval bounds as an additional loss (Morawiecki et al., 2020) (further details in Appendix F.3).
We see that MAML+IBP achieves higher 5-way 1-shot classification accuracy than the 5 contenders on the miniImageNet and tieredImageNet (Ren et al., 2018) datasets.
Moreover, we also illustrate that the feature embedding learned by IBP-aided training exhibits better intra-task compactness in terms of the mean Euclidean distances from the nearest neighbor in the same class for 100 query instances from 600 tasks, in the feature space characterized by f θ S .
Recent works (Ni et al., 2021; Yao et al., 2022) have shown that augmenting the training data with artificial tasks can improve performance in domains with a limited amount of tasks.
Therefore, while IBP-aided training improves the performance of vanilla MAML (as well as other baselines, detailed in Appendix
F.3), we are particularly interested in the added advantage that it lends by facilitating the generation of artificial tasks within the neighborhoods defined by the interval bounds.
4. I NTERVAL BOUND – BASED TASK INTERPOLATION
Since minimizing the additional losses L LB and L UB is expected to ensure that the ϵ -neighborhood around a task is mapped to a small interval in the feature embedding space, artificial tasks formed within such intervals are naturally expected to be close to the task manifold.
Therefore, we create additional artificial tasks by interpolating between an original task and its corresponding interval bounds (i.e., either the upper or the lower interval bound).
In other words, for a training task T i , a corresponding artificial task T ′
i is characterized by a support set D s ′
i
=
( H s ′
i,r , y si,r ) } N s r =1 in the embedding space.
The artificial support instances H s ′ i,r are created as:
[Equation]
(8) where k denotes the class to which x si,r belongs, λ k ∈ [0 , 1] is sampled from a Beta distribution
Beta ( α, β ) , and the random choice of ν k ∈ { 0 , 1 } dictates which of the bounds is chosen randomly for each class.
The labels y si,r for the artificial task remain identical to that of the original task.
The query set D q ′ i for the artificial task is also constructed analogously.
We then minimize the mean of the additional classification loss L ′ CE for the artificial task T
′ i and the classification loss L CE for the original task T
i for query instances (also the support instances in case of meta-learning).
As a reminder, the losses L LB and L UB are also additionally minimized for the query instances.
The complete IBP-based task interpolation or Interval Bound Interpolation (IBI) training setup is illustrated in Figure 4 in Appendix B. Since IBI does not play any part during the testing phase, the testing recipe remains identical to that of vanilla few-shot learning.
The detailed pseudocode of MAML+IBI (along with the IBI variant of ProtoNet) can be found in Appendix B.
Theoretical analysis: The data X i ( i = 1 , 2 , · · · , N ) for tasks T i can be thought of as i.i.d.
observations from a marginal distribution P X defined on a compact subset X of R d ( d ≥ 1 ), paired with corresponding Y i drawn from the marginal distribution P Y .
The map f θ S is bestowed with the task of producing a lower-dimensional representation of the input X .
Let us denote the embedding space by H ⊆ R κ , given that κ ≤ d .
The spaces X and H are endowed with l 2 norm for simplicity and conformity to our convention.
One may observe that f θ S = h 1 ◦ h 2 ◦ · · · ◦ h S , where in general h l ( z )
=
σ ( A l z + b l ) given that A l ∈ R d
l +1 ×
d l and b l ∈ R d l +1 , l = 1 , · · · , S .
The function σ denotes the activation (such as ReLU), applied component-wise.
Evidently, in our notation d 1 = d and d S +1 = κ .
With this setup, we proceed to undertake the theoretical analysis of our approach.
Please find the detailed proofs in Appendix C.
Definition 1 (Perturbation) .
Given any x 1 ∈ X , an ε -perturbation corresponding to x 1 is the set of points x 1 ( ε ) ⊂
X such that ∥ x 1 − x 2 ∥ = ε , ∀ x 2 ∈ x 1 ( ε ) ; ε > 0 .
For the particular choice of the l 2 norm, Definition 1 characterizes ε -perturbation as a hollow ball of radius ε =
ϵ √ d around a given point.
Lemma 1 (Lipschitz networks ensure bounded IBP) .
Let x and x be ε -perturbations of x ∼ P X for an ε > 0 (i.e. x , x ∈ x ( ε ) ).
Given that the activation σ is Lipschitz continuous (such as ReLU) with constant c σ > 0 , there exists a constant D = D
( c σ ; A 1 , A 2 , · · · , A S ; ε ) such that f θ S ( x , ε ) and f θ S ( x , ε ) ) will at most be an ˆ ε -perturbed version of f θ S ( x ) , where ˆ ε = εD .
The minimization objective function of IBI can be rephrased as L =
L CE + ω 1 L LB + ω 2 L UB , where ω 1 , ω 2 ≥ 0 are Lagrangian multipliers.
The forthcoming result, however, relies on the constrained formulation of the objective, given as min {L CE } subject to L LB ≤ t 1 and L UB ≤ t 2 , where t 1 , t 2 ≥ 0 . This is motivated by the fact that the constrained formulation yields solutions upper bounding the ones obtained using its Lagrangian counterpart [(Boyd & Vandenberghe, 2004), Chapter
Lemma 1 implies that the two losses ( L UB and L LB ) appearing in the constraints can always be made arbitrarily small, depending upon ε .
As such, in the constrained regime, the remaining problem is to show that the multi-task sample classification loss can indeed be dealt with.
Theorem 1 (Generalization bound) .
Let ˜ P be the joint distribution of ( f θ S ( X ) , Y ) , supported on H × R .
In the multi-task regime, let I denote the set of tasks, each consisting of
N samples. Define ˆ R ( N, | I | ) = E [Equation]
i ∼
[Equation]
[Equation]
,Y
[Equation]
[Equation]
[ L CE ( f θ L
[Equation]
S ( H ∗ j ) , Y j )] and R = E
T i ∼ p ( T ) E ( X j ,Y j ) ∼T
i
[ L CE ( f θ L − S ( f θ S ( X j )) , Y j )] .
For a bounded loss function L CE : R × R → [0 , a ]( a ≥ 0) , if the neural network-induced map f θ L − S is such that (cid:12)(cid:12) ∇ f θ L − S ( · ) (cid:12)(cid:12) < ∞ , we ensure: [Equation] ˆ [Equation]
N
[Equation]
| [Equation] holds with probability at least 1 − δ , where ˜ λ =
˜ λ
(ˆ ε, λ ) .
E XPERIMENTS
Experiment protocol:
The experiments are conducted on few-task few-shot image classification datasets, viz.
a subset of the miniImageNet dataset called miniImageNet-S (Yao et al., 2022), and two medical images datasets namely DermNet-S (Yao et al., 2022), and ISIC (Codella et al., 2018;
Li et al., 2020).
We begin our experiments with a few analyses and ablations to better understand the properties of our proposed method.
We then empirically demonstrate the effectiveness of our proposed IBI method on the gradient-based meta-learning method MAML (Finn et al., 2017) as well as the prototype-based metric-learner ProtoNet (Snell et al., 2017) to show that IBI can be seamlessly integrated with multiple few-shot learning paradigms.
For our experiments, we employ the commonly used “4-CONV” network (Vinyals et al., 2016) as well as the larger ResNet-12 network (Lee et al., 2019) to demonstrate the scalability of the proposed method (further details on scalability in Appendix
E).
We perform 5-way 1-shot and 5-way 5-shot classification on all the above datasets (except ISIC where we use 2-way classification problems, similar to Yao et al.
(2021), due to the lack of sufficient training classes).
Further discussion on the datasets and implementation details of IBI along with the choice of hyperparameters can be found in the Appendix.
[Table]
Ablation studies on task
interpolation:
We undertake an ablation study to highlight the importance of generating artificial tasks using IBP bound– based interpolation by comparing
IBI with (1) inter-task interpolation on images, (2) inter-task interpolation in the feature embedding learned by f Sθ , (3)
Worst-Case Loss (WCL) on the ϵ -neighborhood
(Gowal et al., 2019) along with IBP losses, (4) inter-task interpolation while minimizing ULBL (Morawiecki et al., 2020), (5) Gaussian noise–based perturbation (GA) in the image space with IBP losses, (6) Gaussian noise–based perturbation in the feature embedding space f θ S with IBP losses, (7) MLTI (Yao et al., 2022), which performs
MixUp (Zhang et al., 2017) at randomly chosen layers of the learner, and (8) IBP bound–based interpolation without minimizing the L UB and L LB while only optimizing L CE (more results in Appendix F.3).
We perform the ablation study on 5-way 1-shot classification with the “4-CONV” network on miniImageNet-S, ISIC, and DermNet-S. From Table 2, we observe that IBI performs best in all cases.
Moreover, inter-class interpolation at the same fixed layer S as IBI and at randomly selected task-specific layers in MLTI shows worse performance, demonstrating the superiority of the proposed interval bound–based interpolation mechanism.
Further, it is interesting to observe that IBI, when performed without minimizing the L UB and L LB , performs the worst.
This behavior is not unexpected as the neighborhoods are no longer guaranteed to be preserved by the learned embedding in this case, thus potentially resulting in the generation of out-of-manifold artificial tasks.
To further check whether the tasks generated by IBI indeed follow the distribution, we undertake an additional comparison based on the similarity of the artificial tasks with the corresponding original tasks.
Concretely, we define the distance between a task and its artificial counterpart as the median of the pairwise distances between the corresponding data instances in the two tasks.
If an artificial task is created by combining two tasks, a la MLTI (Yao et al., 2022), we consider the smaller of the two median distances.
We observe from Table 2, that the average median distance over 600 tasks is smaller for the proposed method compared to MLTI, as well as inter-task interpolation in the feature embedding learned by f Sθ .
This indicates that the tasks generated by IBI are more likely to lie close to the original task distribution.
[Table]
Importance of dynamic loss weighting:
To validate the usefulness of softmax-based dynamic weighting of the three losses for both IBP and IBI, we first find the average weights for each loss in a dynamic weight run and then plug in the respective values as static weights for new runs.
All experiments in Table 3 are conducted on the miniImageNet-S dataset.
From the upper half of Table 3, we can see that the three average weights are always distinct with a definite trend in that L CE gets maximum importance followed by L UB while L LB contributes very little to the total loss L .
This may be due to the particular “4-CONV” architecture used in this study which employs ReLU activations, thus implicitly limiting the spread of the lower bound (Gowal et al., 2019).
Further, the average weights of IBP and IBI are similar for a particular learner highlighting their commonalities, while they are distinct over different learners stressing their learner-dependent behavior.
Further, in the lower half of Table 3, we explore the effect of using static weights as well as the transferability of the loss weights across learners.
In all cases, the softmax-based dynamic weighting outperforms static weighting, thus demonstrating the importance of dynamic weighting.
Results on few-task few-shot classification problems: For evaluating the few-shot classification performance of IBI in few-task situations, we compare against the regularization-based meta-learning methods TAML (Jamal & Qi, 2019), Meta-Reg (Yin et al., 2019), and Meta-Dropout (Lee et al., 2020) for MAML.
We also compare against data augmentation–based methods like MetaMix (Yao et al., 2021),
Meta-Maxup (Ni et al., 2021), and MLTI (Yao et al., 2022) for both MAML and ProtoNet.
The results in Table 4 show that in keeping with the observation in Table 1, IBP without task interpolation can improve upon the corresponding baselines.
The incorporation of IBP-based task interpolation in
IBI generally improves the results even further.
Overall, we observe that both IBP and IBI outperform the other competitors, with the largest gains being observed for the ISIC dataset.
[Table]
Cross-domain transferability analysis: The miniImageNet-S and DermNet-S datasets both allow 5-way 1-shot classification.
Moreover, miniImageNet-S contains images from natural scenes while DermNet-S consists of medical images.
Therefore, we undertake a crossdomain transferability study in Table 5.
We summarize the Accuracy values obtained by a source model trained on DermNet-S but tested on miniImageNet-S and vice-versa (denoted DS → mIS and mIS → DS, respectively).
We can see that in most cases the IBP variant is able to improve upon the corresponding baseline.
Further, the interpolation-based methods, i.e. MLTI [Table] and IBI are able to further improve performance, with IBI achieving the best performance in most cases, thus showing that IBI training can improve cross-domain transferability
[Table]
C ONCLUSION AND FUTURE WORKS
In this paper
[Table]
we attempt to explore the utility of IBP beyond its originally-intended usage for building and verifying classifiers that are provably robust against adversarial attacks.
In summary, we identify the potential of IBP to conserve a neighborhood from the input image space to the learned feature space through the layers of a deep neural network by minimizing the distances of the feature embedding from the two bounds.
This can be effective in few-shot classification problems to obtain feature embeddings where task neighborhoods are preserved, thus enabling easy adaptability to unseen tasks.
Further, since interpolating between training tasks and their corresponding IBP bounds can yield artificial tasks with a higher chance of lying on the task manifold, we exploit this property of IBP to prevent overfitting to seen tasks in the few-task scenario.
The resulting IBI training scheme is shown to be effective in both the meta-learning and metric-learning paradigms of few-shot learning.
We demonstrate in our results that IBI can be effectively scaled to relatively large networks like
ResNet-12 as we typically only need to apply IBP to a few initial layers (see Appendix E).
However, this still adds extra computational cost (see Appendix E for a comparative study) which scales linearly with the number of layers subjected to IBP.
Therefore, to limit the additional complexity and computational cost, a future direction of research may be to investigate the applicability of more advanced provably robust training methods that yield more efficient and tighter bounds (Lyu et al., 2021).
Moreover, few-shot learners can also be improved with adaptive hyperparameters (Baik et al., 2020), feature reconstruction (Lee & Chung, 2021), knowledge distillation (Tian et al., 2020), embedding propagation (Rodr´ıguez et al., 2020), etc.
Thus, it may be interesting to observe the performance gains from these orthogonal techniques when coupled with IBI.
However, this may not be a straightforward endeavor, given the complex dynamic nature of such frameworks.
R EPRODUCIBILITY S TATEMENT
We have included the pseudo-codes and PyTorch based Python implementation for the proposed method in Appendices B and G, respectively.
The description of all datasets used in this study along with other key implementation details is available in Appendices D,
E. The hyperparameter settings for different algorithms along with their tuning strategy are listed in Appendix F. For the theoretical analyses, complete proofs are provided in Appendix C. A copy of the code is available in the Appendix G while the same can also be found at https://anonymous.4open.science/ r/maml-ibp-ibi-D072/ .
A P ROTOTYPE - BASED METRIC - LEARNING :
Metric-based few-shot learning aims to obtain a feature embedding of the task manifold suitable for non-parametric classification.
Prototype-based metric-learning, specifically Prototypical Network (ProtoNet)
(Snell et al., 2017), assigns a query point to the class having the nearest (in terms of
Euclidean distance) prototype in the learned embedding space.
Given the model f θ and a task T i , we first compute class prototypes { c k } Kk =1 as the mean of f θ ( x si,r ) for the instances x si,r belonging to class k :
[Equation]
where D s,ki ⊂ D si represents the subset of N s support samples from class k .
Given a sample x qi,r from the query set, the probability p ( y qi,r = k | x qi,r ) of assigning it to the k -th class is calculated using the distance function d ( ., . )
between the representation f θ ( x qi,r ) and the prototype c k :
[Equation]
Thereafter, the parameters θ for the model f θ can be trained by minimizing cross-entropy loss (2).
In the testing stage, each query sample x qj,r is assigned to the class having the maximal probability, i.e., y [Equation] [Equation] =
[Equation] max k p [Equation] y qj
[Equation]
r =
[Equation]
|
[Equation]
r )
[Equation]
B A LGORITHMS OF MAML AND P ROTO N ET COUPLED WITH IBP AND IBI
The following Figure 4 illustrates a schematic diagram for the training of IBP and IBI variants. [Equation] [Figure] For IBP-based interpolation,
artificial tasks T ′
i are created with instances H ′ formed by interpolating both the support and query instances with their corresponding lower or upper bounds.
The mean of the classification loss L CE for the T i and the corresponding extra loss L ′ CE for T ′
i is minimized.
The steps for MAML+IBP/IBI and ProtoNet+IBP/IBI are respectively presented in Algorithm 1 and 2.
Please consult the main paper for various notations and equations used in the algorithms.
Remark 1.
The way in which the training support set D si informs the loss calculation on the corresponding query set D qi differs between the MAML and ProtoNet variants.
While a limited number of training steps on the support set is undertaken to obtain the model f ϕ i where the loss is calculated on the query set for MAML, the support set is used to calculate the prototypes { c k } Kk = for the loss calculation on the query set for ProtoNet.
Requires: Task distribution p ( T ) , batch size B , learning rates η 0 and η 1 , interval coefficient ϵ .
1: Randomly initialize the meta-learner parameters θ .
2: while not converged do 3:
Sample a batch of B tasks from the distribution ρ ( T ) .
For IBI, randomly sample an index 1 ≤ m ≤ B to perform the interpolation.
5: for all i ∈ { 1 , 2 , · · · , B } do 6:
Initialize base learner to meta-learner state.
Sample a support set D si of data-label pairs { ( x si,r , y si,r ) } N s r =1 from task T i .
Calculate the classification loss L CE using f θ ( x si,r ) and y si,r .
9: if i = m then 10:
Generate interpolated support and query instances H s ′
i,r and H s ′ i,r using (8).
Calculate classification loss L ′ CE using f θ L − S ( H s ′ i,r ) and y si,r .
Set L CE = 12 ( L CE + L ′ CE ) .
13: end if 14:
Update base learner parameters to ϕ i = θ − η 0 ∇ θ L CE .
Sample a query set D qi of data-label pairs { ( x qi,r , y qi,r ) } N q r =1 from task T i .
Calculate the classification loss L CE with f ϕ i ( x qi,r ) and y qi,r .
Calculate L LB and L UB respectively using (4) and (5).
18: if i = m then 19:
Calculate classification loss L ′ CE using f ϕ L − S ( H q ′ i,r ) and y qi,r .
Set L CE = 12 ( L CE + L ′ CE ) .
21: end if 22:
Calculate L by accumulating L CE , L LB and L UB using (6).
23: end for 24:
Update meta-learner parameters θ = θ − η 1 1 B (cid:80) Bi
=1 ∇ θ L .
25: end while
Requires: Task distribution p ( T ) , learning rate η , interval coefficient ϵ .
1: Randomly initialize the learner parameters θ .
2: while not converged do 3:
For IBI, randomly select if interpolation is to be performed.
Sample a support set D si of data-label pairs { ( x si,r , y si,r ) } N s r =1 from task T i .
Calculate the features f θ L ( x si,r ) and find the prototypes { c k } Kk =1 using (9).
6: if interpolation to be performed then 7:
Generate interpolated support and query instances H s ′
i,r and H s ′ i,r using (8).
Calculate features f θ L − S ( H s ′ i,r ) and find prototypes { c ′ k } Kk =1 .
9: end if 10:
Sample a query set D qi of data-label pairs { ( x qi,r , y qi,r ) } N q r =1 from task T i .
Calculate the loss L CE using (10) and (2).
Calculate L LB and L UB using (4) and (5).
13: if interpolation to be performed then 14:
Calculate classification loss L ′ CE with f θ L − S ( H q ′ i,r ) , { c ′ k } Kk =1 and y qi,r by (10) and (2).
Set L CE = 12 ( L CE + L ′ CE ) .
16: end if 17:
Calculate L by accumulating L CE , L LB and L UB using (6).
Update learner parameters θ
= θ − η ∇ θ L .
19: end while
C D ETAILED T HEORETICAL A NALYSIS
Interval bound propagation for networks with affine layer: Let us assume a network f with L layers where the 0 -th layer denotes the initial input. Let us also consider a layer l ≤ L that is not the 0-th input layer. The 0 -th layer of f takes the input along with its perturbed counterparts as shown in Section 3 in the main paper.
If at the end of l − 1 -th layer the activation, upper bound, and lower bound are respectively z l − 1 , z l − 1 and z l − 1 .
If the l -th layer performs an affine transformation (such as a convolutional, fully connected, batch normalization, etc.) followed by a monotonic activation function (such as ReLU, sigmoid, tanh, etc.)
i.e. z l = σ ( A l z l − 1 + b l ) , then as per Gowal et al.
(2019), we can calculate the interval bounds for the subsequent l -th layer as follows:
[Equation]
where ψ l = | A l | ψ l − 1 and µ l =
A l µ l − 1 + b l given µ l − 1 =
z l − 1 + z l − 1and ψ l − 1 =
z l − 1 − z l − 1.
Lemma 1 (Lipschitz networks ensure bounded IBP) .
Let x and x be ε -perturbations of x ∼ P X for an ε > 0 (i.e. x , x ∈ x ( ε ) ).
Given that the activation σ is Lipschitz continuous (such as ReLU) with constant c σ > 0 , there exists a constant D = D
( c σ ; A 1 , A 2 , · · · , A S ; ε ) such that f θ S ( x , ε ) and f θ S ( x , ε ) ) will at most be an ˆ ε -perturbed version of f θ S ( x ) , where ˆ ε = εD .
Proof.
Given that x 1 , x 2 ∈ X
[Equation]
where ∥ A 1 ∥ = sup ∥ x ∥ =1 ∥
A 1 x ∥ .
The inequality 13 is due to the Lipschitz continuity of σ .
Commonly used activation functions, such as ReLU, tend to satisfy this condition.
In particular, for ReLU, c σ = 1 .
As such, the map h 1 also turns out to be Lipschitz continuous.
A similar argument also proves that h l , l = 2 , · · · , S all follow the same trait.
As a result, f θ S also becomes Lipschitz continuous with accompanying constant ( c σ A ) S , where A = max {∥ A l ∥} .
The recurrence relation of extremities in IBP, as suggested by Gowal et al.
(2019), can be written as:
[Equation]
and f [Equation] ( x , ε ) =
σ
[Equation]
, where the | · | operator results in a matrix with all elements replaced by their corresponding absolute values, and l = 1 , 2 , ..., S .
Thus,
[Equation]
[Table]
[Equation]
i.e., the deviation in the first layer can be made arbitrarily small based on ε .
The quantity (cid:13)(cid:13)(cid:13) f θ 1 ( x ) − f θ 1 ( x , ε ) (cid:13)(cid:13)(cid:13) can be shown to be upper bounded using a similar argument.
In other words, both f θ 1 ( x , ε ) and f θ 1 ( x , ε ) are at most ε 1 -perturbed from f θ 1 ( x ) .
By the method of induction we eventually get a D
= D ( c σ ; A 1 , A 2 , · · · , A S ; ε ) > 0 for which the lemma holds.
Theorem 1 (Generalization bound) .
Let ˜ P be the joint distribution of ( f θ S ( X ) , Y ) , supported on H × R .
In the multi-task regime, let I denote the set of tasks, each consisting of
N samples.
Define ˆ R ( N, | I | ) =
E T i ∼ ˆ p ( T ) E ( X j ,Y j ) ∼ ˆ p ( T i ) [ L CE ( f θ L − S ( H ∗ j ) , Y j )] and R = E
T i ∼ p ( T ) E ( X j ,Y j ) ∼T i [ L CE ( f θ L − S ( f θ S ( X j )) , Y j )] . For a bounded loss function L CE : R × R → [0 , a ]( a ≥ 0) , if the neural network-induced map f θ L − S is such that (cid:12)(cid:12) ∇ f θ L − S ( · ) (cid:12)(cid:12) < ∞ , we ensure:
[Equation]
holds with probability at least 1 − δ , where ˜ λ =
˜ λ
(ˆ ε, λ ) .
Proof.
Before beginning with the proof we point out that, based on Definition 1, given ε > 0 and x ∈ X , any x ′ ∈ x ( ε ) can be written as x ′ = x + η ( ε ) .
For example, in the simplest case, η ( ε ) can be a function in the family ± ϵ 1 .
Thus, in case of IBI, the f θ S ( x i , ε ) and f θ S ( x i , ε ) can both be expressed as f θ S ( x i )
+ η
(ˆ ε ) with corresponding η
(ˆ ε ) .
In essence H ∗
i
=
(1 − λ )
f θ S ( x i ) + λ ( f θ S ( x i )
+ η
(ˆ ε )) , where λ ∈ [0 , 1] .
Now, we can observe that,
[Equation]
where η (ˆ ε ) ∈ R κ , ˆ ε being as mentioned in lemma 1.
We obtain (14) by using the Taylor expansion of f θ L − S up to the first order.
Given that (cid:12)(cid:12) ∇ f θ L − S ( · ) (cid:12)(cid:12) < ∞ , the second term λ ∇ f θ L − S (cid:0) f θ S ( x i ) (cid:1) η
(ˆ ε ) can be made arbitrarily small.
The higher-order terms in the expansion all follow suit
[Equation]
which justifies their omission.
Now,
[Equation]
Since our networks use ReLU activation, the map induced by f θ L − S can be shown to be continuous.
Given H is compact the output space also becomes compact.
Restricted to such a space, the crossentropy loss L CE (similarly, regularized cross-entropy loss) turns out to be Lipschitz continuous.
Consequently,
[Equation]
where c L > 0 is the Lipschitz constant associated with L CE .
Without loss of generality we can construct the map f
θ
L − S such that ∥ f θ L − S ∥ ≤ 1 .
Now, in case there are | I | tasks involved, namely {T i } | I | i
=1 (i.e., the multi-task regime), the population risk turns out to be
[Equation]
We are interested in observing the deviation of the same from the realized risk.
In other words, [Equation] where J = E T i ∼ ˆ p ( T ) E ( X j ,Y j ) ∼T i (cid:104)
L CE (cid:0) f
θ L − S ( H j ) , Y j (cid:1)(cid:105) and ˆ p is the empirical counterpart of the task distribution.
Using Jensen’s inequality, ( i ) can be upper bounded by
[Equation]
where we utilize arguments (15) and (16) to reach (18). Using the union bound based on | I | tasks on top of Corollary 3.14 of Wojtowytsch & E (2020) we can show that the second term in the right-hand side of (18) becomes ≾ 2 L − S +1 (cid:113) 2log(2 κ +2) N + a (cid:113) 2log( 2 | I | δ ) N , with probability at least 1 − δ . To put a deterministic upper bound on ( ii ) let us first define the class of functions
[Equation]
where W L − S is the function space induced by networks with L − S hidden layers (Wojtowytsch &
E, 2020).
Let us now calculate the Rademacher complexity of the class functions G :
[Equation]
where (20) is due to the Lipschitz property of L CE ( · , y )
[Lemma 26.9 of Shalev-Shwartz & Ben-David (2014) or Theorem 7 of Meir & Zhang (2003)].
We arrive at (21) using lemma 3.13 of Wojtowytsch
& E
[Equation]
The inequality (19) is based on the fact that sup u [Equation] ] , given the expectation exists for the class of functions U and random variable X . Thus we obtain the deterministic bound on ( ii ) given by
[Equation]
that holds with probability at least 1 − δ .
The bounds on (i) and (ii) together prove the theorem.
D D ETAILS OF DATASETS USED IN THIS STUDY
miniImageNet: The miniImageNet dataset (Vinyals et al., 2016) is a commonly used subset of
ImageNet (Deng et al., 2009) for evaluating few-shot classifiers.
The dataset contains a total ofclasses each containing 600 images of resolution 84 × 84 × 3 .
Following the directives of Vinyals et al.
from the total 100 classes 64 are kept in the Training set, 16 are retained for Validation, and the rest of the 20 classes are used for testing.
tieredImageNet:
In (Ren et al., 2018) the authors proposed a new larger subset of ImageNet (Deng et al., 2009) for addressing the limitations of miniImageNet.
In miniImageNet it is not ensured that the classes used for training are distinct from those contained in the Test set.
Evidently, this contains the risk of information leakage and may not provide a fair evaluation of the few-shot classifier.
As a remedy Ren et al.
(2018) proposed to go higher in the class hierarchy in ImageNet.
This enables tieredImageNet to use higher-level categories in the Training, Validation, and Test sets maintaining significant diversity between the three.
In essence, a total of 608 ImageNet leaf-level classes are considered that can be categorized into 34 groups.
Among these 34 higher-level groups, 20 are used for training, 6 are kept for validation, and the rest 8 are included in the Test set.
miniImageNet-S:
This dataset is created by only using a subset of the original miniImageNet
Training set for training the few-shot learner in a few-task scenario Yao et al.
Training Classes: n03017168, n07697537, n02108915, n02113712, n02120079, n04509417, n02089867, n03888605, n04258138, n03347037, n02606052, n
Validation and Test sets are kept as same as those used in miniImageNet.
DermNet-S: DermNet-S (Yao et al., 2022) is a subset of the ”Dermnet Skin Disease Atlas” publicly available at http://www.dermnet.com/ .
The dataset after discarding the duplicates contains more than 22,000 medical images spread across 625 classes of dermatological diseases.
Following the preprocessing suggested by Prabhu et al.
the authors of (Yao et al., 2022) created DermNet-S by first extracting the 203 classes containing more than 30 images.
Then from the long-tailed data distribution of the 203 disease classes, the top 30 larger classes are kept for training while the smaller 53 bottom classes are considered for meta-testing.
The images are resized to 84 × 84 × 3 to match the resolution of miniImageNet.
We follow the same dataset construction strategy in our case.
Moreover, we use random classes not included in the Training or Test set as the Validation set.
The complete list of classes in the Training and Test sets are listed as follows:
Training Classes:
Seborrheic Keratoses Ruff, Herpes Zoster,
Atopic Dermatitis Adult Phase, Psoriasis Chronic Plaque, Eczema
Hand, Seborrheic Dermatitis, Keratoacanthoma, Lichen Planus,
Epidermal Cyst, Eczema Nummular, Tinea (Ringworm) Versicolor,
Tinea (Ringworm) Body, Lichen Simplex Chronicus, Scabies,
Psoriasis Palms Soles, Malignant Melanoma, Candidiasis large
Skin Folds, Pityriasis Rosea, Granuloma Annulare, Erythema
Multiforme, Seborrheic Keratosis Irritated, Stasis Dermatitis and Ulcers, Distal Subungual Onychomycosis, Allergic Contact
Dermatitis, Psoriasis, Molluscum Contagiosum, Acne Cystic,
Perioral Dermatitis, Vasculitis, Eczema Fingertip
Testing Classes:
Warts, Ichthyosis Sex Linked, Atypical Nevi,
Venous Lake, Erythema Nodosum, Granulation Tissue, Basal Cell
Carcinoma Face, Acne Closed Comedo, Scleroderma, Crest Syndrome,
Ichthyosis Other Forms, Psoriasis Inversus, Kaposi Sarcoma,
Trauma, Polymorphous Light Eruption, Dermagraphism, Lichen
Sclerosis Vulva, Pseudomonas, Cutaneous Larva Migrans, Psoriasis
Nails, Corns, Lichen Sclerosus Penis, Staphylococcal Folliculitis,
Chilblains Perniosis, Psoriasis Erythrodermic, Squamous Cell
Carcinoma Ear, Basal Cell Carcinoma Ear, Ichthyosis Dominant,
Erythema Infectiosum, Actinic Keratosis Hand, Basal Cell Carcinoma
Lid, Amyloidosis, Spiders, Erosio Interdigitalis Blastomycetica,
Scarlet Fever, Pompholyx, Melasma, Eczema Trunk Generalized,
Metastasis, Warts Cryotherapy, Nevus Spilus, Basal Cell Carcinoma
Lip, Enterovirus, Pseudomonas Cellulitis, Benign Familial Chronic
Pemphigus, Pressure Urticaria, Halo Nevus, Pityriasis Alba,
Pemphigus Foliaceous, Cherry Angioma, Chapped Fissured Feet,
Herpes Buttocks, Ridging Beading
ISIC:
Following Yao et al.
(2022) for “ISIC 2018:
Skin Lesion Analysis Towards Melanoma
Detection” (Codella et al., 2018; Li et al., 2020)
we select the third task where 10,015 medical images are categorized into seven classes based on lesion types.
We first resize the images to 84 × 84 × 3 to match the miniImageNet resolution.
Then among the seven classes in the ISIC dataset, we select theclasses containing a higher number of samples for training while considering the rest for meta-testing as per the directives of Yao et al.
Since there are only 4 classes in the Training set setting the number of ways to 2 results in six possible class combinations in a task.
This in consequence offers an extreme few-task scenario.
For hyper-parameter tuning random classes are used as a Validation set following the cross-validation-based approach employed in (Yao et al., 2022).
The list of classes in the Training and the Test sets are listed as follows:
Training Set: Nevus, Melanoma, Benign Keratoses, Basal Cell Carcinoma
Testing Set:
Dermatofibroma, Pigmented Bowen’s, Vascular
E I MPLEMENTATION DETAILS
Scheduling of ϵ :
In their paper Gowal et al.
(2019) recommended starting with an initial perturbation ϵ 0 = 0 and gradually increasing it to the intended perturbation ϵ over the training steps.
In our case, we follow a similar approach for scheduling the value of perturbation ϵ t at the t -th training step.
We have observed that a fast increase in perturbation usually slows down training while a very slow increment fails to aid the learner.
From extensive experimentation with various scheduling techniques such as linear, cosine, etc., we have found that the following strategy works well in practice.
If the maximum allowed number of training steps is set to T then for the step t the perturbation ϵ t is calculated as:
[Equation]
In essence, we linearly increase ϵ t starting from 0 up to ϵ over 90% of the maximum training steps T and keep it fixed at ϵ for the remainder of the training.
Frequency of interpolation for IBI variants: Performing IBP bound–based interpolation for every task during training may not be beneficial and may instead mislead the learner.
For MAML, we have seen that performing interpolation once in every batch of B tasks aids the training process.
In the case of ProtoNet, we have found that performing IBP bound–based interpolation with a 25% probability results in the best outcome.
Modifications to network architecture: We have used two networks for our experiments namely “4-CONV” and “ResNet-12”.
The “4-CONV” network can be seamlessly integrated with IBI for both MAML and ProtoNet.
This network consists of 4 blocks, each having a convolution, batch normalization, max pooling, and ReLU in sequential order.
IBI can be performed after any one of the blocks.
The ResNet-12 network also consists of 4 blocks where a block (except the first one) receives inputs from (1) the output of the preceding block, and (2) the input of the preceding block through a skip connection.
While the idea of applying IBI after any of the blocks seems appealing, the presence of skip connections may hinder a straightforward integration of IBI in this case.
To understand how ResNet-12 can be customized to accommodate MAML+IBP (and consequently
MAML+IBI) we undertake an ablation study on the miniImageNet-S dataset in a 5-way 1-shot classification problem as described in Table 6.
We can observe that in our initial hyperparameter tuning experiment, MAML+IBP can not match the performance of vanilla MAML on ResNet-12.
Moreover, the performance gap increases as IBP is applied deeper into the network.
This may be explained by the fact that the interval bounds become gradually loose as they progress through the network.
Thus, with increasing depth, the magnitude of the bound losses (especially L UB as the
ReLU activations prevent L LB from becoming too large) will largely outscale the classification loss and consequently affect convergence (see Remark 2).
Applying IBP after only the first block still fails to achieve parity with the baseline because IBP induces a distortion in the feature space, due to its regularization effect.
While the sequential part of the blocks after IBP can adapt to this distortion due to their complexity, the simpler skip paths can not do so.
Hence, the effect of the distortion keeps propagating to the deeper blocks via skip connections.
To aid the network in such a situation, we investigate three approaches to modify the skip connection immediately after the block(s) subjected to IBP, viz.
(1) remove the skip connection for the subsequent block (2) introduce additional layers in the skip connection for the subsequent block to make it deeper and more complex (3) use a skip after one or more of the initial sub-block(s) (consisting sequentially of one convolution, one batch normalization, and one ReLU layer) of the next block.
Among the three approaches, we empirically found that MAML+IBP (consequently MAML+IBI) performs best when the skip connection starts after the second sub-block in block 2.
Due to the comparatively powerful learning strategy of ProtoNet, no such modifications to ResNet-12 are necessary for ProtoNet+IBI.
[Table]
Remark 2.
[Scalability of IBI] IBP (and consequently IBI) requires the propagation of the two interval bounds along with the input data.
This introduces a computational overhead, especially in deeper networks.
However, in practice even in a deeper network, we may not need to perform IBP except in the initial few layers, as the bound losses will otherwise overwhelm the classification loss and consequently impact convergence.
To demonstrate this [Table] we plot the losses (up to 5000 training steps for the ease of visualization) in the following Figure 5 for MAML+IBI using a ResNetnetwork for 5-way 1-shot miniImageNet-S classification, when IBP is applied up to blocks 1-4.
We can see that the three losses have comparable scales only when IBP is applied after block 1.
In all other cases, L UB heavily dominates the total loss.
But, due to its sheer magnitude, the optimizer is unable to minimize it.
Thus, in practice, IBP should only be limited to a few initial layers in deeper networks.
Consequently, IBI easily scales to deeper networks despite the computational overhead. Remark 3.
To show that IBP and IBI variants are well-scalable as their vanilla counterpart we list the actual training costs in the following Table 7 in terms of the average time in seconds to execute a single training step of the algorithm. All the experiments are performed in the same environment using a RTX 3090 GPU. From Table 7 we can observe that, in the case of MAML, the IBI and IBP variants only takes about 40%-70% additional time when “4-CONV” is used. The difference in cost reduces further if ResNet-12 is used as the backbone.
This is expected as we only need to apply
[Figure] i [Figure] IBP in the first few layers of ResNet-12 to gain its full advantage.
For ProtoNet, the increment in computational cost for the proposed techniques is observed to be slightly more compared to that of
MAML.
[Table]
F H YPERPARAMETERS USED IN IBP AND IBI F. N AMES AND FUNCTIONS OF HYPERPARAMETERS
The following Table 8 describes the hyperparameters used in the vanilla MAML, MAML+IBP, and
MAML+IBI.
The following Table 9 describes the hyperparameters used in the vanilla ProtoNet, ProtoNet+IBP, and ProtoNet+IBI.
F. H YPERPARAMETER SEARCH SPACE AND TUNING
For hyperparameter tuning, we employ a grid search.
In Table 10, we list the search spaces for each of the hyperparameters used in MAML+IBP and MAML+IBI.
Moreover, in Table 11, we also detail the search spaces for each of the hyperparameters used in ProtoNet+IBP and ProtoNet+IBI.
For all other learners used in Tables 1 and 2 in the main paper, the results are either taken from the corresponding article or reproduced using the originally recommended hyperparameter settings.
In Tables 12 and 13, we report the optimal dataset-specific hyperparameters for MAML+IBP and
MAML+IBI.
Similarly, Tables 14 and 15 detail the optimal dataset-specific hyperparameter choices for ProtoNet+IBP and ProtoNet+IBI.
For Table 3 in the main paper, the methods using static weights share the same hyperparameter settings with their dynamic weighted counterpart except for γ , which is not used for the static weight runs.
For Table 4 in the main paper, all the MAML variants use the same settings as vanilla MAML.
Further, for all the different interpolation strategies Beta distribution is used with the choices of α and β matching those of the MAML+IBI settings. [Table] [Table]
Traditional ProtoNet Snell et
[Table]
phase
[Table]
layer in the network where IBP losses will be calculated
[Table]
hyperparameters introduced in
[Table]
F. F ULL RESULTS
Contenders in Motivating Example: For the contenders in Table 1 the settings are as follows:
MAML+SN on f θ S :
This variant of MAML applies Spectral Normalization (Miyato et al., 2018) up to the S -th layer of the “4-CONV” network.
Here similar to the MAML+IBP the value of S is set to 3. 2.
MAML+SN on f θ : Here Spectral Normalization is applied on the full network.
MAML+GL:
In this variant, we calculate a Gaussian regularization loss instead of IBP.
Here we send the query
set along
with its perturbed version and attempt to minimize their norm
after
the S -th layer alongside L CE . The extra loss L GL can be expressed as follows: [Equation]
[Table]
[Table]
For the “4-CONV” learner containing 4 blocks of Convolution, Batch nor- malization, Max pooling, and ReLU, S is searched at the block level in the set { 1 , 2 , 3 , 4 } .
For example, S = 2 means IBP losses are calculated after the second block.
For the “ResNet-12” network the ablation study in Appendix E provides the optimum choice of S .
Additional
[Table]
MAML+IBI
[Table]
Search space contains three pairs of choices (0 . 1 , 1) , (0 . 25 , 1) [Table] and (0 . 5 , 0 . 5) where a tuple contains the value of α and β in order. [Table]
Additional hyperparameters introduced in ProtoNet+IBP
Interval coefficient ϵ
[Table]
in the set { 0 . 05 , 0 . 1 , 0 . 2 } .
Softmax
[Table]
γ
Searched in the set { 0 . 01 , 1 , 10 } .
Layer S
For the “4-CONV” learner containing 4 blocks of Convolution, Batch nor- malization, Max pooling, and ReLU, S is searched at the block level in the set { 1 , 2 , 3 , 4 } .
For example, S = 2 means IBP losses are calculated after the second block.
For the “ResNet-12” network the ablation study in Appendix E provides the optimum choice of S .
Additional hyperparameters introduced in ProtoNet+IBI α and β
Search space contains three pairs of choices (0 . 1 , 1) , (0 . 25 , 1) [Table] and (0 . 5 , 0 . 5) where a tuple contains the value of α and β in order. [Table]
[Table] [Table] [Table] [Table]
[Table]
where ζ ∼ N (0 , σ ) , and the standard deviation σ is scheduled similar to ϵ with starting from 0 and slowly increasing to ϵ/ 2 .
MAML+ULBL:
Following Morawiecki et al.
we replace the two bound losses with a single one that calculates the distance between the upper and lower interval bounds.
The loss L ULBL in this case can be written as:
[Equation]
The full version of Table 1 is provided in the following Table 17.
Comparison of IBP with other few-shot learners: As contending meta-learning algorithms, we choose the vanilla MAML along with notable meta-learners such as Meta-SGD Li et al.
Reptile Nichol et al.
(2018), LLAMA Grant et al.
R2-D2 Bertinetto et al.
(2019), and BOIL [Table] [Table]
for miniImageNet and tieredImageNet datasets in terms of 5-way 1-shot Accuracy and intra-task compactness. This is the full version of Table 1. [Table]
Oh et al.
Moreover, considering the regularizing effect of IBP and IBI
[Table]
we also include meta-learners such as TAML Jamal & Qi (2019), Meta-Reg Yin et al.
[Table]
and Meta-Dropout
Lee et al.
(2020) which employ explicit regularization.
We further include data augmentation–reliant learners such as MetaMix Yao et al.
Meta-Maxup Ni et al.
(2021), as well as the intertask interpolation method MLTI Yao et al.
In case of metric-learners, we compare against the vanilla ProtoNet in addition to other notable methods like MatchingNet Vinyals et al.
RelationNet Sung et al.
IMP Allen et al.
and GNN Satorras & Estrach (2018).
We also compare against ProtoNet coupled with data augmentation methods such as MetaMix, Meta-Maxup, and MLTI, as done in Yao et al.
While Yao et al.
(2022) had to modify the training strategy of the canonical ProtoNet to accommodate the changes introduced by MetaMix, Meta-Maxup, and
MLTI, the flexibility of IBP and IBI imposes no such requirements.
We summarize the findings in Table 18.
We can observe that either IBP or IBI or both achieve better Accuracy than the competitors in all cases.
The slightly better performance of IBP with ProtoNet seems to imply that IBP-based task interpolation is often unnecessary for ProtoNet when a large number of tasks is available.
[Table]
Notes on contenders used in Table 2: The extra parameter settings required for the contenders in
Table 2 are as follows: MAML+WCL: Here given a task its worst-case loss in the ϵ -neighborhood (Gowal et al., 2019) is added with the original loss.
In essence, this acts similar to augmentation with the worst-case logits. We tune the relative contribution of the original task and the worst-case task to the final L CE following the recommendations made by (Gowal et al., 2019). MAML+GA (image space): Here the original task is perturbed with Gaussian noise to form the augmented task in the image space.
The noise is sampled from a Gaussian with meanand standard deviation σ = ϵ/ 2 . The value of σ is scheduled similarly to ϵ . 3. MAML+GA (at f θ S feature space): Here the embedding of the original task after f θ S is perturbed with Gaussian noise.
Similar to the image space, the mean of the normal distribution used for sampling noise can be set to 0.
However, finding a good σ may not be straightforward as the f θ S feature space is continuously updating.
In our implementation, we take σ as half of the median distance between the original task and its bounds over a MAML+IBI run.
The full version of Table 2 is detailed in Table 19. The full version of Table 4 in the main paper is provided here across Tables 20 and 21.
Moreover, the full version of Table 5 in the main paper is presented in Table 22.
[Table]
[Table]
[Table] [Table]
G C ODES
The following code can be used to run MAML, MAML+IBP, and MAML+IBI.
Please put the data in a folder with the dataset name.
The data should be in “.jpg” image format typically of resolution 84 × 84 .
The data is expected to be arranged in train, validation, and test folders each containing folders for individual classes.
The main code is kept inside the “src” folder while the helper code can reside outside.
The “main” function resides in the “run learner.py”.
Helper code: image data process.py
[Figure]
def read dataset ( data dir , val presence=
True ): # Read the image dataset .
if val presence is True : return tuple ( read classes (os . path .
join ( data dir , x)) for x in [ ’ train ’ , ’val ’ , ’ test ’ ]) else : return tuple ( read classes (os . path .
join ( data dir , x)) for x in [ ’ train ’ , ’ test ’ , ’ test ’ ]) def read classes ( dir path ):
# Read the class directories in a train / val / test directory .
# Images should be in ”.
jpg” format .
return [ ImageProcessClass (os . path .
join ( dir path , f )) for f in os .
listdir ( dir path )] class ImageProcessClass : # Loading and using the image dataset .
# To use these APIs , you should prepare a directory that # contains three sub−directories : train , test , and val .
def init ( self , dir path ): self .
dir path =
dir path self .
cache = {} def sample( self , num images ): # Sample images (as pytorch tensor ) from the class .
names =
[ f for f in os .
listdir ( self . dir path )
if f . endswith ( ’ . jpg ’ )] random .
shuffle (names) images =
[] for name in names [: num images ]: images .
append( self .
read image (name))
return images def read image ( self , name): # For reading images and transformations as necessary .
# Image resolution is set to 84x84 .
if name in self .
cache : return self .
cache [name] with open (os . path .
join ( self .
dir path , name) , ’rb ’)
as in file : img =
Image .
open ( in file ).
resize ((84 , 84)).
convert ( ’RGB’)
img
= np. array (img ).
astype ( ’ float32 ’) / 0xff img =np.
rollaxis (img , 2, 0) self .
cache [name] = torch .
tensor (img) return self .
read image (name)
Main
: run learner.py import random import os import sys import numpy as np import torch import argparse from img data process import read dataset from datetime import datetime from copy import deepcopy from src .
models import NetworkModel from src .
eval model import bulk evaluate from src .
train model import train from src .
learners import Learner def argument parser (): # Get an argument parser for a training script .
[Table] [Table]
if
[Table]
[Table] [Table]
, ’val ’ )
[Table] =
True print (”Validation set is present . ”)
else : val presence =
False print (”Validation set is not found .
Exiting . ”)
sys .
exit () time string = datetime .now
strftime (”%m%d%Y %H:%M:%S”)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
if name
[Table]
Model: “src/models.py”
[Table] [Table]
[Table] [Table]
[Table] [Table]
correct .
item () def sample mini dataset ( dataset , num classes , num shots ): # Sample a few shot task from a dataset .
shuffled = list ( dataset ) random .
shuffle ( shuffled ) for class idx , class obj in enumerate ( shuffled [: num classes ]): for sample in class obj .
sample( num shots ): yield (sample , class idx ) def mini batches (samples , batch size , num batches ): # Generate mini−batches from some data .
samples = list ( samples ) cur batch =
[] batch count = 0 while True : random .
shuffle ( samples ) for sample in samples : cur batch .
append(sample) if len ( cur batch ) < batch size : continue yield cur batch cur batch =
[] batch count +=if batch count =
=
num batches : return def split train test (samples , test shots =1): # Split a few−shot task into a train and a test set .
train set = list ( samples ) test set =
[] labels = set ( item [1] for item in train set )
[Table]
Training function: “src/train model.py”
[Table] [Table]