H IERARCHICAL V ARIATIONAL M EMORY FOR F EW - SHOT L EARNING A CROSS D OMAINS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural memory enables fast adaptation to new tasks with just a few training samples.
Existing memory models store features only from the single last layer, which does not generalize well in presence of a domain shift between training and test distributions.
Rather than relying on a ﬂat memory, we propose a hierarchical alternative that stores features at different semantic levels.
We introduce a hierarchical prototype model, where each level of the prototype fetches corresponding information from the hierarchical memory.
The model is endowed with the ability to ﬂexibly rely on features at different semantic levels if the domain shift circumstances so demand.
We meta-learn the model by a newly derived hierarchical variational inference framework, where hierarchical memory and prototypes are jointly optimized.
To explore and exploit the importance of different semantic levels, we further propose to learn the weights associated with the prototype at each level in a data-driven way, which enables the model to adaptively choose the most generalizable features.
We conduct thorough ablation studies to demonstrate the effectiveness of each component in our model.
The new state-of-the-art performance on cross-domain and competitive performance on traditional few-shot classiﬁcation further substantiates the beneﬁt of hierarchical variational memory.
I NTRODUCTION
Few-shot learning with an external memory is known to learn new concepts quickly with only a few samples, especially when embedded in a meta-learning setting (Santoro et al., 2016).
A common tactic is to store short-term memory (Munkhdalai & Yu, 2017; Munkhdalai et al., 2018; Kaiser et al., 2017) as obtained from the support set of the current task, and to empty it at the end of a task.
Another tactic is to let the memory store long-term knowledge distilled from all the training tasks (Zhen et al., 2020), which provides a conceptual context to learn a new task.
The long-term memory has demonstrated effectiveness in enabling a model to quickly learn new few-shot learning tasks within domains (Zhen et al., 2020).
Nonetheless, for both the short- and long-term memory tactics, the learning ability is limited when facing a task from an unseen domain (Tseng et al., 2020; Guo et al., 2020; Cai & Shen, 2020; Du et al., 2021).
Cross-domain few-shot learning (Tseng et al., 2020;
Guo et al., 2020) is even more challenging than the conventional few-shot learning problem as it also has to take distribution shift into account.
In cross-domain few-shot learning, directly using traditional external memory, which often only stores high-level semantic features (Zhen et al., 2020), is unable to ﬁnd the suitable semantics in memory due to the domain shift causing side effects.
Nonetheless, low-level features like textures, shapes, edges, etc.
from training domains are often still meaningful for a new domain (Adler et al., 2020).
We hypothesize, features at different levels would play distinctive roles in generalizing across domains.
It remains unexplored to investigate hierarchical features in memory that can enable quick adaptation, as humans do, when faced with a few samples from new domains.
In this paper, we make three contributions.
First, we propose hierarchical variational memory, which accrues and stores the long-term semantic knowledge of multiple network layers from past experiences.
Each entry stores features at different hierarchical levels by summarizing the feature representations of class samples, thus generalizing well in presence of a domain shift between the training and test distributions.
We formulate the memory recall at different hierarchical levels as a variational inference of the latent memory, which is an intermediate stochastic variable.
Second, to better deploy the hierarchical memory, we introduce a hierarchical variational prototype model to obtain prototypes at different levels as well.
We also develop the hierarchical variational prototype in a latent model by treating the prototypes at different levels as the latent variable.
In doing so, the model is endowed with the ability to ﬂexibly rely on low-level textures or high-level semantic features if the domain shift circumstances so demand.
As the third contribution, we propose to learn the weights associated with the prototype at each level in a data-driven way, which can explore and exploit the importance of the different feature levels for domain generalization.
We leverage the few training samples in the new domain to generate reasonable weights for each prototype, since these samples already carry sufﬁcient appearance cues about the new domain, which enables the model to adaptively choose the most generalizable features.
Two different hierarchical variational inference problems are included in our optimization objective: 1) the inference of the latent memory, which conditions on the latent memory of previous layers and the memory contents of the current layer; 2) the inference of prototypes, which treats the support set, the latent memory of the current layer and the prototype of previous layers as conditions.
We demonstrate the effectiveness of the proposed hierarchical memory by extensive experiments on the cross-domain few-shot learning classiﬁcation tasks introduced by (Guo et al., 2020) and the more traditional within-domain few-shot learning classiﬁcation tasks (Snell et al., 2017).
We conduct an extensive ablation study to demonstrate the contributions of different components in our model.
We also analyze how hierarchical memory adaptively chooses the weights in the face of domain change and identiﬁes the appropriate layer of the prototype that should be used.
An extensive evaluation of cross-domain few-shot and traditional few-shot classiﬁcation benchmarks reveals that our hierarchical memory consistently achieves results that are better, or at least competitive, compared to other methods.
M ETHODOLOGY 2. P RELIMINARIES
In the following, we present the relevant background on few-shot classiﬁcation and its cross-domain setting, as well as the preliminaries of the (variational) prototypical network and variational semantic memory (Zhen et al., 2020).
Few-shot classiﬁcation In the few-shot classiﬁcation scenario
, we deﬁne the N -way K -shot classiﬁcation problem, which is comprised of the support sets S and query set Q .
Each task also called an episode, is a classiﬁcation problem sampled from a task distribution p ( T ) .
The ‘way’ of the episode refers to the number of classes in the support, while the ‘shot’ of the episode refers to the number of examples per class.
Tasks are drawn from a dataset by randomly sampling a subset of classes, sampling points from these classes, and then partitioning the points into support and query sets.
Episodic optimization (Vinyals et al., 2016) iteratively trains the model by taking one episode update at a time.
Cross-domain setting For few-shot classiﬁcation, the test tasks are typically assumed to come from the same task distribution p ( T ) as the training tasks.
The recently introduced task of cross-domain few-shot learning (Guo et al., 2020) considers the few-shot classiﬁcation challenge under domain shift.
Concretely, the training task comes from a single source domain p ( T 0 ) , while the test tasks come from several unseen domains {T 1 , · · · , T M } .
The goal of cross-domain few-shot learning is to learn a meta-learning model using a single source domain that generalizes to several unseen domains.
Prototypical network Our model builds upon the prototypical network (Snell et al., 2017), which is widely used for few-shot image classiﬁcation.
It computes a prototype z k = 1 K (cid:80) k
f θ ( x k ) for each class through an embedding function f θ , which is realized by neural networks.
It computes a distribution over classes for a query sample x given a distance function d ( · , · ) as the softmax over its distances to the prototypes in the embedding space:
[Equation]
Variational prototype network
The variational prototype network (Zhen et al., 2020) is a powerful model for learning latent representations from small amounts of data, where the prototype z per class is treated as a distribution.
Given a task with a support set S and query set Q , the ELBO takes the following form:
[Equation]
where q ( z | S ) is the variational posterior over z , p ( z | x i ) is the prior, and L z is the number of Monte
Carlo samples for z .
Variational semantic memory Variational semantic memory (Zhen et al., 2020) is proposed to accumulate and store the semantic information from previous tasks for the inference of prototypes of new tasks.
It consists of two main processes: memory recall , which retrieves relevant information that ﬁts with speciﬁc tasks based on the support set of the current task; memory update , which effectively collects new information from the task and gradually consolidates the semantic knowledge in the memory.
Variational semantic memory formulates the memory recall as a variational inference of the latent memory, which is an intermediate stochastic variable.
The objective is written as follows:
[Equation]
where p ( m | S ) is the introduced prior over latent memory m and M denotes the memory content.
The above networks and memory only focus on the problem of few-shot classiﬁcation within domain and their performance would be degrade for cross domain few-shot learning, because of the domain shift problem.
In this paper, we propose hierarchical memory to address few-shot classiﬁcation under domain shift.
2. H IERARCHICAL V ARIATIONAL P ROTOTYPE
In the variational prototype network (Zhen et al., 2020), the probabilistic prototype is obtained by feeding the last layer of features to an amortization network.
Hence, it is not possible to obtain the prototypes of different layers when addressing the domain shift problem, making it relying on the high-level prototype only, which is not necessarily shared between domains.
We introduce the hierarchical variational prototype network to generate different layer prototypes.
Compared to its ﬂat counterpart, it introduces q ( z l | z l − 1 , S ) instead of q ( z | S ) , so we approximate the true posterior by minimizing the KL divergence:
[Equation]
where z l is the prototype of layer l .
By applying the Baye’s rule, we then obtain the following ELBO: log
[Equation]
In practice, the variational posterior q ( z l | z l − 1 , S ) is implemented by an amortization network (Kingma & Welling, 2013) that takes the concatenation of the average feature representations of samples in the support set S and the upper layer prototype z l − 1 and returns the mean and variance of the current layer prototype z l .
The hierarchical probabilistic prototype provides both a more informative representation than the deterministic prototype and the ability to capture different representation levels, making it more suitable for cross-domain few-shot learning.
More importantly, the hierarchical variational prototype also provides a principled way of incorporating the prior knowledge of the different levels from the past experienced tasks.
Therefore, we introduce the external hierarchical memory to enhance the prototype at different levels.
[Figure]
2. H IERARCHICAL V ARIATIONAL M EMORY
We introduce the hierarchical variational memory to accrue and store the long-term semantic knowledge at different levels from previously experienced tasks for the hierarchical inference of prototypes for new tasks.
By seeing more examples of objects at different semantic feature levels, the knowledge in the memory is enriched and consolidated, which allows for quick learning on new tasks.
Concretely, our model has L layers of memory, each having M storage units which store a keyvalue pair in each row of the memory array.
The key in each memory stores the average feature representations of images from the same class and the value is the corresponding label.
We adopt a similar memory mechanism as (Zhen et al., 2020): memory recall and memory update, but we introduce a hierarchical version of memory recall.
We apply hierarchical variational memory into the hierarchical variational prototype network to generate the prototypes at different layers.
From a Bayesian perspective, the prototype posterior can be inferred by:
[Equation]
6) where m l is the latent memory of layer l .
Different from the variational semantic memory (Zhen et al., 2020), we design a hierarchical variational approximation q ( m l | m l − 1 , M l , S ) to the posterior over the latent memory m l of layer l by inferring from M l conditioned on S .
The hierarchical variational inference of prototypes by a hierarchical Bayesian framework is written as :
[Equation]
where a is the addressed categorical variable, M a represents the corresponding memory content at address a , and | M | indicates the memory size.
We ﬁrst leverage the support set S and memory M to generate the addressed categorical variable a , and then use the fetched memory content M a , support set S , and the latent memory m l − 1 of upper layer l − 1 to infer the latent memory m l of layer l .
For the prototype z l , we use the upper layer prototype z l − 1 , the support set S , and the latent memory m l as conditions.
By combining the ELBO in Eq.
(5) for the hierarchical variational network and Eq.
7, we obtain the following ELBO for hierarchical variational inference:
[Equation]
The overall computational graph of our approach is shown in Fig.
Directly optimizing the above objective does not consider the role of the different feature levels for domain generalization.
Thus, we propose to learn the weights associated with the prototype at each level to achieve adaptive prototypes, which enables the model to adaptively choose the most generalizable features.
2. L EARNING TO WEIGH PROTOTYPES
Features from different layers contain different levels of semantic abstraction about an object and should contribute differently to the generalization of new tasks.
Therefore, the importance of established prototypes from different feature levels need to be treated distinctively by depending on the domain from which a new task originates.
To this end, we propose learning to weigh prototypes of different levels in a data-driven way.
Assume we have L prototypes: { z 1 , · · · , z L } , and we can obtain L different prediction logits { ˆ y 1 , · · · , ˆ y L } based on Eq.
We can produce the ﬁnal logit ˆ y by bagging (Breiman, 1996) to vote which category it should be categorized into.
However, this approach does not allow the model to adaptively choose the weight that different level prototypes should occupy based on the new domain.
In general, the ﬁnal logit can be represented as a weighted sum: ˆ y
[Equation]
where we learn the weight α l for each prototype, by conditioning on the support set of the task.
For cross-domain few-shot learning, we have a few samples S when testing on new domains, we can generate a reasonable weight for prototypes at different levels of S , because, intuitively, these samples already carry sufﬁcient domain information.
To generate the weight α l of layer l , we deploy a hypernetwork f lα ( · ) that takes the average gradients (Munkhdalai et al., 2018) of all the support sets S as input, and returns the weight α l by softmax :
[Equation]
where ∇ lS are the average gradients on layer l of all the support sets.
The hypernetwork f lα ( · ) is ﬁrst learned at meta-training time and then directly used as the support set S from the test domain at meta-test time.
Note that on the test domain we do not learn the parameter of f lα ( · ) ; instead, we simply rely on the support gradient of each layer to generate its weight for the ﬁnal result.
We propose to learn the weights associated with the prototype at each level of the hierarchical variational prototype and hierarchical variational memory to adaptively unify different prototypes at different levels, which enables the model to adaptively choose the most generalizable features, thus achieving generalization for cross-domain few-shot learning.
R ELATED W ORKS
Memory Several works augment neural networks with an external memory module to improve learning (Santoro et al., 2016; Pritzel et al., 2017; Weston et al., 2014;
Graves et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017; Munkhdalai et al., 2018; Ramalho & Garnelo, 2019).
(Santoro et al., 2016)
equip neural networks with a neural Turing machine for few-shot learning.
The external memory stores samples in the support set in order to quickly encode and retrieve information from new tasks.
(Bornschein et al., 2017) proposed augmenting generative models with an external memory, where the chosen address is treated as a latent variable.
Sampled entries from training data are stored in the form of raw pixels in the memory for few-shot generative learning.
In order to store a minimal amount of data, (Ramalho & Garnelo, 2019) proposed a surprise-based memory module, which deploys a memory controller to select minimal samples to write into the memory.
An external memory was introduced to enhance recurrent neural network in (Munkhdalai et al., 2019), in which memory is conceptualized as an adaptable function and implemented as a deep neural network.
Semantic memory has recently been introduced by (Zhen et al., 2020) for few-shot learning to enhance prototypical representations of objects, where memory recall is cast as a variational inference problem.
In contrast to (Zhen et al., 2020), our memory adopts a hierarchical structure, which stores the knowledge at different levels instead of the last layer only.
Prototypes The prototypical network for few-shot learning is ﬁrst proposed by (Snell et al., 2017).
It learns to project the samples into a common metric space where classiﬁcation is conducted by computing the distance from query samples to prototypes of classes.
Inﬁnite Mixture Prototypes was proposed by (Allen et al., 2019), which improved the prototypical network by an inﬁnite mixture of prototypes that represents each category of objects by multiple clusters.
Several Bayesian nonparametric methods (Rasmussen, 2000; Hjort et al., 2010) were proposed to infer the number of clusters.
Triantaﬁllou et al.
(2020) combines the complementary strengths of Prototypical Networks and MAML (Finn et al., 2017) by leveraging their effective inductive bias and ﬂexible adaptation mechanism for few-shot learning.
Zhen et al.
(2020) proposed the variational prototypical network to improve the prototypical network by probabilistic modeling of prototypes, which provided more informative representations of classes.
Different from their work, we propose hierarchical variational prototype networks to generate different layer prototypes instead of using one prototype, which could capture information at different levels for few-shot learning across domains.
Meta-learning or learning to learn (Schmidhuber, 1987; Bengio et al., 1991; Thrun & Pratt, 1998), is a learning paradigm where a model is trained on distribution of tasks so as to enable rapid learning on new tasks.
Meta-learning approaches to few-shot learning (Vinyals et al., 2016; Ravi & Larochelle, 2017;
Snell et al., 2017; Zhen et al., 2020;
Finn et al., 2017) differ in the way they acquire inductive biases and adapt to individual tasks.
They can be roughly categorized into three groups.
Those based on metric-learning generally learn a shared embedding space in which query images are matched to support images for classiﬁcation (Vinyals et al., 2016; Snell et al., 2017; Santoro et al., 2016; Oreshkin et al., 2018; Allen et al., 2019).
The second, optimization-based group learn an optimization algorithm that is shared across tasks, which can be adapted to new tasks for efﬁcient and effective learning (Ravi & Larochelle, 2017; Andrychowicz et al., 2016;
Finn et al., 2017; 2018;
Grant et al., 2018; Triantaﬁllou et al., 2017; Rusu et al., 2019).
The memory-based meta-learning group leverages an external memory module to store prior knowledge that enables quick adaptation to new tasks (Santoro et al., 2016; Munkhdalai & Yu, 2017; Munkhdalai et al., 2018;
Mishra et al., 2018; Zhen et al., 2020).
Our method belongs to the ﬁrst and third group, as it is based on prototypes with external hierarchical memory, with the goal to perform cross-domain few-shot classiﬁcation.
Cross-domain few-shot learning The problem is formally posed by (Tseng et al., 2020), who attack it with feature-wise transformation layers for augmenting mage features using afﬁne transforms to simulate various feature distribution.
Then, (Guo et al., 2020) proposed the cross-domain few-shot learning benchmark, which covers several target domains with varying similarities to natural images.
Phoo & Hariharan (2020) introduce a method, which allows few-shot learners to adapt feature representations to the target domain while retaining class grouping induced by the base classiﬁer.
It performed cross-domain low-level feature alignment and also encodes and aligns semantic structures in the shared embedding space across domains.
In contrast, we ﬁrst address cross-domain fewshot learning by a variational inference approach, which enables us to better handle the prediction uncertainty on the unseen domains.
Moreover, we propose a hierarchical variational memory for crossdomain few-shot learning, and to better deploy the hierarchical memory, we introduce a hierarchical variational prototype model to obtain prototypes at different levels.
E XPERIMENTS 4. E XPERIMENTAL SETUP
We apply our method to four cross-domain few-shot challenges and two within-domain few-shot image classiﬁcation benchmarks.
Sample images from all datasets are provided in the appendix A.
Cross-domain datasets The 5-way 5-shot cross-domain few-shot classiﬁcation experiments use mini Imagenet (Vinyals et al., 2016) as training domain and test on four different domains, i.e. , CropDisease (Mohanty et al., 2016) containing plant disease images, EuroSAT (Helber et al., 2019) consisting of a collection of satellite images, ISIC2018 (Tschandl et al., 2018) containing dermoscopic images of skin lesions, and ChestX (Wang et al., 2017), a set of X-ray images.
Results on the 5-way 20-shot and 5-way 50-shot are provided in the appendix.
Within-domain datasets The traditional few-shot within-domain experiments are conducted on mini Imagenet (Vinyals et al., 2016) which consists of 100 randomly chosen classes from ILSVRC2012 (Russakovsky et al., 2015), and tiered Imagenet (Ren et al., 2019) which is composed ofclasses grouped in 34 high-level categories.
In the test stage, we measure the accuracy of 600 tasks sampled from the meta-test set.
In this paper, we focus on 5-way 1-shot/5-shot tasks following the prior work (Snell et al., 2017; Finn et al., 2017; Zhen et al., 2020).
[Table] [Table]
Implementation details We extract image features using a ResNet-10 backbone, which is commonly used for cross-domain few-shot classiﬁcation (Ye et al.,
[Table]
Guo et al., 2020).
For the computation of features at each level, we ﬁrst input the last convolutional layer feature map of each residual block through a ﬂattening operation and then input the ﬂattened features into the two fully connected layers.
For the within-domain experiments, we use two backbones: Conv-4 and ResNet-12.
The
Conv-4 was ﬁrst used for few-shot classiﬁcation by (Vinyals et al., 2016), and is widely used (Snell et al., 2017; Finn et al., 2017;
Sun et al., 2019; Zhen et al., 2020).
To gain better performance,
ResNet-12 (Bertinetto et al., 2019; Gidaris & Komodakis, 2018; Yoon et al., 2018) is also widely reported for few-shot classiﬁcation.
Following the prior works, we conﬁgure the ResNet-12 backbone as 4 residual blocks.
Each prototype is obtained in the same way as the cross-domain few-shot classiﬁcation.
We use a two-layer inference network for the few-shot cross domain experiments.
Following the VSM (Zhen et al., 2020), we use the same three-layer inference network for the few-shot within domain experiments.
As our memory only stores the prototype per category, by a 256-dimensional vector, the total memory size of our model is about 8K. This is negligible compared to the about 10M parameters of our ResNet-10 or ResNet-12 backbones.
Our code will be publicly released.
Metrics
The average cross-domain/ within-domain few-shot classiﬁcation accuracy ( % , top-1) along with 95% conﬁdence intervals are reported across all test images and tasks.
4. R ESULTS
Beneﬁt of hierarchical prototypes To show the beneﬁt of the hierarchical prototypes, we compare hierarchical variational prototypes with variational prototypes (Zhen et al., 2020), which obtains probabilistic prototypes of the last layer only.
Table 1 shows the hierarchical prototype achieves a slightly better or comparable performance to the single prototypes on all cross-domain few-shot classiﬁcation tasks.
More importantly, the experiments conﬁrm that hierarchical prototypes can capture features at different levels, which allows for better deployment of hierarchical memory.
As it may fetch the corresponding information for each level of prototype, which we will demonstrate next.
Hierarchical vs. ﬂat variational memory We ﬁrst show that a ﬂat memory as used in (Zhen et al., 2020) is less suitable for cross-domain few-shot learning in Table 2.
With ﬂat memory, performance even degrades a bit compared to variational prototypes (see Table 1).
One interesting observation is that on ChestX – with the largest domain gap – the performance of ﬂat memory degrades the most compared to the variational prototype.
This is reasonable since features vary for the domains, resulting in the tested domains not ﬁnding the appropriate semantic information.
Our hierarchical variational memory consistently surpasses ﬂat memory on all test domains.
We attribute the improvements to our model’s ability to leverage memory of different layers to generate prototypes.
The hierarchical memory provides more context information at different layers, enabling more information to be transferred to new domains, and thus leads to improvements over ﬂat memory.
Beneﬁt of learning to weigh prototypes We investigate the beneﬁt of adding the learning to weigh prototypes to the hierarchical variational memory.
The experimental results are reported in Table 3.
Hierarchical variational memory with bagging produces the ﬁnal result by bagging (Breiman, 1996) [Table] to vote which category it should be categorized into.
Adding the learning to weigh prototypes on top of the hierarchical variational memory leads to a small but consistent gain under all test domains.
Thus, the learned adaptive weight acquires the ability to adaptively choose the most generalizable features for the new test domain, resulting in improved performance.
Importance of prototypes at different levels We present a visualization of the inﬂuence of the prototypes at different levels, using a varying number of shots, in Fig.
We sample 600 test tasks on each domain to show the value of α l at different blocks, and average α l among all the test tasks.
We choose the last three blocks of ResNet-10 to generate the memory and prototype.
As presented in (Guo et al., 2020), the degree of domain shift between mini ImageNet and target domains is ordered by:
CropDiseases <
EuroSAT < ISIC <
ChestX. From Fig.
2, we can see that the value of α increases along with the increase in domain shift.
Another interesting observation is that as the training samples (shots) in each class increase, the weight of the high-level prototypes also increases.
This is expected as high-level features can already better represent the information of the new domain due to the increase in the number of training samples.
However, the weight of the low-level prototypes is still larger than the weight of the high-level ones.
This experiment demonstrates that the lower-level features may be more useful across different domains than the more specialized high-level concepts.
Visualization of hierarchical variational prototypes To understand the empirical beneﬁt of using prototype at different levels, we visualize the distribution of prototypes obtained by different blocks on the most challenging ChestX domain under the 5-way 5-shot setting in Fig.
The level 1 (low-level) prototypes enable different classes to be more distinctive and distant from each other, while the prototypes obtained by level 3 (high-level) have much more overlap among classes.
This suggests that with a large domain shift between the original source domain and the new target domain, low-level textures are more useful and distinctive.
This again demonstrates that our hierarchical variational memory to leverage prototypes at different levels is suitable for cross-domain few-shot learning
[Table]
[Table]
[Figure]
Few-shot across domains We evaluate hierarchical variational memory on the four different datasets under 5-way 5-shot and 5-way 20-shot in Table 4, the results for 50-shot are provided in the appendix B.4.
Our hierarchical variational memory achieves state-of-the-artperformance on all four cross-domain few-shot learning benchmarks under each setting.
On CropDisease (Mohanty et al., 2016), our model achieves high recognition accuracy under various shot conﬁgurations, surpassing the second best method, i.e. ProtoNet (Snell et al., 2017), by a large margin of 7 . 93% on the 5-way 5-shot.
Even on the most challenging ChestX (Wang et al., 2017), which has a large domain gap with the mini ImageNet, delivers 27 .
15% on the 5-way 5-shot setting, surpassing the second best
ProtoNet (Snell et al., 2017) by 3 . 10% .
The consistent improvements on all benchmarks under various conﬁgurations conﬁrm that our hierarchical variational memory is effective for cross-domain few-shot learning.
Few-shot within domain We also evaluate our method on few-shot classiﬁcation within domains, in which the training domain is consistent with the test domain.
The results using ResNet-12 are reported in Table 5, the results using Conv-4 are reported in the appendix B.5.
Our hierarchical variational memory achieves consistently better performance compared to the previous methods on both datasets.
The results demonstrate that hierarchical variational memory also beneﬁts performance for few-shot learning within domains. improve performance for few-shot learning within domains.
[Table]
C ONCLUSION
In this paper
[Table]
we propose hierarchical variational memory for few-shot learning across domains.
The hierarchical memory stores features at different levels, which is incorporated as an external memory into the hierarchical variational prototype model to obtain prototypes at different levels.
Furthermore, to explore the importance of different feature levels, we propose learning to weigh prototypes in a data-driven way, which further improves generalization performance.
Extensive experiments on six benchmarks demonstrate the efﬁcacy of each component in our model and the effectiveness of hierarchical variational memory in handling both the domain shift and few-shot learning problems.
E THICS S TATEMENT
Despite the rapid development of machine learning, most machine learning and deep learning applications still require very large amounts of annotated data and sufﬁcient sample coverage of each training category, e g., different viewing angles or lighting conditions, to enable the model predictions more accurate.
Moreover, collecting and annotating the training samples is time-consuming, costly, and even impractical in some cases.
For example, annotating medical images is an extremely time-consuming task that requires expert knowledge.
Our method has the capability to handle the problem of domain shift and limited labelled data and also has potential positive impacts in realistic applications e g., medical imaging and automatic driving.
Our method is therefore potentially useful in facing some negative social impacts accompanying applications, e g., lack the annotated medical data or the medical data distribution is inconsistent due to personal privacy issues or equipment issues between different hospitals.
R EPRODUCIBILITY S TATEMENT
We provide detailed descriptions of implementations and training settings in Section 4.1 and the
Appendix.
We will further open-source all code, scripts to reproduce the experiments with the exact hyperparameters, and scripts to calculate the evaluation results at:
https://github.com/ <redacted>/<redacted> .
A D ATASETS
We apply our method to four cross-domain few-shot challenges and two within-domain few-shot image classiﬁcation benchmarks.
Sample images from each dataset are provided in Figure 4.
[Figure]
B M ORE R ESULTS
The more ablation results on the 5-way 20-shot and 5-way 50-shot for cross-domain few-shot classiﬁcation are shown in Sec.
B.1, Sec.
B.2 and Sec.
B.3.
The results on four proposed crossdomain few-shot challenges under 5-way 50-shot setting are shown in Sec.
B.4.
The results of few-shot learning methods on the two benchmark datasets mini Imagenet and tiered Imagenet by the
Conv-4 are shown in the Sec.
B.5.
B. B ENEFIT OF HIERARCHICAL PROTOTYPE
To show the beneﬁt of the hierarchical prototypes, we compare hierarchical variational prototypes with variational prototypes (Zhen et al., 2020), which obtains the probabilistic prototypes of the last layer.
Table 10 and Table 11 show the the hierarchical variational prototypes consistently outperform variational prototypes on all cross-domain few-shot classiﬁcation tasks under the 5-way 20-shot and 5-way 50-shot setting.
[Table]
[Table]
B. H IERARCHICAL VS . FLAT VARIATIONAL MEMORY
We show that a ﬂat memory as used in (Zhen et al., 2020) is less suitable for cross-domain few-shot learning in Table 8 and 9 under the 5-way 20-shot and 5-way 50-shot setting.
Hierarchical variational memory consistently achieves the best performance among all the datasets and settings.
[Table] [Table]
B ENEFIT OF LEARNING TO WEIGH PROTOTYPES
Table 10 and 11 show the results of hierarchical variational memory with bagging vs. hierarchical variational memory for the cross-domain few-shot classiﬁcation under the 5-way 20-shot and 5-way 50-shot setting.
hierarchical variational memory with learning to weigh prototype performs best overall.
[Table]
B. F EW - SHOT ACROSS DOMAIN
We evaluate hierarchical variational memory on the four different datasets under 5-way and 50-shot conﬁgurations in Table 12.
Our hierarchical variational memory achieves the new state-of-the-art performance on all the cross-domain few-shot learning benchmarks under 5-way and 50-shot setting.
B. F EW - SHOT WITHIN DOMAIN
We report the results of few-shot learning methods on the two few-shot within domain benchmark datasets mini Imagenet and tiered Imagenet by the Conv-4 backbones in Table 13.
Under both datasets, hierarchical variational memory consistently outperforms the previous approaches.
C T RAINING S PEED
We plot the training loss versus training iterations for different algorithms in Figure 5.
From Figureand Table 4, we conclude our hierarchical variational memory achieves best training efﬁciency and classiﬁcation accuracy, which demonstrates that our hierarchical variational memory is effective for cross-domain and within domain few-shot learning.
D R ESULTS ON DATA AUGMENTATION
We also provide results for few-shot within domain using a ResNet-12 backbone under data augmentation in the meta-training stage following (Zhang et al., 2021).
The results are shown in Table 14. With [Table] [Table]
data augmentation for few-shot within domain, our model also consistently achieves thecompetitive performance.
E B ENEFITS OF HIERARCHICAL STRUCTURES
We ﬁrst show the beneﬁt of hierarchical structures for few-shot within domain in Table 15.
To show the effect of our hierarchical formulation, we compare with VSM, which does not use the hierarchical structure for the prototype, and with hierarchical variational memory (last level).
We found a performance improvement by using the hierarchical formulation for the prototype.
Also, by
[Table]
achieves competitive performance.
[Table]
comparing the hierarchical variational memory (last level) and hierarchical variational memory, we found that the performance has further improved, which shows the effect of ensembling.
F B ENEFITS OF MODEL SIZE
We also ablate our approach in terms of model size, by varying the backbone.
We report few-shot classiﬁcation within domain by using Conv-4, WRN-28-10, ResNet-12 and ResNet-18 in Table 16.
The performance of our method increases along with the increase in backbone capacity.
As our memory is much smaller than the backbones, the trend in experimental results is not affected by the change of backbone.
[Table]
[Table]