S WITCH TO G ENERALIZE : D OMAIN -S WITCH L EARNING FOR C ROSS -D OMAIN F EW -S HOT C LASSIFICATION
A BSTRACT
This paper considers few-shot learning under the cross-domain scenario.
The cross-domain setting imposes a critical challenge, i.e. , using very few (support) samples to generalize the already-learned model to a novel domain.
We hold a hypothesis, i.e. , if a deep model is capable to fast generalize itself to different domains (using very few samples) during training, it will maintain such domain generalization capacity for testing.
It motivates us to propose a novel DomainSwitch Learning (DSL) framework.
DSL embeds the cross-domain scenario into the training stage in a “fast switching” manner.
Speciﬁcally, DSL uses a single domain for a training iteration and switches into another domain for the following iteration.
During the switching, DSL enforces two constraints: 1) the deep model should not over-ﬁt the domain in the current iteration and 2) the deep model should not forget the already-learned knowledge of other domains.
These two constraints jointly promote fast generalization across different domains.
Experimental results conﬁrm that the cross-domain generalization capacity can be inherited from the training stage to the testing stage, validating our key hypothesis.
Consequentially,
DSL signiﬁcantly improves cross-domain few-shot classiﬁcation and sets up new state of the art.
I NTRODUCTION
This paper challenges a realistic problem, i.e. , the cross-domain scenario, for few-shot learning.
Basically, the few-shot learning task uses a classiﬁer learned on the training set to recognize novel classes with very few support samples.
In real-world applications, there is usually a domain gap between the training samples and the support samples Tseng et al.
Guo et al.
; Chen et al.
The domain gap further imposes a critical challenge: since the support samples are very rare, they do not sufﬁce for mitigating the domain gap between the support set (for novel classes) and the training set.
Consequentially, the domain gap signiﬁcantly compromises the recognition accuracy of the novel classes.
Therefore, learning a model with strong cross-modality generalization capacity is important for cross-domain few-shot learning Tseng et al.
We argue that we may enhance the desired cross-modality generalization by “learning-togeneralize”.
To be more speciﬁc, we hold a hypothesis / intuition that if a deep model learns to fast generalize itself to different domains (using very few samples) during training, it will maintain the good domain generalization capacity for testing.
We model our intuition into a novel Domain-Switch Learning (DSL) framework, as illustrated in
Fig.
DSL uses multiple ( M > 1 ) domains to construct cross-domain training scenario in a “fast switching” manner.
Instead of simply mixing all the domains to construct a mini-batch (Fig. 1 (a)),
DSL includes only a single domain into every training iteration and switches to another domain for the following iteration (Fig. 1 (b)).
Moreover, each iteration contains very few samples per category, so as to imitate the few-shot setting.
Therefore, after every switch, the deep model crosses into a domain different from the former one, yielding the cross-domain few-shot scenario.
[Figure]
learnedknowledge
[Figure]
An important advantage of this domain-switch learning manner is that it can suppress learning from domain-speciﬁc knowledge, as illustrated in Fig.
Speciﬁcally, we consider each domain contains both domain-general and domain-speciﬁc knowledge. If we mix all the domains for each training iteration (Fig. 1 (a)), the deep model may memorize all the domain-speciﬁc knowledge, as well as the domain-general knowledge.
Consequentially, during testing, the domain-speciﬁc knowledge hinders generalization towards the novel domain and thus compromises few-shot classiﬁcation accuracy.
In contrast, in DSL (Fig. 1 (b)), the model only learns from one single domain in an iteration.
Since the domain-speciﬁc knowledge of the current domain does not ﬁt the following domain, the deep model is prone to discarding these domain-speciﬁc knowledge during the following iteration.
To further promote cross-domain generalization in this domain-switch learning scheme, DSL enforces two constraints as follows:
The deep model should not over-ﬁt the domain in the current iteration , because over-ﬁtting the current domain makes the model memorize much of the domain-speciﬁc knowledge.
The ﬁrst constraint is implemented with a domain-speciﬁc prompter module (consisted of multiple prompters).
After the model learns from the i -th training domain D i and gets updated, its prediction accuracy on D i increases and is relatively high.
We store this edition of model as the prompter for D i .
Next time before the training domain switches to D i again, the model just gets updated from D i − 1 and becomes relatively inaccurate on D i .
Given the accurate prompter and the inaccurate learner ( w.r.t. D i ), we average their prediction for supervision so that the penalty on the learner will be suppressed.
The details are to be accessed in Section 3.3.
The deep model should not forget the already-learned knowledge of former domains , so that next time the model crosses into these domains again, it can directly re-use the corresponding knowledge for prediction.
The second constraint is implemented with a domain-general teacher module.
Speciﬁcally, we collect several historical models and average their parameters to get a mean model.
During training, we use the softmax prediction of the mean model as the auxiliary supervising signals for the learner (apart from the ground truth label).
The mean model serves as a teacher distilling the already-learned knowledge of former domains to the learner.
Since this teacher has no obvious bias towards any single domain, we name it as a domain-general teacher.
The details of the domaingeneral teacher module are to be accessed in Section 3.4.
These two constraints achieve complementary beneﬁts for DSL, and jointly reinforce the crossdomain generalization.
Extensive experiments under four cross-domain scenarios show that DSL consistently improves cross-domain few-shot learning and achieves performance on par with the state-of-the-art methods
Our main contributions are summarized as follows:
• We propose a novel Domain-Switch Learning (DSL) framework for cross-domain few-shot learning.
DSL uses multiple domains for training and switches the domain in consecutive training iterations.
It provides a cross-domain learning scenario where the deep model learns to generalize across different domains.
• Under the DSL framework, we integrate two modules, i
e. , the domain-speciﬁc prompter and the domain-general teacher.
These two modules achieve complementary beneﬁts for DSL and jointly reinforce the cross-domain generalization.
• We conduct extensive experiments to validate the effectiveness of the proposed DSL.
Experimental results conﬁrm that the fast generalization capacity can be inherited from training to testing and thus improves cross-domain few-shot classiﬁcation.
On all the four popular benchmarks, DSL achieves performance on par with the state of the art.
R ELATED WORKS
Few-Shot Learning.
Few shot learning methods could be roughly categorized into two branches: meta-learning and ﬁne-tuning methods.
Some meta-learningSatorras & Estrach (2018); Mishra et al.
; Ren et al.
; Snell et al.
; Sung et al.
; Vinyals et al.
(2016) methods map the few-shot samples into a non-linear embedding space and evaluate the similarity between support samples and the query samples.
Recently, ﬁne-tuning methods Chen et al.
; Tian et al.
; Afrasiyabi et al.
(2020) propose to pre-train the model on the training set (typically in a classiﬁcation learning approach), and ﬁne-tune the classiﬁer on the novel classes.
The ﬁnetuning methods achieve competitive performance compared with the state-of-the-art meta-learning methods.
This work considers the few-shot learning task under the cross-domain scenario and adopts the ﬁne-tuning pipeline as the baseline.
Domain Generalization and Adaptation may be viewed as two closely related concepts.
Domain adaptation uses abundant (unlabeled) samples to adapt the already-learned model from the source domain to the target domain Ganin & Lempitsky (2015);
Tzeng et al.
Long et al.
(2017; 2015; 2016); Tzeng et al.
In contrast, domain generalization aims to generalize the knowledge from multiple source domains to the target domain without learning any samples of the target domain during the training stage Balaji et al.
; Li et al.
In cross-domain fewshot learning, there is a domain gap between the training set and the testing set.
To promote the cross-domain generalization, Tseng et al.
(2020) considers the challenge is more of a domain generalization problem than a domain adaptation problem.
This consideration is reasonable because the support samples are very rare and do thus not sufﬁce for a typical domain adaptation.
From the viewpoint of domain generalization, FWT Tseng et al.
(2020) proposes to simulate the cross-domain scenario to improve the domain generalization capacity.
It indeed inspires us of the multi-domain training strategy.
In spite of sharing the multi-domain training strategy, DSL signiﬁcantly differs from FWT:
First, FWT uses changes the training domain epoch-by-epoch.
It does not rely on fast switching for better generalization.
In contrast, DSL is featured for the novel fast switching manner and ﬁnds that this manner promotes learning domain-general knowledge.
Second, DSL further considers two constraints in multi-domain training strategy, i.e., the deep model should not over-ﬁt every single domain and keep recognition on learned knowledge shared with other domains.
Finally, DSL achieves superior cross-domain effect than FWT (Section 4.2).
Catastrophic forgetting.
Catastrophic forgetting plays an important role in the mechanism of DSL.
When the deep model is trained sequentially on multiple tasks (domains), there is a tendency that the knowledge learned on former tasks (domains) is abruptly lost, yielding the so-called catastrophic forgetting Kirkpatrick et al.
; McCloskey & Cohen (1989); French (1999).
In DSL, when the deep model switches from a domain to another, it forgets much of the knowledge on the former domain, due to catastrophic forgetting.
Forgetting the domain-speciﬁc knowledge is beneﬁcial, while forgetting the domain-general knowledge is harmful.
In response, next time the deep model comes across the former domain again, we 1) supplement it with the corresponding knowledge with a domain-speciﬁc prompter to prevent over-ﬁtting the domain-speciﬁc knowledge and 2) use a domain-general teacher to reinforce the common knowledge shared among all the former domains.
In a word, in its mechanism, DSL actually utilizes catastrophic forgetting to improve cross-domain generalization.
M ETHODS 3. P RELIMINARIES
In the cross-domain few-shot learning task, there is a signiﬁcant domain gap between the training set and the testing set.
The testing set consists the support set and query set, which are sampled from the same novel categories (which are unseen in the training set).
The support set provides very few labeled samples for recognizing the unlabeled samples in the query.
Concretely, the support set contains C classes and each class has K samples, which is a C -way K -shot setup in few-shot learning.
Our work is based on a popular ﬁne-tuning pipeline Chen et al.
; Tian et al.
; Afrasiyabi et al.
The ﬁne-tuning baseline uses the training set D to learn a deep model consisted of a feature encoder F and a feature classiﬁer Y .
F is parameterized with θ and Y is parameterized with a weight matrix W .
During training, given a sample x , the feature encoder ﬁrst maps x into a feature F ( x ; θ ) ∈ R d .
Then the feature classiﬁer Y predicts the softmax probability of x belonging to each training category as P ( i | x ; θ , W ) = [Equation]
F [Equation] x ; θ [Equation]
) ( w i is the weight vector for the i -th category).
The training is supervised with a popular classiﬁcation loss function, i.e. , the cross-entropy loss, which is formulated as:
[Equation]
where y is the label for x , w y is the weight vector corresponding to the ground-truth category.
During the testing stage, we replace the feature classiﬁer Y with a new linear classiﬁer L .
L is a C -way classiﬁer, corresponding to the total number of classes in the support set.
We freeze the parameters of the feature encoder F and ﬁne-tune the parameters of L with samples from the support set.
Finally, we use the feature encoder F and the learned L as the predictor for the query.
This paper focuses on the training stage, i.e. , learning the feature encoder F and the classiﬁer Y on the training set.
3. O VERVIEW OF D OMAIN -S WITCH L EARNING
Domain-Switch learning (DSL) employs M datasets to construct the training set, i.e. , D = { D 1 , D 2 , ..., D M } .
D i is an individual domain, which may be selected from some popular fewshot datasets, e.g. , CUB Welinder et al.
Cars Krause et al.
, Places Zhou et al.
Plantae Horn et al.
These switchable training sets are relatively small, and does not sufﬁce for learning a discriminative feature encoder.
Therefore, following Tseng et al.
(2020), we employ a relatively large-scale dataset mini-ImageNet Deng et al.
(2009) as a “basic” training set.
During training, each mini-batch actually contains samples from the basic mini-ImageNet and a switchable training set D i .
We note that such implementation does NOT contradict our motivation of using a single domain for each training iteration, because we may view the mixture of mini-ImageNet and a
D i as a single “compound” domain.
During training, DSL uses samples from a single switchable domain D i (and the mini-ImageNet) for each iteration, as illustrated in Fig. 2.
Since mini-ImageNet appears for every iteration, we omit it in Fig.
2 for simplicity.
The domain-switching process is formulated as:
[Equation]
where B j is the training batch at the j -th iteration, D i is the i-th domain, (cid:98)·(cid:99) is the rounding down operation.
We deﬁne a enumeration of all the M domains as a switching round .
Learner 𝐷learner softmax prompter softmax 𝜆 𝑝 + 1 − 𝜆 𝑝 𝒫 𝑝 𝒫 𝑝 𝐷 1 F 𝐷 2 F 𝐷 𝑀 F Teacher past iterations Ƹ𝑝 model updated by 𝐷 𝑁 … … … 𝐷 2 F iteration ( i +1) model updated by 𝐷model updated by 𝐷… Prompter
Binary KLD Loss
Reweighted CE Loss …
Cross-Entropy Loss iteration i (current) iteration ( i -1) iteration ( i - M +1) iteration ( i - M ) …
Reweighted CE Loss weighted average average
F Feature encoder
In order to further improve the cross-domain generalization in the above “fast switching”, DSL enforces two constraints: 1) the deep model should not over-ﬁt the domain in the current iteration and 2) the deep model should not forget the knowledge of other domains.
These two constraints are implemented with a Domain-Speciﬁc Prompter (Section.3.3) and a Domain-General Teacher (Section.3.4) module, respectively.
3. D OMAIN -S PECIFIC P ROMPTER
Deﬁnition of the prompter : Let us consider in the j -th iteration, the model has already learned from
D i and gets updated.
We store this edition of model as the prompter for the ( j + M ) -th iteration (when the learner is to learn from D i again), as illustrated in Fig.
The prompter aims to prevent the learner from over-ﬁtting the current domain D i .
Given a sample x in current domain, DSL averages the softmax prediction of the prompter ( P ( y | x ; θ P , W P ) , where the upper-script P denotes the prompter) and the softmax prediction of the learner ( P ( y | x ; θ , W ) ), which is formulated as:
[Equation]
where λ is a hyper-parameter.
The condition arg max i
P ( i | x ; θ P , W P ) =
y is to ensure the prompter makes the correct prediction.
With the re-weighted softmax prediction ˆ P , the cross-entropy loss for supervising the classiﬁcation is transformed into a Re-weighted Cross-Entropy loss (RCE loss in Fig.
[Equation]
We explain how the domain-speciﬁc prompter prevents the learner from over-ﬁtting the current domain with two following remarks (Remark1 and Remark2).
Remark1 : Compared with the learner, the prompter has relatively higher predicted probability on the ground-truth category, i.e. , P ( y | x ; θ P , W P ) ≥ P ( y | x ; θ , W ) . [Figure]
Remark1 is intuitive.
Without losing generality, we assume the current learning domain in Fig.is D 1 .
The domain-speciﬁc prompter is the former model updated by D 1 and thus ﬁts D 1 well.
In contrast, the current learner is newly updated by D M and forgets some knowledge of D 1 , due to the well-known “catastrophic forgetting”.
Therefore, the prompter has higher predicted probability on the ground-truth category than the learner does.
We empirically validate the above analysis in Fig.
3, from which we clearly observe that the prompter has higher prediction accuracy than the learner.
Remark2 :
Since P ( y | x ; θ P , W P ) ≥ P ( y | x ; θ , W ) ( i.e. , Remark1), the re-weighted procedure in
Eq.
3 alleviates the penalty on the learner, which is beneﬁcial to exclude domain-speciﬁc knowledge.
Please refer to the Appendix for the conceptual proof.
Consequentially, the prompter counters overﬁtting the current domain.
3. D OMAIN -G ENERAL T EACHER
Deﬁnition of the domain-general teacher : We recall that a switching round is a period within which all the M domains are enumerated.
As illustrated in Fig.
2, we average the parameters of all the M encoders updated in the past switching round to generate the domain-general teacher for the current domain D i , which is formulated as:
[Equation]
The teacher aims to preserve and distill the domain-general knowledge learned in former domains to the current learner.
We note that the domain-general teacher is different from the popular temporal moving average for semi-supervised and unsupervised task Tarvainen & Valpola (2017);
He et al.
Grill et al.
In temporal moving average, the teacher absorbs parameters from all the historical models (with higher weighting factor for the more recent model).
In contrast, our domain-general teacher absorbs parameters from only M historical models with equal weighting factor.
Remark3 :
When the learner gets updated from D i , it has higher prediction accuracy on D i than the teacher does.
Meanwhile, the teacher has higher prediction accuracy on former domains D j ( j (cid:54) = i ) .
Remark3 is intuitive.
It is because the teacher absorbs knowledge from M historical learners, therefore preserving historical knowledge of the former domains.
We empirically validate this intuition in Fig.
3 (b), which is consistent with Remark3.
Moreover, since the teacher averages the parameters of all the M historical learners, it has no obvious bias towards any single domain.
Therefore, the teacher has relatively smaller variation of accuracy on all the domains, as shown in Fig.
3 (b).
Binary Kullback-Leibler divergence loss.
Given a sample x in the current domain, we use the output of the teacher P ( y | x ; θ T , W T ) as an auxiliary supervising signal for the learner.
To this end, we propose a novel Binary Kullback-Leibler divergence (KLD) Loss as follows: [Table] [Table]
[Equation]
where P y = P
( y | x ; θ , W ) and P T y
= P ( y | x ; θ T , W T ) .
The difference between the proposed BKLD loss and the popular KLD loss is: KLD aims to make the student consistent with the teacher at all the entries of the softmax prediction.
In contrast, the proposed BKLD loss only focuses on the consistency at the ground-truth entry.
In the experiment in
Section 4.4, we empirically show BKLD loss is better than the KLD loss for DSL.
Overall Training.
We recall that besides the switchable domains, mini-ImageNet is employed as a “basic” training set, which is sampled into all the training iterations.
For mini-ImageNet, we adopt the cross-entropy loss as a basic loss L Basic .
The overall training loss is as follows:
[Equation]
where α is a weighting factor.
E XPERIMENTS 4. E XPERIMENTAL S ETUPS
We evaluate the proposed DSL on ﬁve datasets: i.e., 4 ﬁne-grained datasets (CUB Welinder et al.
(2010), Cars Krause et al.
, Places Zhou et al.
(2018), and Plantae Horn et al.
andpopular large scale dataset mini-ImageNet Deng et al.
We use ResNet-10 He et al.
without pretraining as the backbone network.
Please refer to the Appendix for more experimental setups and implementation details.
[Table]
4. C OMPARISON WITH THE STATE OF THE ARTS
We compare DSL with the existing state of the arts on 4 benchmarks.
We adopt the leave-one-out setting, i.e. , one of the ﬁne-grained datasets is chosen as a testing set and the other 3 datasets (out of the 4 datasets) are used as the training sets.
Meanwhile, we employ mini-ImageNet as a “basic” training set, which is available during the whole training stage.
The results of 5-way 1-shot and 5-way 5-shot are summarized in Table 1 and Table 2 , respectively.
From Table 1, we clearly observe the superiority of DSL under 5-way 1-shot classiﬁcation.
First, comparing DSL with all the single-domain training methods, we ﬁnd that DSL achieves signiﬁcant superiority.
For example, DSL outperforms the most competing single-domain method (“RelationNet + ATA (s)”)
Wang & Deng (2021) by +7.13%, +5.34%, +2%, +7.45% on CUB, Cars, Places and Plantae, respectively.
Second, while the multi-domain training methods achieve considerable improvement (over their single-domain counterpart), DSL still outperforms them with a clear margin.
For example, DSL outperforms the most competing multi-domain method (“RelationNet + LFT (m)”) by +1.77%, +4.92%, +2.42%, +6.17% on CUB, Cars, Places and Plantae, respectively.
From Table 2, we clearly observe that DSL achieves competitive accuracy under the 5-way 5-shot classiﬁcation.
Most of the competing methods already appear in the comparison of 5-way 1-shot setting (Table 1).
Compared with all these methods (except NASE Liang et al.
, DSL maintains signiﬁcant superiority on all the four benchmarks.
For example, DSL surpasses “RelationNet + LFT(m)”Tseng et al.
(2020) by +8.58%, +15.09%, +6.75%, +11.71% on CUB, Cars, Places and
Plantae, respectively.
NASE in Table 2 is a very recent and competitive single-domain training method.
It splits the support samples of the same category into two sub-sets for similarity learning.
Therefore, it requires more than 1 support samples for each category and is incompatible to the 1-shot setting.
With this unique similarity-learning manner, it achieves very competitive performance ( e.g. , 68.51% on CUB).
Compared with NASE, DSL obtains consistent superiority For example, DSL surpasses NASE by 5 .
06% on CUB.
4. T HE E FFECTIVENESS OF THE D OMAIN -S WITCH O PERATION
A key characteristic of DSL is the domain-switch operation.
In Table 3, we compare the singledomain baseline (on mini-ImageNet), the domain-mix and the proposed domain-switch scheme (without domain-general teacher and the domain-speciﬁc prompter ) on CUB and Cars.
We draw two following observations:
First, compared with single-domain training, the domain-mix learning scheme only brings slight improvement.
Although it employs multiple domains (as well as multiple datasets) for training, the beneﬁt of more training data is trivial (less than 1% improvement).
We infer it is because in the domain-mix learning scheme, the disadvantage of learning domain-speciﬁc knowledge largely offsets the advantage of more training data.
Second, comparing “domain-mix” with “domain-switch”, we observe that domain-switch is a superior multi-domain learning manner.
Speciﬁcally, domain-switch surpasses domain-mix by 1 .
60% accuracy under 5-way 5-shot setting, respectively.
It is consistent with our conceptual analysis, i.e. , the domain-switch learning manner better ignores the domain-speciﬁc knowledge, therefore improving the cross-domain generalization.
4. A BLATION S TUDIES
Ablation on the prompter and teacher module.
While the domain-switch framework already improves cross-domain generalization, DSL uses the domain-speciﬁc prompter and the domaingeneral teacher to prevent over-ﬁtting domain-speciﬁc knowledge and to reinforce the domain-
BKLD
KLD no teacher 73.
BKLD
KLD no teacher 50.
1 round 2 rounds 3 rounds 73.
1 round 2 rounds 3 rounds 5-way 1-shot 5-way 5-shot 5-way 1-shot 5-way 5-shot
[Figure] [Figure]
[Table]
All (DSL)
[Table]
general knowledge, respectively.
We investigate their beneﬁts through ablation in Table 4, from which we draw two observations:
First, both the domain-speciﬁc prompter and the domain-general teacher are beneﬁcial.
Based on the domain-switch operation, we ﬁnd that adding the prompter or the teacher brings accuracy improvement.
For example, under the 5-way 5-shot setting, adding the prompter and the teacher increases the accuracy by 4 .
80% and 4 .
12% , respectively.
Second, the domain-speciﬁc prompter and the domain-general teacher are complementary to each other.
While both modules already bring independent improvement, combining them (“All (DSL)”) yields even higher accuracy.
It validates that these two modules achieve complementary beneﬁts ( i.e. , countering domain-speciﬁc knowledge and promoting domain-general knowledge).
Analysis on some important conﬁgurations of the teacher module.
We investigate two conﬁgurations: 1) using a novel Binary KLD loss (instead of the popular KLD loss) for knowledge distillation and 2) using the historical models within a single switching round for generating the teacher.
Fig.
4 (a) shows that the domain-general teacher favors the Binary KLD loss than the canonical KLD loss.
Under the 5-way 5-shot setting, the canonical KLD barely improves over “no teacher”, while the BKLD surpasses KLD by 1 .
69% accuracy.
We infer it is because the knowledge from other domains is not accurate for the non-target classes.
Therefore, forcing the learner to approximate the teacher w.r.t.
all the non-target classes is inappropriate.
Fig.
4 (b) shows that using more switching rounds gradually compromises the effectiveness of the teacher, compared with using a single round.
We infer it is because increasing switching rounds includes some severely out-of-date models, whose knowledge largely diverges from the up-of-date model.
Therefore, we use a single switching round for generating the teacher model.
C ONCLUSION
This paper proposes a novel Domain-Switch Learning (DSL) method for cross-domain few-shot classiﬁcation.
DSL embeds the cross-domain scenario into the training stage in a fast switching manner.
We show that during the domain switching procedure, the deep model favors domaingeneral knowledge and is prone to ignoring the domain-speciﬁc knowledge, so as to fast adapt itself to different domains.
Moreover, DSL employs a domain-speciﬁc prompter and a domain-general teacher module to further promote the cross-domain generalization capacity.
Experiments conducted on multiple benchmarks demonstrate that DSL improves cross-domain few-shot classiﬁcation and the achieved results are on par with the state of the arts.
E THICS S TATEMENT
This paper can help to improve the model generalization capacity in recognizing novel classes with limited labeled samples ( e.g., the rare birds, the rare plants,etc) under a signiﬁcant domain gap between training sets.
It may be applied to some environmental protection projects which require monitoring on the diversity of species.
Speciﬁcally, it can help recognize some rare species in the wild reserve.
We will explore more application scenarios of few-shot learning, as well as crossdomain few-shot learning.
Moreover, we will try to improve the reliability of few-shot learning systems to reduce the potential problems.
R EPRODUCIBILITY S TATEMENT
The DSL is reproducible.
In the main text, we describe the utilized datasets in DSL, i e., 4 ﬁnegrained datasets (CUB, Cars, Places, and Plantae) and 1 popular large scale dataset mini-ImageNet.
We provide the details about the experimental implementation, the proof of the proposed remark and the analysis of some hyper parameters in appendix
A A PPENDIX
In the appendix, we supply the details which are not described in the main text due to space limitation.
In Section A.1, we provide the details about the experimental implementation.
In Section A.2, we prove the Remark2 in Section 3.3.
In Section A.3, we investigate DSL on different backbones.
In the main experiments, DSL and all the competing methods adopt the standard inductive inference.
In Section A.4, we further provide some comparison under the transductive setup ( e.g. , with a label propagation method Liu et al.
In Section A.5, we analyze the impact of some hyper-parameters.
A. I MPLEMENTATION D ETAILS .
We use ResNet-10 He et al.
without pretraining as the backbone network.
When training the model on the training set, we use the Adam Kingma & Ba (2015) optimizer.
We train the model with 400 epochs and set the initial learning rate as 1e-3.
In each training iteration, the mini-batch size of switchable domain and the “basic” training set are both 64.
In each mini-batch , we randomly sample 16 classes from each domain (4 images per class).
We evaluate the classiﬁcation accuracy over 1000 experiments on the test set.
When ﬁne-tuning the model on the support set, we randomly sample C classes and K samples per class, according to the
C -way K -shot setting( e.g., C =5, K =1 or 5).
We use the SGD Qian (1999) to optimize the linear classiﬁer layer and the initial learning rate is 0.01.
The weight decay of SGD is 0.001 and the SGD momentum is 0.9.
A. R EMARK 2 PROOF .
We ﬁrst recall the domain-speciﬁc prompter in Section 3.3.
Given a sample x in current domain, we average the softmax predictions of the prompter P ( y | x ; θ P , W P ) and learner P ( y | x ; θ , W ) to get re-weighted softmax prediction ˆ P .
It can prevent the learner from over-ﬁtting the current domain.
Remark2 :
Given P ( y | x ; θ P , W P ) ≥ P ( y | x ; θ , W ) , we have the re-weighted softmax prediction ˆ P ≥ P ( y | x ; θ , W ) .
It thus reduces ﬁnal prediction errors (on the current domain), alleviating the penalty on the learner.
The proof is as illustrated by:
[Equation]
where | · | denotes the value of vector.
λ ∈ [0 , 1] is a hyper parameter.
[Figure]
A. DSL ON VARIOUS BACKBONES .
We further analyze the adaptation of our methods on Conv4 and ResNet18.
The performance on
CUB are shown in Fig.
We adopt “domain-mix” learning manner as the baseline.
We observe that the proposed method can still achieve considerable improvements on both shallow or deeper backbone settings, which proves the robustness of the proposed method.
A. A DDITIONAL EXPERIMENTS .
We adopt a simple label propagation method Liu et al.
(2020) for DSL and compare “DSL+LP” with label-propagation based methods.
[Table]
From Table 5, we observe that when all the methods use transductive setup ( e.g. , label propagation),
DSL achieves competitive accuracy under the 5-way 5-shot classiﬁcation. ’
DSL+LP’ outperforms
GNN + LFT by +3.61%, +12.33%, +0.05%, +6.86%on CUB, Cars, Places and Plantae, respectively.
A. H YPER - PARAMETERS ANALYSIS .
We analyze the impact of three important hyper-parameters,i.e., λ in re-predicted procedure(Eq. 3 ), the α in overall loss(Eq. 7 ) and the temperature coefﬁcient µ in computing the predictions P T y =
P ( y | x ; θ T , W T ) of domain-general teacher.
[Figure]
In Fig.
6, we evaluate the impact of hyper-parameter λ , which controls the weight of predictions from domain-speciﬁc prompter in Eq.
We observe that the accuracy ﬁrst increases (when λ increases from 0 to 0.4) and then decreases (when λ further increases to 1.0).
We set λ = 0.4 as the weight factor.
[Figure]
In Fig.
7, we evaluate the impact of hyper-parameter α , which controls the weight of Binary KLD loss and RCE loss in Eq.
We observe that the accuracy ﬁrst increases (when α increases fromto 0.7) and then decreases (when α further increases to 1.0).
We set α = 0.7 as the weight factor.
[Figure]
In Fig. 8, we evaluate the impact of hyper-parameter µ , which denotes the temperature coefﬁcient in computing predictions P T y =
P ( y | x ; θ T , W T ) of domain-general teacher.
When the temperature coefﬁcient µ ≥ 1, we consider that the predictions of domain-general teacher are more smooth and thus transfer more general knowledge shared with other domains.
We set µ to vary from 1 to 2.
It is observed that the accuracy undergoes an increase (when µ increases from 1 to 1.25) and then a decrease (when µ > 1.25).
Therefore, we set µ = 1.25 as the optimized threshold for domain-general teacher.