DSP: D YNAMIC S EMANTIC P ROTOTYPE FOR G ENERATIVE Z ERO -S HOT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Conditional generative models ( e.g. , generative adversarial network (GAN)) have advanced zero-shot learning (ZSL).
Studies on the generative ZSL methods typically produce class-speciﬁc visual features of unseen classes to mitigate the issue of lacking unseen samples based on the predeﬁned class semantic prototypes.
As these empirically predeﬁned prototypes are not able to faithfully represent the actual semantic prototypes of visual features ( i.e. , visual prototypes), existing methods limit their ability to synthesize visual features that accurately represent real features and prototypes.
We formulate this phenomenon as a visual-semantic domain shift problem.
It prevents the conditional generative models from further improving the ZSL performance.
In this paper, we propose a dynamic semantic prototype learning (DSP) method to align the empirical and actual semantic prototypes for synthesizing accurate visual features.
The alignment is conducted by jointly reﬁning semantic prototypes and visual features so that the conditional generator synthesizes visual features which are close to the real ones.
We utilize a visual → semantic mapping network (V2SM) to map both the synthesized and real features into the class semantic space.
The V2SM beneﬁts the generator to synthesize visual representations with rich semantics.
The visual features supervise our visual-oriented semantic prototype evolving network (VOPE) where the predeﬁned class semantic prototypes are iteratively evolved to become dynamic semantic prototypes.
Such prototypes are then fed back to the generative network as conditional supervision.
Finally, we enhance visual features by fusing the evolved semantic prototypes into their corresponding visual features.
Our extensive experiments on three benchmark datasets show that our DSP improves existing generative ZSL methods, e.g. , the average improvements of the harmonic mean over four baselines ( e.g. , CLSWGAN, f-VAEGAN, TF-VAEGAN and FREE) by 8.5%, 8.0% and 9.7% on CUB , SUN and AWA2, respectively.
I NTRODUCTION
Zero-shot learning (ZSL) recognizes the unseen classes, by transferring semantic knowledge from some known classes to unknown ones.
Recently, conditional generative models ( e.g. , generative adversarial networks (GANs) Goodfellow et al.
and variational autoencoder (VAE) Kingma & Welling (2014)) have been successfully applied in ZSL and achieved promising performance.
They synthesize the class-speciﬁc images or visual features of unseen classes to mitigate the lack of unseen samples based on the condition of the predeﬁned class semantic prototypes Arora et al.
; Xian et al.
(2018; 2019b); Chen et al.
(2021a;b).
Because of the high quality synthesis of GAN, studies arise to generate unseen samples in the CNN features to beneﬁt ZSL Yan et al.
; Xian et al. (2019b).
The class-speciﬁc unseen sample generations are conditioned on the semantic prototypes.
However, there is a limitation that the empirically predeﬁned semantic prototypes are not able to faithfully represent the actual semantic prototypes of visual features ( i.e. , visual prototypes).
As shown in Fig.
1, i) the predeﬁned class semantic prototypes are annotated by human, inevitably resulting in some inaccurate annotations ( e.g. , the attribute “bill color orange” in Sample-2 in Fig. 1(a)), and ii) the visual images are with multi-views, resulting in the phenomenon that some importantly annotated attributes do not appear in the visual representations ( e.g. , the attribute “bill color orange” in Sampleand attribute “leg color black” in Sample-4 in Fig. 1(a)).
That is, some visual representations of one
[Figure]

[Figure]
mapped with their the features synthesized by existing generative ZSL methods are far away from their corresponding real visual features and visual prototypes (as shown in Fig. 1(b)), which heavily limits their classiﬁcation performance.
We formulate this phenomenon as a problem of Visual-Semantic Domain Shift .
Therefore, it is essential to reﬁne the empirical deﬁned semantic prototype.
Using an accurate prototype beneﬁts the generator supervision to improve the generative ZSL performance.
In light of these observations, we argue that the predeﬁned semantic prototypes can be reﬁned to become consistent to the actual semantic prototypes ( i.e. , visual prototypes) according to the visual information.
As such, the reﬁned semantic prototypes can capture the visual samples exactly and act as an accurate conditional signal for the generator, enabling generative ZSL methods to learn a desirable semantic → visual mapping ( i.e. , generator).
Targeting this goal, in this paper, we propose a dynamic semantic prototype learning (DSP) method to align the empirical and actual semantic prototypes for synthesizing reliable visual features.
The alignment is conducted by jointly reﬁning semantic prototypes and visual features so that the conditional generator synthesizes class-speciﬁc visual features which are close to the real ones, as shown in Fig.
1(c).
Accordingly, our DSP well tackles the visual-semantic domain shift problem and advances the generative ZSL methods.
Speciﬁcally, DSP consists of a visual → semantic mapping network (V2SM) and visual-oriented semantic prototype evolving network (VOPE).
Cooperated with the generator, V2SM maps visual features into the class semantic space, enabling the conditional generator to synthesize class-speciﬁc visual samples with rich semantic information and convey visual information into the VOPE.
Under the supervision of visual information, VOPE iteratively evolves and reﬁnes the predeﬁned class semantic prototypes to become dynamic semantic prototypes, which are fed back to the generative network as a conditional supervision to encourage the generator for reliable visual feature synthesis.
Finally, DSP concatenates the evolved semantic prototypes into visual features for enhancement, enabling the visual features to be closer to their corresponding semantic prototypes and alleviate the cross-dataset bias problem Chen et al. (2021a).
Extensive experimental results demonstrate consistent performance gains over the state-of-the-art generative ZSL methods on three challenging benchmark datasets, i.e. , CUB Welinder et al.
SUN Patterson & Hays (2012) and AWA2 Xian et al. (2019a).
For example, the average improvements of harmonic mean over four baselines ( e.g. , CLSWGAN Xian et al.
(2018), f-VAEGAN Xian et al.
(2019b), TF-VAEGAN Narayan et al.
and
FREE Chen et al.
(2021a)) are 8.5% , 8.0% and 9.7% on CUB, SUN and AWA2, respectively.
Notably, our DSP is ﬂexible and effective to be entailed in any generative ZSL method.
R ELATED W ORK
Embedding-based Zero-Shot Learning.
Embedding-based ZSL methods were popular early, they learn a visual → semantic projection on seen classes that is further transferred to unseen classes Akata et al.
(2016; 2015); Huynh & Elhamifar (2020a)
; Chen et al.
; Huynh & Elhamifar (2020b),
Considering the cross-dataset bias between ImageNet and ZSL benchmarks Chen et al.
(2021a), embedding-based methods were recently proposed to focus on learning the region-based visual features to enhance the holistic visual features 1 using attention mechanism Yu et al.
Zhu et al.
; Xie et al.
; Huynh & Elhamifar (2020a); Xu et al.
However, these methods depends on the attribute features for attribute localization.
As such, Wang et al.
(2021) introduces
DPPN to iteratively enhance visual features using the category/visual prototype as supervision.
Differently, we jointly and mutually reﬁne the visual features and semantic prototype, enabling signiﬁcant visual-semantic interactions in generative ZSL.
Generative Zero-Shot Learning.
Since embedding-based methods learn the ZSL classiﬁer only on seen classes, inevitably resulting in the models overﬁtting seen classes.
To tackle this challenge, generative ZSL methods employ the generative models ( e.g. , VAE and GAN) to generate the unseen visual features for data augmentation Arora et al.
; Xian et al.
; Shen et al.
(2020), ZSL is then converted to a supervised classiﬁcation task further.
As such, the generative ZSL methods have shown signiﬁcant performance and become very popular recently.
However, there are lots of challenges that need to be tackled in generative ZSL.
To improve the optimization process of
ZSL methods, Skorokhodov & Elhoseiny (2021) introduces class normalization for the ZSL task.
Çetin et al.
(2022) introduces closed-form sample probing for generative ZSL.
To avoid ZSL models overﬁtting to seen classes, Liu et al.
(2021) introduces an isometric propagation network to strengthen the relation between classes within the visual and semantic spaces.
Chou et al.
(2021) designs a generative scheme to simultaneously generate virtual class labels and their visual features.
To enhance the visual feature in generative ZSL methods, Felix et al.
(2018), Narayan et al.
and
Chen et al.
(2021a) reﬁne the holistic visual features by learning a visual → semantic projection, which encourages conditional generator to synthesize visual features with more semantic information.
Orthogonal to these methods, we propose a novel dynamic semantic prototype learning method to tackle the visual-semantic domain shift problem and advance generative ZSL further.
Visual-Semantic Domain Shift vs Projection Domain Shift.
ZSL learns a projection between visual and semantic feature space on seen classes, and then the projection is transferred to the unseen classes for classiﬁcation.
However, data samples of the seen classes (source domain) and unseen classes (target domain) are disjoint, unrelated for some classes, and their distributions can be different, resulting in a large domain shift Fu et al.
; Wan et al.
; Pourpanah et al.
This challenge is the well-known problem of projection domain shift.
To tackle this challenge, inductive-based methods incorporate additional constraints or information from the seen classes
Zhang et al.
; Jia et al.
; Wang et al.
Besides that, several transductive-based methods have been developed to alleviate the projection domain shift problem Fu et al.
; Xu et al.
; Wan et al.
We should note that the visual-semantic domain shift is different from the projection domain shift.
Visual-semantic domain shift denotes that the two domains (visual and semantic domains) share the same categories and only the joint distribution of input and output differs between the training and test stages.
In contrast, the projection domain shift can be directly observed in terms of the projection shift, rather than the feature distribution shift Pourpanah et al. (2022).
As shown in Fig.
1, we demonstrate that visual-semantic domain shift is a bottleneck challenge in generative ZSL and is neglected by existing methods.
D YNAMIC S EMANTIC P ROTOTYPE L EARNING
Problem Setting: The problem setting of ZSL is formulated as follows.
Assume that we have seen class data D s = { ( x si , y si ) } with C s seen classes, where x si ∈ X denotes the i -th visual feature with 2048-dim extracted from a CNN backbone ( e.g. , ResNet101 He et al.
(2016)), and
y si ∈ Y s is the corresponding class label.
Note that D s is divided into training set D str and test set D ste according to
Xian et al.
(2019a).
Another set of unseen classes C u is D ute =
{ ( x ui , y ui ) } , where x ui ∈ X are the visual features of unseen classes, and y ui ∈
Y u are the corresponding labels.
A set of class semantic vectors (semantic values annotated by humans according to attributes) of the class c ∈ C s ∪ C u with
[Figure]
Evolved with + = …
[Figure]

[Figure]
Then, DSP concatenates visual features with their corresponding ˜ z s/u to enhance visual feature representations, which are utilized for classiﬁcation. | A | attributes, denoted as z c = [ z c (1) , . . .
, z c ( A )] (cid:62) ∈ R | A | . In the conventional zero-shot learning (CZSL) setting, we aim to learn a classiﬁer for unseen classes, i.e. , f CZSL : X → Y U . In contrast, we aim to learn a classiﬁer for seen and unseen classes in the generalized zero-shot learning (GZSL) setting, i.e. , f GZSL : X → Y U ∪ Y S .
In the following, we ﬁrst brieﬂy introduce the generative ZSL model. Then, we speciﬁcally demonstrate our dynamic semantic prototype learning method, its framework is shown in Fig. 2). 3. G ENERATIVE ZSL M ODEL
In our work, we focus on generative ZSL Xian et al. (2018; 2019b). The main goal of generative ZSL is to learn a conditional generator G: Z × O → ˜ X , which takes the class semantic vectors/prototypes z ∈ Z and Gaussian noise vectors o ∈ O as inputs, and synthesizes the class-speciﬁc visual feature samples ˜ x ∈ ˜ X . Once such a conditional generative model is learned, we can take it to synthesizes a large number of class-speciﬁc visual feature samples for unseen classes based on the semantic vector z u , and the ﬁnal classiﬁer over y ∈ Y can be obtained using any standard supervised classiﬁer ( e.g. , softmax).
3. V ISUAL → S EMANTIC M APPING N ETWORK
To encourage the conditional generator (G) to synthesize class-speciﬁc visual samples with enriched semantic information and conduct visual information for evolving the semantic prototype, we design a Visual → Semantic Mapping Network (V2SM) that is entailed on the G of the conditional generative model.
V2SM is a multilayer perceptron (MLP), which has a residual block to avoid too much information loss when mapping visual features of 2048-dim to semantic features of | A | -dim.
The network details are presented in Appendix A. Speciﬁcally, V2SM maps the real/synthesized visual features x/ ˆ x as semantic vectors ˆ z real / ˆ z syn , respectively, formulated as:
[Equation]
where ˆ z real ∪ ˆ z syn =
ˆ z , and ˆ z ∈ R | A | .
We can ﬁnd that the conditional generator ( i.e. , semantic → visual mapping) and V2SM conduct a semantic cycle network, i.e. , z → G ( o, z )
=
ˆ x and ˆ x → V2SM (ˆ x ) =
ˆ z , where o is a random Gaussian noise vector with | A | -dim.
As such, V2SM encourages the conditional generator to synthesize visual features with more semantic information.
To enable V2SM to map visual features into semantic space effectively, we employ the semantic cycle-consistency loss ( L Scyc ) during training.
This loss can be written as:
[Equation]
where z k is the dynamic semantic prototype learned by VOPE at the k -th step, which will be introduced in the next subsection.
This is different from TF-VAEGAN Narayan et al.
(2020) and
FREE Chen et al.
(2021a) that use the predeﬁned semantic prototypes as semantic supervision in their semantic decoders.
Since the dynamic semantic prototype is more close to its corresponding visual prototype, it provides more accurate supervision signals for the V2SM and generative models.
As such, DSP can signiﬁcantly improve the performance of TF-VAEGAN and FREE .
We ﬁnd that z , ˆ z and z k are closely affected by each other. ˆ z = V2SM ( x ) is mapped from the visual features in V2SM and used to convey visual signal to supervise VOPE, enabling z to be progressively evolved as dynamic semantic prototype z k = αz +(1 − α ) VOPE ( z k − 1 ) ( z k = z when k=0).
As such, z k IS closer to its corresponding visual prototype, and serves as the accurate condition information that encourages generator to synthesize reliable visual features.
3. V ISUAL -O RIENTED S EMANTIC P ROTOTYPE E VOLVING N ETWORK
We further introduce a visual-oriented semantic prototype evolving network (VOPE) to reﬁne the semantic prototype under the supervision of visual information.
As such, the dynamic semantic prototype can be closer to its corresponding visual prototype and the visual-semantic domain shift can be alleviated, as shown in Fig.
1(c).
Considering that the predeﬁned semantic prototypes have rich accurate information annotated by humans, we progressively update it progressively based on the visual information guidance.
Targeting this goal, VOPE is designed as an MLP network with a residual block that acts as a routing gate mechanism implemented by channel attention, which evolves the speciﬁc attribute value of semantic prototypes according to the corresponding visual information.
The network details are presented in Appendix B.
Speciﬁcally, VOPE iteratively evolves the initialized semantic prototype z k ( z k is the predeﬁned semantic prototype z annotated by humans when k =0) to another semantic prototype space ˜ z , formulated as: [Equation]
[Equation]
To encourages ˜
z k +1 evolved with visual supervision information, we conduct the visual signals from
V2SM, e.g. , ˆ z .
We formulate ˆ z as supervision using a visual-to-semantic loss:
[Equation]
Here, we use cosine similarity as a weak constraint for VOPE to conduct the information transfer from the visual domain to the semantic domain.
The reason is that we do not desire the visual information to be over strong, helping the semantic prototype evolve progressively.
Simultaneously, we also use the semantic reconstruction loss to constrain VOPE, deﬁned as:
[Equation]
where z k +1 is the dynamic semantic prototype at the k + 1 -th step, formulated as follows:
[Equation]
where α is a combination coefﬁcient for fusing the dynamic semantic prototype z k evolved in the previous step and the mapped semantic vectors ˜
z k +1 of the current step in VOPE.
Thus, z k is dynamically evolved for each training step and fed back to VOPE as input.
Considering that the empirically predeﬁned semantic prototype provides rich semantic priori, α is set to relatively large ( i.e. , α = 0 . 9 ) for preserving its original information.
Such a setting i) encourages the dynamic semantic prototype to evolve progressively under the supervision of visual information, and ii) facilitates model optimization .
3. O PTIMIZATION
Our DSP is jointly optimized with the generative model, e.g., CLSWGAN Xian et al.
(2018), fVAEGAN Xian et al.
(2019b), TF-VAEGAN Narayan et al.
and
FREE Chen et al.
(2021a), with the following objective function:
[Equation]
where L G is the loss of generative model, λ Scyc , λ V 2 S , λ S 2 S are the weights that control the importance of the related loss terms.
Since λ Scyc and λ S 2 S are semantic reconstruction losses, we set λ
Scyc
= λ S 2 S .
With the similar measurement loss terms and model components designed for DSP, we empirically observe that our model is robust and easy to train on various ZSL benchmarks when it is entailed on various generative models.
3. I NFERENCE P HASE
Synthesizing Visual Features for Unseen Classes: After training, we ﬁrst take the pre-trained VOPE to update the predeﬁned semantic prototypes to be dynamic semantic prototypes for unseen classes using Eq. 3 and Eq. 6, i.e. , ...
z u = αz u + (1 − α ) VOPE ( z u ) .
Then, we take the dynamic semantic prototypes of unseen classes as conditional supervision in generator (G) to synthesize class-speciﬁc visual features for unseen classes:
[Equation]
where o is a random Gaussian noise vector with | A | -dim.
Enhancing Visual Features for All Classes: Finally, we concatenate visual features (including the real visual features of seen classes and real/synthesized visual features of unseen classes) and their corresponding evolved semantic prototypes ˜
z = VOPE ( z ) to enhance them, enabling the visual features to be closer to their corresponding semantic prototypes and alleviate the cross-dataset bias Chen et al. (2021a).
It is formulated as:
Seen
[Equation]
where ˜ z s ∪ ˜ z u = ˜
z , x tr is the real visual feature in the training set D str , x te is the real visual feature in the test sets D ste and D ute .
As such, the enhanced visual features are closer to their corresponding semantic prototypes, and they are more discriminative as the cross-dataset bias is alleviated.
These enhanced visual features are used for classiﬁcation later.
ZSL Classiﬁcation: We employ ˜ x str and ˜ x usyn to lean a classiﬁer ( e.g. , softmax), i.e. , f czsl : ˜ X
→
Y s ∪ Y u or f gzsl : ˜ X
→ Y s ∪ Y u .
Once the classiﬁer is trained, we use ˜ x ste and ˜ x ute to test the model further.
Note that our DSP is an inductive method as we do not use the real visual features of unseen classes for model optimization.
E XPERIMENTS
Datasets.
We evaluate the effectiveness of our DSP on three well-known ZSL benchmark datasets, i.e. , two ﬁne-grained datasets ( CUB Welinder et al.
(2010), SUN Patterson & Hays (2012) and FLO Nilsback & Zisserman (2008)) and one coarse-grained dataset (AWA2 Xian et al. (2019a)).
CUB consists of 11,788 images of 200 bird classes (seen/unseen classes = 150/50) captured by 312 attributes.
SUN includes 14,340 images of 717 scene classes (seen/unseen classes = 645/72) described by 102 attributes.
AWA2 contains 37,322 images of 50 animal classes (seen/unseen classes = 40/10) captured by 85 attributes.
FLO inludes 8,189 images of 102 ﬂower classes (seen/unseen classes = 82/20) captured by 1024 attributes.
Implementation Details.
We use the training splits proposed in Xian et al. (2019a).
Meanwhile, the visual features of images (input size 224 × 224 ) are extracted from the 2048-dimensional top-layer pooling units of a CNN backbone (i.e., ResNet-101 He et al.
pre-trained on ImageNet.
The V2SM and VOPE are MLPs with residual blocks, network details of which can be found in Appendix
A and Appendix B, respectively.
Since f-VAEGAN Xian et al.
(2019b) is a popular generative ZSL method, we take it as a baseline and entail our DSP on it for conducting the ablation study, qualitative experiment and hyper-parameter analysis.
Following Xian et al. (2019a), we also use the same validation split on each dataset to conduct the hyperparameter setting.
We synthesize 150, [Table] and 3400 visual features per unseen class to train the classiﬁer for SUN, CUB and AWA2 datasets, respectively.
The combination coefﬁcient for updating semantic prototypes is set to 0.9 for all datasets.
The loss weight of λ Scyc = λ S 2 S is set to 0.1, 0.01 and 0.001 for CUB, SUN and AWA2, respectively.
The loss weight of λ V 2 S is set to 0.6/0.6 and 1.0 for CUB/AWA2 and SUN, respectively.
The speciﬁc hyper-parameter settings of our DSP entailed on other baselines are in Appendix C.
Evaluation Protocols.
During testing (ZSL classiﬁcation), we follow the uniﬁed evaluation protocols proposed in Xian et al. (2019a).
We calculate the top-1 accuracy of the unseen class for the CZSL setting (denoted as Acc ).
In the GZSL setting, we measure the top-1 accuracy on seen and unseen classes, denoted as S and U , respectively.
Their harmonic mean (deﬁned as
H = (2 × S × U ) /
( S + U ) ) are a better protocols in the GZSL setting.
As such, we typically take the H to discuss the performance of ZSL methods in the GZSL setting.
4. E XPERIMENTAL R ESULTS
Comparing with the SOTA.
We ﬁrst compare our f-VAEGAN+DSP with the state-of-the-art methods in the GZSL setting.
Table 1 shows the performance of various ZSL methods, including nongenerative methods ( e.g. , SGMA Zhu et al.
, AREN Xie et al.
APN Xu et al.
DAZLE Huynh & Elhamifar (2020a), CN Skorokhodov & Elhoseiny (2021), I2DFormer Naeem et al.
and generative methods ( e.g. , f-VAEGAN Xian et al.
(2019b), TF-VAEGAN Narayan et al.
(2020), AGZSL Chou et al.
(2021), GCM-CF Chou et al.
HSVA Chen et al.
(2021b),
ESZSL Çetin et al.
Compared to the popular non-generative methods, our f-VAEGAN+DSP achieves the best results on all datasets, i.e. , CUB ( H = 67 . 4 ), SUN ( H = 48 . 1 ), AWA2 ( H = 74 . 2 ) and FLO ( H = 75 . 2 ).
Thus, our DSP effectively helps generative ZSL methods beat the embeddingbased methods.
Our method also achieves the best results with signiﬁcant improvements compared to the other generative methods.
For instance, our f-VAEGAN+DSP outperforms the latest generative method ( i.e. , TF-VAEGAN+ESZSL Çetin et al.
by a large-margin, resulting in the improvements of harmonic mean by 10.8%, 6.4% and 10.7% on CUB, SUN and AWA2, respectively.
These results consistently demonstrate the superiority and great potential of our DSP in generative ZSL.
[Table]
Furthermore, we also compare our DSP with the popular generative methods in the CZSL setting, as shown in Table 2.
f-VAEGAN+DSP achieves competitive performances of harmonic mean with 62.8, 68.6 and 71.6 on CUB, SUN and AWA2, respectively.
These results indicate that DSP also effectively tackles the semantic-visual domain shift problem of the generative ZSL in the CZSL setting.
Ablation Study.
Based on the baseline ( i.e. , f-VAEGAN Xian et al.
(2019b))
, we conduct ablation studies to evaluate the effectiveness of our DPS in terms of the L Scyc , L V 2 S , L S 2 S , semantic prototype evolution ( i.e. , Eq. 6) and enhancing visual feature ( i.e. , Eq. 9 and Eq. 10).
Our results are shown in Table 3.
When DSP is without the constraints of L Scyc , its performance is dropped [Table] as the generator in the generative model lacks semantic cycle-consistency supervision.
Meanwhile [Table] the performance of DSP is dropped slightly without the optimization of L S 2 S .
In contrast, if DSP can not be evolved under the supervision of visual information using L V 2 S , DSP achieves very poor results compared to its full model,
i
[Table]
e. , the harmonic mean drops by 7.5% and 5.4% on CUB and SUN, respectively.
These results demonstrate that evolving semantic prototypes according to the visual information is effective for reﬁning the predeﬁned semantic prototypes aligned to the visual prototypes, helping DSP tackle the visual-semantic domain shift problem in generative ZSL.
Notably, if DSP does not dynamically evolve the predeﬁned semantic prototypes for generative learning, i.e. , DSP w/o Eq.
6, it almost loses its effectiveness and only achieves little improvement for the baseline with additional constraints ( i [Table]
e. , L Scyc , L V 2 S and L S 2 S ).
For instance, its performance of harmonic mean drops by 12.6% and 6.0% on CUB and SUN, respectively, compared to its full model.
Enhancing visual feature also effectively improves the performance of DSP.
These results show that the dynamic semantic prototype is effective to tackle the visual-semantic domain shift. [Table]
Generative ZSL Methods with DSP.
To evaluate our dynamic semantic prototype learning method as a general technique to improve generative model, we entail it into four recent popular generative ZSL methods, i.e. , CLSWGAN Xian et al.
(2018), f-VAEGAN Xian et al.
(2019b), TFVAEGAN Narayan et al.
and FREE Chen et al.
(2021a).
We use the ofﬁcial repositories shared by respective authors of these works to implement experimental results.
Since TF-VAEGAN and
FREE have the semantic decoder, we only add the VOPE with L V 2 S and S 2 S to enable them to learn dynamic semantic prototypes.
The speciﬁc hyper-parameter settings of our DSP entailed on these baselines are in Appendix C. As shown in Table 4, DSP consistently improves the performances of all baselines on all benchmark datasets by a large-margin.
For example, the average performance gains of harmonic mean are 8.5%, 8.0% and 9.7% on CUB, SUN and AWA2, respectively.
These results demonstrate that i) the visual-semantic domain shift is an important bottleneck challenge and neglected by existing generative ZSL, and ii) our dynamic semantic prototype learning method effectively tackles this problem.
Qualitative Performance.
To intuitively show the effectiveness of our DSP, we qualitatively investigate the improvements over baseline ( i.e. , f-VAEGAN Xian et al.
(2019b)).
We conduct t-SNE visualization Maaten & Hinton (2008) of the real/synthesized visual features learned by the fVAEGAN and f-VAEGAN+DSP on 10 classes randomly selected from CUB.
As shown in Fig.
3(Left), the visual features synthesized by f-VAEGAN Xian et al.
(2019b) are far away from their corresponding real visual features, caused by the visual-semantic domain shift problem.
Meanwhile, since the cross-dataset bias problem Chen et al. (2021a), the real visual features directly extracted [Figure] [Figure] from the CNN backbone are poor quality.
contrast, our DSP dynamically evolves the prototypes to accurately the visual representations generator to synthesize reliable visual features for all classes (as shown in Fig.
3(Right)).
Furthermore, DSP enhances the real/synthesize visual features by enriching semantic information into visual representations.
As such, DSP obtains signiﬁcant performance gains over f-VAEGAN Xian et al.
(2019b), i.e. , considerable improvements of harmonic mean with 14.5%, 6.8% and 10.7% on CUB, SUN and
AWA2, respectively.
Hyper-Parameter Analysis.
We conduct extensive experiments to investigate the effects of various hyper-parameter on CUB, i.e. , combination coefﬁcient α in Eq. 6, synthesizing number of per unseen classes N syn , the loss weights λ
Scyc and λ V 2 S .
Since L Scyc and L S 2 S are the semantic reconstruction loss, we set λ
Scyc
= λ S 2 S and take one loss weight into the analysis.
From the results in Fig. 4(a), DSP is not insensitive to the combination coefﬁcient for dynamic semantic prototypes evolution, and DSP achieves better results with a relatively large value of α ( i.e. , 0.9).
The reason is that the dynamic semantic prototypes can adapt to the evolution according to the visual-semantic interactions during optimization.
The results shown in Fig. 4(b) indicate our DSP is robust to the
N syn when N syn is not set too large.
Results in Fig. 4(c) and Fig. 4(d) show the best effects of our f-VAEGAN+DSP can be achieved when λ Scyc and λ V 2 S is set to 0.1 and 0.6, respectively.
Additionally, we ﬁnd that λ V 2 S should be set to larger than λ Scyc and λ S 2 S .
This is because the dynamic semantic prototypes should mainly depend on the visual information during evolution.
Overall, our model is robust to all hyper-parameters and easy to optimize.
C ONCLUSION
In this work, we are the ﬁrst to discuss that the predeﬁned semantic prototype results in the visualsemantic domain shift problem in generative ZSL.
To tackle this problem and advance generative
ZSL, we propose a dynamic semantic prototype learning method, which jointly and mutually reﬁnes the semantic prototypes and visual features enabling the generator to synthesize reliable visual features.
Our DSP is plug-and-play to be entailed on various baselines and consistently achieves largemargin improvements on three ZSL benchmark datasets.
We believe that our work will facilitate the development of stronger zero-shot systems and motivate their deployment in real-world applications.
Reproducibility statement.
We discuss the problem of visual-semantic domain shift in generative
ZSL and propose dynamic semantic prototype learning to tackle it.
We intuitively demonstrate that it is an important bottleneck problem in generative ZSL.
We also clearly introduce our model details, implementation details and hyper-parameter settings, which we use in our experiments.
We will provide our source code on a public repository once our paper is accepted
APPENDIX
Appendix organization:
• Appendix A: Network details of V2SM.
• Appendix B: Network details of VOPE.
• Appendix C:
The hyper-parameter settings of our DSP entailed on four baselines.
• Appendix D: t-SNE Visualization on SUN and AWA2.
A N EWORK D ETAILS OF V2SM
As shown in Fig.
5, the network of V2SM is an MLP with a residual block, which avoids too much information loss when mapping the visual feature of 2048-dim into semantic space with | A | -dim.
[Equation]
[Figure]
Addition
B N EWORK D ETAILS OF VOPE
As shown in Fig.
6, the network of VOPE is an MLP with a residual block (fusing with Hardamad
Product).
Essentially, the network of VOPE is a routing gate mechanism implemented by channel attention, which is effective in selectively evolving the speciﬁc attribute value of semantic prototypes according to the corresponding visual information.
As such, the predeﬁned semantic prototypes can be progressively evolved and aligned to the visual prototypes.
[Figure]
C T HE H YPER -P ARAMETER S ETTINGS OF O UR DSP E NTAILED ON V ARIOUS B ASELINES
We present the hyper-parameter settings of our DSP entailed on various baselines ( i.e. ,
CLSWGAN Xian et al.
(2018), f-VAEGAN Xian et al.
(2019b), TF-VAEGAN Narayan et al.
and
FREE Chen et al.
(2021a))
on CUB, SUN and AWA2.
Including the synthesizing number of per unseen classes N syn , the loss weights λ
Scyc , λ V 2 S and combination coefﬁcient α in Eq.
We empirically observe that our DSP is robust and easy to train when it is entailed on various generative models.
Based on these hyper-parameter settings, our DSP achieves signiﬁcant performance gains
[Table]
over the various popular generative models on all datasets.
For instance, the average performance gains of harmonic mean are 8.5%, 8.0% and 9.7% on CUB, SUN and AWA
[Table]
respectively.
Please refer to Table 4.
D T -SNE V ISUALIZATION ON SUN AND AWA
As shown in Fig.
[Table]
t-SNE visualizations of visual features learned by the f-VAEGAN Xian et al.
(2019b) (Left) and f-VAEGAN+DSP (Right) on SUN and AWA2.
Analogously, the visual features generated by f-VAEGAN are also far away from their corresponding real ones, and the discrimination of these real/synthesized visual features is undesirable.
In contrast, our DSP help f-VAEGAN synthesize visual features close to their corresponding real ones and enhance these real visual features.
As such [Figure] our DSP signiﬁcantly improves the performances of f-VAEGAN Xian et al.
(2019b) on
SUN and AWA2.
[Figure]