Make an Omelette with Breaking Eggs: Zero-Shot Learning for Novel Attribute Synthesis
AffiliationAddress email
Abstract
Most of the existing algorithms for zero-shot classification problems typically rely on the attribute-based semantic relations among categories to realize theclassification of novel categories without observing any of their instances.
However,training the zero-shot classification models still requires attribute labeling for eachclass (or even instance) in the training dataset, which is also expensive.
To thisend, in this paper, we bring up a new problem scenario: “ Can we derive zero-shotlearning for novel attribute detectors/classifiers and use them to automaticallyannotate the dataset for labeling efficiency? ”.
Basically, given only a small set ofdetectors that are learned to recognize some manually annotated attributes (i.e., theseen attributes), we aim to synthesize the detectors of novel attributes in a zero-shotlearning manner.
Our proposed method, Z ero- S hot L earning for A ttributes (ZSLA),which is the first of its kind to the best of our knowledge, tackles this new research problem by applying the set operations to first decompose the seen attributes intotheir basic attributes and then recombine these basic attributes into the novel ones.
Extensive experiments are conducted to verify the capacity of our synthesizeddetectors for accurately capturing the semantics of the novel attributes and showtheir superior performance in terms of detection and localization compared toother baseline approaches.
Moreover, we demonstrate the application of automaticannotation using our synthesized detectors on Caltech-UCSD Birds-200-2011dataset.
Various generalized zero-shot classification algorithms trained upon thedataset re-annotated by ZSLA shows comparable performance with those trainedwith the manual ground-truth annotations.
Introduction
Zero-shot learning (ZSL) algorithms for classification aim to recognize novel categories withoutobserving any of their instances during model training; thus, the cost of collecting training samples forthe novel categories can be eliminated.
Typically, the core challenge behind zero-shot classificationlies in associating novel categories with the seen ones during training.
Various existing approachesleverage different auxiliary semantic information to construct such associations across categories,thus being able to generalize the learned models for classifying novel categories [14, 1, 21, 13, 3, 25]or synthesize the training samples for each novel category [27, 28, 22, 30, 19].
Among differenttypes of auxiliary semantic information adopted for ZSL, defining a group of attributes shared amongcategories becomes one of the most popular choices, where each category is described by multipleattributes (i.e., multi-labeled by the attributes), and the attribute-based representations are discriminative across categories.
However, it comes with the expensive cost of manually annotating the samplesin the dataset their attribute labels at a much granular level.
For example, CUB dataset [24], one ofthe most widely-used benchmarks for learning zero-shot classification, is built by spending a greatdeal of time and effort to label 312 attributes for 11788 images.
As motivated by the issue of annotation efficiency on attribute labels, this paper aims to develop ZSLon known attributes to annotate novel attributes for a dataset automatically .
That is, analogous tothe zero-shot classification scenario, we now advance to annotate novel attributes for a dataset viautilizing the knowledge from a few types of seen/given manual attributes, as illustrated in Figure1.
Specifically, we take the well-known CUB dataset [24] as our main test-bed and have a deepinvestigation on its attributes.
We discover that, many attributes in CUB dataset (e.g. “red head" or“blue belly") follow the form of combinations over base attributes (e.g. “red", “blue", “head" and“belly" respectively).
Building upon such observation, given a defined set of attributes in the formof the ones used in the CUB dataset and labels of a few seen attributes (where the number is farless than that of overall defined attributes), we propose Z ero- S hot L earning for A ttributes (ZSLA), amethod of training the seen attribute detectors and then tackle the ZSL problem to synthesize unseenattribute detectors via a decompose-and-reassemble manner.
In detail, the seen attribute detectorsare firstly decomposed into base attribute representations, in which they are further reassembledwith novel combinations into novel attribute detectors, as illustrated in Figure 1.
Here, both thedecomposition and reassembly steps are achieved via set operations (i.e., the interaction and unionoperators, respectively).
Together with the seen ones, the novel attribute detectors can be utilized toannotate the attribute labels for the dataset automatically.
To demonstrate the efficiency of ZSLA, we synthesize 207 novel attribute detectors by leveragingonly 32 seen ones from the CUB dataset.
These novel attribute detectors are shown to be effectivein capturing their corresponding semantic information and benefit both the attribute detection andlocalization for the samples in CUB dataset.
Besides, we also synthesize α -CLEVR dataset by [11]for conducting the controlled experiments to further discuss the influence of noisy seen attributelabels.
The results show that ZSLA can provide more robust annotations than the other baselinemethods under the noisy scenario.
Below, we highlight the contributions of this paper:
• To the best of our knowledge, we are the first to propose ZSL for attributes to automatically annotateattribute labels for the zero-shot classification datasets.•
We propose a novel decompose-and-reassemble approach to single out the base attribute representations by applying intersection on the seen attribute detectors and synthesize the unseen ones byhaving the union operation over the base attributes representations.
• In Section 4.2, we show on the CUB dataset that, given only 32 attributes with manual annotations,ZSLA can synthesize novel attribute detectors to provide high-quality annotations for the dataset.
By using the auto-annotated attributes, generalized zero-shot classification algorithms can alsoachieve comparable or even better performance than that using 312 manually-annotated attributes
Related Works
Zero-shot learning (ZSL) was originally proposed to tackle the specific classification problem, wherethe model is expected to be capable of classifying the samples belonging to the novel categories whichare not seen previously during training.
The problem setup has been extended to other applicationssuch as detection [5, 20, 7] and segmentation [6, 32].
Here we provide a brief review of the worksof zero-shot classification [14, 21, 1, 2, 13, 29, 27, 28, 22, 30, 19].
Without loss of generality
the ZSL approaches rely on using the auxiliary information (such as attributes, word embeddings, or text descriptions) as the basis for describing the categories and building the semantic relationamong seen and unseen categories; and the existing methods can be roughly categorized into twogroups:
the embedding-based methods [1, 2, 13, 22, 29] and generative methods [27, 28, 22, 30, 19].
The embedding-based methods basically aim to learn a latent space that connects between thefeature representations of training samples and the embeddings of their corresponding auxiliaryinformation (e.g., the visual features and the embeddings of attribute labels for the training imagesin the CUB dataset), such that the test samples can be classified as the novel categories once theirfeature representations are close to the embeddings of novel categories (which are defined uponauxiliary information without requiring any additional training samples).
The generative methodsinstead utilize the deep-generative models (e.g., generative adversarial networks [9], variationalautoencoder [12], or their hybrids/variants) for learning to synthesize the samples or features ofthe unseen categories based on their auxiliary semantic information.
Though saving the effortof collecting the training samples to recognize novel categories via ZSL techniques, manuallyannotating the auxiliary semantic information for the samples in the zero-shot training dataset isstill quite expensive and time-consuming.
In turn, our proposed task of zero-shot learning for novelattributes helps to reduce such costs for the scenario of zero-shot classification where the auxiliaryinformation is defined on attributes.
Aside from the typical zero-shot classification problem, recently there exists another specific zero-shottask that our work is also conceptually related to: compositional zero-shot learning (CZSL) [16, 18,4, 15, 10, 17].
As also known as state-object compositionality problem, CZSL aims to recognizethe novel compositions (e.g. “ripe tomato”) given the seen visual primitives of states(e.g.
“ripe”,“rotten”) and objects (e.g. “apple”, “tomato”) in the training dataset.
Our proposed problem scenario
ZSLA is distinct from CZSL under several perspectives: (1) An image in our problem scenario wouldhave multiple attributes while there usually exists only a single state-object composition for CZSL;(2)
Our synthesized attribute detectors are able to provide labels of novel attributes (i.e. these novelattributes do not have any manually labeled samples in the training set) for the images thus leading tomore detailed descriptions for all the categories, while CZSL typically aims to increase the numberof categories (i.e. each novel composition is treated as a new fine-grained class).
Please kindly referto our supplementary materials for the extensive discussions and experiments with respect to CZSL.
ZSLA: Proposed Method
Given a zero-shot classification dataset { X , Y , A s } , each image x ∈ X has its class label y ∈ Y andthe multi-attribute labels ϕ
s ( x ) , where ϕ s ( x ) is a binary vector with its each element denoting if xhas a certain attribute a ∈ A s .
ZSLA starts with using { X , A s } to train the detectors M s for all theattributes in A s , which are treated as seen attributes, then it adopts the seen attribute detectors M s tosynthesize the detectors M u for the unseen attributes A u via a decompose-and-reassemble procedure,where A s ∩ A u = ∅ .
For ease of understanding, we use the most popular zero-shot classificationdataset, CUB [24], to illustrate how these steps are realized as follows.
3.1 Training Seen Attribute Detectors
Our attribute detectors are built on top of the image feature space produced by the image featureextractor f .
Given an input image x and its feature map f ( x ) ∈ R W × H × C where each C -dimensionalfeature vector at position ( i, j ) of f ( x ) , denoted as f ( x )[ i, j ] , is the feature representation of thecorresponding image patch on x , the attribute detectors M s ∈ R C × N s (in which N s denotesthe number of attributes in A s ) aim to give high response on the image patches containing thevisual appearance related to the attributes in A s .
Specifically, each column in M s is acting as theembedding of a certain attribute.
We use m sk to indicate the k -th column of M s .
The response ofthe corresponding k -th attribute in A s with respect to the patch-wise feature vector f ( x )[ i, j ] iscalculated by a specific form of their cosine similarity cos ( | m sk | , f ( x )[ i, j ]) , where | m sk | denotesapplying element-wise absolute-value operator on m sk .
We have | m sk | in our cosine similaritycomputation due to the reason that: Each dimension along channels of f ( x ) is considered to capture aspecific visual pattern.
Our | m s k | hence acts as to apply the weighted combination over these variousvisual patterns for representing the characteristics of the k -th attribute in A s , and the absolute-valueoperator over m sk is to ensure the combination weights are non-negative.
We denote R s ( x ) ∈ R W × H
× N s as the response map which has included the cosine similarities of allthe seen attributes A s at each position on f ( x ) .
Note that, as our feature extractor f adopts the ReLU activation function in its last layer (similar to most image feature extractors based on the convolutionalnetworks), the values in f ( x ) become non-negative.
Furthermore, as both | m sk | and f ( x )[ i, j ] arenon-negative vectors, all entries of R s results to be within the range [0 , 1] .
Following the populartricks for ZSL and deep learning pointed out in [23], where adopting scaled cosine similarity in logitscomputation is important to achieve better model training, we use the computation below to calibratethe value of elements in R s ( x ) :
[Equation]
where the calculation within brackets shifts and expands the values in R s ( x ) towards [ − 1 , 1] to matchthe typical value range of cosine similarity, and the hyperparameter γ is set to 5 as suggested by [23].
Then, we perform the max-pooling operation on ˜ R s ( x ) and obtain the image-wise attribute response˜ r s ( x ) ∈ R N s .
Such logits over attributes thus are able to drive the model training (i.e. optimizationover M s and f ) via the error between the attribute detection results and the ground-truth attributelabels ϕ s ( x ) .
The objective function L bce to evaluate the error between the logits of attribute detectionresult ˜ r s ( x ) and the ground-truth attribute labels ϕ
s ( x ) is defined via the binary cross-entropy:
[Equation]
where ϕ sk ( x ) and ˜ r sk ( x ) denote the k -th elements in ϕ s ( x ) and
˜ r s ( x ) respectively, and σ is the sigmoidfunction.
In addition to the L bce loss, we introduce another objective function L umc to place the uni-modalconstraint [31] on the response map ˜ R s ( x ) , which encourages the response map for a certainattribute (e.g. ˜ R sk ( x ) , the k -th channel of ˜ R sk ( x ) ) to be uni-modal and concentrated.
In other words,we regularize a detector to focus on a single location or a small region in the image x .
We provide afurther discussion of relaxing this assumption in our supplementary materials.
[Equation]
where ˘ i k , ˘ j k = arg max
i,j ˜ R sk ( x )[ i, j ] and ∥ · ∥ denotes the Euclidean norm.
The overall objective to train the feature extractor f and the seen attribute detectors M s is illustratedin the left portion of Figure 2 and summarized as: L bce + λ L umc , where the hyperparameter λcontrols the balance between losses and is set to 0 .
2 in our experiments.
Moreover, we are aware that in CUB dataset the additional annotations indicating the ground-truthlocations for the attributes which an image x has are also available (e.g. we know where the attribute“brown wing” appears on an image of “gadwall”).
Hence, in addition to max-pooling the responsemap R s ( x ) to obtain the image-wise response r s ( x ) for attributes, we experiment another way toobtain r s ( x ) : (1) If ϕ sk ( x ) is true, the k -th element in r s ( x ) , i.e. r sk ( x ) , is assigned by R s ( x )
[ i, j ]where the centre of the ground-truth location for the k -th attribute in A s is located on the patchrelated to the position ( i, j ) of R s ; (2) If ϕ sk ( x ) is false, r sk ( x ) is assigned by having the averagepooling over the k -th channel of R s ( x ) .
In supplementary materials we provide the analysis for theimpact of using such additional annotations of attribute location on the performance of ZSLA.
3.2 Decompose-and-Reassemble for Synthesizing Novel Attribute Detectors
After obtaining the seen attribute detectors M s , we now aim to perform the decompose-andreassemble procedure (as shown in the right-half of Figure 2) for generating the detectors M u ∈ 167 R of the novel attributes A u (where N u is the number of attributes in A u ) by leveraging M s .
First, we observe that most of the attributes in CUB dataset (the most popular zero-shot classificationdataset and also our test-bed in this work) follow the form of “ adjective + object part ”, for instance:“black eye”, “brown forehead”, “red upper-tail”, or “buff breast”.
Starting from such observation, wedefine two disjoint sets of base attributes , B c and B p , representing the adjectives and object partsused in the seen attributes, respectively (e.g. “blue”, “yellow”, “solid”, and “perching-like” for B c ;“leg”, “beak”, “belly”, and “throat” for B p ).
Please note that the concepts behind adjectives B c in
CUB dataset include not only color but also texture, shape, and others.
Formally, given an attribute a ,we use β c ( a ) and β p ( a ) to denote its corresponding base attributes on the adjective and object part,respectively (i.e. β c ( a ) ∈ B c and β p ( a ) ∈ B p ), where β c ( · ) and β p ( · ) are functions to indicate thebase attributes in B c and B p for an attribute a , respectively.
Now, given two seen attributes a k and a l ∈ A s in which a k = { β c ( a k ) , β p ( a k ) } and a l ={ β c ( a l ) , β p ( a l ) } , if a k and a l have common ground in either the base attribute of adjectives (i.e.β c ( a k ) =
β c ( a l ) ∈ B c ) or the one of object parts (i.e. β p ( a k ) =
β p ( a l ) ∈ B p ) but not both, thenwe can use the intersection operation I to extract such common base attribute from a k and a l :
[Equation]
For instance, I is able to extract the base attribute “red” from the seen attributes “red wing” and “redbreast”; or the base attribute “tail” from the seen attributes “buff tail” and “black tail”.
Once we obtain the base attributes via intersection over seen attributes, we further adopt the unionoperation U to create novel attributes.
Given two pairs of seen attributes { a k , a l } and { a k ′ , a l ′ }in which β c ( a k ) =
I ( a k , a l ) and β p ( a k ′ ) =
I ( a k ′ , a l ′ ) , i.e. { a k , a l } share the same base attributeof adjective while { a k ′ , a l ′ } share the same base attribute of object part, a novel attribute ˜ a can besynthesized by combining β c ( a k ) and β p ( a k ′ ) , i.e. ˜ a = U ( β c ( a k ) , β p ( a k ′ )) .
In particular, if suchcombination of base attributes has been seen in A s , i.e. there exists an attribute
a ∈ A s whereβ c ( a )
=
β c (˜ a ) and β p ( a ) =
β p (˜ a ) , we say the seen attribute a is reconstructed by ˜ a .
Otherwise, ifnone of the seen attributes has the identical combination as our synthesized ˜ a , we denote ˜ a a novelattribute and ˜ a ∈ A u .
In summary, extracting base attributes from seen attributes via intersection,followed by combining the base attributes into novel attributes via union, holistically forms ourdecompose-and-reassemble procedure to synthesize the novel attributes.
In practice, our implementation of the intersection function I (please refer to its illustration providedin the supplementary materials) is built based on the encoder architecture of vision transformer [8](ViT), in which its input is the embeddings of the seen attributes, i.e. the transformer takes m skand m sl from M s as input when performing I ( a k , a l ) , where a k , a l ∈ A s .
To be detailed, thereare several modifications in our transformer for intersection I with respect to the original ViT: (1) We remove the position embedding in order to fulfil the commutative property of intersection, i.e. 201 I ( a k , a l ) =
I ( a l , a k ) ; (2) We attach a learnable token named “intersection head” to the input sequenceof transformer, which is similar to the extra class embedding in ViT.
The corresponding output ofthis intersection head after going through the transformer encoder represents the embedding of theresultant base attribute, where we apply the element-wise absolute-value operation on it to make it anon-negative vector (being analogous to what we did for the seen attributes).
Please note that, theembedding of a base attribute is also a C -dimensional vector.
Regarding our union function U , we simply adopt the average operation for its implementation, that is: Given two base attributes b c ∈ B cand b p ∈ B p , we obtain the embedding ˜ m of the synthesized attribute ˜ a = U ( b c , b p ) by averagingthe embeddings of b c and b p .
Specifically, such C -dimensional embedding ˜ m is also defined uponthe image feature and acts as the detector for the synthesized attribute ˜ a .
The training of our decompose-and-reassemble procedure for synthesizing novel attributes is simplybased on the reconstruction loss of the seen attributes L rec .
Given a synthesized attribute ˜ a , if thereexists a seen attribute a k ∈ A s with having β c ( a k ) =
β c (˜ a ) and β p ( a k ) =
β p (˜ a ) , the embedding ˜ mof ˜ a and the embedding m sk of a k are expected to be identical to each other, L rec is thus defined as:
[Equation]
Note that, as our union function U has no trainable parameters (since it is simply an average operation),the gradient of L rec is propagated to focus on learning the parameters of our transformer for theintersection function I .
In other words, we expect that the transformer is so powerful to be capableof extracting the base attributes where their averages are informative enough to act as the detectorsfor the synthesized attributes.
Note that in order to fully leverage the seen attributes for training ourdecompose-and-reassemble procedure, we have a particular training scheme where its algorithm andthe implementation details are shown in the supplementary materials.
Moreover, we also provide oursource code in supplementary materials for the reviewing purpose.
Experimental Results
Dataset.
Our experiments are mainly conducted on the Caltech-UCSD Birds-200-2011 dataset [24](usually abbreviated as CUB) for zero-shot classification.
CUB dataset collects 11,788 images of 200bird categories, where each image is annotated with 312 attributes.
We select 32 attributes as our seenattributes A s , which can be decomposed into 15 base attributes of adjective B c and 16 base attributesof object part B p , and we use these base attributes to synthesize 207 novel attributes A u .
We followthe setting proposed by [26] for the task of generalized zero-shot learning (GZSL) to split the CUBdataset, where such training and testing sets are used to train and evaluate our proposed scenario of
ZSL on attributes, respectively.
Furthermore, we base on [11] to create a synthetic dataset, namedα -CLEVR, for providing extensive analysis on the robustness of our proposed ZSLA against thenoisy seen attributes (which are related to an issue hidden behind CUB dataset).
α -CLEVR contains24 attributes (i.e. types of toy bricks) which are the combinations of 8 colors (i.e., base attributesof adjective B c ) and 3 shapes (i.e., base attributes of object part B p ).
Among them, 16 attributes,which can be decomposed into the 11 base attributes, are selected as seen attributes A s for trainingour proposed ZSLA, where we are then able to synthesize novel attribute detectors to annotate theimages of α -CLEVR dataset.
Noting that, classes in our α -CLEVR dataset are defined by the specificcombinations of toy bricks (where toy bricks with different color-shape combinations are treated asdifferent attributes) For performing the GZSL task on α -CLEVR and evaluate the annotation qualityof our synthesized attribute detectors, we create 160 classes in α -CLEVR, where each class has 30images; 80 classes are set as seen data, and the other 80 are set as unseen data.
For GZSL inference,testing images from both seen and unseen classes are used.
Baselines.
As our task of ZSL on attributes for dataset annotation is novel, there is no prior work thatwe can directly make a comparison with.
However, as the task of ZSL on attribute has a hierarchybetween attributes and base attributes, we adapt two representative methods (which explicitly have the class–attribute hierarchy behind their formulation) to be our baselines,by using the analogy between two hierarchies (i.e. our attribute–base attribute versus their class–attribute).
These two baselines are ESZSL [21] and LAGO singleton [3] (note that both of themrealize classification with the help of attribute prediction), in which we particularly rename theiradaptions to our scenario of ZSL on attributes as A - ESZSL and A - LAGO respectively for avoidingconfusion.
Details of implementing baselines are provided in the supplementary materials.
Notethat, in the following experiments, both the baselines and ZSLA use the additional ground-truth ofattribute locations (i.e. knowing where an attribute appears on the image) provided by CUB to trainthe seen attribute detectors, unless stated otherwise.
Limitation and discussion.
Our proposed method (ZSLA) tackles zero-shot learning for unseenattributes and enables the efficient attribute annotations when constructing the new datasets (latershown in Sec 4.2) as long as the attribute format can be re-factored (e.g. blue wing) for assemblingnovel attributes via the decompose-and-reassemble approach.
Although such requirement on theattribute format restricts the direct application of our ZSLA on some ZSL datasets (e.g. AWA2and SUN datasets), the particular attribute format adopted in our ZSLA actually better fits thehuman’s intuition for describing the discriminative attributes for the “fine-grained” classes.
Hence,we would not see this format requirement on attributes as the limitation of our proposed method butthe restriction of the mainstream ZSL datasets for not having the fine-grained attributes.
4.1 Evaluation of Unseen Attributes
We design three schemes to evaluate the quality of the synthesized novel attribute detectors learntby ZSLA:
(1) Attribute Classification .
Based on ground-truth attribute annotation of the testimages (note that each image typically has multiple attributes), we measure the performance of oursynthesized attribute detectors on recognizing their corresponding attributes in the test images.
Weadopt the area under receiver operating characteristic (AUROC) as our metric for the classificationaccuracy of each attribute, and we report the average over AUROCs (denoted as mAUROC) ofall synthesized attribute detectors; (2) Attribute Retrieval.
We rank the test images according totheir image-wise responses as to a given attribute detector, to simulate the application scenario ofretrieving the images which are most likely to own the target attribute from an image set.
Note that theimage-wise response is computed by max-pooling over the responses of patch-wise image featureswith respect to the attribute detector.
For each attribute detector we compute the average precision(AP) of its top 50 retrieved images, and report the average AP (denoted as mAP@50) of all detectorsas the metric; (3) Attribute Localization.
As in CUB the ground-truth locations that an attributeappears on the test images are available, we introduce the localization accuracy (LA) to measure howwell the location having the highest response to an attribute detector matches with the ground-truthones (counted as correct if they are located on the same or neighboring patches).
We average over the
LA of each attribute as the metric (denoted as mLA).
Table 1 summarizes the performance in terms of mAUROC, mAP@50, and mLA obtained bybaselines and ZSLA, with the number N s of seen attributes A s set as { 32 , 64 , 96 } .
It is clear to seethat ZSLA provides superior performance in comparison to the baselines on all the settings of N sand evaluation schemes, particularly the localization accuracy.
Moreover, by using merely 32 seenattributes to perform the synthesis of novel attribute detectors, ZSLA can achieve comparable resultswith the baselines of using 64 or 96 seen attributes.
Qualitative examples for showing the results ofmAUROC mAP
A-ESZSL
N s
A
[Table]
[Table]
[Table]
[Table]
[Table]
attribute retrieval and attribute localization for the novel attributes synthesized by ZSLA are providedin Figure 3.
Besides these quantitative and qualitative results demonstrating the efficacy of ZSLA onnovel attributes, we also provide some qualitative examples in Figure 4 to showcase the localizationand retrieval ability of our base attribute representations extracted from the seen attribute detectors.
4.2 Automatic Annotations for Learning Generalized Zero-Shot Image Classification
To further access the quality of our synthesized attribute detectors, we adopt the 32 seen attributedetectors and the 207 novel attribute detectors (i.e. N s =32, N u =207) learned by ZSLA to re-annotatethe attribute labels for the whole CUB dataset to simulate the labeling process during constructinga new dataset, and name the resultant new dataset “ δ -CUB”.
Then we adopt δ -CUB to train andevaluate four representative GZSL algorithms, i.e. ALE [1], ESZSL [21], CADAVAE [22], and
TFVAEGAN [19] using the settings proposed by [26] (i.e. for δ -CUB and CUB, training with samplesfrom the 150 seen classes, then evaluating the performance on all 200 classes including the 50 unseenones).
Note that, the class-attribute matrix, which shows the composition of attributes for each classand is needed for GZSL (i.e. the semantic information of classes), is computed by the statistics inδ -CUB.
Similarly, we also use A - ESZSL and A - LAGO baselines to re-annotate CUB dataset andperform GZSL under the same aforementioned setting.
The results related to ZSLA and baselines aresummarized in the row shaded by the orange color of Table 2.
Moreover, we additionally experimenton training the four GZSL algorithms by using only 32 attributes or using all 312 attributes obtainedfrom the original CUB dataset as the semantic information, where their results are summarized in therows shaded by the blue and green color of Table 2, respectively.
From the results, we observe that using δ -CUB for training, where our ZSLA automatically annotatesall the attribute labels, can largely benefit the performance of GZSL algorithms.
By treating theharmonic mean over the accuracy numbers on both seen and unseen categories as the metric for
GZSL, δ -CUB is superior to those datasets annotated by baselines or even the one using manualannotations.
Specifically, the gain obtained by using our δ -CUB with respect to the setting of using32 manually-labeled attributes (i.e., the blue-shaded row of Table 2) demonstrates the practicalvalue of our proposed problem scenario of ZSL on attributes: Without additional cost for collectingannotation, we provide more attribute labels via synthesizing novel attribute detectors from theseen ones, and thus different categories can be better distinguished by more fine-grained/detailedattribute-based representations.
Moreover, regarding the results that our automatic re-annotationleads to better performance than the manual one (i.e., the green-shaded row of Table 2), we believethat this is mainly due to the biased semantic information caused by noisy labels stemming from theinconsistency between different human annotators when building CUB dataset.
In comparison, ourattribute detectors can produce consistent attribute annotations as we use the same set of attributedetectors for labeling all images; it eventually contributes to a more suitable semantic for learningzero-shot classification.
We provide more discussions on such issues in the supplementary materials.
[Table]
4.3 Robustness Against Noisy Attribute Labels
Due to the preference bias among different annotators mentioned in Section 4.2, it is hard to obtainperfect seen attribute labels for training.
Thus, it is interesting to discuss the effect of the noisy levelof seen attribute labels (used for training) on the final annotation quality produced by different autoannotation methods.
Thus, we conduct the controlled experiments using α -CLEVR to understand theeffect of the noisy labels.
In detail, we adjust the amount of noisy labels for the analysis.
To measurethe performance drop caused by noisy seen attribute labels, we define the wrong attribute label rate(abbreviated as WALR ) to represent the noisy level of attribute labels.
For example, when WALR isset to 0.3, any toy brick in the training images has a 30% chance of inaccurately annotating (e.g., ablue cube is annotated as a red sphere).
As the mAUROC curves shown in Figure 5(a), we can observe that: (1) ZSLA outperforms thebaselines in attribution classification, no matter how noisy the training data is; (2) the performancedrop of ZSLA with respect to WALR is much smaller than that of the baselines; (3) baselines have alarger variance than ours, i.e., they are more sensitive to different combinations of the noisy labelseven the noise level is the same.
The observations prove the robustness of ZSLA against the noisylabels of training attributes.
We also show in our supplementary materials that mAP@50 and mLA(for attribute retrieval and location) have a similar trend as mAUROC.
Moreover, similar to the experimental setting as Section 4.2, we use the novel attribute detectors, synthesized by different methods under various WALR settings, to automatically re-annotate the dataset.
The resultant dataset is used for learning four GZSL algorithms (i.e., CADAVAE, TFVAEGAN, ALE,and ESZSL).
The average of their harmonic means is reported in Figure 5(b).
We can observe thesuperior quality in terms of automatic re-annotation produced by our ZSLA (i.e., the red curve)compared to the other baselines (i.e., the blue and green curves for A-ESZSL A-LAGO, respectively)under all WALR settings.
Specifically, we also simulate the situation where humans annotate allattribute labels for the dataset while maintaining the corresponding WALRs (i.e., the purple curve).
Itleads to a similar observation as we find in the CUB dataset.
Once WALR is high (i.e., quite noisylabeling), the performance of GZSL algorithms trained with the semantic information provided byour ZSLA (i.e., the red curve) becomes superior to the one trained with the noisy manual labels.
Conclusion
This paper proposes a new method of developing zero-shot learning on novel attributes to reducethe attribute annotation cost for constructing a zero-shot classification dataset.
By leveraging thetrained detectors of seen attributes, our model learns to decompose them into base attributes to furthersynthesize novel unseen attributes by reassembling pairs of base attributes.
Experimental resultsshow that our method is able to exploit the information embedded in the seen attributes to generatehigh-quality unseen attributes, validated by various evaluation schemes for attribute classification,retrieval, and localization.
We also demonstrate that the semantic information based on our automaticre-annotation is beneficial for the GZSL task.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the “ Limitation and discus-
sion " part in Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No] To the best
of our understanding, there are no potential negative societal impacts in our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Please refer to
our supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please refer to our code and our supplementary materials.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
For the experiment result that strongly impacted by the
random sampling process, we repeat the experiment for several rounds and report the
mean performance as well as the 95% confident region.
See Section 4.3.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the “ Dataset "
part in Section 4.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] We use the public dataset with referencing to the original
paper/resource.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] To the best of our understanding, there are no
related issue in the applied dataset.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]