L EARNING TO S EPARATE D OMAINS IN G ENERALIZED Z ERO -S HOT AND O PEN S ET L EARNING : A PROBABILISTIC PERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies the problem of domain division which aims to segment instances drawn from different probabilistic distributions.
This problem exists in many previous recognition tasks, such as Open Set Learning (OSL) and Generalized Zero-Shot Learning (G-ZSL), where the testing instances come from either seen or unseen/novel classes with different probabilistic distributions.
Previous works only calibrate the conﬁdent prediction of classiﬁers of seen classes (WSVM Scheirer et al.
or taking unseen classes as outliers Socher et al.
In contrast, this paper proposes a probabilistic way of directly estimating and ﬁne-tuning the decision boundary between seen and unseen classes.
In particular, we propose a domain division algorithm to split the testing instances into known , unknown and uncertain domains, and then conduct recognition tasks in each domain.
Two statistical tools, namely, bootstrapping and KolmogorovSmirnov (K-S) Test, for the ﬁrst time, are introduced to uncover and ﬁne-tune the decision boundary of each domain.
Critically, the uncertain domain is newly introduced in our framework to adopt those instances whose domain labels cannot be predicted conﬁdently.
Extensive experiments demonstrate that our approach achieved the state-of-the-art performance on OSL and G-ZSL benchmarks.
I NTRODUCTION
This paper discusses the problem of learning to separate two domains which include the instances sampled from different distributions.
This is a typical and general research topic that can be potentially used in various recognition tasks, such as Open Set Learning (OSL) and Generalized Zero-Shot
Learning (G-ZSL).
Particularly, OSL can break the constraints of the closed set in supervised learning and aim at recognizing the testing instances from one of the seen classes ( i e., known domain ), and the novel class ( i.e., unknown domain) .
The novel classes include the testing instances which have different distributions from that of the seen ones.
In contrast, G-ZSL targets at distinguishing the labels of instances from the seen and unseen classes.
Only the seen classes have the training instances, but unseen classes do not.
Note that OSL does not explicitly give the class labels for those instances categorized as the novel class, but G-ZSL requires predicting the class labels of unseen classes.
To address G-ZSL, semantic attributes or vectors are introduced as the intermediate representations; each (seen/unseen) class has one semantic prototype that contains class level information.
Speciﬁcally, a reasonable solution of OSL and G-ZSL is via dividing the known and unknown domains.
For training classes, the predictors are constructed to map visual features to the class label space (OSL), (or semantic space (G-ZSL)).
Testing is performed on each separated domain to identify seen classes and the novel class (OSL), or both seen and unseen classes (G-ZSL).
The key question of OSL and ZSL is how to deal with the newly introduced novel class/unseen classes efﬁciently in the testing time.
This is different from the conventional Zero-Shot Learning (ZSL) task which assumes that, in the testing stage, seen classes would not be misclassiﬁed as unseens, and vice versa; ZSL only uses the unseen classes for testing.
Unfortunately, the predictors learned on training classes will inevitably make OSL or G-ZSL approaches tend to be biased towards the seen classes, and thus leading to very poor classiﬁcation results for the novel class (OSL) or unseen classes (G-ZSL)
Xian et al.
; Chao et al.
We show an example in Fig.
On aPY dataset (described in Sec. 6.1) Farhadi et al.
(2009), we employ t-SNE van der Maaten &
[Figure] [Figure]
Then we can recognize instances in each domain intraclass interclass distances
We compute the empirical density the pairwise distance aPY dataset (described in Sec.
There is a large overlapping of the distribution of the intraclass and interclass
Hinton (2008) to visualize the distributions of the testing instances of the ResNet-101 features in Xian et al.
(2017) (Fig.
(a)), and semantic features learned by SAE Kodirov et al.
(2017) (Fig.(b)).
We categorize the SAE prediction as known or unknown domain labels and compare with the ground-truth in Fig.
1(c).
We show that a large portion of unseen instances being predicted as one of the known classes.
A natural recipe for addressing this problem is to learn to separate domains by the distributions of instances; and different classiﬁers can be directly applied in each domain.
However, there are still two key problems.
First, visual features alone are not discriminative enough to help to distinguish the seen and unseen/novel classes.
As Fig.
2 (a), bicycle and motorbike, respectively , are one of the seen and unseen classes 1 in aPY dataset (described in Sec. 6.1).
We can observe that there is a large overlapping region between their t-SNE visualized feature distributions.
That is, the visual features may not be representative enough to differentiate these two classes; the instances of motorbike (circled as the uncertain domain) may be taken as the bicycle, or vice versa; Second, the predictors trained on seen classes may be not trustworthy.
A not well-trained predictor may negatively affect the recognition algorithms.
Third and even worse, the performance of classiﬁers in each domain is still very sensitive to the results of domain separation: should the domain of one testing instance be wrongly divided, it would never be correctly categorized by the classiﬁers.
To tackle the aforementioned issues, our key insight (see Fig. 2(a)) is to introduce a novel domain – uncertain domain that accounts for the overlapping regions of testing instances from seen or novel/unseen classes.
Thus, the visual or semantic space can be learned to be divided into known , unknown and uncertain domains.
The recognition algorithms will be directly employed in each domain.
Nonetheless, how to divide the domains based on known information is also a non-trivial task.
Though the supervised classiﬁers can learn the patterns of known classes, not all classes encountered during testing are known.
Formally, we propose exploiting the distribution information of seen and novel/unseen classes to efﬁciently learn to divide the domains from a probabilistic perspective.
Our domain separation algorithm has two steps: the initial division of domains by bootstrapping, and ﬁne-tuning by the
Kolmogorov-Smirnov test.
Speciﬁcally, according to extreme value theory Scheirer et al.
(2014), the maximum/minimum conﬁdence scores predicted by the classiﬁer of each class can be taken as an extreme value distribution.
Since we do not have the prior knowledge of the underlying data distributions of each class; bootstrapping is introduced here as an asymptotically consistent method in estimating an initial boundary of known classes.
Nevertheless, the initial boundary estimated by bootstrapping is too relaxed to include novel testing instances as is illustrated in Fig. 2(b).
To ﬁnetune the boundary, we exploit the K-S Test to validate whether the learned predictors are trustworthy in a speciﬁc region.
The uncertain domain introduced thus accounts for those testing instances whose labels are hard to be judged.
Recognition models can be conducted in each domain.
Contributions: The main contribution is to present a systematic framework of learning to separate domains by probabilistic distributions of instances, which is capable of addressing various recognition tasks, including OSL and G-ZSL.
Towards this goal, two simple, most widely used, and very effective tools – bootstrapping and the Kolmogorov-Smirnov test, are employed to ﬁrstly initially estimate
and then ﬁne-tune the boundary.
In particular, we introduce an uncertain domain, which encloses the instances which can hardly be classiﬁed into known or unknown with high conﬁdence.
We extensively evaluate the importance of domain division on several zero-shot learning benchmarks and achieved signiﬁcant improvement over existing ZSL approaches.
R ELATED WORKS
One-Class Classiﬁcation (OCC) .
It is also known as the unary classiﬁcation or class-modeling.
The OCC assumes that the training set contains only the positive samples of one speciﬁc class.
By learning from such positive instances, OCC aims at identifying the instances belonging to that class.
The common algorithms of OCC include One-class Support Vector Machine (OCSVM)
Sch ¨Alkopf et al.
Local Outlier Factor (LOF) Breunig et al.
OCSVM leverages Support Vector
Data Description (SVDD) to get a spherical boundary in feature space.
It regularizes the volume of hypersphere so that the effects of outliers can be minimized.
The LOF measures the local deviation of the density of a given instance comparing to other instances, namely locality.
The locality represents the density of the area.
The instances in low-density parts can be taken as outliers.
Note that all OCC algorithms just considered and build a boundary for the positive instances of one class.
Open Set Learning (OSL).
It judges whether the instances belong to known/seen classes Sattar et al.
Scheirer et al.
(2014; 2013); Bendale & Boult (2015) or a novel unknown class.
Both the OCC and OSL are able to divide the instances into known and unknown domains and recognize the known classes from the known domain.
OSL aims at discriminating the instances into seen classes and instances beyond these classes are categorized into a single novel class.
Critically OSL does not have the semantic prototypes of unseen classes to further give the class label of those instances in the novel class.
Both the OCC and OSL are able to divide the instances into known and unknown domains and recognize the known classes in the known domain.
Intrinsically, their key difference lies in whether leveraging the information of different seen classes in building the classiﬁers.
Speciﬁcally, OCC only utilizes the instances of one class to learn its class boundary, whilst OSL can use the instances of different seen classes.
Zero-Shot Learning (ZSL).
ZSL aims at recognizing the novel instances which have never been trained before.
It transfers the knowledge learned from known source classes to recognize the testing instances from unknown target classes.
The knowledge can be formulated as semantic attributes
Farhadi et al.
Lampert et al.
Fu et al.
semantic word vector Norouzi et al.
(2013); Frome et al.
; Fu & Sigal (2016); Xu et al.
(2017), or semantic ontology Rohrbach et al.
However, ZSL usually assumes that the unseen classes cannot be mis-classiﬁed as seen classes and vice versa.
This has greatly simpliﬁed the learning task.
Generalized Zero-Shot Learning .
Chao et al.
Changpinyo et al.
(2016) realized that it is nontrivial to directly utilize the existing Zero-Shot Learning algorithms in a more general setting, i.e. , G-ZSL.
In such a setting, the testing instances can come from either the seen or unseen classes.
A thorough evaluation of G-ZSL is further conducted in Xian et al.
Their results show that the existing
ZSL algorithms do not perform well if directly applied to G-ZSL.
The predicted results are inclined to be biased towards seen classes.
L EARNING TO CATEGORIZE INSTANCES INTO DIFFERENT DOMAINS 3. P ROBLEM SETUP
In learning tasks, we are given the training dataset , i.e. , seen classes, of n s instances, D s = { x i , y i , l i } n s i =1 : x i ∈ R n is the feature of i th instance with the class label l i ∈ C s , where C s is the source class set; n cs is the number of instances in seen class c .
Analogous to standard ZSL setting, we introduce the target label classes C t with C s (cid:84) C t = ∅ and the total class label set C
=
C s ∪ C t .
y i is the semantic attribute vector of instance x i .
In general, the y i of instances in one class should be the same Lampert et al.
We simplify y c as the semantic prototype for all the instances in class c .
Given one test instance x i , our goal is to predict its class label c
i .
We discuss two tasks: (1)
Open set recognition: c i ∈ {C s , (cid:48) novel class (cid:48) } ;
(2) Generalized zero-shot learning: c i ∈ {C s , C t } .
The semantic prototype is predeﬁned for each class in C t .
The ’ novel class ’ is an umbrella term referring to any class not in C s .
3. E XTREME VALUE DISTRIBUTIONS
We ﬁrstly introduce the background of modeling the extreme values ( i.e. , minimum/maximum scores) computed from one supervised classiﬁer as the extreme value distributions.
In particular, by using each source class c , we can train a binary predictor function, e.g . SVM, z c = f c ( x ) : R n → R where z is the conﬁdence score of instance x belonging to the class c .
In Extreme Value Theory (EVT) , the extreme values (i.e., maximum / minimum conﬁdence scores) of the score distribution computed by the predictor function f c ( · ) can be modeled by an EVT distribution.
Speciﬁcally, for instance set { x } that belong to class c ; the minimum score z c min = min f c ( { x } ) follows the Weibull distribution,
[Equation]
where G ( · ) is the Cumulative Distribution Function (CDF) of Weibull distribution ( z > ν ):
G ( z c ; λ c , ν c , κ c ) = 1 − exp (cid:16) − (cid:16) z c − ν c λ c (cid:17) κ c (cid:17) .
Critically, Eq (1) models the lower boundary distribution of instance conﬁdence scores belonging to class c .
On the other hand, for instance set { x } NOT belonging to class c , the maximum score z c max = max f c ( { x } ) should follow the reverse
Weibull distribution Scheirer et al.
[Equation]
(2) where rG ( · ) is the CDF of reverse Weibull distribution:
[Equation]
Eq (2) models the upper boundary distribution of conﬁdence scores NOT belonging to class c .
The scale parameters λ c , λ (cid:48) c , shape parameters κ c , κ (cid:48) c , and location parameters ν c , ν (cid:48) c are estimated by
Maximum Likelihood Estimator ﬁtted from the training data.
Critically, Eq.
(2) models the upper boundary distribution of instance conﬁdence scores NOT belonging to class c .
3. W EAKNESS OF CATEGORIZING INSTANCES INTO KNOWN / UNKNOWN DOMAINS
One can conduct the OSL and G-ZSL by directly learning to separate the domains of seen and novel/unseen classes.
The idea of learning to divide testing instances to known or unknown domains has been studied in W-SVM Scheirer et al.
(2014) (OSL), and CMT Socher et al.
(2013) (G-ZSL).
Particularly, given a test instance x i , the supervised binary algorithm can compute the conﬁdence score
z c i =
f c ( x i ) belonging to the class c .
Here we introduce two events:
[Equation]
the other seen classes
[Equation]
The distributions of extreme values deﬁned in Eq (1) and Eq (2) can actually give us the boundary of each event above happened in a probabilistic perspective.
Thus we can have the probability
P G ( E 1 ) =
1 − exp (cid:16) − (cid:16) z [Equation] (cid:17) κ c (cid:17) ; and P rG ( ¬ E 2 ) =
exp (cid:18) − (cid:16)
[Equation]
c (cid:19) .
Thus, to determine if one testing instance x i belong to class c , W-SVM Scheirer et al.
(2014) computes the statistic m c ( x i ) by,
[Equation]
W-SVM introduces a threshold δ c to determine whether the instance i belongs to the class c as,
[Equation]
) where δ c is a ﬁxed value Scheirer et al.
The instance x i rejected by all the seen classes by
Eq (5) is labeled as the unknown domain.
Generalizing to C s class is straightforward by training multiple prediction functions { f c ( x ) } , c = 1 , · · · , |C s | .
However, there are several key limitations in directly utilizing Eq (4) and Eq (5) of learning the division of domains: (1) Eq (4) directly multiplies two terms and this indicates that there is a potential hypothesis that no correlation exists between E 1 and ¬ E 2 , which is generally not the case.
In multiple seen classes, the instances may derive from many different classes.
It is hard to determine a single ﬁxed δ in Eq (5) for each class.
Furthermore, we give an illustration of the non-negligible overlapping between the intra-class and inter-class distances of each pair of instances on the aPY dataset (described in Sec. 6.1,Farhadi et al.
As shown in the feature space of Fig.
2 (b), we compute the pairwise L 2 distances over (1) the instances within the same classes (intra-class), and (2) the instance from different classes (inter-class) on aPY dataset Farhadi et al.
We use the empirical density to show the results in Fig.
2 (c).
Practically, it is hard to predict the class labels of instances of the overlapped region in the known/unknown domain.
Due to the large overlapped region, the instances ( e.g. , CMT in Socher et al. (2013)) whose domains are wrongly labeled will never be correctly categorized.
P ROPOSED DOMAIN DIVISION ALGORITHM 4. D ETERMINING THE INITIAL BOUNDARY BY BOOTSTRAPPING
The W-SVM in Eq (5) and Eq (4) estimates the conﬁdence scores by a ﬁxed threshold empirically per-deﬁned for any data distributions in the known domain.
However, intrinsically, it can be taken as a model selection task in estimating the boundary by Eq (4).
In this paper, we tackle the question of constructing the boundary of the known domain via the bootstrap approach Efron (1979).
The bootstrapping 2 is a strategy of estimating the standard errors and the conﬁdence intervals of parameters when the underlying distributions are unknown.
Its procedures are closely related to the other methods such as cross-validation, and jackknife.
Bootstrapping is the most widely used tool in approximating the sampling distributions of test statistics and estimators.
To facilitate the discussion, we denote the training set of class c as { x c tr } ; the testing set whose instances are mostly conﬁdently predicted as class c , as { x c te } .
Thus the corresponding conﬁdence score set on training and testing data are { z c tr } = f c ( { x ctr } ) and { z cte } =
f c ( { x cte } ) respectively.
The whole algorithm is shown in Alg.
Till now, we had a sketch of our domain division algorithm.
Speciﬁcally, the training instances of seen classes are utilized to learn the f c ( · ) , c ∈ C s ;
For any given testing instance x i , we compute its conﬁdence score f c ( x i ) , c ∈ C s .
To determine whether the class of a instance x i is seen or unseen, we calculate the statistic m c ( x i ) in Eq (4) and Eq (5) with the threshold δ c estimated by the bootstrapping algorithm in Alg. 1.
The instances computed in the known and unknown domain will be categorized by supervised, or zero-shot classiﬁers respectively.
There are still two difﬁculties in the above framework.
(1) The whole framework relies on the classiﬁer f c ( · ) , c ∈ C s which is supposed to be robust and well-trained.
However, empirically, we can not always train good classiﬁers for all classes.
For example, some class has small number of labeled training instances which are insufﬁcient in training the classiﬁer; some outliers may affect the predictor; the hyper-parameters of the classiﬁers are wrongly tuned.
The naive bootstrapping in Alg.
1 generally provides the bad approximations of the distributions of empirical quantiles in practice Falk & Reiss (1989).
Practically, in our tasks, we observe that the estimated δ c may be consistently too relaxed to determine the boundary of the known domain.
We illustrate such a phenomenon in Figure 2(a): the low-density of seen class bicycle instances (blue points) in the northwest part extends the decision boundary.
The relaxed boundary could inadvertently classify unseen instances (red points) as the false positives.
Unfortunately, in the framework above, once one testing instance in unseen class is wrongly labeled as the known domain, this instance will never be correctly classiﬁed.
To address these two problems, we suggest a shrinking step in updating the initial boundary of bootstrapping in the next subsection.
4. S HRINKING THE BOUNDARY OF THE KNOWN DOMAIN BY K OLMOGOROV -S MIRNOV T EST
The key idea of updating initial boundary of bootstrapping is to validate whether the learned classiﬁer f c ( · ) , c ∈ C s is trustworthy.
Generally, assume the instances of class c independent and identically distributed and provided training samples sufﬁcient, a ideal classiﬁer f c ( · ) should produce the similar conﬁdence score distributions of training and testing instances of class c .
The Kolmogorov-Smirnov (K-S) test is an efﬁcient, straightforward, and qualiﬁed choice method for comparing distributions Massey Jr (1951); Miller (1956);
Wang et al.
Remarkably, K-S test is a distribution free test, and the statistics of K-S test is effortless to compute.
We deﬁne the null and alternative hypothesis as
[Equation]
from the same distribution
[Equation]
We introduce a distance measure K c =
[Equation]
F ctr
[Equation]
z ) − F cte
[Equation]
z ) (cid:107) where F ctr = ecdf( { z ctr } ) and
F cte = ecdf( { z cte } ) ; and the ecdf ( · ) is the empirical distribution function.
The null hypothesis would be rejected at the signiﬁcant level α when,
[Equation]
When H 0 is accepted, it indicates that the f c ( · ) is trustworthy, and the conﬁdence scores of training and testing instances in class c come from the same distribution.
We are certain that a large portion of testing instances { z cte } = f c ( { x cte } ) should be indeed in the class c .
On the other hand, when
H 0 is rejected, we are not sure whether f c ( · ) is well learned; and the class labels of these testing instances are uncertain.
To this end, we introduce a new domain – uncertain domain to include these instances.
Uncertain Domain.
The labels of instances in the uncertain domain should be labeled as the most likely seen class, or one of unseen classes.
Speciﬁcally, we can compute the { z c = f c ( x ) } |C s | c =1 over all C s classes; and we can obtain,
[Equation]
The mapping function g ( · ) is learned on the known domain from features x i to its corresponding semantic attributes y i .
Given one testing instance x i : if z (cid:63)i is very high, we can conﬁdently predict x i belonging to one of seen classes; otherwise, the label of x i is either in the uncertain or unknown domain.
We thus have,
[Equation]
seen domain
[Equation]
unseen domain
[Equation]
9) where y c is semantic prototype of class c ; c (cid:63) is the most likely known class to which x i belongs to.
Note that in OSL, we only know the y c ( c ∈ C s ) of seen classes; We can dynamically construct a C t set by randomly generating y i by making sure (cid:107) y
i − y j (cid:107) > (cid:15) ( ∀ y j ∈ C s ), and (cid:15) =
min y i , y j ∈C s ; i (cid:54) =
j (cid:107) y
i − y j (cid:107) .
The sample size is usually the same with the number of target classes.
[Equation]
ECOGNITION MODELS IN KNOWN / UNKNOWN / UNCERTAIN DOMAINS
We can apply different recognition algorithms in each domain.
In known domain, the standard supervised classiﬁers can be learned and applied.
In unknown and uncertain domains, we propose a simple yet effective feature prototype embedding recognition algorithm as our plain implement.
Feature prototype embedding.
Once the domain is well separated, we can use the ZSL algorithms to set up the mapping from feature space to semantic/attribute space.
In order to conﬁrm that our main contribution is the domain division part, we do not use very complicated ZSL algorithms.
Only the simplest linear predictor is utilized here to recognize the unseen classes.
Particularly, we use feature prototypes to replace all the instances of each class to avoid the unbalance sample size among classes.
We learn a linear predictor to predict the attribute/word vector g ( x ) = w T · x .
The feature prototype embedding is computed as,
[Equation]
where (cid:101) x c = 1 n cs Σ l i =
c x i is the feature prototype of class c ; y c is the semantic prototype of class c .
When we tackle an instance in the unknown or uncertain domain, we need to embed features into semantic space with g , which can infer the class labels of instances:
[Equation]
(11) where c (cid:63) is the most likely seen class for x i which is computed by the supervised classiﬁer and y c is the semantic prototype.
Also, the experiment in each domain can be done with ANY other ZSL.
For instance, we report the implement with f-CLSWGAN (f-C)
Xian et al.
so that G-ZSL can be done with f-CLSWGAN within a single domain.
E XPERIMENTS 6. D ATASETS AND SETTINGS
Datasets.
Animal with Attribute (AwA) Dataset Lampert et al.
(2014) has 50 classes and 30, images in total, with 85 class-level attributes annotated.
We use 40 source training classes (including 13 classes as validation); the rest as testing.
CUB Dataset Wah et al.
(2011) includes 200 classes and 11,788 ﬁne-grain images with 312 class-level attributes annotated.
The training set has 150 classes (including 50 classes as validation).
(3) aPY Dataset Farhadi et al.
(2009) has 15,339 images inclasses with 64 class-level annotated attributes.
We use 20 classes for training (including 5 validation classes).
For the AwA, CUB and aPY, we use ResNet-101 features and the class split contributed by Xian et al.
(4) ImageNet 2012/2010 dataset is proposed in Fu & Sigal (2016).
As the large-scale dataset, we use the split as Fu & Sigal (2016): 1000 training classes with full training instances in ILSVRC 2012; and 360 testing classes in ILSVRC 2010, non-overlapped with ILSVRC 2012 classes.
We address two recognition tasks: OSL and G-ZSL.
Experimental settings.
Our model is validated in OSL and G-ZSL settings.
OSL identiﬁes whether an image belongs to the one of seen classes or the novel class.
G-ZSL gives the class label of testing instances either from seen or unseen classes.
We set the signiﬁcance level α = 0 .
05 to tolerate 5% Type-I error.
By default, we use SVM with RBF kernel with parameter cross-validated, unless otherwise speciﬁed.
6. R ESULTS OF O PEN S ET L EARNING
We compare against the competitors, including Attribute Baseline (Attr-B), W-SVM Scheirer et al.
One-class SVM
Sch ¨Alkopf et al.
(2001), Binary SVM, OSDN Bendale & Boult (2016)
and
LOF Breunig et al.
The attribute baseline is the variant of our task without using domain division algorithm.
Particularly, the Attr-B uses the same semantic space and embedding as our model, but does not leverage domain division step, i.e. , use negative samples and prototypes to identify projected instances directly (Fig. 1 (c)).
We use the metric – F1-measure, which is deﬁned as the harmonic mean of seen class accuracy (speciﬁc class) and unseen prediction accuracy (unnecessary to predict the speciﬁc class).
The results are compared in Tab. 1.
Signiﬁcant performance gain over existing approaches has been observed, in particular for AwA, aPY and ImageNet.
This validates the effectiveness of our framework.
We attribute the improvement to the newly introduced uncertain domain which help better differentiate whether testing instances derive from known or unknown domain.
[Table]
[Table]
R ESULTS OF G ENERALIZED Z ERO -S HOT L EARNING
Settings : We ﬁrst compare the experiments on G-ZSL by using the settings in Xian et al.
The results are summarized in Tab. 2.
In particular, we further compare the separate settings; and top-1 accuracy in ( % ) is reported here: (1) S → T : Test instances from seen classes, the prediction candidates include both seen and unseen classes; (2) U → T : Test instances from unseen classes, the prediction candidates include both seen and unseen classes.
(3) We employ the harmonic mean as the main evaluation metric to further combine the results of both S → T and U → T , as
H = 2 · ( Acc ( U → T ) ×
Acc ( S → T )) /
( Acc ( U → T ) + Acc ( S → T )) .
Competitors .
We compare several competitors.
(1) DAP
Lampert et al.
(2014), trains a probabilistic attribute classiﬁer and utilizes the joint probability to predict labels; (2) ConSE Norouzi et al.
(2013), maps features into the semantic space by convex combination of attributes; (3) CMT Socher et al.
[Table]
(2013), projects features into unsupervised semantic space and uses LOF to detect novel classes; (4)
SSE Zhang & Saligrama (2015), regards novel classes as mixtures of seen proportions to measure the instance similarity.
(5) Latem Xian et al.
(2016), is a novel latent embedding for ZSL and G-ZSL.
(6) ALE Akata et al.
(2016), embeds labels into the attribute space by learning a function to rank the likelihood of each class.
(7) DeViSE Frome et al.
(2013), uses both unsupervised information and annotated attributes to classify classes in an embedding model; (8) SJE Akata et al.
(2015) is a hierarchical embedding to learn an inner product gram matrix between features and attributes.
(9) ESZSL Romera-Paredes & Torr (2015), focuses on the regularization term in the projection from features to semantic space.
(10) SYNC Changpinyo et al.
(2016), aligns the semantic space to feature space by manifold learning.
(11) SS-VOC Fu & Sigal (2016), optimizes the triplet loss to learn the projection from features to semantic space.
(12) SAE Kodirov et al.
(2017) is an auto-encoder to combine feature and semantic space.
(13) SE-GZSL Verma et al.
(2018) leverages VAE Kingma & MaxWelling (2014) as the generator of pseudo instances to train the mapping.
(14-15) cycleCLSWGAN (cycle-C) & f-CLSWGAN (f-C)
Felix et al.
; Xian et al.
(2018): both of them use
W-GAN Arjovsky et al.
(2017) to reconstruct features and cycle-C adds another regularizer.
PTMCA Long et al.
(2018) uses transfer approach and embedding way to reduce bias and variance.
[Table]
Results .
As seen in Tab. 2, our harmonic mean results are signiﬁcantly better than all the competitors on almost all the datasets.
This shows that ours can effectively address the G-ZSL tasks.
Particularly, (1) Our plain results can beat other competitors by a large margin on AwA and aPY dataset, due to the efﬁcacy of our domain division algorithm.
Also, thanks to the power of f-CLSWGAN (f-C) , our implement with it on both CUB and AwA dataset are impressive.
The key advantage of the proposed framework is learning to better divide the testing instances into known , uncertain and unknown domains.
In the known domain, we use the standard SVM classiﬁer.
In unknown/uncertain
[Table]
domains, we directly embed feature prototypes into semantic space and match the most likely class in the candidate pool.
This is the most simple and straightforward recognition method.
Thus our good harmonic mean performance on G-ZSL largely comes from the good domain division algorithm.
Additionally, we also highlight that the other advanced supervised or zero-shot algorithms are orthogonal and potentially be useful in each domain if we want to further improve the performance of G-ZSL.
Our framework is also applied to large-scale datasets in Tab. 3.
We compare several state-of-the-art methods that address G-ZSL on the large-scale dataset.
We use the SVM with the linear kernel on this dataset, due to the large data scale.
Our harmonic mean results surpass the other competitors with a very signiﬁcant margin.
We notice that other algorithms have very poor performance on U → T .
This indicates the intrinsic difﬁculty of G-ZSL on large-scale dataset.
In contrast, our domain division algorithm can better separate the testing instances into different domains; thus achieving better recognition performance.
Additionally, we found that the prediction of
ConSE Norouzi et al.
(2013) is heavily biased towards known classes which is consistent with the results in small datasets.
This is due to the probability of unseen classes are expressed as the convex combination of seen classes.
Usually, there is no higher probability would be assigned to unseen classes than the most probable seen class, especially for large datasets.
6. A BLATION STUDY
In the ablation study, we report the F1-measure and Harmonic mean for OSL and G-ZSL respectively with our plain implement.
As is illustrated in Fig.
2, we notice that although the distance statistic shows the different histogram patterns in feature space, the overlapping part is not negligible.
Importance of bootstrapping the initial threshold .
We introduce a variant A of our framework by replacing bootstrapping step (Sec. 4.1) by using Eq (4) and Eq (5) to ﬁx the threshold ( i.e. , W-SVM
Scheirer et al.
i.e ., K-S test ( √ ), and Bootstrap ( × ).
As in Tab. 4, the results of variant A are signiﬁcantly lower than our framework on all three datasets.
This actually directly validates the importance of determining the initial threshold by bootstrapping.
Improvements of ﬁne-tuning the threshold by K-S test.
We deﬁne the variant B is to only use step without ﬁne-tuning the boundary by K-S Test (in Sec. 4.2).
Table 4 directly shows the improvement with/without ﬁne-tuning the threshold, i.e ., K-S test ( × ), and Bootstrap ( √ ).
In particular, we note that variant B has signiﬁcant lower results on OSL and G-ZSL than variant A and our framework.
One reason is that our bootstrapping step actually learns to determine a very wide boundary of the known domain, to make sure the good results in labeling testing instances as unknown domain samples.
The ﬁne-tuning threshold step will further split the individual known domain into known/uncertain domain by shrinking the threshold.
Without such a ﬁne-tuning step, variant B may wrongly categorize many instances from unseen classes as one of the known classes.
Thus, we can show that the two steps of our framework are very complementary to each other and they work as a whole to enable the good performance on OSL and G-ZSL.
Finally, we introduce the variant C in
Tab. 4 , by using W-SVM to do OSL, and then use our ZSL model for G-ZSL, i.e. , K-S test ( × ), and
Bootstrap ( × ).
The performance of variant C is again signiﬁcantly lower than that of ours, and this demonstrates the efﬁcacy of our model.
C ONCLUSION
This paper learns to divide the instances into known, unknown and uncertain domains for the recognition tasks from a probabilistic perspective.
The domain division procedure consists of bootstrapping and K-S Test steps.
The bootstrapping is used to set an initial threshold for each class; we further employ the K-S test to ﬁne-tune the boundary.
Such a domain division algorithm can be used for OSL and G-ZSL tasks, and achieves remarkable results.