D EEP HIERARCHICAL MODEL FOR HIERARCHICAL SELECTIVE CLASSIFICATION AND ZERO SHOT LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Object recognition in real-world image scenes is still an open problem.
With the growing number of classes, the similarity structures between them become complex and the distinction between classes blurs, which makes the classiﬁcation problem particularly challenging.
Standard N-way discrete classiﬁers treat all classes as disconnected and unrelated, and therefore unable to learn from their semantic relationships.
In this work, we present a hierarchical inter-class relationship model and train it using a newly proposed probability-based loss function.
Our hierarchical model provides signiﬁcantly better semantic generalization ability compared to a regular N-way classiﬁer.
We further proposed an algorithm where given a probabilistic classiﬁcation model it can return the input corresponding super-group based on classes hierarchy without any further learning.
We deploy it in two scenarios in which super-group retrieval can be useful.
The ﬁrst one, selective classiﬁcation, deals with the problem of low-conﬁdence classiﬁcation, wherein a model is unable to make a successful exact classiﬁcation.
The second, zero-shot learning problem deals with making reasonable inferences on novel classes.
Extensive experiments with the two scenarios show that our proposed hierarchical model yields more accurate and meaningful super-class predictions compared to a regular N-way classiﬁer because of its signiﬁcantly better semantic generalization ability.
I NTRODUCTION
Object recognition from images in real world scene, which requires drawing boundaries between groups of objects in a seemingly continuous space, is a highly complex problem.
The visual world is populated with a vast number of diverse objects, where each object instance may pose many visual forms.
Furthermore, with the growing number of classes, the similarity structures between them become complex and the distinction between classes blurs.
Because of the high complexity of this task, most existing work focus on simplifying the problem to a supervised single-label classiﬁcation problem.
Unfortunately, since one-vs-all classiﬁers treat all classes as unrelated, visual recognition systems cannot transfer semantic information about learned classes.
One way of dealing with this issue is to respect the natural continuity of visual space instead of artiﬁcially partitioning it into disjoint categories.
Classes have varying degrees of discriminability.
Some classes have unique features while other classes might share similar features and are hence harder to distinguish from each other.
Such similarity structures in the data are very valuable information that could potentially lead to improving classiﬁers (Frome et al., 2013; Rippel et al., 2015;
Bilal et al., 2018).
Such classiﬁer has a potential to make semantically reasonable errors.
For example, in the upper image of Figure 1 the standard model choose the wrong class ’stingray’ while a more reasonable error can be ’hammerhead’ or ’great white shark’.
Moreover, such semantic classiﬁers may be less biased to irrelevant context as illustrated in the example.
The standard model is biased to the wrong class because the image has people.
We found that in the training set many images of stingray are with people.
In order to conﬁrm this statement we occlude 1 the people.
By doing this we mange to classify the image correctly with the standard classiﬁer (Zeiler & Fergus, 2014).
On the other hand, our semantic model identify the true class even in the original image.
Furthermore, after occluding the error cause our model ranks the top-5 predicated classes according to their relevance to the true class measured by the taxonomy distance.
We propose an approach that addresses these shortcomings by learning a hierarchical inter-class relationship model.
We present a novel hierarchical probability-based loss function, which we call soft-NLL .
This loss function gives a probability weight according to the inter-class taxonomy graph distances.
We evaluate the model semantic generalization ability, which means for example that the model makes more semantically reasonable errors.
Our model shows a signiﬁcant improvement on the hierarchical semantic metric in an order of magnitude over existing methods, while it guarantees a minor decrease in the common top-k accuracy compared to the widely used hard-NLL loss.
Our approach is independent of the speciﬁc prediction model and thus can always beneﬁt from design progress.
Furthermore, we develop an algorithm where given a probabilistic model, based on its top-k predictions our algorithm returns the input corresponding super-group based on classes hierarchy without any further learning.
We present two use cases for super-group retrieval and show that our soft-NLL outperforms in both applications.
The ﬁrst one, selective classiﬁcation, deals with the problem of low-conﬁdence classiﬁcation, wherein a model is unable to make a successful exact classiﬁcation.
In this case, our algorithm returns a corresponding closest super-class.
In the second scenario, the proposed method is used for the zero-shot learning problem.
In this case, given a novel input, the algorithm returns its hierarchically related group, rather than generating a true unseen group.
R ELATED WORK
Vision researchers have sought to exploit external semantic knowledge.
Taxonomies, or class hierarchies were the ﬁrst to be explored (Grifﬁn & Perona, 2008; Marszalek et al., 2008).
Taxonomy depends on a human-deﬁned criteria, and thus implicitly consists of semantic knowledge.
Early works (Bengio et al., 2010; Gao & Koller, 2011; Grifﬁn & Perona, 2008; Marszalek et al., 2008) exploit the tree structure for efﬁcient branch-and-bound training and prediction, while a few use taxonomies as sources of relational knowledge between categories (Deng et al., 2014; Hwang, 2013).
Our method shares the same goal with the latter group of work.
Deep Convolutional Neural Networks which learn high-level features and a N-way ﬂat softmax classiﬁer have recently gained enormous popularity for their impressive performance on a number of visual recognition tasks.
Since Krizhevsky et al.
(2012) won the ImageNet ILSVRC challenge
2012 (Deng et al., 2009), many variants of this model have been proposed.
GoogLeNet (Szegedy et al., 2015), Resnet (He et al., 2015) and VGGNet (Simonyan & Zisserman, 2014) focus on increasing the network depth by adding more convolutional layers to the original model.
These models generalize better than smaller ones.
Our goal is to beneﬁt form the progress of CNN designs.
Some existing work has explored various approaches to learning semantic information in deep neural networks.
One related approach is to learn a joint representation of images and labels in an embedding space.
Frome et al.
; Norouzi et al.
(2013) learn semantic relationships between classes using a semantic word embedding model, where each class represented by an embedding vector.
In parallel, they pre-train a deep neural network for visual object recognition.
Then, both methods develop a speciﬁc mapping of the images into the semantic embedding space.
Both methods evaluated their semantic ability using a semantic metric.
Moreover, they employed their models for zero-shot learning.
However, Frome et al.
(2013) show modest performance in its semantic relevance meaning because the word2vec embeddings are weak in the sense of capturing the inherent hierarchical semantics.
Furthermore, it imposes structure only on the output space, and structure on the learned space is not explicitly enforced.
However, our goal is to transfer direct inter-class hierarchical data to the learned space.
Deng et al.
(2014) introduce a new formalism which captures semantic relations between any two labels applied to the same object:mutual exclusion, overlap and subsumption.
In the experiments part they used Wordnet heirachy and deﬁned exclusion relation between any two labels unless they share a descendant.
Each image got a single label and in this scope some leaf examples was relabeled only to theirs
immediate parent(s) internal node label.
In different to this method, we propose to formalize real mutual exclusion and subsumption relations in a continuous similarity meter calculated from taxonomy.
The similarity level of labels is getting lower according to their distance from the true class.
We show that their exclusion approach is too strict.
That is we show that giving similarity weight even to labels which share predecessor is beneﬁcial to performance in Section 4.2.
Secondly, our similarity concept which gives a soften-label allow to share similarity structure on all images.
Moreover, the authors did not present an evaluation of how predictions (i.e. even the incorrect ones) are semantically meaningful which is our focus in this work.
Yan et al.
(2015) propose a two-staged CNN architecture named HD-CNN, which leverages the taxonomy to separate the categories into easy coarse-grained ones and confusing ﬁne-grained ones, trained in separate networks.
However, such separate learning for coarse and ﬁne-grained categories results in larger memory footprints, while it improves performance with only a modest rate.
Recently Bilal et al.
(2018) designed a hierarchy-aware CNNs that accelerate model convergence and alleviate overﬁtting.
They select AlexNet (Krizhevsky et al., 2012) as a reference architecture and created branches from multilevel deep features to perform group-level classiﬁcation and backpropagate group error.
The number of groups is gradually increased along the depth of the inner layers.
However this scheme demands extensive analysis and consequently it cannot easily beneﬁt from the progress of CNN designs.
Moreover, the latter two works did not present an evaluation of how predictions are semantically meaningful.
Our goals are to directly exploit the class hierarchy information while beneﬁt form the progress of
CNN designs.
We focus on large scale multi-class problems.
In contrast to most previous works we evaluate the semantic ability of our model according to a semantic metric.
We further show that such a metric is meaningful when we compare our soft semantic model to a regular hard-model on two applications.
T HE PROPOSED METHOD 3. S OFT NEGATIVE LOG LIKELIHOOD LOSS FORMULATION
The problem we address in this paper is a supervised multiclass classiﬁcation.
Following common setup, we assume the input X ∈ X and label Y ∈ Y = { 1 , ..., C} can be modelled by a joint distribution
π ( X, Y ) .
The labels are organized in a taxonomy graph G , where each directed edge represents an ’is a’ relation, and the distance between nodes reﬂects a semantic relationship between them.
Our goal is to model these semantic relationships, so that its top-k predictions will be closely related to the true label according to G .
The standard objective formulation used in the context of deep learning for a multiclass classiﬁcation problem is the negative log likelihood (NLL) loss, which is also referred to as the cross entropy loss with the probability class indicator.
Given a probabilistic model ˆ π ( Y | X ) and a sample ( x 0 , y 0 ) ,
NLL loss is deﬁned by
[Equation]
, where π ( y | x
[Equation]
This formulation assigns the whole probability weight to a single class with all other classes having zero weight.
This approach artiﬁcially partitions the visual space into disjoint categories and does not take into account semantic relationships between classes.
We propose to use a ’soft’ (instead of ’hard’) model π ( y | x ) as a semantic relationship measure.
This model can be learned as the class graph taxonomy G .
More formally, let Π ∈ R C×C represent a semantic probability taxonomy relationship (or distance) map, where Π i,j (cid:44) π ( i | j ) .
We deﬁne the soft NLL loss as:
[Equation]
Π is a class row-wise i.e. each ith row resembles the ith true-class probabilities, which are inversely proportional to the on-the-graph distance deﬁned, d [Equation] class node i [Equation]
class
[Equation]
j [Equation] shortest path length between nodes Details about hyper-parameters selection is described in Section 4.1.
3. S UPER - GROUP RETRIEVAL ALGORITHM
In this section we present a super-group retrieval algorithm.
Returning a super-group can be useful in hard scenarios for example, when given object samples with high confusion between similar classes or when making an inference for novel classes.
Soft-NLL has a better semantic generalization ability as we show below in Section 5.1.
Its top-k predictions are more accurate with respect to the true class.
Therefore, they can be used as a coarse-grained classiﬁer and retrieve a better super group, compared to standard models.
Given the model top-k predictions the algorithm follows these steps:
Clean the top-k predictions from less relevant predictions: The model top-k predictions may be noisy.
Based on the graph taxonomy we can generate a cleaner subset of the top-k predictions.
More speciﬁcally, given the top-k predictions
we calculate a subset of them with each ith class l-hCorrectSet
as
follows,
[Equation]
Where for a given class its l-hCorrectSet is the l nearest set of classes gathered from the graph taxonomy as deﬁned by Frome et al.
We choose the highest matching set, S C ( l, k ) where C = argmax i
[Equation]
Generate super-group candidates: The set { A S } equals to super-group nodes which are ancestors for each class in S C ( l, k ) .
Choose the most speciﬁc super-group: The most speciﬁc super group is the a ∈ { A S } which is the lowest common ancestor (LCA) of S C ( l, k ) generated by LCA = min a ∈{ A S } (cid:80) s ∈ S C [Equation]
,k [Equation] d G
[Equation] a [Equation] For example, suppose we have a simple taxonomy as illustrated in Figure 2, where the leaf nodes are valid classes and the other nodes are theirs super-classes.
Suppose we got a sample from class 1, and our top-3 predictions are: 0,1 and 6.
Each class has its own 3-hCorrestSet, where the true class hCorrestSet gives the maximal intersection set with the top3 predication i.e. S 1 (3 , 3) = 0 , 1 .
We have three super-group candidates: A,B,D where D is the LCA generated super-group.
In Section 4.2 we discuss ways of choosing the algorithm’s hyper-parameters k and l .
E XPERIMENTS
The objective of this work is to develop a method for generating a semantic vision model i.e. a model which makes semantically relevant predictions even when it makes errors.
Moreover, we show its advantage in super group generation in two different scenarios of selective classiﬁcation and zero shot learning.
DATASET ImageNet is the largest publicly available labeled image dataset, encompassing more than 14 million images that belong to more than 21K object categories (Deng et al., 2009).
The object categories are nouns in the WordNet database of the English language (Miller, 1995) .
A fundamental property of WordNet is its semantic hierarchical organization of concepts.
In our experiments we adopt a subset of ImageNet the ILSVRC12 dataset, which gather 1K classes that are randomly selected according to certain criteria that aim to reduce ambiguity.
Theseclasses are mutually exclusive leaf nodes of WordNet that has 820 internal nodes.
It consists of approximately 1.3 million training images and 50k validation labeled images from each category.
As illustrated in Figure 3, ILSVRC12 taxonomy is a complex directed graph, there may be multiple routes from the root to the leaf nodes.
Its minimal spanning tree which contains the leaf nodes is highly not balanced, where routes depth range is between 6 and 18.
METRICS We use several metrics in the evaluation process, each was averaged on test images.
The ﬂat hit@k is a standard metric 0/1 error used in large scale classiﬁcation problems which returns a success if the true label resides in the top k predictions.
To measure the semantic quality of predictions beyond the true label, we also evaluate with the hierarchical-precision@k ( hp@k ) introduced in Frome et al.
(2013), which is a semantic relevance of the model top-k predictions.
It is computed as a fraction of the top-k predictions that overlap with the true class k-hCorrectSet :
[Equation]
(cid:88) n = number of model’s top-k predictions in k-hCorrectSet for image
i k where for a true class k-hCorrectSet is the k nearest classes set gathered from the graph taxonomy as described in detail in Frome et al. (2013).
In order to evaluate super-group (SG) generation succession we adopt two metrics, SG - hit is a 0/ error metric, where given a predicated label, it returns success if there is a path between the SG candidate and the true class.
The latter metric may be useless because the algorithm may return the root, therefore we calculate how much a candidate is speciﬁc by measuring its distance to the true class i.e. SG - specificity =
d G ( SG, truelabel ) .
The last two metrics are based on the graph taxonomy.
We check the SG generation succession in two scenarios: selective classiﬁcation and zero shot learning.
TRAINING DETAILS
In our experiments we are comparing semantic ability of different loss paradigms while using the same core vision model.
We train Resnet50 proposed by He et al.
(2015) and
Alexnet presented in Krizhevsky et al.
(2012) covnvolutional networks as our core vision models.
We use the data preprocessing, training procedures, and hyperparameters as described in these papers.
4. SOFT HIERARCHICAL BASED PROBABILITIES : CHOOSING HYPER - PARAMETERS
In this section we deal with determination of soft labels hierarchical probabilities hyper-parameters Π which deﬁned in Equation 1.
Given Π ith row which indicates the true class record.
The probabilities of all jth corresponding classes Π i,j satisfy
[Equation]
where Π i,i is the true class probability.
That is, the probabilities of all jth classes which share a equal distance d to the ith true class, are degrade by a constant shrink factor f d in relation to the true ith class probability.
f d values are in direct relation to the on-graph distance metric.
For instance, Table 1 describes the probability values given for the class ’tabby’ with the shrink vector [ f 2 , f 3 , 4 , f 5 , 6 ] =
[10 , 100 , 1000] , where f 3 , 4 indicates that both distance level 3 and level 4 gets the same value.
We empirically investigate the impact of different sets of f values on the ﬂat-hit and hierarchicalprecision metrics on the ILSVRC 2012 1K benchmark.
To provide a baseline for comparison, we compared the performance of our model to a hard-NLL model, where all models trained with
Renset50.
Table 2 shows results for the mentioned models on both metrics.
Figure 4 (left) visualizes these results.
Comparing 1-3 models to the baseline 0 shows that adding weight probabilities to far classes increases the semantic metric performance hp@k for growing k values.
In other words, even labels which share common predecessor have valuable semantic information in different what was deﬁned in Deng et al.
(2014) which deﬁned mutual subsumption only between labels share an immediate parent.
Secondly, if we skip distance levels the performance is degraded as can be showed in model 4, where we skip all classes with distance level d=2 giving these classes a zero probability.
Furthermore, using small weights yields a model which has only a minor decrease in ﬂat-hit while still boosting the hp@k as can be seen in model 5 relative to model 3. An evidence stems from model error analysis gives validity to this mechanism in Appendix 7.1.
4. S UPER - GROUP RETRIEVAL : CHOOSING HYPER - PARAMETERS
As we mentioned an interesting and useful cases of super-group generation is for hard scenarios where the model is probable to miss its top-1 or when making an inference for novel classes.
In this section we focus on the ﬁrst case.
According to this we take into consideration only the samples where the standard softmax-NLL makes an error in it’s top1 prediction.
In Section 5.2 we discuss of how to predict the failure cases and formalize a selection mechanism of returning a speciﬁc prediction when we have enough conﬁdence and otherwise return the super-category.
The algorithm for super-group generation incorporates two hyper-parameters k and l , i.e the number k of the top-k predictions and the minimal extent of the hCorrectSet.
Clearly, increasing both k and l improves SG-hit while hurts SG-Speciﬁcity.
In this section we investigate the impact of [Table] [Table] combinations of these parameters on super-group performance.
The experiments were performed with the same trained soft-NLL model used for results in Section 5.1.
As baseline we compared the performance of our soft model to a standard softmax model.
Both models were trained with resnet50 topology.
Our ﬁrst observation is that, k should not be ﬁxed as a uniform value for all cases.
It should be determined adaptively according to the uncertainty of the model top-k predictions.
This uncertainty can be measured by the model top-k probability coverage deﬁned,
[Equation]
where ˆ π i is the model softmax response for the ith class, and the subscript θ indicates that π θ is a threshold .
That is, if only small k say k = 5 gives high probability coverage say π θ = 0 .
95 then the top-5 predictions may be considered with high certainty relative to the case when we need k =to get the same probability coverage.
This adaptive approach is needed because if we have high conﬁdence for small k
but nonetheless we choose to increase it more, then SG - Specificity may be degraded.
On the other hand, if we decrease k in cases when π θ is small i.e. low conﬁdence we may miss the super-group.
Thus, in the experiments we choose to take k ≥ 5 while demanding that π θ > π tresh , where π thresh is a ﬁxed threshold.
Figure 5 displays both metrics SG - Specificity vs SG - hit for different values of k-probability coverage and l values for both hard and soft models as described in the Figure caption.
For both models we show that we can get super-category predictions with more than 80 % hit-rate when the standard model fails in returning the speciﬁc category.
Moreover, these curves can give a systematic way of choosing the best k and l .
Given a demand on one metric dictates the sets of parameter to get the best in the other metric.
That is for SG - hit = 0 .
7 the best lowest SG - Specificity is given for soft-NLL with l = 20 and for SG - Specificity = 3 the highest SG - hit is given for soft-NLL with l = 20 .
Furthermore, Soft-NLL outperformed the standard softmax model with better hit and speciﬁcity for each such demand.
A handful of selected examples from ILSVRC validation dataset shows a qualitatively illustration of the beneﬁts of super-group generation by using our proposed algorithm and soft-model in Appendix 7.2.
R ESULTS 5. S EMANTIC RELEVANCE
This section presents ﬂat and hierarchical results on the ILSVRC 2012 1K dataset.
We compare our soft-NLL to standard softmax-NLL using two architectures Resnet50 and Alexnet, and to DeVise approach presented in Frome et al.
(2013) trained with Alexnet.
The hyper-parameter search setup used with soft-NLL loss is speciﬁed in details in Section
Table 3 shows results for the mentioned models on both the ﬂat and hierarchical metrics.
Figure(right) visualizes these results.
The soft-NLL shows a signiﬁcant improvement in its semantic generalization ability while exhibiting only a minor decrease in the top-k accuracy compared to standard softmax-NLL on a same deep topologies.
For Resnet50 at k=5 the soft-NLL gives 0.615 while hardNLL gives 0.423 model which is about 45% relative improvement, at k=10 soft-NLL and hard-NLL gives 0.614 and 0.377 respectively which is a 63% relative improvement.
Our relative improvement
[Table]
order of given DeVise Vise Frome et al ) method gives a relative improvement of about 2% and 5% respectively over the standard model.
A handful of selected examples from ILSVRC12 validation dataset shows a qualitatively illustration of our model
5. H IERARCHICAL S ELECTIVE CLASSIFICATION
When we consider a large-scale problem ( | Y | (cid:29) 1 ), the standard approach is to consider the top-k predictions where k > 1 .
But in practice, given an image a desired need may be to get a single label and not k noisy predictions.
However, considering only the top-1 prediction includes many mistakes even in state-of-the-art models.
As we discuss in Section 4.2 Soft-NLL can be used to make higher quality reasonable super-category inferences given hard samples because of it has a better semantic generalization ability as shown before in Section 5.1.
In this section we discuss about a selection mechanism of returning only a single high probable prediction.
That is return the top-1 speciﬁc prediction when we have enough conﬁdence and otherwise returning the super-class using owe preferred model.
In other words, we formalize this mechanism and discuss of how to identify if the top-1 prediction is improbable to make hit.
A selection with guaranteed risk control problem is formalized in Geifman & El-Yaniv (2017), where given a classiﬁer f a selective-classiﬁer returns f ( x ) prediction if it guaranteed a desired error and otherwise it abstains the prediction.
In our case instead of abstaining the predication if it has low conﬁdence we propose to retrieve the super-group as follows,
[Equation]
where, f NLL ( x ) , f soft − NLL ( x ) are the standard and semantic classiﬁers and g : X → { 0 , 1 } is a selection function which calculated according to the desired risk.
As showed in Geifman & El-Yaniv (2017) the top-1 softmax model response can be used to control the guaranteed error that is,
[Equation]
otherwise where θ is a threshold determined according the desired error of top-1 predictions.
That is, suppose the model gives a total 75% top-1 accuracy while we would like a desired error of 5% (or 95% topaccuracy) this demand impose θ o to guarantee this risk.
In other words, we coverage only a subset of our data distribution such that we fulﬁll satisfactory demands.
Given a classiﬁer a risk-coverage curve presents the computed risk for many θ values.
We calculated risk-coverage curve for ILSVRC12 validation dataset.
For each θ value we split the set.
In the guaranteed-controlled set we calculated top-1 error and in the other complementary split we calculated the super group generation ability.
We inverse the x-axes of the latter split set for alignment reasons.
As shown in 6.
For example, for 65% coverage NLL gives about 92% top-1 accuracy, and in the complementary set (35% coverage)
we get SG - hit of 87% and 3.45 SG - specificity using soft-NLL model, while without this mechanism we can obtain only 75% total accuracy on the full coverage set.
Those super-categoty inferences can then be used with ﬁned-grained classiﬁer to obtain a more speciﬁc category.
Moreover, our soft-NLL model outperforms standard NLL in both
SG metrics for all θ -coverage values.
This results show that our semantic soft model returns higher quality super-group classes.
5. H IERARCHICAL Z ERO SHOT LEARNING
Soft-NLL model can be generalized as a coarse-grained descriptor to return a better super group.
In this manner, our model can make reasonable inferences about candidate it has never visually observed and given a novel class return its super-class, we call this inference Hierarchical Zero shot learning .
Our deﬁnition is easier from the common one which aims to accurately recognize the unseen classes as in Frome et al.
; Norouzi et al.
; Changpinyo et al.
We think that this super-category inference can be reasonable if we can get satisfactory results in our mode relative to the poor results obtained even with state of the art methods with the common deﬁnition (Xian et al., 2017; Changpinyo et al., 2017).
To test this hypothesis, we extracted images from the ImageNet 2011 21K dataset with labels that were not included in the ILSVRC 2012 1K dataset on which our model was trained.
The zero-shot experiments were performed with the same trained Soft-NLL model used for results in Section 5.1.
To provide a stronger baseline for comparison, we compared the performance of our model to standard softmax model.
We again evaluate super group quality with SG-hit and SGspeciﬁcity metrics.
To quantify the performance of the model, we constructed from ImageNet 2011 21K zero-shot data two test data sets as deﬁned in Frome et al.
The datasets are deﬁned with increasing difﬁculty based on the image labels’ graph distance from ILSVRC 2012 1K labels. ’
2-hop’ is the easy dataset which comprised of 1,548 labels that are within two tree hops of the training labels, making them visually and semantically similar to the training set.
A more difﬁcult ’3-hop’ dataset comprised of 5,989 labels was constructed in the same manner.
The datasets contains 1.3 million and 4.6 million images respectively.
Figure 7 compared between soft-NLL and hard-NLL SG - Specificity , SG - hit metrics for different values of k-probability coverage and l values.
Both models can give more than 70% SG-hit with reasonable SG-speciﬁcity even in the harder benchmark, while state of the art methods (Xian et al., 2017; Changpinyo et al., 2017) gets less then 5% on the harder set trying to hit the true label.
We think that it is apparently better to give a high level inference with good correctness than trying to give very speciﬁc inference with very low conﬁdence.
Moreover, based on ours super-group predictions a more speciﬁc class can be generated using a ﬁned-grained mechanism.
Secondly, our soft-NLL outperformed hard-NLL on both metrics on both datasets.
That is, it gets a better hit for each demand in the SG-speciﬁcity metric and a better speciﬁc super-class for each demand in the hit meter.
Futhermore, in the harder ’3-hop’ dataset we get some degradation in the SG-hit metric but get a comparable results on the SG-speciﬁcity metric relative to ’2-hop’ dataset.
The speciﬁcity of predictions is reasonable taking into consider the high degree of classes as shown in 3.
C ONCLUSION
In this work we have shown that our hierarchical probability based soft-NLL loss can be trained to give a comparable performance to the state-of-the-art hard-NLL softmax based model on ﬂat classiﬁcation metric, while simultaneously making more semantically reasonable errors, which is indicated by its signiﬁcant improved performance on a hierarchical label metric relative to existing methods (Frome et al., 2013).
Our model’s learning process has no additional cost in training and classiﬁcation time.
This approach is independent of the speciﬁc deep architecture and thus can beneﬁt progress.
Furthermore, we present an algorithm where given a probabilistic model, based on its top-k predictions our algorithm returns the input corresponding super-group based on classes hierarchy without any further learning.
Using this technique we can take into account only a single high probable prediction instead of taking top-k predictions for multi-class large-scale problems.
We have shown that our soft-NLL retrieve a higher-quality coarse category on two use cases than the standard model.
The ﬁrst case is when the regular model is probable to make a miss.
We show that our algorithm obtains more than 80 % hit-rate when the standard model fails in returning the speciﬁc category.
The second is for zero-shot scenario which deals with making an inference for novel classes.
In this case our soft-model success in returning a high-quality with success of more then 75 % while the state of the art methods gets less than 5 % trying to return a speciﬁc super-class.
Based on those super-group predictions a more speciﬁc class can be generated using a ﬁned-grained mechanism.
Moving forward, we are experimenting cases where the taxonomy is unknown.
Bilal et al.
(2018) proposed method to constructing the class hierarchy based on identifying block pattern in the confusion matrix, then reﬁning the hierarchy recursively.
Such taxonomy is not perfect and probably contains noises.
We will deal how these signiﬁcantly affect the models performance.
Moreover,
We are experimenting an adaptive use of our loss function according to visual similarity between classes resides in the confusion matrix.
In some cases a hierarchical closeness may not coincide with visual closeness which is the integral issue when trying to differentiate between visual objects.
Another interesting work we are going to to work on is whether our semantic inferences are less easily fooled relative to standard models (Nguyen et al., 2015) and thus can help to get higher conﬁdence networks.
A PPENDIX 7. A PPENDIX A: E XPLANATION OF SEMANTIC WEIGHTING TECHNIQUE
We found an interesting evidence which gives validity and intuition to our semantic empirical weighting technique proposed in Section 4.1. This evidence stems from analyzing the error hardmodel behavior. We found that this behavior is common to different deep architectures where in this section the details regarded to resnet50 topology trained on ILSVRC12-1K dataset.
All labels in this dataset are deﬁned as leaf nodes in the taxonomy graph as mentioned in Section 4.
Figure(a) presents the Top-1 hard model error prediction occurrences at each distance level where the sum of all columns gives the total model error.
Although we expect that this graph will decrease monotonically according to the distance 3 it has much complex behavior.
As expected there is a maximum in level-2 which is the closest related leaf nodes distance level, means that the the maximum confusion corresponds to taxonomy distance similarity.
Moreover, starting from level-7 there is a monotonic decrease behavior.
However, in level-3 there is a decrease and an immediate salient increase in level-4 which is opposed to intuition.
Nevertheless, We found that the deep model is truly semantic and acts according to our intuition.
The salient increase in level-4 is a biased term stems directly from dataset distribution.
As shown Figure(b) given a leaf node
, there is a signiﬁcant relative increase in the occurrences of leaf neighbours in level-4 relative to level-3 (about 3 times).
A dataset bias term can be expressed by a multiplication argument.
That is, the model behaviour can be formalize as,
[Equation]
where, M k is the empirical model probability of error at distance d, which is biased according to data as we just claim.
This term is shown in Figure 8 (a).
N k is the dataset probability occurrence at distance d and f d is the true model preferring factor of level d excluding the dataset bias term.
Figure 9a presents the behaviour of f. f decreases approximately exponentially in d according: f = 6 . 46 · 10 − 0 .
135 · d , which proves our claim.
This fact is an evident for the ability of CNN models to make a semantic errors in average, although they did not make any use in hierarchical information, because visual object categories are naturally hierarchical (Deng et al., 2010; Bilal et al., 2018).
Nevertheless, using directly semantic hierarchy can improve classiﬁcation systems.
f can be interpreted as a similarity metric between categories at different distance level and thus it can be used as a guideline of to deﬁne Π the similarity probability weighting matrix presented in Section 3.1.
Each class gets the maximum weight, and the similarity weight is divided according to distance level.
Because of the fast decay of f, labels which are in distance level-7 can be viewed as labels which are fully disjoint categories and therefore get zero weights.
Furthermore, We tried to use f values directly to calculated Π values and got weights that are close to what we found through experiments.
We even get some improvement in the hierarchical metric as shown in Figure 9b.
We assume that our weighting mechanism can be applied to other cases because our mechanism was deduced from the largest publicly dataset and we further remove biased inﬂuence.
-1K is a huge and diverse benchmark, therefore averaging on the dataset is probably to give strong unbiased claims
[Figure]
7. A PPENDIX B: S EMANTIC HAND -NLL VS . SOFT -NLL SAMPLES
Figure 10 qualitatively illustrate our soft model.
We selected examples from ILSVRC12 validation dataset and make a comparison between our soft-model to the hard-model.
Both models were trained with resnet50 topology, where our model is the same trained model used for results in Section 5.1.
For each example we show the top-5 label predictions and theirs corresponded softmax responses.
Next to each class we indicates taxonomy distance from true class to that class.
In all cases our model predictions are generally more semantically close to the desired label.
Moreover, The soft top-5 ranks are mostly arranged according to their relevance to the true class.
Figure 10 (a-c) shows cases where our model accurately predict the true class in the top-1 place where hard-NLL missed it.
In Figure 10a the hard-model top-5 classes are: a kind of a fox, the true class i.e. a siamese cat and three dog breeds while our model returns different types of domestic cat.
Furthermore, our model outputs are less biased to irrelevant context.
It is very reasonable that the hard-model choose arctic-fox mostly because of the white background which is similar to snowy environment.
Figure 10b shows a case where our model identify the true class in topalthough the texture similarity to other top-5 classes.
Figure 10c image contains two objects where both are resides in the set.
Our model gives to both these labels high score where the hard-model give most of probability weight to one of the classes.
Figure 10 (d-e) shows cases where the hardmodel accurately predict the true class in the top-1 place where our model missed it.
In Figure 10d although our model identify that the class is a camera it confuses between the two camera types.
This is reasonable because through learning a speciﬁc features of a class our model inﬂuence other close related classes too.
Figure 10e is an image which contains two relevant objects.
Our model predict with high conﬁdence the class ’punching bag’ which is a kind of a ball in the taxonomy, because many punching bags in training images color is red like the ball in the image.
Nevertheless, it still identify the true class in its top-5 predictions.
7. A PPENDIX C: S UPER - GROUP RETRIEVAL IN HARD CLASSIFICATION CASES SAMPLES
Figure 11 qualitatively illustrates the signiﬁcant beneﬁt of our proposed super-group retrieval algorithm presented in Section 3.2.
We selected examples from ILSVRC12 validation dataset where the hard-model miss its top-1, a,d,e are cases where it miss its top-5.
We used our soft-model as the coremodel of our proposed algorithm, because its improvement over the hard-model as shown in Section 4.2.
We choose l − hCorrectSet extent to be 10 and adaptive k such that k ≥ 5 while demanding that π θ > 0 . 85 with those hyper-parameters we can get SG - hit = 0 .
7 and SG - Specificity =as shown in Figure 5.
Both hard and soft models were trained with resnet50 topology, where our model is the same trained model used for results in Section 5.1.
For each example we show the hard-model top-5 label predictions and theirs corresponded softmax responses.
Next to each class we indicates taxonomy distance from true class to that class.
Moreover we indicates in the table the true class next to the soft-SG prediction label, SG hierarchical distance to the true class and the number of soft predictions used in order to generate the SG.
All samples are cases where it is not trivial to guess what is the object given the top-5 predictions.
For example in Figure 11a it may be a snake or a lizard, while our method succeed in identify even the speciﬁc breed of the snake.
In Figures d-e the conﬁdence is very low such that we scale the limits of the responses accordingly.
In those cases we cannot get even a clue about the inner object based on the hard-model predictions.
Our soft-model used k=50 predictions and succeed in predicting a very reasonable super-class outputs. [Figure]
[Figure]
Target
Green snake
Soft-SG
Colubridsnake
H-distnace k
[Figure]
Target
Water ouzel
Soft-SG
Bird
H-distnace k
[Figure]
Target
Chiffonier
Soft-SG
Furniture
H-distnace k
[Figure]
Target
Backpack
Soft-SG
Container
H-distnace k
[Figure]
Target
Airship
Soft-SG
Craft
H-distnace k