Paper:
Object recognition in real-world image scenes is still an open problem. With the growing number of classes, the similarity structures between them become complex and the distinction between classes blurs, which makes the classification problem particularly challenging. Standard N-way discrete classifiers treat all classes as disconnected and unrelated, and therefore unable to learn from their semantic relationships. In this work, we present a hierarchical inter-class relationship model and train it using a newly proposed probability-based loss function. Our hierarchical model provides significantly better semantic generalization ability compared to a regular N-way classifier. We further proposed an algorithm where given a probabilistic classification model it can return the input corresponding super-group based on classes hierarchy without any further learning. We deploy it in two scenarios in which super-group retrieval can be useful. The first one, selective classification, deals with the problem of low-confidence classification, wherein a model is unable to make a successful exact classification. The second, zero-shot learning problem deals with making reasonable inferences on novel classes. Extensive experiments with the two scenarios show that our proposed hierarchical model yields more accurate and meaningful super-class predictions compared to a regular N-way classifier because of its significantly better semantic generalization ability.

Review:
SUMMARY
The paper presents a method for classification which takes into account the semantic hierarchy of output labels, rather than treating them as independent categories. In a typical classification setup, the loss penalizes the KL-divergence between the model’s predicted label distribution and a one-hot distribution placing all probability mass on the single ground-truth label for each example. The proposed method instead constructs a target distribution which places probability mass not only on leaf category nodes but also on their neighbors in a known semantic hierarchy of labels, then penalizes the KL-divergence between a model’s predicted distribution and this target distribution. This model is used for classification on ImageNet-1k, and for zero-shot classification on ImageNet-21k where a model must predict superclasses seen during training for images of leaf categories not seen during training.
Pros:
- Method is fairly straightforward
- Modeling relationships between labels is an important problem
Cons:
- Missing references to key prior work in this space
- Minimal comparison to prior work
- Confusing experimental setup
- Paper is difficult to read
MISSING REFERENCES
This paper is far from the first to consider the use of a semantic hierarchy to improve classification systems; see for example:
Deng et al, “Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition”, CVPR 2012
Deng et al, “Large-scale object classification using label relation graphs”, ECCV 2014 (Best Paper)
Jiang et al, “Exploiting feature and class relationships in video categorization with regularized deep neural networks”, TPAMI 2017
None of these are cited in the submission. [Deng et al, 2014] is particularly relevant, as it considers not just “is-a” relationships as in this submission, but also mutual exclusion relationships between categories. Without citation, discussion, and comparison with some of these key pieces of prior work, the current submission is incomplete.
COMPARISON TO PRIOR WORK
The only direct comparison to prior work in the paper is the comparison to DeViSE on ILSVRC12 classification performance in Table 3. However since DeViSE was intended to be used for zero-shot learning and not traditional supervised classification, this comparison seems unfair.
Instead the authors should compare their method against DeViSE and ConSE for zero-shot learning. Indeed, in Section 4.3 the authors construct a test set “in a [sic] same manner defined in Frome et al” but do not actually compare against this prior work.
I suspect that the authors chose not to perform this comparison since unlike DeViSE and ConSE their method cannot predict category labels not seen during training; instead it is constrained to predicting a known supercategory when presented with an image of a novel leaf category. As such, the proposed method is not really “zero-shot” in the sense of DeViSE and ConSE.
EXPERIMENTAL SETUP
From Section 3.1, “we adopt a subset of ImageNet the ILSVRC12 dataset which gather [sic] 1K classes [...]”. The 1000 category labels in ILSVRC12 are mutually exclusive leaf nodes; when placed in the context of the WordNet hierarchy there are 820 internal nodes between these leaves and the WordNet root. As a result, for the method to make sense I assume that all models must be trained to output classification scores for all 1820 categories rather than the 1K leaf categories. This should be made more explicit in the paper, as it means that none of the performance metrics reported in the paper are comparable to other results on ILSVRC12 which only measure performance on the 1K leaf categories.
The experiments on zero-shot learning are also confusing. Rather than following the existing experimental protocol for evaluating zero-shot learning from [Frome et al, 2013] and [Norouzi et al, 2013] the authors evaluate zero-shot learning by plotting SG-hit vs SG-specificity; while these are reasonable metrics, they make it difficult to compare with prior work.
POOR WRITING
The paper is difficult to follow, with confusing notation and many spelling and grammatical errors.
OVERALL
On the whole, the paper addresses an important problem and presents a reasonable method. However due to the omission of key references and incomplete comparison to prior work, the paper is not suitable for publication in its current form.