Paper:
This paper presents a novel deep learning architecture for classifying structured objects in ultrafine-grained datasets, where classes may not be clearly distinguishable by their appearance but rather by their context. We model sequences of images as linear-chain CRFs, and jointly learn the parameters from both localvisual features and neighboring class information. The visual features are learned by convolutional layers, whereas class-structure information is reparametrized by factorizing the CRF pairwise potential matrix. This forms a context-based semantic similarity space, learned alongside the visual similarities, and dramatically increases the learning capacity of contextual information. This new parametrization, however, forms a highly nonlinear objective function which is challenging to optimize. To overcome this, we develop a novel surrogate likelihood which allows for a local likelihood approximation of the original CRF with integrated batch-normalization. This model overcomes the difficulties of existing CRF methods to learn the contextual relationships thoroughly when there is a large number of classes and the data is sparse. The performance of the proposed method is illustrated on a huge dataset that contains images of retail-store product displays, and shows significantly improved results compared to linear CRF parametrization, unnormalized likelihood optimization, and RNN modeling.

Review:
Summary:
This paper introduces a new dataset consisting of images of various objects placed on store shelves that are labeled with object boundaries and what are described as “ultrafine-grained” class labels. The accompanying task is to predict the labels of each object given the individual images as well as their spatial layout relative to each other. To solve this task, a deep structured model is used consisting of CNN features for each image which are fed into a linear-chain CRF. To better deal with the large number of classes, pairwise potentials are represented as the multiplication of two lower-rank matrices which represent a sort of “class embedding” for each potential label. Training efficiency is improved by considering an objective based on a form of piecewise pseudolikelihood, which allows for training-time inference to be conducted with linear complexity relative to the number of labels. This objective also allows for easy use of batch normalization for the input features to the CRF model. This model/training procedure are compared against a number of models/training procedures to demonstrate its utility.
Comments:
Arguably, the primary contribution of this paper is the introduction of a new “ultrafine-grained” classification dataset which additionally allows for context to be utilized during prediction. This an interesting task, and it’s clear where being able to make such classifications is useful. The task is somewhat limited in scope, however. It’s unclear to me how models developed for this specific task would contain insights or be useful for other tasks - the utility of any models developed for this task seem limited to this exact task. If you have any other examples where inputs might be structured in this way, this would be good to add to the paper.
The model introduced is interesting, but its novelty is limited. It’s mostly a synthesis of ideas from previous work - CNN-based features, using a CRF to model correlations among labels, and approximating the full likelihood with pseudolikelihood. The interesting additions to these ideas are the fact that an “embedding” is learned for each class and that using the pseudolikelihood during training allows for batch norm to be applied in an easy way. Neither of these is a ground-breaking insight, but they are interesting nonetheless. I am somewhat surprised that the use of batch norm during training but not during testing did not hurt performance - a discussion of why this is the case would be good to have. For the most part, I think the experimentation is sufficiently rigorous - comparisons are made against a variety of baselines, and the new model trained with the specified training procedure outperforms the other alternatives. The one additional comparison I would have liked to see would have been against a model that pairwise potentials from the input features using a neural network-based model (for example, the one used in [1] - this seems like a rather glaring omission.
Other Comments:
-Since you ran a cross-validation, you should add confidence intervals to your reported numbers
-One additional dataset detail I was hoping to see that you didn’t provide is the mean/standard deviation of the number of instances per class,
-Your appendix contains a number of interesting ablation studies - you really should report the numbers for these as well
-The title of your paper is somewhat misleading - it’s hard to argue that the form of class embedding you use is a “deep” class embedding since it’s just a matrix of parameters that are learned during training.
Overall, I’m not convinced the model/training procedure by themselves would be fully worthy of publication, but the fact that a new dataset is introduced with a challenging variant of standard classification tasks adds merit to this work.
[1] Ma, Xuezhe, and Eduard Hovy. "End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF."
REVISION:
The other reviewers raised some concerns that I had overlooked (especially regarding novelty of using matrix factorization to generate your potentials). Given these, I do not think that this paper is in a state where it is ready to be accepted. Proper citations and analysis of your approach will be needed first.