Paper:
Deep Structured Prediction with Nonlinear Output Transformations
Deep structured models are widely used for tasks like semantic segmentation, where explicit correlations between variables provide important prior information which generally helps to reduce the data needs of deep nets. However, current deep structured models are restricted by oftentimes very local neighborhood structure, which cannot be increased for computational complexity reasons, and by the fact that the output configuration, or a representation thereof, cannot be transformed further. Very recent approaches which address those issues include graphical model inference inside deep nets so as to permit subsequent non-linear output space transformations. However, optimization of those formulations is challenging and not well understood. Here, we develop a novel model which generalizes existing approaches, such as structured prediction energy networks, and discuss a formulation which maintains applicability of existing inference techniques.

Review:
This paper studies the problem of training deep structured models (models where the dependencies between the output variables are explicitly modelled and some components are modelled via neural networks). The key idea of this paper is to give up the standard modelling assumption of structured prediction: the score (or the energy) function is the sum of summands (potentials). Instead of using the sum the paper puts an arbitrary non-linear (a neural network) transformation on top of the potentials. The paper develops an inference (MAP prediction) technique for such models which is based on Lagrangian decomposition (often referred to as dual decomposition, see details below). The training of the model is done by combining this inference technique with the standard Structure SVM (SSVM) objective. The proposed approach is evaluated in 4 settings: OCR, multilabel classification, image tagging, sematic segmentation and is compared against several baselines.  Overall the paper is well written (I think I understood the method without too much effort) and there is a novel component. However, I don't think that the paper is significant enough to warrant a NIPS publication. My skepticism is based on the following three points:  1) The proposed method has no theoretical justification on the inference side. The Lagrangian relaxation technique usually works well when the duality gap is small (Kappes et al., 2015), but can provide arbitrarily bad results otherwise. There is no study of whether the inference problem is solved well or not.  2) The paper uses non-standard datasets and does not compare to any baselines reported in the literature. For OCR, there is a standard dataset of Taskar et al. (2003) and a lot of baselines (see, "Lacoste-Julien et al., Block-Coordinate Frank-Wolfe Optimization for Structural SVMs, ICML 2013" for linear baselines; "Perez-Cruz, Ghahramani, Pontil, Conditional Graphical Models, 2006, http://mlg.eng.cam.ac.uk/zoubin/papers/CGM.pdf" for kernel-based baselines, "Leblond et al., SEARNN: Training RNNs with Global-Local Losses, 2017, https://arxiv.org/abs/1706.04499" for some RNN-based baselines). I do not understand the reasoning for not using the standard dataset on the same tasks.  3) The novelty of the paper is somewhat limited, in part because learning with SSVM and Lagrangian dual of the inference procedure was done before. In particular, the work of Komodakis (2011) did this, but connections to it are not discussed at all (the paper is cited however). The work of Komodakis&Paragios (2009) used an inference method, which is very similar to the one proposed, and is directly comparable in some settings. For example, the OCR datasets (both the version used for experiments and the standard one of Taskar et al. (2003)) contain only 50 different words, this ideally matches the patter-based potentials used by Komodakis&Paragios (2009). I do agree that the proposed method is not identical to the one of Komodakis&Paragios (2009), but I would insist on fair discussion and comparison.   === after the response I would like to thank the authors for addressing some of my questions and hope the corresponding clarifications would be added to the next revision. However, I still think that the contribution is somewhat incremental because the setting is quite limited, but anyway I'm raising my score to 6.