  We propose Nester, a method for injecting neural networks into constrained
structured predictors. The job of the neural network(s) is to compute an
initial, raw prediction that is compatible with the input data but does not
necessarily satisfy the constraints. The structured predictor then builds a
structure using a constraint solver that assembles and corrects the raw
predictions in accordance with hard and soft constraints. In doing so, Nester
takes advantage of the features of its two components: the neural network
learns complex representations from low-level data while the constraint
programming component reasons about the high-level properties of the prediction
task. The entire architecture can be trained in an end-to-end fashion. An
empirical evaluation on handwritten equation recognition shows that Nester
achieves better performance than both the neural network and the constrained
structured predictor on their own, especially when training examples are
scarce, while scaling to more complex problems than other neuro-programming
approaches. Nester proves especially useful to reduce errors at the semantic
level of the problem, which is particularly challenging for neural network
architectures.Sub
