Paper:
The knowledge that humans hold about a problem often extends far beyond a set of training data and output labels. While the success of deep learning mostly relies on supervised training, important properties cannot be inferred efficiently from end-to-end annotations alone, for example causal relations or domain-specific invariances. We present a general technique to supplement supervised training with prior knowledge expressed as relations between training instances. We illustrate the method on the task of visual question answering (VQA) to exploit various auxiliary annotations, including relations of equivalence and of logical entailment between questions. Existing methods to use these annotations, including auxiliary losses and data augmentation, cannot guarantee the strict inclusion of these relations into the model since they require a careful balancing against the end-to-end objective. Our method uses these relations to shape the embedding space of the model, and treats them as strict constraints on its learned representations. In the context of VQA, this approach brings significant improvements in accuracy and robustness, in particular over the common practice of incorporating the constraints as a soft regularizer. We also show that incorporating this type of prior knowledge with our method brings consistent improvements, independently from the amount of supervised data used. It demonstrates the value of an additional training signal that is otherwise difficult to extract from end-to-end annotations alone.

Review:
The paper argues for encoding external knowledge in the (linguistic) embedding layer of a multimodal neural network, as a set of hard constraints. The domain that the method is applied to is VQA, with various relations on the questions translated into hard constraints on the embedding space. A technique which involves distillation is used to satisfy those constraints during learning.
The question of how to encode external knowledge in neural networks is a crucial one, and the limitations of end-to-end learning with supervised data is well-made. Overall I feel that this is a potentially interesting paper, addressing an important question in a novel way, but I found the current version a highly-frustrating read (and I read the paper carefully a number of times); in fact, so frustrating that it is hard for me to recommend acceptance in its current form. More detailed comments below.
Major comments
--
The main problem I have with the paper lies with the first part of section 3, which is a key section describing the main method by which the constraints are satisfied during learning. This is very confusing. The need for the two-step procedure, in particular, and the importance of distillation needs much more explanation, and not relegated to the Appendix (which reviewers are not required to read - see call for papers). I'm not suggesting that the whole of the appendix needs moving to the body of the paper, but I would suggest perhaps 1/2 a page.
A related comment is the use of the distillation technique. This looks crucial, but I don't believe distillation is mentioned at all until the end of the related work section, and even there it comes as a bit of a surprise since there's no mention anywhere of this technique in the introduction.
I would say a little more about the distinction between the embedding space and parameter space, since you say that the external knowledge is encoded in the former and not the latter, and this is important to the overall method. Since embeddings are typically learned (or at least fine-tuned) it's not clear where the boundary is here. Another comment is that embedding space in this paper means the linguistic embedding space. Since this is ICLR and not, eg, ACL, I would make clear what you mean by embedding space.
I don't understand the diagram in Fig. 3 of the architecture, nor the explanation. What's an operation here? Is it *, or *6? I don't get why 3 is embedded by itself in the diagram, and then combined with the remainder using the MLP. Why not just run the RNN over the sequence?
Why are the training instances {3,+1...} and {4,*2,...} equivalent. I stared at this a while, and still have no idea. Also, how are these "known to be equivalent" - what's the procedure?
Minor comments including typos etc.
--
The paper has the potential to be really nicely written and well-presented. Currently it reads like it was thrown together just before the deadline (which only adds to the overall frustration as a reader).
In fig. 1 the second equivalent question example is interesting, since strictly speaking "box" and "rectangular container" are not synonyms (e.g. boxes can be round). Since strict synonymy is hard to find, does that matter? (I realise the dataset already exists and was presented elsewhere, but this might be worth a footnote).
missing (additional) right bracket after Herbert (2016)
Not sure footnote 1 needs to be a footnote. It's already been said, I think, but if it does need repeating it probably deserves to be in the body of the text.
between pairs questions
see Fig.3 -> figure 2?
see Fig.1 -> Tab. 1? (on p.5)
footnote 1 missing a right bracket
usually involve -> involves
+9]) - extraneous bracket
Fig. 4.1 -> Fig. 3? (p.6)
p.7 wastes a lot of space. In order to bring some of the appendix into the main body, I would do away with the very large bulleted list. (I don't mean lose the content - just present it more efficiently)
Remember than
Finally in Fig. 4.2 - some other figure
due of the long chains