Paper:
Deep convolutional neural networks trained for image object categorization have shown remarkable similarities with representations found across the primate ventral visual stream. Yet, artificial and biological networks still exhibit important differences. Here we investigate one such property: increasing invariance to identity-preserving image transformations found along the ventral stream. Despite theoretical evidence that invariance should emerge naturally from the optimization process, we present empirical evidence that the activations of convolutional neural networks trained for object categorization are not robust to identity-preserving image transformations commonly used in data augmentation. As a solution, we propose data augmentation invariance, an unsupervised learning objective which improves the robustness of the learned representations by promoting the similarity between the activations of augmented image samples. Our results show that this approach is a simple, yet effective and efficient (10 % increase in training time) way of increasing the invariance of the models while obtaining similar categorization performance.

Review:
The paper proposes to explicitly improve the robustness of image-classification models to invariant transformations, via a secondary multi-task objective. The idea is that the secondary objective makes intermediate-layer representations invariant to transformations of the image that should lead to the same classification. The paper also establishes that the typical models do not actually learn such representations by themselves.
This is an interesting idea (I am not specialist enough on image classification to know whether this idea has been tried before.) and highly relevant to this conference. The paper is g. well-written, easy to read, and correct.
I do, however, rate it as Weak Accept only for one reason: I would expect that making the model more robust should improve classification accuracy. But according to the paper, accuracy does not improve (and even degrades slightly). The paper does not experimentally demonstrate that the proposed methods objectively improves the model.
In a sense, it is to be expected, since the model is no longer optimized 100% for the classification task.
I can think of three changes to the paper that would flip my review to a Strong Accept:
* Try using the multi-task objective as a pre-training, followed by fine-tuning without multi-task objective. This should foster robust internal representations while allowing to fully optimize for the classification task. Alternatively, you could anneal alpha to 0. Try whether this alleviates the losses on the tests that got worse, and leads to higher gains on the others.
* Maybe the robust models, while being worse on benchmarks, are better on real-life data, e.g. where training and test mismatches are higher. Can you find a test set that demonstrates a larger, reliable accuracy improvement from robustness?
* "However, note that the hyperparameters used in all cases were optimized to maximize performance in the original models, trained without data augmentation invariance. Therefore, it is reasonable to expect an improvement in the classification performance if e.g. the batch size or the learning rate schedule are better tuned for this new learning objective." -- Then that should be done.
Besides this, I have a few detailed feedback points:
Eq. (2): Using sigma for "invariance", which is the opposite of the usual meaning of sigma... I wish you had used a different symbol. Not a dealbreaker, but if you have a chance, it would be great to change to a different one.
"we normalize it by the average similarity with respect to the *other* images in the (test) set" -- If you use only *other* images, I think it is theoretically possible that sigma becomes negative. I think you should include the numerator image in the denominator as well. I understand that in practical terms, this is never going to be a problem, so it should be OK, no need to rerun anything.
"we first propose to perform in-batch data augmentation" -- This increases correlation of samples within the batch, and may therefore affect convergence. To be sure that this is not the cause of the degradation, it would be good to rerun the baseline with the same in-batch augmentation (but without the additional loss). Was that already done?
Figure 5: I was a little confused at first because I read this as the invariance score (like Figures 3 and 4), not the invariance loss. They seem to be opposite of each other. So I wondered why the right panel would show poorer invariance score as training progresses. Do you actually need the concept of "invariance score" at all? Or can you redefine the invariance score as a non-invariance score (1- of it), so that its polarity is the same as the invariance loss? If not, an easy fix would be to add "(lower=better)" to the caption of Fig. 5, and likewise ("higher=better") to Fig 3 and 4.
"DATA AUGMENTATION INVARIANCE" -- You really want more. You want to be robust to all valid transformations of the object in the images. Obviously it is not possible to augment data for that, but it would be good to state this somewhere as the idealized goal of this work, which you approximate by data augmentation.