Paper:
Deep networks have shown great performance in classification tasks. However, the parameters learned by the classifier networks usually discard stylistic information of the input, in favour of information strictly relevant to classification. We introduce a network that has the capacity to do both classification and reconstruction by adding a "style memory" to the output layer of the network. We also show how to train such a neural network as a deep multi-layer autoencoder, jointly minimizing both classification and reconstruction losses. The generative capacity of our network demonstrates that the combination of style-memory neurons with the classifier neurons yield good reconstructions of the inputs when the classification is correct. We further investigate the nature of the style memory, and how it relates to composing digits and letters.

Review:
This paper proposes to train a classifier neural network not just to classifier, but also to reconstruct a representation of its input, in order to factorize the class information from the appearance (or "style" as used in this paper). This is done by first using unsupervised pretraining and then fine-tuning using a weighted combination of the regular multinomial NLL loss and a reconstruction loss at the last hidden layer. Experiments on MNIST are provided to analyse what this approach learns.
Unfortunately, I fail to see a significantly valuable contribution from this work. First, the paper could do a better job at motivating the problem being addressed. Why is it important to separate class from style? Should it allow better classification performance? If so, it's never measured in this work. If that's not the motivation, then what is it?
Second, all experiments were conducted on the MNIST dataset. In 2017, most would expect experiments on at least one other, more complex dataset, to trust any claims on a method.
Finally, the results are not particularly impressive. I don't find the reconstructions demonstrated particularly compelling (they are generally pretty different from the original input). Also, that the "style" representation contain less (and I'd say slightly less, in Figure 7 b and d, we still see a lot of same class nearest neighbors) is not exactly a surprising result. And the results of figure 9, showing poor reconstructions when changing the class representation essentially demonstrates that the method isn't able to factorize class and style successfully. The interpolation results of Figure 11 are also underwhelming, though possibly mostly because the reconstructions are in general not great. But most importantly, none of these results are measured in a quantitative way: they are all qualitative, and thus subjective.
For all these reasons, I'm afraid I must recommend this paper be rejected.