Paper:
Texture Synthesis Using Convolutional Neural Networks
Here we introduce a new model of natural textures based on the feature spaces of convolutional neural networks optimised for object recognition. Samples from the model are of high perceptual quality demonstrating the generative power of neural networks trained in a purely discriminative fashion. Within the model, textures are represented by the correlations between feature maps in several layers of the network. We show that across layers the texture representations increasingly capture the statistical properties of natural images while making object information more and more explicit. The model provides a new tool to generate stimuli for neuroscience and might offer insights into the deep representations learned by convolutional neural networks.

Review:
In this submission, the authors develop a new texture synthesis algorithm, that begins with the ideas and general methodology of the classical Portilla-Simoncelli texture synthesis algorithm, and augments it with the rich feature set provided by convolutional neural nets. This procedure overcomes a major limitation of Portilla-Simoncelli -- its dependence on a hand-crafted feature set. The convnets, while trained on some other supervised objective (eg classification on imagenet), provide a possible way forward. The authors demonstrate compelling texture synthesis, and some simple and intuitive observations of how generated textures depend on choices of architecture and layer.For clarity, originality, and significance, this paper gets a thumbs up. For quality, I've given this paper a 6, but I would certainly bump it up higher if the authors convincingly address and fix a few crucial points. In no particular order:(1) At the moment, the paper is framed in terms of its potential for studying the ventral stream. I can see why this is a tempting proposition, given how Portilla-Simoncelli has fed into to the Freeman et al V2 studies. However, the authors aren't really doing anything in this direction in the body of the paper. The closest they come to perception is in Section 6, wherein they tack a classifier onto the summary statistics collected at a particular layer of VGG. Yet the improvement here with depth seems more to me a statement about the representation that VGG is learning in order to perform well on imagenet, and less about textures and the brain. This could end up a useful tool for studying the internal representations in deep nets, but the connection to the ventral stream is too vague. I think it would be better off if this link was less central, and made into a discussion point, with the texture synthesis in its own right being the focus of the paper. If the authors wish to propose that the summary statistics collected at different layers of a convnet will be useful for studying hierarchical representations in the brain, then they must acknowledge that these two architectures engage different tradeoffs between scale and complexity. By scale here I mean the size of the receptive field; by complexity, I mean the degree of nonlinearity. In a convnet like VGG, these two factors are correlated: as one goes deeper, units are integrating over a larger spatial region, and have cumulatively computed a more complex nonlinearity. In the ventral stream, this is not the case: as early as V1, the representation is multiscale, including simple cells with lower spatial frequency tuning (i.e. larger RFs). Portilla-Simoncelli is useful as a tool for probing the V1->V2 transformation precisely because it parallels the multiscale feature of the physiology; a convnet is designed with different architectural constraints that may make it less analogous to physiology in this regard. (2) The authors make a point that the features in Portilla-Simoncelli are hand-crafted, but those proposed in this paper are not. This is mostly true, *except* the choice of the correlation summary statistic *is* hand-crafted. There is no explicit reason for them to choose this, other than it works. For the record, the term "correlation" is misleading here (perhaps in signal processing terms it's a correlation, but it's not normalised, so I would have described it as a covariance). (3) It's really important to tell us how many (cumulative) summary statistics are being enforced. If I get the numbers right (from inspecting VGG), by pool1 there are 3 * 64*63/2 = 6048 parameters collected, this goes up to 30k by pool2, 194k by pool3, 0.8M by pool4, and 1.5M by pool5. These numbers should go in Figure 2. That's a *lot* of parameters (especially when the number of pixels is ~50k?) and it makes it hard to interpret these as "summary" statistics. The authors should comment on what this means. (4) I'm at a loss as to why conv5 and/or pool5 are not shown in Figure 2. They're clearly part of VGG, and it's weird that they are omitted without mention. (5) I'm confused as to why the yellow boxes in the bottom left hand corner of each P-S source texture are showing up in the bottom left hand corner of each synthesised image. If the statistics are stationary, and the seed is white noise, then there's no reason for this feature to be generated precisely here. This looks like a boundary condition artefact. If boundaries are handled in some special way, or ignored, we need to know. (6) It's pretty topical, so the authors could relate their approach to the recent google inception/deepdream syntheses to make it a more attractive paper. Clearly this work was developed in parallel, and the overall objectives are different. But there is an overlap. For example, my understanding of the deepdream algorithm is that it places pressure on a summary statistic of the current image, namely the variance of one feature map. This is precisely one of the statistics being collected in the algorithm presented here.  (7) As one goes deeper in the net, the texture model is holding on to all the summary statistics computed in previous layers, so the representation is cumulative. This seems like an important point, and yet it is not clearly justified.  ** Following author rebuttal: with less emphasis on the biological link, and the authors' proposed amendments, this is an interesting paper. I don't agree with all the interpretations, but it's discussion-provoking and relevant to many communities.