Paper:
Recent image style transferring methods achieved arbitrary stylization with input content and style images. To transfer the style of an arbitrary image to a content image, these methods used a feed-forward network with a lowest-scaled feature transformer or a cascade of the networks with a feature transformer of a corresponding scale. However, their approaches did not consider either multiscaled style in their single-scale feature transformer or dependency between the transformed feature statistics across the cascade networks. This shortcoming resulted in generating partially and inexactly transferred style in the generated images. To overcome this limitation of partial style transfer, we propose a total style transferring method which transfers multi-scaled feature statistics through a single feed-forward process. First, our method transforms multi-scaled feature maps of a content image into those of a target style image by considering both interchannel correlations in each single scaled feature map and inter-scale correlations between multi-scaled feature maps. Second, each transformed feature map is inserted into the decoder layer of the corresponding scale using skip-connection. Finally, the skip-connected multi-scaled feature maps are decoded into a stylized image through our trained decoder network.

Review:
SUMMARY
The paper is concerned with the problem of arbitrary feed-forward style transfer, where a feed-forward model receives a content image and a style image as input, and must produce as output an image matching the content of the former and the style of the latter. The approach roughly follows that of [Li et al, NIPS 2017]: An encoder network (VGG pretrained on ImageNet) is used to extract features from both the style and content image; the features from the content image are adjusted to match the statistics from the style image, and the adjusted features are passed to a decoder network which generates the output image.
Compared to prior work, the main innovations are:
- Considering correlations between features from different encoder layers rather than only correlations within a single layer in both the feature adjustment step as well as in the loss
- An improved encoder / decoder architecture which uses skip connections in the decoder, allowing for a single encoder / decoder pair rather than a cascade of encoder / decoder pairs for each layer.
PROS
- Considering correlations between features from different encoder layers is a good idea
- The improved encoder / decoder architecture is significantly more efficient than the cascaded approach of [Li et al, NIPS 2017]
CONS
- Somewhat incremental
- Limited experimental evaluation
- Qualitative results not clearly better than existing methods
- Missing citation for multi-scale losses
LIMITED EXPERIMENTAL EVALUATION
One of the key claims of the paper is that “our method with inter-scale
(fig.7(f)) or intra-scale feature transform (fig.7(g)) are more similar to the target style than those of single-scale style transfer without considering inter-channel correlation” (Figure 7 caption); this claim is substantiated primarily by qualitative results in Figures 4, 7, and 8. Personally I don’t find the results with inter-feature correlations to be much better than those with only intra-feature correlations or the results from prior work. All recent style transfer methods depend on a host of hyperparameters like style and content weight, learning schedule, etc; in my experience differences in these hyperparameter settings can have large effects on the qualitative appearance of the generated images, and by varying these hyperparameters it is common to see qualitative differences similar to those shown in Figure 4 and 5. From the small number of qualitative results presented I do not think that the benefits of inter-scale correlations have been clearly demonstrated. 
I appreciate that the authors tried to quantify their results by comparing loss values (Table 1) but unfortunately it’s hard to know how much the values of different losses correlate with human judgement of style quality.
In addition to selected qualitative results I would have liked to see a user study demonstrating human preference for images generated using inter-scale correlation losses, ideally across a range of different hyperparameter settings for each method.
GATYS BASELINE
From Table 1, the proposed method with intra-scale features achieves lower content loss than the direct optimization baseline (a) from [Gatys et al.] This is very surprising to me - typically direct optimization leads to much lower losses than any feedforward methods. From Section 4.4.1 this baseline uses Adam; in my experience using L-BFGS tends to achieve lower losses which may explain the results from Table 1.
MISSING CITATION FOR MULTI-SCALE STYLE TRANSFER
Instead of computing correlations between features from different encoder layers, [Wang et al, CVPR 2017] define a loss that considers the generated and style image at multiple spatial scales. This should be discussed in relation to the proposed method.
ENCODER / DECODER DETAILS
There are some missing details about exactly how the encoder and decoder are initialized and trained. I assume that the encoder was pretrained on ImageNet; is it updated during training or kept fixed? Is the decoder initialized randomly or does it mirror the pretrained ImageNet weights?
TYPOS / FORMATTING
There are minor typos throughout, e.g. “verity”, “sinlge” in Section 4.3, Paragrah 1. I also found the citation style to be somewhat jarring, especially in the introduction; parenthetical citations and better spacing may improve readability.
OVERALL
On the whole I feel that the paper is somewhat incremental. The inter-scale loss seems intuitively like a good idea, but I don’t think the paper presents sufficient experimental evidence to justify it. On the other hand the proposed decoder architecture seems like a clear improvement over the cascaded approach from [Li et al, NIPS 2017], as it is significantly more efficient without sacrificing quality. However I don’t feel that this alone is enough novelty for ICLR, so I lean slightly toward rejection.