Paper:
FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction
The basic principles in designing convolutional neural network (CNN) structures for predicting objects on different levels, e.g., image-level, region-level, and pixellevel, are diverging. Generally, network structures designed specifically for image classification are directly used as default backbone structure for other tasks including detection and segmentation, but there is seldom backbone structure designed under the consideration of unifying the advantages of networks designed for pixellevel or region-level predicting tasks, which may require very deep features with high resolution. Towards this goal, we design a fish-like network, called FishNet. In FishNet, the information of all resolutions is preserved and refined for the final task. Besides, we observe that existing works still cannot directly propagate the gradient information from deep layers to shallow layers. Our design can better handle this problem. Extensive experiments have been conducted to demonstrate the remarkable performance of the FishNet. In particular, on ImageNet-1k, the accuracy of FishNet is able to surpass the performance of DenseNet and ResNet with fewer parameters. FishNet was applied as one of the modules in the winning entry of the COCO Detection 2018 challenge. The code is available at https://github.com/kevin-ssy/FishNet.

Review:
The paper proposes a new backbone architecture more suited to various tasks, e.g. image-, region- and pixel-level tasks, than traditional pre-trained backbones. It allows direct backpropagation from deep layers to shallow layers by solving the issue caused by isolated convolutions happening in common architectures, e.g. ResNet or DenseNet. Experimental validation confirms that FishNet backbone is better than comparable ResNet and DenseNet backbones on image classification, and on both object detection and instance segmentation.          The paper is rather clear and easy to follow. The motivations are exposed well and the justification of the idea to solve the issue of gradient propagation in common backbones is appealing. However, the mathematical notations are cumbersome, and not consistent across equations, e.g. the superscripts 'u' disappear between equations (4) and (5). They are also different notations between the equations and Figure 2, see k and K, or different fonts for M function. It could also help to write the name of the variables directly in Figure 2, making the link between the two easier to follow.          Regarding the experimental section, the results presented are convincing on the usefulness of FishNet, especially for object detection and segmentation, where the use of recent frameworks (FPN, Mask R-CNN) is welcome. It would be interesting to have comparisons with deeper versions of ResNe(X)t backbone on MS COCO dataset, as the 101-layer versions are the most common used. It would also be good to evaluate FishNet on the test sets for easier comparisons with the literature, and to have the results from other state-of-the-art methods in the same tables, for both classification on ImageNet and object detection/segmentation on MS COCO. The ablation study seems a bit light, and does not show the relative importance of the main parts introduced, i.e. the fish body and head.        ************* Post-rebuttal recommendation The rebuttal is a bit disappointing since the authors only answer to one of my concern. This said, I like the proposed method and the experiments are overall convincing. I thus stick on my weak accept recommendation.