Paper:
Doubly Convolutional Neural Networks
Building large models with parameter sharing accounts for most of the success of deep convolutional neural networks (CNNs). In this paper, we propose doubly convolutional neural networks (DCNNs), which significantly improve the performance of CNNs by further exploring this idea. In stead of allocating a set of convolutional filters that are independently learned, a DCNN maintains groups of filters where filters within each group are translated versions of each other. Practically, a DCNN can be easily implemented by a two-step convolution procedure, which is supported by most modern deep learning libraries. We perform extensive experiments on three image classification benchmarks: CIFAR-10, CIFAR-100 and ImageNet, and show that DCNNs consistently outperform other competing architectures. We have also verified that replacing a convolutional layer with a doubly convolutional layer at any depth of a CNN can improve its performance. Moreover, various design choices of DCNNs are demonstrated, which shows that DCNN can serve the dual purpose of building more accurate models and/or reducing the memory footprint without sacrificing the accuracy.

Review:
The paper proposes to increase parameter sharing in convolutional layers by making all filters part of larger "metafilters". Filters are obtained by convolutionally extracting overlapping parts from the metafilters, hence the name "doubly convolutional neural networks". Models with this form of parameter sharing are compared with regular CNNs and some other variants on the CIFAR-10, CIFAR-100 and ImageNet datasets.
Added after rebuttal / discussion:  Although I still contend that the CIFAR-10 baseline network used in the paper is unnecessarily weak, I appreciate that results on a large-scale dataset such as ImageNet (on which the method is shown to be quite competitive) are much more relevant, and I no longer consider it a major issue.  I still disagree with the presentation of the idea (asserting that the filters are "translated versions of each other", when the reason the method works is precisely because they are _not_ exact translated versions of each other, only approximately so), but I guess that can be put down to a matter of taste.  That leaves the problem with the CyclicCNN baseline, which I maintain is unnecessarily crippled by removing its ability to use relative orientation information. My issue was not with the fact that this is not explained in enough detail (as the rebuttal seems to imply), but rather that the model is wrong. This form of parameter sharing is pointless, except in very rare cases where relative orientation information is not relevant to the task at hand (I can't think of any situations where this is true, but there might be some). In natural image classification problems like the ones tackled here, I think it is clear that relative orientations between low-level features encode crucial information that should not be discarded.  I really feel that this should be addressed, either by fixing the model architecture of the CyclicCNN so it actually benefits from the parameter sharing, rather than being crippled by it (i.e. not pooling after every layer), or by simply removing this model from the paper altogether. As it stands it doesn't really add anything because there are other, more interesting baselines to compare to, and it unnecessarily weakens the paper.  The paper probably should not be held back by an issue with an additional baseline. In light of all this I have decided to increase the technical quality score to 3, in the hope that the CyclicCNN issue will be addressed.   ---------   I'm not sure if the averaged maximum 1-translated correlation shown across different layers in Figure 2 is all that meaningful. A stroke detecting filter such as the one in the top-left corner of Figure 1, and its horizontal flip, will presumably also have a relatively high correlation compared to random filters. It would be useful to compare with something like this as well, because it would be more meaningful than the comparison with random filters (arguably a filter and its horizontal flip are not translated versions of each other, so if they also have high correlation it means the measure is not particularly meaningful).  The paper states that "it is interesting to see for most filters, there exist several filters that are roughly translated versions of each other". I agree that this is interesting, surprising even, considering that convolution involves exactly filter translation. I would have expected a bit more in-depth discussion of why convnets tend to learn approximately translated filters despite this. My intuition after reading the paper is that the value of the proposed approach seems to be in masking larger filters in various ways. Clearly these are _not_ translated versions of each other (only approximately so), else the resulting feature maps would be redundant (the would also be shifted versions of each other). So the fact that they are only _approximately_ translated versions of each other seems to be key, and this is not really stressed in the paper.  Although the experimental section is quite extensive, which is commendable, there are a few issues with the evaluation:  - The baseline models for CIFAR-10 and CIFAR-100 have the same number of units in all convolutional layers. This is non-standard and it constitutes an inefficient use of the parameter budget. It means that the total size of the representation (number of feature maps * height * width) shrinks considerably throughout the network. Usually, later layers are given more feature maps to compensate for smaller feature map sizes. This is reflected in the fairly poor performance of the baseline CNN (10.02% without / 9.59% with data augmentation), which reduces the impact of the results.  - The CyclicCNN model differs significantly from the models described in "Exploiting Cyclic Symmetry in CNNs" by Dieleman et al., so its name is a bit misleading. The CyclicCNN has rotation invariant layers (because after each layer, the activations are pooled across different orientations), whereas in the original paper, the entire network is invariant but the individual layers are instead equivariant (local vs. global invariance). This is an important difference because it means that the CyclicCNN is unable to learn high-level features that depend on relative orientations of low-level features. This cripples the network compared to the baseline, so it is not surprising that it performs worse. The CyclicCNN is actually more similar to the models described in "Flip-rotate-pooling Convolution and Split Dropout on CNNs for Image Classification" by Wu et al. than to any of the models described in the cyclic symmetry paper (although their framework could arguably be used to construct this type of model).  - At the end of Section 4.2.1 it is stated that, to ensure a fair comparison, the number of layers and the number of units per layer is kept constant across models. However, this does not ensure a fair comparison at all: instead, the number of parameters should be kept (approximately) constant. In Table 2, the DCNN is shown to outperform all other models, but it has almost twice as many parameters so this comparison is not fair. Instead, the number of feature maps in each layer of the DCNN should be reduced so the number of parameters becomes comparable.    Minor comments:  - A nonstandard symbol (circle with middle dot) is used for the convolution operation. This symbol is usually used for elementwise multiplication instead, so it's a bit confusing.  - Regarding MaxoutDCNN, it would be interesting to also evaluate other pooling function (mean pooling in particular, possibly after the application of a nonlinearity).  - The related work section should also discuss "Learning Invariant Features through Topographic Filter Maps" by Kavukcuoglu et al., which details another approach for grouping similar filters together (through a group sparsity penalty rather than explicit weight tying).  - Regarding [11] it is said that "this explores the scale invariant property of images", but I don't see how this is the case unless the weights across different layers are tied.  - [15] is said to "investigate the combination of more than one transformation of filters", but in fact STN transform the input, not the filters.  - In section 4.2.1, for MaxoutCNN it is said that the stride is k. Is this a stride or a window size? Or are both equal?