Paper: RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNets (ICLR-2024)

Reviewer comment (Evaluation):
Observe that the ImageNet experimental results have about 1% performance improvement on many models, but also a lot more Params.

Author response:
Although RefConv introduces extra params, these params are conducted on the basis weights instead of the feature maps, thus the extra Flops and memory cost introduced in training are negligible as stated in Sec.3.2 and quantitively shown in Table 1.
Moreover, these extra params can be equivalently transformed after training, thus no extra params are introduced in inference.

Reviewer comment (Experiments):
The network architectures that come into play are generally early CNN models such as ResNet, DenseNet, MobileNet family, etc. For modern convolutional architectures such as SlaK, RepLKNet, HorNet, etc., the effect is currently unknown.

Author response:
We also test the effectiveness of RefConv on two modern architectures, ConvNeXt [1] and FasterNet [2], the improvements are 0.96% and 1.15%, as shown in Table 1.\
[1] A convnet for the 2020s. (CVPR 2022)\
[2] Run, don’t walk: Chasing higher flops for faster neural networks. (CVPR 2023)

Dear Reviewer 3dDh,

Thank you for your response. We understand your concerns but unfortunately, the author response phase is nearing its end, and we regretfully acknowledge our inability to promptly finish the experiments with larger models, which require several days for completion. Nevertheless, we are committed to addressing your concerns and will continue to conduct these requested experiments with RepLKNet, SLaK, and bigger ConvNeXt/FasterNet models. **We promise that the results will be added to the revised paper.**

We kindly request your understanding of our inability to finish the experiments before the author response phase deadline, because we provided a detailed response to your concerns (which were mostly focused on the mechanisms of the proposed method) **on Nov 14th**. Without your timely feedback to our last response, **we did not expect you would request experiments with larger models or TIMM-trained models**. 

While we may not be able to complete such large-scale experiments before the end of the author response phase, we are lucky that Reviewer Z4v5 had also requested additional experiments in the initial review, which we have conducted throughout the rebuttal phase (so that we do not have enough GPU resources for RepLKNet, SLaK, and bigger ConvNeXt/FasterNet models)  and included in the response to Reviewer Z4v5. 

Specifically, we have conducted experiments of ResNet-34 and MobileNetv3 **with the official training scripts of TIMM**. For the well-trained ResNet-34, RefConv improves the accuracy from 75.12% to 75.86%. Similarly, for MobileNetv3, RefConv enhances the accuracy from 74.28% to 74.91%.

These experiments support the claims made in our paper and **align with your required experiments that use ResNet trained with TIMM as the baseline**. We believe these results should be duly considered for their relevance.

Thank you for your attention and consideration. 

Please let us know if there is anything we can do to convince you to further raise the score.

Best Regards,

Authors

Reviewer comment (Theory):
For the base weight W_b, one of the points claimed by the refocusing technique is the possibility of establishing links between its individual channels. Why is this necessary? Each channel of the base weight kernel has its own role, so to link them?

Author response:
DW conv was originally proposed to reduce the model size and FLOPs but at the cost of representational capacity. Consider a two-channel input feature map whose shape is (2, H, W), assume the conv layer is a two-channel DW conv whose kernel shape is (2, 1, K, K). The first channel of output only relates to the first channel of the input, and the second channel of output only relates to the second channel of input. But if the kernel is a regular (dense) conv, the kernel shape will be (2, 2, K, K), so that each of the output channel relates to all the input channels. From this perspective, we can see that a regular conv's representational is higher than a DW conv, i.e., it comes at a cost to let "each channel of the base weight kernel have its own role". 

As evidence, in practice, if we only consider the performance, regular conv is always better than DW conv (namely, the accuracy will become higher when simply replacing DW conv with regular conv). Therefore, it's expected that implicitly connecting the channels of the DW conv can narrow the representational gap between the DW conv and the regular conv.

We explain how RefConv works. Taking the aforementioned two-channel DW conv for example, we denote its input, output, and kernel by X, Y, and W, respectively, so that using pytorch-style pseudo code, this layer can be represented by\
Y = conv2d(X, W)

RefConv can link the channels by re-parameterizing the kernel and such a re-parameterization is realized with a convolution on W. So that\
Y = conv2d(X, conv2d(W_b, W_r))\
Note that conv2d(W_b, W_r) can be seen as a re-parameterization of the original W. Since conv2d(W_b, W_r) is a dense conv, each channel of the output directly relates to each channel of W_b, so that each output channel of Y **indirectly** relates to each channel of W_b. Note that we omit the "identity mapping" (Eq.1 in the paper) here for simplicity.

Except for the improvements in performance, in this paper, we have also provided evidence that linking the channels brings benefits. In a DW conv, since there are no connections between the channels of DW conv, it is easy to result in redundancy. The evidence is shown in Section 4.5. In contrast, RefConv reduces the redundancy, as shown in Figure 4. This is because by linking the channels with W_r, the re-parameterized kernel W_t is encouraged to learn new representations through the learnable combination of the channels of W_b, which were originally independent.

Reviewer comment (Experiments):
During refocus training, the result after convolution of the base weight W_b with its previous features can be seen as a new "feature". The transform weight W_t can be seen as trainable to process this new "feature". This process is equivalent to fine-tuning the convolution after "injecting" new parameters. I would like to ask if any experiments with other models (e.g. new convolutional networks) have found that this degrades performance?

Author response:
So far we have not encountered the performance degradation.\
As discussed above, in the procedure of refocus training, the basis weights W_b dose not conduct convolution on the feature maps, it actually act as the input"features" of the refocusing transformation with W_r. Then the output generated by the refocusing transformation is the transformed weights W_t, which replaces the W_b and conducts the convolution on the feature maps. To conclude, in refocus training, W_b does not serve as a kernel to convolve on anything, W_r conducts refocusing transformation on W_b and generates W_t, and W_t is the kernel of convolution on the input feature maps and generates the output feature maps.\
We reckon that RefConv is more than just simply fine-tuning the original conv layer. It uses learnable transformations to establish the connections between convolution kernels, learning new representations through combining the learned representations by the basis kernels, thus strengthening the model representational capacity.\
Moreover, due to the use of "identity mapping" (Eq.1 in the paper), the representational capacity of W_t should be no lower than the original basis kernel.