Paper: Branch-level Network Re-parameterization with Neural Substitution (ICLR-2024)

Reviewer comment (Writing):
The paper is not well written. The high level goal of this work should be laid out clearly but instead the paper assumes that the reader is intricately familiar with most of the existing works on re-parameterization.

Author response:
Following the reviewer’s comments, we have included more explanations to improve the readability in understanding the proposed re-parameterization method. We thank you for the constructive comment.

Reviewer comment (Experiments):
While I appreciate the interest in branch-level reparameterization, eventually the paper must be able to make a convincing case that the effort is worth it. The gains are marginal at best and don't appear to match up with other works (e.g., RepVGG). The choice of baselines is not explained.

Author response:
***RepVGG***: ACNet and DBB are the improved version of RepVGG model. Therefore, we show that ours performs better than ACNet and DBB, and thus we can guess that ours performs well compared to RepVGG. RepVGG mentioned that using the activation function in the re-parameterization method can improve performance, but they were not able to use the non-linear activation function in their method. However, we propose a re-parameterization method based on branch-level connectivity with the non-linear activation function, resulting in further improved performance compared to block-level connectivity based re-parameterization methods.

Reviewer comment (Evaluation):
The technical contribution (neural substitution, guided activation) are somewhat minor. By themselves, this would not be a weakness if the experiments were comprehensive enough to show their value. Latency, memory footprint and other relevant issues that are central discussion points in other papers on this topic appear to be omitted.

Author response:
***Resource usage***: Since the re-parameterization methods merge the branched convolutional layers into a single one after training is completed, they use the same network architecture as the original backbone for inference. Therefore, all re-parameterization methods use the same latency and memory footprint as the original backbone.

Reviewer comment (Novelty):
I find the general goal of this work relevant but find the degree of novelty in the main contributions and the overall rigor/comprehensiveness of the experiments (e.g., compared to RepVGG) underwhelming. Please explain if possible why these concerns are misplaced or incorrect.

Author response:
***RepVGG***: ACNet and DBB are the improved version of RepVGG model. Therefore, we show that ours performs better than ACNet and DBB, and thus we can guess that ours performs well compared to RepVGG. RepVGG mentioned that using the activation function in the re-parameterization method can improve performance, but they were not able to use the non-linear activation function in their method. However, we propose a re-parameterization method based on branch-level connectivity with the non-linear activation function, resulting in further improved performance compared to block-level connectivity based re-parameterization methods.

Reviewer comment (Writing):
In the introduction, the discussion of contributions is excessively verbose, obscuring the method's branch-level connectivity advantages.

Author response:
We shorten the sentences in the introduction to make it easier to read. Additionally, we revised our paper to give a comprehensive explanation for the proposed method in `Sections 1 and 2`.

Reviewer comment (Writing):
While the Neural Substitution section is eloquently articulated, explaining a sequence shuffle-based technique for diminishing the dimensions of branch-level connectivity, the explanation within the Guided Activation Function segment lacks specificity.

Author response:
In addition, we thought that our explanation of guided activation was too brief, so we added more equations to clarify the necessity of guided activation. We updated equation in `Section 4.2`  to make it easier for readers to understand.

Reviewer comment (Reproducibility):
Regarding Neural Substitution, can you clarify whether you perform random shuffling of $\mu$ and $\nu$ during batch training, as mentioned in lines 12 to 14 of Algorithm 1? If so, could you provide insights on how to apply $\mu$ and $\nu$ during inference?

Author response:
***Shuffle operation***: In `Figure. 3`, a kernel that uses b1 as input would never encounter $a_4$ to $a_9$ during training if shuffle wasn't used, and $b_1$ would always equal $a_1+a_2+a_3$. Therefore, less diversified topologies will be used in the training stage due to this fixed connection. To make diverse connections, we use shuffle operation so that $b_1$ can use $a_1$ through $a_9$ during the training stage. Following training, we integrate all convolution layers into a single layer as illustrated in `Figure. 2`. Therefore, we do not use the shuffle operation for inference. Moreover, the line 12~14 of `Algorithm. 1` produces the index mapping matrix to shuffle input features.

Reviewer comment (Evaluation):
Merely reporting accuracy is not sufficient to demonstrate the superiority of your method. Considering that accuracy and speed can often be a trade-off in network design, it is advisable to include additional metrics, such as speed, parameters, and FLOPs (in reference to RepVGG [R1]).

Author response:
***Resource specification***: All networks use identical resources, such as FLOPs, parameters, and throughput, after re-parameterization. In the similar vein of the shuffle operation, we merge all branched convolution layers into a single layer after re-parameterization, so all the re-parameterization networks exploit the same resources.

Reviewer comment (Evaluation):
Even when focusing on accuracy alone, the current evidence may not be adequate to establish the superiority of your method. In many settings, the improvement appears to be marginal, often less than 1%.

Author response:
***Performance***: In the rebuttal phase, we constructed our own architecture to show further improved performance in `Table.1`. Therefore, it is shown that the performance is much better than the submitted version of the paper and performs favorably against existing methods.

We have also applied our method to Vision Transformer and found that the performance is also improved in `Table. 6`.

Reviewer comment (Presentation):
The paper would benefit from enhancing its clarity and readability. It currently lacks a "Related Work" section, which is essential to contextualize the research and provide a comprehensive background.

Author response:
We have polished our paper to improve readability and to make our claims clearer. We appreciate the reviewer for the precious comments. Specifically, we have added more detailed explanations in `Section. 2`. Related work and `Figure. 4` to make it easier for the reader. We have also updated the formulas for the clear understanding.

Reviewer comment (Novelty):
The main idea is rather simple; more importantly, advancements over existing methods seem marginal, with many improvements being within a 0.3% accuracy range, which could be considered insignificant.

Author response:
***Performance***: For both Cifar and ImageNet dataset, we show that our proposed method consistently 1) **outperforms existing network re-parameterization methods**, and 2) **provides significant improvement over the original baseline network**. Further, in the rebuttal phase, we introduce new architecture for our re-parameterization method. Thus, we show that ours further improve the performance of Cifar dataset. While we didn't confirm the performance improvements on ImageNet dataset due to the limited time of the rebuttal phase. However, we are confident that we can improve performance on the ImageNet dataset and will continue to experiment after the rebuttal phase.

Reviewer comment (Experiments):
The applicability of the proposed technique is demonstrated solely on CNN architectures. Its effectiveness on other architectural designs, such as fully connected layers or Transformers, is not addressed, raising concerns about its generalizability.

Author response:
***Vision Transformer***: We apply the proposed branch-level re-parameterization method for the Vision Transformer on the vision language model. We confirm that ours improves the performance of Vision Transformer in `Table. 6`.

Reviewer comment (Experiments):
It would be valuable to see the generalization on different CV tasks in experiments beyond classification.

Author response:
***Vision Transformer***: We have further demonstrated that the proposed method improves the performance of the Vision Transformer in `Table. 6`, using the vision language model.

Reviewer comment (Experiments):
I notices some differences between the performance of baseline methods in the paper and the ones reported in the DBB paper. For example, in DBB paper, the results on ImageNet with MobileNet, ResNet-18 and ResNet50 are (72.88, 70.99, 76.71) for DBB-Net and (72.14, 70.53, 76.46) for ACNet. In Table 2, the corresponding results are (70.78, 70.13, 77.50) for DBB-Net and (71.12, 70.02, 77.30) for ACNet. Some of the values are higher but the others are lower. I think the gap may derive from the configuration difference. It would be highly recommended to report the results according to the configuration of baseline method (e.g., DBB) as well, enabling a more fair comparison.

Author response:
***Performance of existing networks***: The reason why the performance of existing re-parameterization networks such as DBB is different from the original paper is that we performed our experiments in the same experimental environment by ourselves.

Each existing network has a slightly different experimental environment, such as hyperparameters, and we made them all the same so that we could perform all the experiments and report the results for a fair comparison. In our experiments, the performance of the existing networks of our experiments closely resembles that of the original study, as does the overall trend.