From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposed to extend the existing self-supervised learning methods for blockwise training scheme based on the Barlow Twins self-supervised learning framework. Reviewers raised some concerns about the limited novelty and weak empirical studies (e.g., overall improvements do not seem very convincing, only evaluated on a single SSL method and a single DNN backbone, lacking evaluations on state-of-the-art baselines and larger models/dataests, insufficient ablation studies, etc), etc. The authors have partially addressed some of the questions during the rebuttal but did not provide sufficient results and answers to convince the reviewers. Overall, the paper contains some interesting idea that may be worth publication, but the quality of the current manuscript is not strong enough for publication. 
justification_for_why_not_higher_score: N/A
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper5227/Authors
title: Thank you for your interesting suggestions and for acknowledging our work
comment: We are thankful to the reviewer for the interesting comments which helped us improve the paper. We are also thankful to the reviewer for acknowledging our contributions!

From: ICLR.cc/2023/Conference/Paper5227/Reviewer_JnWG
title: Thank you
comment: Thank you for addressing my comments.

Showing performance of Block wise Barlow twin on ImageNet is not trivial and I hope that other reviewers would value this achievement.
I think the amount of work is very impressive and the paper provides useful reference points and experiments when working to scale up localized self-supervised learning algorithms. Hence I believe it deserves publication. I do not change my grade since it already reflected this opinion.

From: ICLR.cc/2023/Conference/Paper5227/Authors
title: Response to reviewer JnWG (2/2)
comment: 3. I find it very curious that the network with lower top1 accuracy at early layers perform well for later layers. This curiosity is also put forward in the paper as an interesting side result. This might indeed hide some important effect which explain the performance difference. A deeper analysis of this might make the paper stronger, why does it happen with the block-wise barlow twins with the specific pooling strategy? Is that an effect of the pooling loss only or the loss function too?
- Response: This is indeed also one of the most interesting results for us, and that we would like to understand better. We consider this effect to be a particular artifact of the way self-supervised learning works. Goodhart’s law states that: “when a measure becomes a target, it ceases to be a good measure”. The metric is analogous to the self-supervised objective in this case, which is a proxy for learning good representations, but not the end goal. Therefore, trying too hard to optimize this goal at the initial layers makes the initial layers extremely task specific. Since most SSL methods can be considered as variants of learning invariance between the different augmentations of the input, forcing hard to learn this invariance at the initial layers can result in premature suppression of information, which could have been leveraged by the blocks on top to improve recognition. A big part of our investigation was inherently motivated to understand these effects (specifically the ones covered in the appendix A). We started with plotting the invariance (on-diagonal) and redundancy reduction (off-diagonal) terms in the Barlow Twins loss function to gain a deeper understanding of this effect. It is important to note that we plot the terms by directly performing global spatial pooling on the feature maps i.e. without any projector. The reason for this is because the end-to-end trained model has no projector. However, when we apply the Barlow Twins loss, it is always applied on the output of the projector. The results for this experiment were presented in Fig. 8. The figure shows that blockwise training is making the model very task specific right at the start, which will not be fixed even with a different loss function. Pooling scheme is one possible way to adjust the amount of supervision provided to the actual block. Therefore, we explored different pooling schemes. Another possible route is to adjust the target for the on-diagonal and off-diagonal terms within the Barlow Twins loss to match that of end-to-end trained model. Since the off-diagonal terms are already near zero, we just adapted the target for the on-diagonal terms. We added a latest experiment on this in Appendix A.1 in the updated draft. We used invariance targets of 0.6, 0.75, 0.9, and 1.0 for block-1, block-2, block-3 and block-4 respectively instead of 1.0 for all the blocks. However, this fails to induce any improvement in performance. Please check Appendix A.1 for more details. Please let us know of any further experiment we could run to improve our understanding of this effect.

4. The noise addition is likely to have an effect similar to drop out. It is unfortunate that we do not have a comparison between the suggested noise model against regular dropout.
- Response: Thank you for the suggestion. We do agree that multiplicative noise is also interesting to be evaluated alongside the additive noise that we evaluated. We will attempt to include this in the next version of the paper since we don’t have enough time to complete this experiment within this short period of rebuttal time. However, note that the main point of our paper is not to promote a particular method for noise injection, but just to show that even with a powerful augmentation strategy, it still leads to only modest gains in performance, and the performance of the model is still inferior to end-to-end finetuning trained without additive noise.

From: ICLR.cc/2023/Conference/Paper5227/Authors
title: Response to reviewer JnWG (1/2)
comment: We are thankful to the reviewer for a very detailed, comprehensive, and insightful review. The paper was summarized beautifully and elegantly by the reviewer, including why we think it is exciting and interesting! We respond to the questions raised by the reviewer below.

1. Barlow twins was already shown to work very well on ImageNet in the original paper
- Response: Barlow Twins paper shows the effectiveness of this loss when trained using end-to-end backpropagation. Therefore, this serves as a target in our case i.e. the maximum performance that we can expect in general when using a blockwise training protocol. Therefore, it is essential for us to employ a loss function which has been already demonstrated to work well on large-scale datasets, and evaluate the performance of the model when restricting the length of the backpropagation path.

2. Lowe et al. 2019 demonstrated than block-wise CPC worked well. Lowe et al. 2019 inspected some training variants including simultaneously or sequential trainings of the blocks.
- Response: Thank you for flagging this reference, we now cite it in the text and in Fig. 1 to make our specific contribution more explicit. The reviewer correctly points out that the work from Lowe et al. is very much related to our investigation. However, it is worth noting that Lowe et al. only evaluated their approach on small datasets. Furthermore, they used the InfoMax objective for training. In contrast, we demonstrate high performance of our approach on ImageNet, as well as evaluate the use of different recent SSL objectives. We leverage all recent advances in SSL literature for our investigation. Our results are different from the ones obtained by Lowe et al, where they demonstrated performance improvement instead of degradation with greedy blockwise training. On the other hand, we show that for large-scale datasets, there is still is a gap in performance between end-to-end and blockwise trained models. Even with careful consideration of different hyperparameters, closing this gap is highly non-trivial and difficult. We have updated the related work section in the paper as well as Fig. 1 to make this connection explicit.

From: ICLR.cc/2023/Conference/Paper5227/Authors
title: Response to reviewer Y92E (2/2)
comment: 3. The local learning strategy is not superior to the BP optimization method. In addition, the model is more sensitive to each block after the model is blocked, especially the first block. More additional corrections are needed to improve the performance and robustness of the model, although still lower than BP's method.
- Response: There seems to be some confusion about our motivation judging from this question. End-to-end backpropagation leverages the gradients from the upper layers to guide the adjustment of weights in the lower layers. However, this is very unlikely to happen in the human brain i.e. it is biologically implausible. When we remove this end-to-end path, there is no explicit feedback that aligns the modules to work in harmony with each other. As an example, the model can discard information at the first block which might be relevant for the block at the second layer for classification. This selection naturally happens via end-to-end backpropagation. Therefore, removing end-to-end feedback should be naturally detrimental towards performance. However, at the same time, ability to train modules independently, and stitch them together without finetuning will enable drastic explosion in new capabilities. We consider this as a small step in that direction, where we would like to move towards local learning. Ultimately, we should be able to cover that gap. However, covering that gap is highly non-trivial since we will be losing a very significant amount of information that is leveraged by the model in the form of end-to-end backpropagation paths.

4. Experimental results show that simultaneous blockwise training is better than sequential blockwise training. But the simultaneous blockwise training strategy cannot limit memory.
- Response: We agree that we haven’t demonstrated a memory advantage of our approach at this stage. There are two types of memory advantages we can expect from refinement of our local approach in the future: (1) As far as the point regarding simultaneous blockwise training is concerned, the memory advantage comes from the fact that we can split up the backpropagation path into smaller pieces, which reduces the burden on memory. However, it is worth noting that we have new projectors which offset that gain in memory. Therefore, the gain is not directly observed at this point. As these local training methods get better, the computational gains will also be apparent. (2) This kind of local computation is particularly well-suited for neuromorphic hardware. In such hardware, the information with which a neuron can update its weights is solely local, making the computations much faster and more energy efficient, but incompatible with end-to-end back-prop. Finally, we are investigating local learning rules also to model and understand how biological neural networks can learn, as we know that they do not rely on backprop but on local learning rules (Illing et al. 2021).

5. The blockwise training strategy relies on a special network structure like the block structure of the ResNet-50 model.
- Response: We resort to blockwise training as a natural transition towards purely layerwise learning. The block is just defined to be a set of layers. Therefore, any architecture can be segmented into blocks based on just combining layers together. We just choose a very particular way to segment these blocks following prior work (Lowe et al. 2019), which is not essential to the blockwise/layerwise training framework.

6. There are some writing errors in the paper, such as "informative informative" on page 5 and "performance" on page 1, which lacks a title.
- Response: Thanks for highlighting these errors. We fixed them in the recent version of the paper.

From: ICLR.cc/2023/Conference/Paper5227/Authors
title: Response to reviewer Y92E (1/2)
comment: We are thankful to the reviewer for spending his/her time reviewing our manuscript and for a detailed and comprehensive review. The reviewer highlighted several limitations of our work, which we attempt to clarify below:

1. Motivated by the fact that local learning can limit memory when training the network and the adaptive nature of each individual block, the paper extends local learning to the ResNet-50 to handle large datasets. However, it seems that the results of the paper do not demonstrate the benefits of doing so. 
- Response: We agree with the reviewer that we are not able to demonstrate any gains in memory efficiency at this stage. The primary reason for this limitation is that the local computations we propose would be mostly beneficial in combination to neuromorphic hardware, which by allowing only local computations would be significantly faster and more energy efficient than current hardware. However, we do not have access to such hardware, and the fact that we still need to use backpropagation in the different blocks of layers of the network prohibits the use of such purely local hardware at this stage. That said, we believe that our study provides a valuable step in the direction of discovering purely local learning rules, which would be a paradigm shift for the field of machine learning if they came to bear. Moreover, being able to train blocks locally and stitching them together without finetuning has far-reaching implications beyond computational gains: indeed, human brains are thought to rely on purely local learning rules, and they do not require strong supervision. Having fully self-supervised networks, where each layer can learn on its own without requiring consolidated feedback in the form of gradients from all the layers, is thus particularly exciting for understanding how our brains might learn. In summary, we hope the reviewer understands that our approach, while not yielding immediate computational benefits, could have far-reaching implications for both our understanding of the brain as well as for the design of purely local learning rules which could in the future be implemented with great computational benefits on bio-inspired neuromorphic hardware.

2. The method proposed in the paper essentially differs very little from the traditional BP method. The main contribution of the paper is adding the stop gradient operation between blocks, which appears to be less innovative.
- Response: The paradigm we propose is drastically different than training a network end-to-end, as we train each module locally without global backward interactions. Using stop-gradient is by no means the single contribution of our paper. The main contribution of our paper is to demonstrate that with careful implementation, locally trained modules can reach a performance very similar to their end-to-end counterparts on large-scale datasets. However, closing this gap is hard as well as essential. It is hard because end-to-end learning automatically tunes blocks below to be aligned with what is expected by the blocks above, while local training can’t. On the other hand, closing this gap is essential because purely local learning is layerwise. We demonstrated that there is a considerable drop in performance even when splitting the network into just 4 blocks. Therefore, when going towards pure layerwise learning, it is expected to incur a significant drop in performance as compared to our current results.

From: ICLR.cc/2023/Conference/Paper5227/Authors
title: Response to reviewer Voxk
comment: We are thankful to the reviewer for a detailed and insightful review. We respond to each of the questions/objections raised by the reviewer below:

1. The objective formulation for blockwise training is somewhat unclear to me. How do the authors define the loss in sequential or simultaneous blockwise training? Similarly, what is the formulation for other frameworks, such as SimCLR?
- Response: Thanks for highlighting this confusion. The loss functions we use are the same as in BarlowTwins / SimCLR / VicReg; our main innovation is where we apply them in the architecture of the network. In end-to-end training, the loss is applied after the last layer i.e. the output of the complete model. In our model, we apply the loss at the output of every block of layers. Therefore, since we divide the network in 4 blocks, we compute 4 different loss values. Each of these loss values are used to backpropagate within the block of layers and there are no interactions between the different blocks. The distinction between sequential and simultaneous is unrelated to the loss function, but instead, is related to the training procedure. In sequential training, we train all the blocks sequentially i.e. we train the first block, freeze it, attach the next block on top of it, train it and continue until the last block. On the other hand, for simultaneous blockwise training, we compute all the 4 loss values at the same time and update all the blocks together, but the gradients are only backpropagated for each block locally. Hope that clarifies the confusion. We reformulate some of these details in the manuscript for clarity.

2. It lacks experiment details. For example, what training parameters, such as epoch and batch size, are associated with the results in section 4? How do authors conduct supervised tasks, e.g., loss and model change?
- Response: Thanks for highlighting this insufficiency of details. We use the same training parameters as original Barlow Twins w/ 300 epochs of training. All these results are computed for the model based on 300 epochs of training on ResNet-50 using LARS optimizer. Batch size is set to be 2048. We use cosine learning rate decay with an initial learning rate of 0.2. The projector is a three-layered MLP with 8192 hidden as well as output units. For supervised training, we use the same training protocol as Barlow Twins, but instead of using an output of size 8192 from the projector, and applying Barlow Twins loss on top, we use the same 8192 hidden units for the projector, but use 1000 output units corresponding to the scores of the 1000 classes of ImageNet. We then apply a simple cross-entropy loss to the model for classification. The difference in terms of architecture between a regular ResNet-50 and this model is that a classification head is lightweight in ResNet-50, with a direct mapping to 1000 classes, while we use the SSL projector for classification. We also use Barlow Twins augmentations for training, which are more diverse as compared to regular transformations used to train supervised models. We have also updated the paper to include all these details.

3. It would be interesting to extend the current results to different model architectures. e.g., different CNN/transformer-based models, and include more results on different SSL frameworks, such as SimCLR, and MoCo.
- Response: This is indeed true. We wanted to apply this method to different SSL objectives as well as architectures. However, each run of this self-supervised training is expensive. Therefore, although we do explore different architectural choices and three SSL objectives (Barlow Twins, SimCLR and VicReg), doing a more exhaustive comparison is prohibitive in terms of compute. We do mention in the paper that Masked Auto-Encoders based on Vision Transformers are particularly suited towards local learning, but are beyond the scope of this work.

4. The authors mentioned one of the main advantages of this learning strategy is its ability to handle larger neural networks, it would be interesting to include more comparisons by varying the model size to support this claim.
- Response: Agreed, and we do highlight a prior application of such an approach on videos (Wu et al 2021). However, doing this study on very large networks is computationally prohibitive for us. Moreover, the local computations we propose would be particularly well-suited for neuromorphic hardware which would allow only local computations and be significantly faster and more energy efficient than current hardware. However, we do not have access to such hardware, and the fact that we still need to use backpropagation in the different blocks of layers of the network prohibits the use of such purely local hardware at this stage. That said, we believe that our study provides a valuable step in the direction of discovering purely local learning rules, which would be a paradigm shift for the field of machine learning if they came to bear.

From: ICLR.cc/2023/Conference/Paper5227/Reviewer_Voxk
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposed a blockwise learning strategy based on the Barlow Twins framework as an alternative to replacing standard backpropagation that is commonly used in existing SSL algorithms. Compared to related works, their work addresses the challenge of large-scale datasets. In addition, they compared several spatial and feature pooling strategies for building intermediate features and evaluated a number of strategies to customize the training procedure for blockwise training. Through their numerical studies, they demonstrate that blockwise training can achieve competitive performance compared to the original framework.
strength_and_weaknesses: Strength 
- This paper proposed an alternative approach to training SSL models that can be used for training large neural networks with a limited memory budget.
- They evaluated a variety of strategies to customize the training procedure at the different blocks and the proposed strategies are also transferable to other SSL frameworks.
- They conducted various ablation studies to investigate individual component of their framework, such as comparing different feature pooling strategies for the intermediate block output.

Weaknesses
- The objective formulation for blockwise training is somewhat unclear to me. How do the authors define the loss in sequential or simultaneous blockwise training?  Similarly, what is the formulation for other frameworks, such as SimCLR?
- It lacks experiment details. For example, what training parameters, such as epoch and batch size, are associated with the results in section 4? How do authors conduct supervised tasks, e.g., loss and model change?
- It would be interesting to extend the current results to different model architectures. e.g., different CNN/transformer-based models, and include more results on different SSL frameworks, such as SimCLR, and MoCo.
- The authors mentioned one of the main advantages of this learning strategy is its ability to handle larger neural networks, it would be interesting to include more comparisons by varying the model size to support this claim. 


clarity,_quality,_novelty_and_reproducibility: This paper discusses an alternative learning strategy for SSL, which provides some insights for future research in SSL. However, the quality of the paper should be improved for publication consideration and more systemic comparison should be conducted to generalize the current conclude.  The proposed approaches seem to be reproducible based on existing publicly available codebases.

summary_of_the_review: Please see my comments in Strength And Weaknesses.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5227/Reviewer_JnWG
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper describes multiple tricks to push the performance and of the self-supervised Barlow Twins step in a block-wise setting. Most prominently, it is reported than a block-wise training of a ResNet architecture with Barlow twins leads to a high top-1 accuracy of a down-stream classifier (70+% which is one percent away from end-to-end training). To reach this performance a new method to extract a low-dimensional vector from the ResNet feature map and a noise injection strategy are described.
strength_and_weaknesses: The biggest result of this paper is certainly to demonstrate how competitive can be the Barlow Twins in this block-wise setting. Although this had been demonstrated for the CPC loss by Lowe et al, this result had not been reproduced with other losses on a dataset as hard as ImageNet as far as I know. This alone is likely to be useful intermediate step torwards back-prop-free self-supervised learning method which perform well on ImageNet.

Although it certainly took a lot of work to make it work, I do not see big conceptual innovation but this is not always necessary to make a good paper. Let's recap why I think some results were already published by other authors:
- Barlow twins was already shown to work very well on ImageNet in the original paper,
- Lowe et al. 2019 demonstrated than block-wise CPC worked well,
- Lowe et al. 2019 inspected some training variants including simultaneously or sequential trainings of the blocks,
- The noise addition is likely to have an effect similar to drop out. It is unfortunate that we do not have a comparison between the suggested noise model against regular dropout.

The combination of feature expansion and pooling is probably the most creative innovation here. It is indeed expected that some kind of reshaping and post-processing is necessary to transform the feature map into a low-dimensional latent variable that is relevant for the loss function. Inspecting which variant is best is certainly useful for future work.

I find it very curious that the network with lower top1 accuracy at early layers perform well for later layers. This curiosity is also put forward in the paper as an interesting side result. This might indeed hide some important effect which explain the performance difference. A deeper analysis of this might make the paper stronger, why does it happen with the block-wise barlow twins with the specific pooling strategy? Is that an effect of the pooling loss only or the loss function too?
clarity,_quality,_novelty_and_reproducibility: The paper is clear, as argued before, the conceptual novelty is limited but this is counter balanced by the good performance. I hope that the code will be published to lead the field forward towards back-prop-free self-supervised learning working on ImageNet.
summary_of_the_review: Most of the results are very clearly written. It is very valuable to see some tricks which make Barlow twins competitive in this setting. This paper would be probably more impactful or important if the experiments were targeted to support one strong result with a central story line rather than a list of somewhat unrelated tricks.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5227/Reviewer_Y92E
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The manuscript proposes a strategy to train the network in a blockwise manner. Specifically, this strategy explores two local learning methods for blockwise BP training based on the basic blocks in the ResNet-50 model, i.e. simultaneous blockwise training and sequential blockwise training. In addition, the paper also explores some design details, such as pooling methods, parameter settings, and augmentation methods. At last, the manuscript presents experiments and ablations of the proposed strategies.
strength_and_weaknesses: Strength
The paper explores two strategies for blockwise training based on the Barlow Twins method. The authors perform more detailed experiments and ablation of the proposed methods. Overall, the paper describes the proposed method in detail and validates the method with experiments, and the paper is clearly written and well organized.
Weaknesses
Motivated by the fact that local learning can limit memory when training the network and the adaptive nature of each individual block, the paper extends local learning to the ResNet-50 to handle large datasets. However, it seems that the results of the paper do not demonstrate the benefits of doing so. The detailed weaknesses are as follows:
1)The method proposed in the paper essentially differs very little from the traditional BP method. The main contribution of the paper is adding the stop gradient operation between blocks, which appears to be less innovative.
2)The local learning strategy is not superior to the BP optimization method. In addition, the model is more sensitive to each block after the model is blocked, especially the first block. More additional corrections are needed to improve the performance and robustness of the model, although still lower than BP's method.
3)Experimental results show that simultaneous blockwise training is better than sequential blockwise training. But the simultaneous blockwise training strategy cannot limit memory. 
4)The blockwise training strategy relies on a special network structure like the block structure of the ResNet-50 model.
5)There are some writing errors in the paper, such as "informative informative" on page 5 and "performance" on page 1, which lacks a title.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: The structure of the paper is clear. The paper provides a detailed description of the proposed method. The proposed method is compared with state-of-the-art self-supervised methods and a detailed ablation analysis is provided. However, there are some writing errors in the text that need to be further checked by the author.
Novelty: The blockwise training strategy proposed in the paper lacks innovation. Specifically, ResNet networks with block structure are trained in a blockwise way, i.e., using the BP optimization method within blocks and the stop-gradient operation between blocks.
Reproducibility: Some key settings descriptions are missing in the paper. It is not clear whether the augmentation method for the dataset in the blockwise training strategy and the parameter setting, , in the loss objective are the same as in Barlow Twins.
summary_of_the_review: The blockwise training strategy that uses a stop-gradient operation between blocks is not novel. This approach reduces the performance of the Barlow Twins method and the robustness of the model. It also does not achieve the purpose of limiting the memory and improving the adaptive capability of the model.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough