Paper: Augmenting Negative Representation for Continual Self-Supervised Learning (ICLR-2024)

Reviewer comment (Evaluation):
There is a significant discrepancy in performance between the proposed method and standard joint training. It raises the question of whether the proposed method offers any resource savings compared to conventional training approaches. Additionally, it's pertinent to question why a user would choose to sample exclusively from current task data instead of the entire dataset.

Author response:
**Weakness 1: The scenario of continual self-supervised learning and advantages of CSSL in resource saving**

Thank you for your insightful question. To provide a detailed response to the reviewer's inquiry, we will reiterate the setting of continual self-supervised learning (CSSL) outlined in Section 3.1 and Figure 1(b). As depicted in Figure 1(b), the CSSL scenario considers a situation where the training dataset is incrementally added. Specifically, it allows sequential access to each task dataset ($D\_t$) during the training of the $t$-th task. **However, it explicitly acknowledges the impossibility of accessing previous task datasets ($D\_{1, ..., t-1}$).** In this context, the challenge arises from the unavailability of data sampling from the entire dataset ($D\_{1, ..., t}$), leading to the Low Diversity of Negative Samples problem, a key motivation in CSSL as discussed in Section 3.2 and illustrated in Figure 1(b).

Training with the CSSL scenario has two benefits: **1) reducing the demand for memory resource and 2) minimizing GPU cost for training.** In the same CSSL scenario mentioned earlier, consistently training on the entirety of data (Joint) can be considered a conventional training approach. Consequently, when the size of each task dataset is constant (e.g., $\|D\_t\| = N$), retraining the Joint model at each $t$ task incurs linear increases in both memory and GPU resources (e.g., $N\*t$). Considering the billing structure of major cloud services (e.g., AWS) based on resource usage over a unit period, the total cost for memory and GPU resources until obtaining the $t$-th Joint model is approximately $N\*(t+1)\*t/2$. In contrast, for CSSL, the total memory cost is $N\*t$ to obtain the $t$-th model since it uses only the new $t$-th task dataset for each task $t$. Regarding GPU resources, with an additional cost $\alpha$ introduced by the CSSL algorithm, the approximate cost becomes $N + (N + \alpha)\*(t-1)$.

In summary, when considering the cost of memory and GPU resources in each learning scenario, **CSSL algorithms offer a significant cost reduction compared to the Joint**. Furthermore, the cost efficiency of CSSL algorithms becomes more pronounced as the size of each task's training dataset increases (i.e., as N increases). Taking this cost efficiency into account, we believe that **achieving performance levels of the proposed AugNeg compared to the Joint, from approximately 93% (AugNeg(MoCo) on CIFAR-100, Class-IL 5T) to 96% (AugNeg(BYOL) on DomainNet, Domain-IL 6T), is a substantial and meaningful outcome.**

Reviewer comment (Experiments):
The primary goal of continual self-supervised learning (CSSL) is to conserve resources, for instance, by reducing the need for large mini-batch sizes. However, it's crucial to determine whether the proposed method maintains its efficacy in datasets with an extensive array of classes, such as ImageNet-1K.

Author response:
**Weakness 2: Additional experiments in ImageNet-1k**

We agree that presenting additional experimental results on ImageNet-1k would contribute significantly to validating the efficiency of our algorithm. We are currently conducting them as supplementary experiments. However, given our current limited GPU resources, obtaining the complete set of experimental results may take considerable time. If we manage to complete any partial results before the end of the author response period, we commit to sharing those results.

Reviewer comment (Writing):
The main weakness for me of this paper is seeing it for the second time without small changes. I've spent some of my time to help the authors to improve it the last time and they do not even find enough time to do the good text replacement from Sy-con to AugNeg.

Author response:
Firstly, we regret that the extent of our paper's updates might not meet the reviewer's satisfaction. We appreciate all the feedback received so far and have made diligent efforts to enhance the paper based on the provided comments. The updates in this submission include:

1. **Title and Algorithm Names:** revised the paper title and the names of the proposed algorithm.
2. **Introduction Enhancement:** Improved the final paragraph in the Introduction.
3. **Method Section:**
   - Attenuated the discussion on an issue with conventional loss function and introduced a discussion on the problem of the low diversity of negatives.
   - Newly introduced the idea of extending AugNeg to non-contrastive learning in Section 3.4. Analytical details on existing gradients and other specifics have been relocated to the Supplementary Material.

4. **Experimental Section Changes:**
   - Re-performed AugNeg experiments on CIFAR-100, with results reported in Figure 3 and Table 1.
   - Tried to address the reproduce issue of CaSSLe in Table 3, essentially re-conducting the main experiments. As a result, achieved slightly improved results consistently compared to the previous version.
   - Added more experimental scenarios for downstream task.
   - Included experiments in the Supplementary Materials covering the size of the Proj(MLP) layer, results based on the queue size in AugNeg(MoCo), and experiments with an increased mini-batch size.

Also, we will further enhance our paper by incorporating the comments received in this review through additional revisions.

Reviewer comment (Reproducibility):
Main changes: They've removed one unfavorable plot (CaSSLe was always better on it for two out of three methods) and add Table 2, changed Fig.4, added "SyCON (BYOL)" **(!) (authors own writting)**, and added why they cannot reproduce CaSSLe results (Challenges of reproducing CaSSLe’s results - I've checked the issue page as well, 2% changed at the end in #12).

Author response:
We made diligent efforts to reproduce the CaSSLe experiments in the manuscript, but unfortunately, we were unable to achieve results identical to those reported in the CaSSLe paper. With the Github issue #12 concerning BYOL in the ImageNet-100 experiment, there is  issue #10 discussing reproducibility concerns in the ImageNet-100 experiment. Note both indicate a common trend of obtaining results 5-6% lower than those reported in the original paper, aligning with the tendencies observed in our experimental results. Also, these issues remain unresolved, and the authors have not provided a comprehensive solution.

We hope the reviewer has taken into account the current challenging situation regarding reproducibility. Additionally, we conducted all experiments of our paper in the unified environment and have made the experimental code publicly available. We would appreciate it if these considerations could be taken into account in the final evaluation.

Reviewer comment (Evaluation):
Improvements presented in Table 1 (CSSL baselines) – taking into account the results variability is not always significant (see std. dev. reported there for AugNeg vs CaSSLe).

Author response:
While the numeric improvement of AugNeg over CaSSLe may appear modest, we believe it gains a competitive edge when considering performance of Joint. For example, AugNeg(MoCo) shows a relatively modest increase of 3.6% over MoCo + CaSSLe. However, considering that MoCo + CaSSLe already achieves performance close to the upper bound (Joint) at around 89.8%, the fact that AugNeg(MoCo) reaches the Joint's performance of 93.2% with a similar training cost and low standard deviation is deemed a meaningful improvement. Moreover, we assert that our AugNeg variants demonstrate superior performance in other experiments, including ImageNet-100, DomainNet, and downstream tasks. We would appreciate the reviewer considering these points.

Reviewer comment (Reproducibility):
The results for CaSSLe are still lower from ones presented in the original paper. The pointed issue on the github is mainly about BYOL method.

Author response:
We made diligent efforts to reproduce the CaSSLe experiments in the manuscript, but unfortunately, we were unable to achieve results identical to those reported in the CaSSLe paper. With the Github issue #12 concerning BYOL in the ImageNet-100 experiment, there is  issue #10 discussing reproducibility concerns in the ImageNet-100 experiment. Note both indicate a common trend of obtaining results 5-6% lower than those reported in the original paper, aligning with the tendencies observed in our experimental results. Also, these issues remain unresolved, and the authors have not provided a comprehensive solution.

We hope the reviewer has taken into account the current challenging situation regarding reproducibility. Additionally, we conducted all experiments of our paper in the unified environment and have made the experimental code publicly available. We would appreciate it if these considerations could be taken into account in the final evaluation.

Reviewer comment (Experiments):
For a good comparison in Fig.4 the right figure should be MoCo with just finetuning. We can then compare all three methods better and can be a good sanity check. Right now, what we can say is that AugNeg hurts the performance on the first task a bit (why?) and is better in the following. Do we have the same queue size and all hyper-params here between the methods? (see my questions).

Author response:
Thank you for your valuable feedback. We have updated Figure 4 in the manuscript and made corresponding modifications according to the reviewer's comments. Additionally, in the Supplementary Materials, we included the same analysis on AugNeg(BYOL), so please take a look. The discrepancy in Task 0 performance between MoCo + CaSSLe and AugNeg(MoCo) in the original graph was due to a typo. As evident in Figure 3, for all experiments in our paper, we initiated training with the same Task 0 model, conducting both CaSSLe and AugNeg, and reported the results accordingly. We apologize for the typo and have corrected it.

Reviewer comment (Theory):
There is no clear message why AugNeg works in each of the scenario with each method (MoCo / BYOL).

Author response:
Dear Reviewer Tkp5,

Thank you once again for providing insightful comments. We have made every effort to respond to all your comments within the limited discussion period and have completed the second revision, incorporating all reviewer’s comment into the manuscript.

We want to reiterate that while our proposed AugNeg is a simple idea, its core contribution lies in identifying issues from a different perspective in the field of CSSL research (specifically, pointing out the problem of both 1) the conventional loss form and 2) the low diversity of negative samples). We have introduced a novel loss function to overcome these challenges via model-based augmentation of negative representations. Additionally, through diverse experiments using not only CIFAR-100 but also ImageNet-100, DomainNet and downstream tasks, we experimentally demonstrated that AugNeg achieves consistent performance improvement with little difference in training cost compared to CaSSLe. Furthermore, we believe that the analysis suggested by the reviewer clearly highlights the advantages of AugNeg, confirming its ability to achieve enhanced both plasticity and stability, as aligned with the Method Section.

As the discussion period is coming to a close, we would greatly appreciate it if the reviewer could review our response (and ideally the revised manuscript) and provide feedback. Thank you again.

Reviewer comment (Experiments):
Why do not adjust other hyper-parameters, when changing some crucial ones, e.g. batch-size for cassle?

Author response:
We already reported experimental results on various dimensions of MLP, various queue sizes, and large mini-batch size in the Supplementary Materials.

Reviewer comment (Evaluation):
Why AugNeg for Domain-IL is 43 (Tab.3) when SyCON for the same seting was 46?

Author response:
As mentioned before, we have re-conducted most of the experiments while writing the current paper, so we redrew the figures from scratch. Upon reviewing the previous version of the paper, it seems there was a typo in the table. We apologize for this typo. We have double-checked all results, and there are no typos in the currently drawn tables and figures.

Reviewer comment (Experiments):
More experimental results are expected. CaSSLe performs experiments with all SSL methods mentioned in all CSSL settings. However, the authors only selected two SSL methods, though with exploratory experiments on CIFAR-100, to compare with the baselines. It is worth noting that different SSL methods may have different effects on different CSSL settings and datasets. The goal of various SSL methods is to demonstrate that the loss can universally improve CSSL, given any existing SSL methods and potentially future methods.

Author response:
**Weakness 1: Applicability of AugNeg to other methods**

We appreciate the valuable suggestion. **The rationale behind conducting experiments focused on MoCo V2, SimCLR, BarLow, and BYOL is rooted in the findings of the CaSSLe paper, where these algorithms demonstrated superior performance across the majority of datasets, with MoCo V2 and BYOL particularly excelling.** In alignment with the reviewer's suggestion, we conducted additional experiments by selecting one contrastive and one non-contrastive learning-based algorithm each to explore the applicability of AugNeg to various continual representation learning scenarios.

Given that results for prominent contrastive learning-based algorithms (SimCLR, MoCo V2) were already presented in the manuscript, **we introduced Supervised Contrastive Learning(SupCon) [1] as an additional algorithm** to investigate the effectiveness of AugNeg in scenarios where labels are available. To accomplish this, we modified the implementation of SimCLR in Equation (2) of the manuscript, adapting it to utilize supervised labels, specifically incorporating additional negatives in SupCon + CaSSLe.

For the Non-contrastive learning-based algorithm, **we selected the recent VICReg [2] algorithm** to explore the potential application of AugNeg to various SSL algorithms. To implement AugNeg, we added additional regularization proposed in Equation (5) of the manuscript and conducted experiments accordingly.

We conducted experiments with the CIFAR-100 and ImageNet-100 datasets, employing the Class-IL (5T, 10T) scenario as our experimental setting, and the results are as follows.

| Dataset (5T, ResNet-18) | Alg. | SupCon | VICReg |
| -------- | -------- | -------- | -------- |
| CIFAR-100 | CaSSLe | 60.38 | 53.17 |
| CIFAR-100 | AugNeg | 60.73     |  55.40    |
| ImageNet-100 | CaSSLe | 66.26 | 59.18 |
| ImageNet-100 | AugNeg | 66.96 | 61.88     |

| Dataset (10T, ResNet-18) | Alg. | SupCon | VICReg |
| -------- | -------- | -------- | -------- |
| CIFAR-100 | CaSSLe | 55.38 | 47.54 |
| CIFAR-100 | AugNeg | 56.16     |  50.78    |
| ImageNet-100 | CaSSLe | 60.48 | 49.00 |
| ImageNet-100 | AugNeg | 60.96     |  51.82    |


Based on the results of the above experiments, we observe that **AugNeg can be applied to additional contrastive and non-contrastive learning algorithms, leading to consistent performance improvement compared to CaSSLe**. In particular, we observed successful application of our proposed AugNeg to VICReg, one of the latest SSL algorithms, achieving consistent performance improvements of 2-3% in both CIFAR-100 and ImageNet-100 experiments. We believe that these experimental findings suggest the potential for the proposed AugNeg to be widely applicable to future various contrastive and non-contrastive learning algorithms. 

[1] Khosla, Prannay, et al. "Supervised contrastive learning." Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020.

[2] Bardes, Adrien, Jean Ponce, and Yann LeCun. "VICREG: VARIANCE-INVARIANCE-COVARIANCE REGULARIZATION FOR SELF-SUPERVISED LEARNING." 10th International Conference on Learning Representations, ICLR 2022. 2022.

-- revision of the response --

We discovered a error in the experiment with AugNeg(VICReg) on ImageNet-100. After rectifying the error and re-running the experiment, we have input the revised results into the table and adjusted the response accordingly. We apologize for any inconvenience caused by this.

Reviewer comment (Presentation):
The presentation of the paper needs to be improved, as most of the captions of the tables and figures do not contain useful information.

Author response:
**Weakness 2: Captions and Figures**

We have revised and updated the captions for all Figures and Tables in the manuscript.

Reviewer comment (Theory):
The loss needs a more intuitive explanation. From the current presentation, it seems like the design lacks meaning and is more of an engineer labor. See the question below.

Author response:
**Weakness 3 and Q1) The more intuitive explanation for AugNeg Loss**

Thank you for the question. We have summarized our response regarding the role of additional negatives of the proposed AugNeg Loss in Common Response 3, so please check that for details.

Additionally, **we would like to clarify that $\mathcal{N_{t-1}}$ is not $\mathcal{N_{t}}$ employed in $\mathcal{L}_1$ during the previous task (t-1) training.** As stated in the manuscript, these refer to negatives obtained from $t$ and $t-1$ models during training task $t$. Their implementation varies based on the contrastive learning algorithm; for SimCLR, all samples in the current mini-batch, excluding the identical sample, are considered as negatives. While for MoCo, the entire queue containing negatives used in previous iterations as a positive serves as negatives for the current iteration. At each training iteration for each task, even with the same input mini-batch, the $t$ and $t-1$ models have distinct output features for negative samples (as the $t$ model continues to update, while the $t-1$ model remains frozen). The core idea behind the AugNeg Loss proposed in Equation (2) is to concurrently consider these diverse features in CSSL.

Reviewer comment (Theory):
While I can understand that additional embeddings $\mathcal{N}_{t-1}(i)$ are introduced to $\mathcal{L}_1$, I am curious about the effect of this operation in the representation space and the specific property that $\mathcal{L}_1$ aims to encourage.

Author response:
**Weakness 3 and Q1) The more intuitive explanation for AugNeg Loss**

Thank you for the question. We have summarized our response regarding the role of additional negatives of the proposed AugNeg Loss in Common Response 3, so please check that for details.

Additionally, **we would like to clarify that $\mathcal{N_{t-1}}$ is not $\mathcal{N_{t}}$ employed in $\mathcal{L}_1$ during the previous task (t-1) training.** As stated in the manuscript, these refer to negatives obtained from $t$ and $t-1$ models during training task $t$. Their implementation varies based on the contrastive learning algorithm; for SimCLR, all samples in the current mini-batch, excluding the identical sample, are considered as negatives. While for MoCo, the entire queue containing negatives used in previous iterations as a positive serves as negatives for the current iteration. At each training iteration for each task, even with the same input mini-batch, the $t$ and $t-1$ models have distinct output features for negative samples (as the $t$ model continues to update, while the $t-1$ model remains frozen). The core idea behind the AugNeg Loss proposed in Equation (2) is to concurrently consider these diverse features in CSSL.

Reviewer comment (Experiments):
Are the current negative samples in $\mathcal{L}1$ (previous negative samples in $\mathcal{L}2$) so extensively utilized that the inclusion of the negative samples from another source is necessary?

Author response:
**Weakness 3 and Q1) The more intuitive explanation for AugNeg Loss**

Thank you for the question. We have summarized our response regarding the role of additional negatives of the proposed AugNeg Loss in Common Response 3, so please check that for details.

Additionally, **we would like to clarify that $\mathcal{N_{t-1}}$ is not $\mathcal{N_{t}}$ employed in $\mathcal{L}_1$ during the previous task (t-1) training.** As stated in the manuscript, these refer to negatives obtained from $t$ and $t-1$ models during training task $t$. Their implementation varies based on the contrastive learning algorithm; for SimCLR, all samples in the current mini-batch, excluding the identical sample, are considered as negatives. While for MoCo, the entire queue containing negatives used in previous iterations as a positive serves as negatives for the current iteration. At each training iteration for each task, even with the same input mini-batch, the $t$ and $t-1$ models have distinct output features for negative samples (as the $t$ model continues to update, while the $t-1$ model remains frozen). The core idea behind the AugNeg Loss proposed in Equation (2) is to concurrently consider these diverse features in CSSL.

Reviewer comment (Theory):
The variable $z_{i,t} = Proj(h_{\theta_t}(x_i))$ in CaSSLe is designed for the invariant feature of the current and previous tasks. Does this apply to the proposed algorithms as well?

Author response:
**Q2) Projection layer used in CaSSLe**

Indeed, we utilized it. However, in Section 3.3, $z_{i, t} = Proj(h_{\theta_t}(x_i))$ represents the standard Projection layer commonly employed in SSL. We defined the Predictor for CSSL as $\tilde{z}\_{i, t} = Pred(z\_{i,t})$ and employed them to express the AugNeg Loss.