Paper: Backdoor Contrastive Learning via Bi-level Trigger Optimization (ICLR-2024)

Reviewer comment (Reproducibility):
The setting of K and J are not included in the submission, and their relationship with N remains unclear. It is also uncertain whether K is large enough to ensure the convergence of the surrogate backdoored model.

Author response:
Thank you for your sugguestion. We added all these details in the revised paper for better clarity:

* In Algorithm 1, K and J depend only on the batch size and the total number of data points in the involved dataset. K is calculated via (len of backdoored dataset)/(batch size), and J is calculated via (len of clean dataset)/(batch size). On the other hand, N is independent from K and J (in our default settings, N is 400). We also provide a typical plot (inner and outer loss objectives with N steps) during the bi-level trigger optimization in Appendix C.7, and we can observe that such a design already leads to a converging loss curve.

Reviewer comment (Presentation):
It is unclear whether the x-axis in Figure 4 refers to N. If it includes the inner loop updates (N*J), this may make the comparison of loss curves somewhat unfair.

Author response:
Thank you for your sugguestion. We added all these details in the revised paper for better clarity:

* All data in Figure 4 reflects the victim's training stage: the victim is using the backdoored dataset to pre-train encoder via CL. The x-axis in Figure 4 refer to the **victim's training epoch**. At this stage, **the attacker has finished the trigger optimization and cannot intervene the victim's training process**.  Therefore, it is **fair** to make the comparison between our attack and others, as the difference only lies in different poisoning data injected in the victim's training set.

Reviewer comment (Writing):
The expression *regularly re-initialize the surrogate feature extractor* (in section 4) is ambiguous. It is not clear if it implies that, after the initialization (line 2 in Algorithm 1), there is a subsequent re-initialization at some point.

Author response:
Thank you for your sugguestion. We added all these details in the revised paper for better clarity:

* For "regularly re-initialization", we meant to re-initialize the surrogate feature extractor every N bi-level optimization steps and repeat the procedure for M times. In our experiment, we repeat for M=2 times. The reason to perform re-initialization is to prevent the trigger generator from overfitting to one particular surrogate models (so we reset it and retrain another surrogate model after certain time).

Reviewer comment (Experiments):
The factors determining the superior transferable ability of the chosen surrogate CL framework (SimSiam) are unclear. There may be a need for further analysis and clarification in ablation experiments, such as exploring the impact of different data augmentation choices.

Author response:
Note that although we pick SimSiam as our default surrogate CL framework, our method could also work with other CL frameworks. Here we provide the attack performance (on CIFAR-10) with another two CL frameworks (BYOL and SimCLR) as surrogate models.

| Surrogate Strategy |        |        | SSL Method |        |         |        |
|:------------------:|:------:|:------:|:----------:|:------:|:-------:|:------:|
|                    | SimCLR |        |    BYOL    |        | Simsiam |        |
|                    |  ACC   |  ASR   |    ACC     |  ASR   |   ACC   |  ASR   |
|      SimSiam       | 90.10% | 91.27% |   91.21%   | 94.78% | 90.18%  | 84.63% |
|        BYOL        | 89.83% | 71.95% |   90.70%   | 76.40% | 90.06%  | 74.08% |
|       SimCLR       | 89.27% | 91.45% |   90.13%   | 78.58% | 88.36%  | 87.62% |

We can observe that the attack transferability is not limited to a specific surrogate CL method. We have include these experimental results in Appendix C. 6 (Table 10). We believe the key reason is that though CL frameworks differ with each other in work flows (negative pairs, loss function designs, etc), their core mechanisms are similar (i.e., alignment and uniformity), as discussed in Section 3. This allows our trigger generator fits other CL frameworks easily once it succeed in backdooring one particular CL framework (i.e., the surrogate CL framework in our algorithm). Such a transferable ability across different CL frameworks is also reported by existing work [2].

Reviewer comment (Experiments):
The BLTO procedure contains both a backdoor generator and a backdoored surrogate model θ. It is unclear if the co-training surrogate backdoored model performs as well as a backdoored model actually trained on the poisoned data.

Author response:
Thanks for the question. We provide the surrogate model's performance in the following (framework: SimSiam, dataset: CIFAR-10, backbone: resnet18). For comprison, we also showed victim model's performance after the same training epoch as the surrogate model.


|  | Surrogate Model | Actual Model| 
|:---:|:---:|:---:| 
| BA | 86.98% | 87.31% | 
| ASR | 96.72% | 92.74% | 

Here are our observation and analysis based on our above experimental records.

* **For ASR,** the surrogate backdoored model performs better than an backdoored model actually trained on the poisoned data. This is easy to understand since the trigger generator is directly optimized from the surrogate backdoored model, making it fit the surrogate model better.

* **For backdoor accuracy (BA),** the surrogate backdoored model performs similar with an backdoored model actually trained on the poisoned data, as they both learn knowledge from those clean data points.

Reviewer comment (Experiments):
Although the proposed attack targets the feature extractor, the victim settings in the experiments are limited to the classification task. It could be extended to more tasks to better demonstrate the effectiveness of the proposed attack.

Author response:
In terms of target learning task, we mainly focus on the classification task following the common practice in this area [3, 4, 5] and we would be happy to extend it into other tasks in our future work.

Reviewer comment (Experiments):
The motivation could be better articulated. The authors claim that the fixed trigger design leads to limited ASR. However, in the methodology, not only is the trigger generator adopted, but a bi-level optimization strategy is also used to optimize the trigger generator. This raises the question: Is the trigger generator alone sufficient for the success of the proposed attack? If not, it suggests that the fixed trigger is not the primary cause of the current limitation. I recommend that the authors conduct an ablation study on this matter and be cautious with their claims.

Author response:
Thanks for the insight. In fact, **our ablation study in Appendix C.3 (Table 7) was designed to answer this question**. Specifically, in Table 7, the case w/o inner optimization exactly corresponds to your desired case: the generator is used but the bi-level formulation is disabled (i.e., the generator is only optimized on a pretrained clean encoder). We copy the results as follows:

|  |  | SSL Method |  |
|:---:|:---:|:---:|:---:|
|  | SimCLR | BYOL | Simsiam |
|  | ASR | ASR | ASR |
| bi-level optimization | 91.27% | 97.17% | 84.63% |
| w/o inner optimization | 71.98% | 60.13% | 17.44% |

Observe that without bi-level optimization, the generator still presented a certain level of backdoor effectiveness. This partially suggests the ineffectiveness of "fixed trigger design". However, simply adopting the generator is still not satisfactory and we found that utilizing bi-level optimization can further boost the performance.

We have also adjusted expressions about "fixed trigger design" in the paper to avoid possible misunderstanding. In our context, "fixed trigger design" indicates those triggers that are not optimized, such as fixed patch in SSL backdoor [1].

Reviewer comment (Novelty):
The comparison with other recently developed works could enhance the contribution:
(1) PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learning.
(2) CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learning.
Notably, in "PoisonedEncoder," a bi-level optimization strategy is also employed to formulate the attack. How does this work differ from theirs?

Author response:
Thanks for pointing out these works. We have added corresponding discussions of them in related works, preliminaries and methdologies.

* **PoisonedEncoder[2]**: We notice that this work's threat model is different from ours: it intends to fool the victim's encoder to misclassify **attacker's select inputs** (not all inputs attached with trigger) into target categories. Therefore, it is not a backdoor attack method and thus the objectives in loss functions are actually different. Indeed, PoisonedEncoder also proposed a bi-level optimization fomulation. However, it finally resort to non-iterative heuristic solutions, potentially due to the challenges in solving their corresponding bi-level problem. We have included this discussion in related works section.

* **CorruptEncoder[3]**: This is indeed a revelent work and we have **already discussed it in our related works**. We did not compare with CorruptEncoder mainly due to their unique triggered image generation design: they need to combine background images with objects in various ways. Such a design make it difficult to perform on small size image datasets, such as CIFAR-10 and CIFAR-100. Also they have not yet published the code or processed object/background images and thus it is nontrivial to reproduce the method at the current stage.

Reviewer comment (Experiments):
Is the trigger generator alone sufficient for the success of the proposed attack?

Author response:
Thanks for the insight. In fact, **our ablation study in Appendix C.3 (Table 7) was designed to answer this question**. Specifically, in Table 7, the case w/o inner optimization exactly corresponds to your desired case: the generator is used but the bi-level formulation is disabled (i.e., the generator is only optimized on a pretrained clean encoder). We copy the results as follows:

|  |  | SSL Method |  |
|:---:|:---:|:---:|:---:|
|  | SimCLR | BYOL | Simsiam |
|  | ASR | ASR | ASR |
| bi-level optimization | 91.27% | 97.17% | 84.63% |
| w/o inner optimization | 71.98% | 60.13% | 17.44% |

Observe that without bi-level optimization, the generator still presented a certain level of backdoor effectiveness. This partially suggests the ineffectiveness of "fixed trigger design". However, simply adopting the generator is still not satisfactory and we found that utilizing bi-level optimization can further boost the performance.

We have also adjusted expressions about "fixed trigger design" in the paper to avoid possible misunderstanding. In our context, "fixed trigger design" indicates those triggers that are not optimized, such as fixed patch in SSL backdoor [1].

Reviewer comment (Novelty):
How does this work differ from the "PoisonedEncoder"?

Author response:
Thanks for pointing out these works. We have added corresponding discussions of them in related works, preliminaries and methdologies.

* **PoisonedEncoder[2]**: We notice that this work's threat model is different from ours: it intends to fool the victim's encoder to misclassify **attacker's select inputs** (not all inputs attached with trigger) into target categories. Therefore, it is not a backdoor attack method and thus the objectives in loss functions are actually different. Indeed, PoisonedEncoder also proposed a bi-level optimization fomulation. However, it finally resort to non-iterative heuristic solutions, potentially due to the challenges in solving their corresponding bi-level problem. We have included this discussion in related works section.

* **CorruptEncoder[3]**: This is indeed a revelent work and we have **already discussed it in our related works**. We did not compare with CorruptEncoder mainly due to their unique triggered image generation design: they need to combine background images with objects in various ways. Such a design make it difficult to perform on small size image datasets, such as CIFAR-10 and CIFAR-100. Also they have not yet published the code or processed object/background images and thus it is nontrivial to reproduce the method at the current stage.

Reviewer comment (Experiments):
If the reference data $x_r$ is randomly sampled from the target class? does it need to be included in the downstream dataset to ensure the success of the attack?

Author response:
The reference data is randomly sampled from the target class. For instance, suppose "truck" is the target class, we randomly sample 500 "truck" images (from CIFAR-10) as reference data. We added more classifications to the procedure in **Appendix A.2**. 

The reference data is not required to be included in the victim's downstream dataset to ensure the success of the attack. To verify this, we conduct a new experiment where we changed the downstream dataset to STL-10 dataset while the reference data is from CIFAR-10. The victim could use SimCLR, BYOL or SimSiam to train their ResNet-18 backbone. The downstream classifier is a linear DNN, similar to BadEncoder [4]. We added this evaluation in **Appendix C.6 (Table 15)** and list the results as follows:

| victim's CL methods | SimCLR | BYOL | SimSiam|
|:---:|:---:|:---:|:---:|
| ACC/BA | 76.73% | 79.04% |80.47%|
| ASR | 94.74% | 96.19% |87.55%|

The results indicate that our attack doesn't require the overlapping between reference data and downstream dataset to obtain a good attack performance.

Reviewer comment (Experiments):
The paper lacks analysis or discussion on the impact of using more accurate Hessian approximations to solve the proposed bi-level optimization, relying only on a discrete solution.

Author response:
> Q2. Discuss on using more accurate Hessian approximations to solve the proposed bi-level optimization.


We sincerely thank you for the suggestion. We added a **convergence plot Figure 11** for solving this bi-level optimization problem, and we can observe that our adopted interleaving update strategy is already effective and leads to convergent training loss. Note that such a design is natural and widely adopted in many different problems (such as [5][6]). In fact, [7] actually showed that omitting the second derivatives does not lead to major performance change on the bi-level solution. While Hessian approximations could potentially provide a more accurate solution, it could also bring a higher computation complexity and time cost due to the calculation on the second derivatives, which limits the attack practicability on large-scale datasets. We add this discussion in **Appendix C.7**.

Reviewer comment (Experiments):
The baseline implementations and results seem questionable based on inconsistencies with original papers and recent related work — the attack success rates for SSL backdoor and CTRL differ notably from prior reported values.

Author response:
> Q1. The baseline implementations and results seem questionable based on inconsistencies attack success rates for SSL backdoor and CTRL from prior works.

We are sorry for the confusion and we believe it is just a misunderstanding. We have provided detailed discussion and make it clear in **Appendix C.8**. 

In fact, **the ASR difference is due to the contrastive learning (CL) implementation adopted by the victim**, which can be seen from the model accuracy on clean data: using our CL implementation, the victim encoder can achieve almost $90\%$ BA (i.e., ACC) on CIFAR-10; while with the original CTRL's CL implementation, the encoder achieves $80\%$ ACC on CIFAR-10 (i.e., Table 1 in CTRL [1]). 

In the following, we provide detailed evidences (e.g., CL implementation difference, related works and codes) to demonstrate the fidelity of our experiment on SSL backdoor and CTRL. We hope this can clarify your doubts about our implementation and result. 

1. **Difference in CL implementation that results in different ASRs and ACCs**: our CL implementation for victim follows standard contrastive learning practices (i.e., SimCLR [4]), which differs from CTRL's CL implementation in 1) **data normalization**, and 2) **batch-wise data augmentation strategy**. Specifically, in CTRL's official implementation, it did not use instance-specific data augmentation within the batch during training and data normalization during testing, which leads to an encoder with a **lower ACC/BA and a higher ASR**.


2. **Reason of not using CTRL's CL implementation**: note that in our threat model, the attacker cannot control how the victim train the actual CL model. In practice, victims would prefer adopting CL that produces a more accurate encoder with higher ACC. In fact, our CL implementation follows standard practices with matching ACC: SimCLR (Table B.5 in [4]) reports **90.6% ACC** for ResNet on CIFAR-10, which is close to our reported **90% ACC / BA** in Table 1. However, the SimCLR implemented in CTRL only achieves **80.5% ACC** (Table 1 in CTRL [1]). Besides, CTRL's CL on ImageNet-100 has **42.2% ACC**, which is even lower than ACC on the whole ImageNet (around **69% ACC** in Table 6 of [4]). Therefore, we believe that we actually considered a more practical scenario for the victim's CL implementation.  

3. **Existing work [3] reports similar ASR as ours (on CTRL and SSL backdoor):** our reported ASR of CTRL and SSL backdoor on ImageNet-100 is consistent with those in [3]. To be specific, Table 1 and Table 2 in [3] report **CTRL** on ImageNet100 with **28.8% ASR** and **SSL backdoor** with **14.3% ASR**, and their **ACCs / BAs** are about **70%**. These results in general match ours on ImageNet-100, e.g., CTRL with **35.62% ASR**, SSL backdoor with **13.10% ASR**, and their **70% ACC / BA**). Note that  **70% ACC / BA** in our works and [3] matches common CL practices, higher than **40% ACC / BA** in CTRL (Table 1 of CTRL [1]).

4. **An anonymous GitHub repository is provided to reproduce our results and CTRL's results**, please check ***https://anonymous.4open.science/r/CTRL_correctness-9D51***. Here in the victim contrastive learning procedure, the victim uses SimCLR to train a ResNet18 backbone on **backdoored** CIFAR-10 (as in CTRL [1] Table 1 and ASSET [2] Table 5). Directly excuting the following Python files in the repository can reproduce CTRL's and our results:
    - `main_train.py`: CTRL's contrastive learning implementation following their GitHub repository "https://github.com/meet-cjli/CTRL/tree/master," which can report a **similar ASR (80+%)** and **similar ACC (80+%)** in CTRL [1]. 
    - `main_train_my.py`: our contrastive learning implementation, which reports CTRL's performance with similar **ASR (30+%)** and **ACC (90+%)** in our paper. The major differences in our CL implementation are detailed above and in this anonymous repository.

Reviewer comment (Experiments):
The related work review and experiment scope is too narrow: recent attacks using similar target-class-based trigger synthesis are not compared to.

Author response:
> Q3. Related work discussion on backdoor attack [8] and backdoor sample detection [2].

Thank you for the suggestions. We have added discussion to these works [2, 8] in **Section 2**.

**Comparison with Narcissus [8]**: Narcissus is primarily designed for clean-label backdoor attack in **supervised learning**, which is different from our contrastive learning setting. Specifically, **this attack synthesizes triggers using not only the target class, but also a supervised surrogate model (obtained from labeled datasets)**, while our threat model assumes no access to such a supervised surrogate model by default but trains a surrogate encoder via contrastive learning on an unlabeled dataset. Due to this difference in attack knowledge, this work is not directly comparable of our work but we have discussed it in the updated paper.

**Evaluation under ASSET [2]**: while we have provided an evaluation against multiple backdoor defenses in Section 5.3, we are happy to further evaluate under this suggested one and include it in Section 5.3 and Appendix C.5. Specifically, on our backdoored CIFAR-10 data (with poisoning rate 1%), we used ASSET to sift backdoored data points, and caluclated True Positive Rate **TPR = 5.39%** and False Positive Rate **FPR = 32.17%**. This low TPR and high FPR indicate that our backdoor can still effectively resist ASSET's detection.

Reviewer comment (Experiments):
The related work review and experiment scope is too narrow: a relevant defense for detecting backdoor samples in contrastive learning is not discussed.

Author response:
> Q3. Related work discussion on backdoor attack [8] and backdoor sample detection [2].

Thank you for the suggestions. We have added discussion to these works [2, 8] in **Section 2**.

**Comparison with Narcissus [8]**: Narcissus is primarily designed for clean-label backdoor attack in **supervised learning**, which is different from our contrastive learning setting. Specifically, **this attack synthesizes triggers using not only the target class, but also a supervised surrogate model (obtained from labeled datasets)**, while our threat model assumes no access to such a supervised surrogate model by default but trains a surrogate encoder via contrastive learning on an unlabeled dataset. Due to this difference in attack knowledge, this work is not directly comparable of our work but we have discussed it in the updated paper.

**Evaluation under ASSET [2]**: while we have provided an evaluation against multiple backdoor defenses in Section 5.3, we are happy to further evaluate under this suggested one and include it in Section 5.3 and Appendix C.5. Specifically, on our backdoored CIFAR-10 data (with poisoning rate 1%), we used ASSET to sift backdoored data points, and caluclated True Positive Rate **TPR = 5.39%** and False Positive Rate **FPR = 32.17%**. This low TPR and high FPR indicate that our backdoor can still effectively resist ASSET's detection.

Reviewer comment (Theory):
The proposed bi-level optimization formulation is solved using a discrete approximation without much discussion on the impact of using more efficient yet accurate Hessian approximations or evaluation of the convergence of the proposed solution. Bi-level optimization often benefits from analyzing such approximations rather than directly providing a discretized solution.

Author response:
> Followup Q2. Approach to solve bi-level optimization.

We want to highlight that this is not an optimization work. Our focus is to identify the bottleneck of existing works in backdooring CL and provide our trigger optimization objective. We have explained that our adopted interleaving update is a common practice in large-scale machine learning. It is well-known that hypergradient based solutions results in high complexity with limited improvements in complicated deep learning tasks. Therefore, our choice is easy to understand: a practical attack cannot afford such large computational cost of hypergradient approach. Given that our solution already achieves satisfactory convergence and improved empirical results, we believe this general discussion of our choice is enough. Detailed analysis on the convergence of different bi-level optimization solutions should be the focus of an optimization work, instead of ours backdoor work.

Reviewer comment (Evaluation):
The evaluation results comparing against baselines may contain erroneous implementations. In particular, the reported attack success rate (ASR) for SSL backdoor differs notably from values in the original paper, which, because of the low efficacy of their work, even used a different metric based on the number of misclassified samples.

Author response:
> Followup Q1. Implementation of SSL backdoor and the CTRL experiments.

First, we want to point out that the comment "all three studies utilize the same SimCLR" is not true. As we explained in the implementation details, prior works that evaluate SSL backdoor and CTRL did not include instance-specific data augmentation and data normalization in their SimCLR implementation for victims, which we find to significantly reduce ASR of these attacks. In our evaluation, we follow these standard designs in victims's CL implementation and report ACC and ASR for all attacks. 

Second, per the request, we also provide additional results in early epochs as follows, comparing CRTL's performance in our CL implementation and its original CL (with directly removing data normalization and instance-specific augmentation in our implementation). The detailed running log can also be found in the provided anonymous reporsitory: check `log/original_setting.log` for original CRTL's victim CL, and `log/change_train_dataset_testset.log` for our victim's CL.

| CRTL on our CL implementation | 101epoch | 301epoch | 1000epoch |
|:---:|:---:|:---:|:---:|
| BA | 81.40% | 85.42% | 90.50% |
| ASR | 25.17% | 16.91% | 18.26% |

| CTRL on its CL implementation | 513epoch | 1000epoch |
|:---:|:---:|:---:|
| BA | 80.23% | 83.13% |
| ASR | 83.87% | 80.35% |

We want to highlight that ASR is not necessarily related to BA. In our previous discussion, low ACCs compared with standard CL practices is an indication of the issues in some prior works' CL implementation, which does not suggest that low ACCs should correspond to high ASRs. Our point is that since some prior works' CL implementation does not uses the aforementioned standard practices, they may report over-optimistic ASR while our results (and another recent work [4]) revealed such issue. 

If you still have doubts, feel free to check and try the provided codes and let us know exactly the places you find problematic. Otherwise, we would appreciate you reconsidering the ethic flag raised, which is a serious accusation and we believe we have provided sufficient evidences to clarify.

Reviewer comment (Evaluation):
The reported ASR for CTRL is much lower than results from recent works (e.g., the original paper and [1]) that show CTRL can achieve above 80% ASR with SimCLR on CIFAR-10, contrasting the authors' significantly lower values.

Author response:
> Followup Q1. Implementation of SSL backdoor and the CTRL experiments.

First, we want to point out that the comment "all three studies utilize the same SimCLR" is not true. As we explained in the implementation details, prior works that evaluate SSL backdoor and CTRL did not include instance-specific data augmentation and data normalization in their SimCLR implementation for victims, which we find to significantly reduce ASR of these attacks. In our evaluation, we follow these standard designs in victims's CL implementation and report ACC and ASR for all attacks. 

Second, per the request, we also provide additional results in early epochs as follows, comparing CRTL's performance in our CL implementation and its original CL (with directly removing data normalization and instance-specific augmentation in our implementation). The detailed running log can also be found in the provided anonymous reporsitory: check `log/original_setting.log` for original CRTL's victim CL, and `log/change_train_dataset_testset.log` for our victim's CL.

| CRTL on our CL implementation | 101epoch | 301epoch | 1000epoch |
|:---:|:---:|:---:|:---:|
| BA | 81.40% | 85.42% | 90.50% |
| ASR | 25.17% | 16.91% | 18.26% |

| CTRL on its CL implementation | 513epoch | 1000epoch |
|:---:|:---:|:---:|
| BA | 80.23% | 83.13% |
| ASR | 83.87% | 80.35% |

We want to highlight that ASR is not necessarily related to BA. In our previous discussion, low ACCs compared with standard CL practices is an indication of the issues in some prior works' CL implementation, which does not suggest that low ACCs should correspond to high ASRs. Our point is that since some prior works' CL implementation does not uses the aforementioned standard practices, they may report over-optimistic ASR while our results (and another recent work [4]) revealed such issue. 

If you still have doubts, feel free to check and try the provided codes and let us know exactly the places you find problematic. Otherwise, we would appreciate you reconsidering the ethic flag raised, which is a serious accusation and we believe we have provided sufficient evidences to clarify.

Reviewer comment (Reproducibility):
In addition to the original papers, another separate work [1] evaluated these two attacks and also confirms the potential erroneous implementations in this work.

Author response:
> Q1. The baseline implementations and results seem questionable based on inconsistencies attack success rates for SSL backdoor and CTRL from prior works.

We are sorry for the confusion and we believe it is just a misunderstanding. We have provided detailed discussion and make it clear in **Appendix C.8**. 

In fact, **the ASR difference is due to the contrastive learning (CL) implementation adopted by the victim**, which can be seen from the model accuracy on clean data: using our CL implementation, the victim encoder can achieve almost $90\%$ BA (i.e., ACC) on CIFAR-10; while with the original CTRL's CL implementation, the encoder achieves $80\%$ ACC on CIFAR-10 (i.e., Table 1 in CTRL [1]). 

In the following, we provide detailed evidences (e.g., CL implementation difference, related works and codes) to demonstrate the fidelity of our experiment on SSL backdoor and CTRL. We hope this can clarify your doubts about our implementation and result. 

1. **Difference in CL implementation that results in different ASRs and ACCs**: our CL implementation for victim follows standard contrastive learning practices (i.e., SimCLR [4]), which differs from CTRL's CL implementation in 1) **data normalization**, and 2) **batch-wise data augmentation strategy**. Specifically, in CTRL's official implementation, it did not use instance-specific data augmentation within the batch during training and data normalization during testing, which leads to an encoder with a **lower ACC/BA and a higher ASR**.


2. **Reason of not using CTRL's CL implementation**: note that in our threat model, the attacker cannot control how the victim train the actual CL model. In practice, victims would prefer adopting CL that produces a more accurate encoder with higher ACC. In fact, our CL implementation follows standard practices with matching ACC: SimCLR (Table B.5 in [4]) reports **90.6% ACC** for ResNet on CIFAR-10, which is close to our reported **90% ACC / BA** in Table 1. However, the SimCLR implemented in CTRL only achieves **80.5% ACC** (Table 1 in CTRL [1]). Besides, CTRL's CL on ImageNet-100 has **42.2% ACC**, which is even lower than ACC on the whole ImageNet (around **69% ACC** in Table 6 of [4]). Therefore, we believe that we actually considered a more practical scenario for the victim's CL implementation.  

3. **Existing work [3] reports similar ASR as ours (on CTRL and SSL backdoor):** our reported ASR of CTRL and SSL backdoor on ImageNet-100 is consistent with those in [3]. To be specific, Table 1 and Table 2 in [3] report **CTRL** on ImageNet100 with **28.8% ASR** and **SSL backdoor** with **14.3% ASR**, and their **ACCs / BAs** are about **70%**. These results in general match ours on ImageNet-100, e.g., CTRL with **35.62% ASR**, SSL backdoor with **13.10% ASR**, and their **70% ACC / BA**). Note that  **70% ACC / BA** in our works and [3] matches common CL practices, higher than **40% ACC / BA** in CTRL (Table 1 of CTRL [1]).

4. **An anonymous GitHub repository is provided to reproduce our results and CTRL's results**, please check ***https://anonymous.4open.science/r/CTRL_correctness-9D51***. Here in the victim contrastive learning procedure, the victim uses SimCLR to train a ResNet18 backbone on **backdoored** CIFAR-10 (as in CTRL [1] Table 1 and ASSET [2] Table 5). Directly excuting the following Python files in the repository can reproduce CTRL's and our results:
    - `main_train.py`: CTRL's contrastive learning implementation following their GitHub repository "https://github.com/meet-cjli/CTRL/tree/master," which can report a **similar ASR (80+%)** and **similar ACC (80+%)** in CTRL [1]. 
    - `main_train_my.py`: our contrastive learning implementation, which reports CTRL's performance with similar **ASR (30+%)** and **ACC (90+%)** in our paper. The major differences in our CL implementation are detailed above and in this anonymous repository.

Reviewer comment (Novelty):
The related work could be expanded and compared more thoroughly. Some recent attacks [2] also leverage synthesized triggers using solely the target class, similar to the proposed approach, which is worth incorporating and comparing.

Author response:
> Followup Q3. Discussion and Comparsion with Narcissus.

We believe there are some factual errors in reviewer's comment about Narcissus. We would like to clarify them, and emphasize that these two works are very distinct in attack knowledge and technique. We haved discussed Narcissus in Related Work, Appendix C.3 and Table 16. 

First, Narcissus uses more than a surrogate feature extractor: it requires a labeled dataset to train a surrogate classifier to perform the attack. On the contrary, we only require an unlabeled dataset to obtain a surrogate encoder. Therefore, Narcissus aims to backdoor a classifier in supervised setting, while we target on backdooring an encoder in unsupervised learning setting. Since **our attack knowledge and threat model are totally different**, their comparison is meaningless. In fact, all the works in this line of research (e.g., CTRL, SSL backdoor, CorruptEncoder) does not compare with Narcissus.
 
Second, the comment "each method employs discrete inner and outer optimization processes" is incorrect: Narcissus does not formulate any bi-level optimization problem, thus there is no so-called inner/outer optimization processes. Instead, it optimizes the trigger directly on a pre-trained surrogate classifier.  

Even if we ignore all those differences and evaluate the trigger given by [1] (in their github repository) on directly poisoning the CL pre-training dataset, its attack performance is not as good as ours. Assume the victim is conducting contrastive learning on the backdoored CIFAR-10 dataset (with poisoning rate 1% ), using ResNet18 backbone and CL strategies SimCLR, BYOL and SimSiam. The resulting BA (i.e. ACC) and ASR are shown as follows:

| Narci ($\epsilon=32/255$) | SimCLR | BYOL | SimSiam |
|:---:|:---:|:---:|:---:|
| BA | 88.05% | 90.49% | 88.57% |
| ASR | 73.01% | 58.64% | 3.48% |

The trigger generated by Narcissus presents a certain effectiveness in unsupervised CL scenarios, which is expected as the trigger is associated with the target class using a surrogate classifier. However since it ignores the uniformity effect of CL, its attack performance is less effective and stable compared with our work (even with its higher $\epsilon=32/255$). We have included this evaluation in Appendix C.3.

Reviewer comment (Experiments):
A recent backdoor sample detection method [1] demonstrates effectiveness in detecting poisoned samples in contrastive learning unlabeled datasets, which is relevant but not discussed or evaluated.

Author response:
> Q3. Related work discussion on backdoor attack [8] and backdoor sample detection [2].

Thank you for the suggestions. We have added discussion to these works [2, 8] in **Section 2**.

**Comparison with Narcissus [8]**: Narcissus is primarily designed for clean-label backdoor attack in **supervised learning**, which is different from our contrastive learning setting. Specifically, **this attack synthesizes triggers using not only the target class, but also a supervised surrogate model (obtained from labeled datasets)**, while our threat model assumes no access to such a supervised surrogate model by default but trains a surrogate encoder via contrastive learning on an unlabeled dataset. Due to this difference in attack knowledge, this work is not directly comparable of our work but we have discussed it in the updated paper.

**Evaluation under ASSET [2]**: while we have provided an evaluation against multiple backdoor defenses in Section 5.3, we are happy to further evaluate under this suggested one and include it in Section 5.3 and Appendix C.5. Specifically, on our backdoored CIFAR-10 data (with poisoning rate 1%), we used ASSET to sift backdoored data points, and caluclated True Positive Rate **TPR = 5.39%** and False Positive Rate **FPR = 32.17%**. This low TPR and high FPR indicate that our backdoor can still effectively resist ASSET's detection.

Reviewer comment (Experiments):
The hyper-parameters such as the learning rates of the models might also influence the performance of the proposed method. For example, if the learning rates and the batch sizes of the surrogate model are significantly different from those used by the victim models, then the attack success rates might also be reduced. It is suggested to add the discussion about this.

Author response:
> Q2. Discussion on varying hyper-parameters used in surrogate model and victim model. 

Thanks for pointing out this. We have added a suggested analysis in **Appendix C.6** and list the results as follows.

**Surrogate model**: when optimizing the trigger, by default, we train a surrogate ResNet18 model on CIFAR-10 via Simsiam. We set batch size=512, and use a dynamic learning rate (with a cosine scheduler, base lr=0.03, final lr=0).

**Victim model**: suppose the victim trains a ResNet18 backbone on CIFAR-10 via SimCLR. To evaluate whether different hyper-parameters used by victim will influence the attack effectiveness, we vary the batch size of victim in {256, 512, 1024} and scale surrogate model's learning rate by {$\times 0.5$, $\times 1$, $\times 2$}:

| Batch size | 256 | 512 |1024|
|:---:|:---:|:---:|:---:|
| ACC/BA | 86.05% | 90.10% |90.85%|
| ASR | 91.48% | 91.27% |89.39%|

| learning rate | $\times$ 0.5 | $\times$ 1 | $\times$ 2|
|:---:|:---:|:---:|:---:|
| ACC/BA | 88.05% | 90.10% |88.76%|
| ASR | 91.80% | 91.27% |92.54%|

From above results, we find that our attack can retain effectiveness even though the victim's CL methods, hyper-parameters differ from the attacker's.

Reviewer comment (Evaluation):
The robustness under Feng et al. [6] is not discussed. Is the backdoor samples in the proposed method have high cosine similarity between each other?

Author response:
> Q5. The robustness under Feng et al. [6] is not discussed. Is the backdoor samples in the proposed method have high cosine similarity between each other?

Thanks for poiting out this possible defense. We are happy to further evaluate our attack under this defense method via trigger inversion in **Section 5.3 and Appendix C.5**. Specifically, we examined our backdoored ResNet18 encoder (pretrained on the backdoored CIFAR-10 via SimCLR using the same configuration in our Table 4 and Table 5) and report the results for L1-norm and P1-norm here. 

|  | clean encoder (SimCLR) | backdoored encoder (SimCLR) | clean encoder (BYOL) | backdoored encoder (BYOL) |
|:---:|:---:|:---:|:---:|:---:|
| L1-norm | 582.56 | 719.58 | 638.37 |  604.38|
| P1-norm (backdoored when < 0.1) | 0.19 | 0.23 | 0.20 | 0.20 |

According to the default threshold in [6], if the P1-norm $<0.1$, it's likely that the encoder is "backdoored". The results show that P1-norm can not really differentiate clean and backdoored encoder. We argue that this is because our backdoor trigger is image-specific (generator-based), which somehow provide the robustness over these trigger inversion defenses (including [7, 8] covered in our paper).

Reviewer comment (Novelty):
The usages of the surrogate models and the bi-level optimization is not new in the field of poisoning attacks and backdoor attacks [7,8], which somehow weaken the contributions of this paper.

Author response:
> Q6. The usages of the surrogate models and the bi-level optimization is not new [9, 10], which somehow weaken the contributions of this paper.

We agree that these techniques are indeed widely used, even in a broader field than ML security. However, we believe **our main contribution is identifying the weakness of existing CL attacks** (e.g., the failure of compromising CL's uniformity mechanism discussed in Section 3 and Section 5.4), **which naturally motivates us to formulate our trigger optimization problem**. The surrogate model and bi-level optimization techniques are just tools for us to realize our novel motivation. Therefore, in our viewpoints, our contribution doesn't lie in the adoption of "surrogate models and bi-level optimization" themselves, but in how we formulate our desired goals and use these tools to solve them.