Paper: How Hard is Trojan Detection in DNNs? Fooling Detectors With Evasive Trojans (ICLR-2024)

Reviewer comment (Evaluation):
The effects of specificity loss are poorly understood.

Author response:
Thanks to your suggestion, we have run a preliminary evaluation with the Sun et al. method using their original codebase. Unfortunately, their method does not perform well on our standard trojans, possibly because we use different datasets and triggers. The original paper requires high-resolution images, which are challenging in our setting where we have to train over 6,000 networks. It also only uses colored patch triggers. By contrast, we use smaller resolution images and randomized black-and-white patch triggers to automatically sample from a distribution of triggers. We have not observed any artifacts caused by our specificity loss. To the contrary, not using the specificity loss causes the specificity-based detector to obtain very high scores (SB in Table 2). Thus, we are confident that our specificity loss is a genuine contribution that increases evasiveness against this type of detector. If we have addressed the thrust of your concerns, we kindly ask that you consider raising your score.

Reviewer comment (Experiments):
Removal-based defenses can be just as effective.

Author response:
(As an aside, please note that MNTD does not require a clean dataset. It requires a dataset of clean and trojaned *networks*, but the queries are optimized from scratch.)

Some research in the trojan detection community focuses on developing new attacks to evade the entire pipeline of trojan defenses. However, so far these works have not considered a wide range of model-level detectors, including training-based detectors like MNTD. Our paper does not attempt to evade the entire pipeline of defenses, including input-level, dataset-level, and model-level detection, and trojan removal. Rather, we are studying the specific problem of evading model-level detection, which hasn't been extensively studied before.

Thus, we do not include an extensive comparison to other types of defenses, including dataset-level or trojan removal defenses. In fact, since we fix the triggers in our experiments to highly visible triggers, we are confident that simple dataset-level methods like Activation Clustering would perform quite well on our attacks. Similarly, it is very possible that our method makes trojans easier to remove. However, this is not the focus of our contribution. Combining our model-level evasion method with methods for evading other types of defenses would be a large undertaking, so we leave this to future work.

Reviewer comment (Novelty):
The idea of model-level distribution matching is new and interesting but it is specifically designed against model-level defenses, and there are already many works that explore distribution matching in the latent space, apply more stealthy poisoning attacks, or aim to reduce artifacts of backdoors.

Author response:
The three methods you cite ([1], [2], [3]) are similar to WaNet and LIRA. They aim to reduce the distance between clean and trigger-embedded inputs (either in input-space or latent-space) with the goal of making triggers stealthy, but not with the goal of making parameters look similar to clean parameters. This is very different from our work, which reduces the distance between clean and trojaned *parameters*. Our results in Appendix B suggest these two objectives may be at odds, which would be an interesting direction for future work to explore.

Reviewer comment (Experiments):
In the appendix, the authors show that their attack performs similarly to TaCT, and combining with TaCT improves stealthiness, but there is no reason other existing, more advanced, stealthy attacks cannot also be combined with TaCT.

Author response:
As we mention above, we do compare to strong stealthy trigger attacks, including WaNet and LIRA, finding that they are trivially easy to detect with model-level detectors like MNTD. We also compare to TaCT, and we find that it surprisingly improves evasiveness against a wide range of model-level detectors, which was not previously known (in the original TaCT paper, the authors conducted a more limited evaluation with far fewer detectors). Combining our method with TacT yields stronger results, so we are able to outperform TaCT in source-specific settings; in all-to-one settings TaCT is not applicable.

Reviewer comment (Evaluation):
The results on backdoor detection show no significant improvement, except against the model-level defenses, over simple baseline attacks.

Author response:
It's important to note that **to the best of our knowledge, our attack is state-of-the-art for evading model level detectors**. We are not aware of any recent stealthy trigger attacks that evaluate on model-level detectors like MNTD that leverage a training set of networks (including the three that you cite), and our results on WaNet and LIRA suggest that these attacks would perform poorly. We also compare to specific prior works that do try to evade model-level detectors in Appendix B.2, and our method outperforms them. Given these factors, we do feel justified in claiming state-of-the-art, although research on this problem is still in very early stages. More importantly, we are the first to show that evading a wide range of model-level detectors is possible in the first place, using principled parameter-level distributional distances.

Reviewer comment (Novelty):
The submitted paper does not bring a novel and significantly more effective idea; the evaluation of prior attacks could be more thorough, and simple non-adaptive baseline attacks should be removed from the main text in favor of evaluation against stronger attacks, as weak baselines make results look much better.

Author response:
Reasons for focusing on the patch and blended attack in the main experiments.

Characterizing the patch and blended baselines as "very weak" isn't entirely accurate. While they are weak against input-level and dataset-level detectors, they remain a standard choice of attack in the model-level detection literature. While we do compare to other attacks like WaNet and LIRA, we are not aware of any other attacks that are widely used in model-level detection papers. The patch (BadNets) and blended attacks remain the norm, so using them for our main experiments is a natural choice.

Reviewer comment (Experiments):
There are recent improvements over K-ARM, and more recent methods might be more effective against the proposed attack; the authors should do a better literature search to find the most effective SOTA defenses.

Author response:
Please note that **we do actually include the PixelBackdoor method of Tao et al. [6] that you recommend including.** This is one of the many detectors in our main evaluations. In total, we include 8 model-level detectors, including 5 published detectors. This is far more than any other work we are aware of, and includes a broad range of different detection strategies, including scanning methods like Neural Cleanse and PixelBackdoor, baseline detectors like a specificity check, and adaptive methods that train against our trojaned networks including MNTD and Param. Thanks to your suggestion, we will add the Wang et al. [7] attack to the camera-ready paper, which is similar to the Neural Cleanse, ABS, and K-Arm, and PixelBackdoor detectors that we already include.

Reviewer comment (Experiments):
There is no evaluation of removal-based defenses such as NC or MNTD, which are promising even against strong backdoor attacks, and which are within the paper’s threat model; it would be important to see whether increased effectiveness against reverse engineering makes the attack easier to remove.

Author response:
(As an aside, please note that MNTD does not require a clean dataset. It requires a dataset of clean and trojaned *networks*, but the queries are optimized from scratch.)

Some research in the trojan detection community focuses on developing new attacks to evade the entire pipeline of trojan defenses. However, so far these works have not considered a wide range of model-level detectors, including training-based detectors like MNTD. Our paper does not attempt to evade the entire pipeline of defenses, including input-level, dataset-level, and model-level detection, and trojan removal. Rather, we are studying the specific problem of evading model-level detection, which hasn't been extensively studied before.

Thus, we do not include an extensive comparison to other types of defenses, including dataset-level or trojan removal defenses. In fact, since we fix the triggers in our experiments to highly visible triggers, we are confident that simple dataset-level methods like Activation Clustering would perform quite well on our attacks. Similarly, it is very possible that our method makes trojans easier to remove. However, this is not the focus of our contribution. Combining our model-level evasion method with methods for evading other types of defenses would be a large undertaking, so we leave this to future work.

Reviewer comment (Experiments):
The datasets used to evaluate the method are too small.

Author response:
Most neural trojan papers only train a handful of models, or a few hundred at most. Investigating model-level detection requires training datasets of models. We train over 6,000 models in total, with each individual number in our tables representing hundreds of trained models. Each model requires approximately an hour to train (more for Vision Transformers on GTSRB). Thus, for computational reasons we decided to focus on 32x32 datasets that are commonly used in the model-level detection literature.

Reviewer comment (Experiments):
The model architecture for evaluation is also unclear. The parameter distance between two architectures is computed in your new evasive loss (Sec. 4.1) and I don’t think this is computationally scalable for large models.

Author response:
The evasion loss is very computationally efficient, requiring only an L2 distance computation for each parameter vector. This is comparable to the compute required for L2 regularization.

The evasion loss does incur a memory overhead for loading the model, since it requires loading an "anchor model" fixed to the initialization. For the ResNet-50 scale models that we use (described in Section 5), this is inconsequential. However, for LLM-scale models it could present some moderate challenges, although it would certainly still be a viable attack.

Reviewer comment (Experiments):
Only 4 datasets are given in the evaluation, and all of them have input size less than 32x32 (GTSRB is downsampled to 32x32 as mentioned on Page 6). I don’t think this is enough to prove the method is effective.

Author response:
Most neural trojan papers only train a handful of models, or a few hundred at most. Investigating model-level detection requires training datasets of models. We train over 6,000 models in total, with each individual number in our tables representing hundreds of trained models. Each model requires approximately an hour to train (more for Vision Transformers on GTSRB). Thus, for computational reasons we decided to focus on 32x32 datasets that are commonly used in the model-level detection literature.

Reviewer comment (Evaluation):
A lot of redundant evaluations and figures in the paper. For example, Figure 3, Table 1, and Table 2 are telling the same thing with minor changes in evaluation metrics. Also, the abnormal performance that exists in the results is not fully explained. For example, why `Param` shows better detection rate in `standard trojan` compared to `evasive trojan`?

Author response:
You are correct that Figure 3 and Table 2 show two views of the same data. Table 2 is a table of AUROC values, and Figure 3 shows individual ROC curves for three representative experiments in this table. It's standard to show ROC curves alongside AUROC values, as the full curves can provide helpful context as to the shape of the tradeoff and whether one method is always better than the other, or whether this is only true for certain recall and FPR values.

Table 1 shows different numbers: the ASR of the trojans and accuracy of the trojaned models on clean inputs without triggers. This demonstrates that our evasive trojans have similar ASR and accuracy to standard trojans and clean networks, which is important for ensuring that the comparisons between standard and evasive trojans are fair. If we have addressed the thrust of your concerns, we kindly ask that you consider raising your score.

**Reason for Param detector's high performance.**

In Appendix B.1, we show that removing our randomization loss causes the Param detector to obtain 100% AUROC. In fact, the Param detector's strong performance in preliminary experiments is what led us to develop the randomization loss and discover the issue of emergent coordination that we show in Figure 4. Even after including our randomization loss, the Param detector performs better on our Evasive Trojans than on Standard Trojans, although the difference is much smaller than before. We have updated the paper to clarify this. Thank you for your suggestion.