Paper: Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective (ICLR-2024)

Reviewer comment (Theory):
The established theoretical framework predominantly caters to kernel-based dataset distillation methods, and its application seems restricted primarily to the initially proposed KIP method. Given the complexities associated with computing the NTK, KIP's practicality has been empirically questioned. While subsequent kernel-based dataset distillation methods capable of distilling comprehensive datasets have emerged (e.g., [1]), this paper falls short of validating their compatibility with the introduced framework. This oversight not only raises concerns about the paper's soundness but also limits the practicability of the introduced attack method.

Author response:
We understand KIP/NTK has known computational drawbacks. However,
1. Through KIP/NTK, we can theoretically understand the effect of data distillation and backdoor. Our new findings can measure the performance of each trigger with a closed-form formula and induce two theory-driven triggers which can survive from the dataset distillation. Advanced kernel based methods lose theoretical clarity.

2. Practically, we find that our findings on this theoretical framework extend to more advanced approaches, such as FRePo (Dataset Distillation using Neural Feature Regression. NeurIPS, 2022) and DM (Dataset condensation with distribution matching. WACV, 2023). The experimental results show that our triggers can successfully transfer to the FRePo and DM.

| Trigger-type | Dataset | IPC (Image Per Class)} | Distillation | Model | CTA (%) | ASR (%) |
| ------------ | ------- | ---------------------- | ------------ | ----- | ------- | ------- |
| Cifar10 | simple-trigger | 10 | FRePO | ConvNet | 60.32 | 83.10 |
| Cifar10 | relax-trigger | 50 | FRePO | ConvNet | 68.34 | 81.61 |


| Trigger-type | Dataset | IPC (Image Per Class)} | Distillation | Model | CTA (%) | ASR (%) |
| ------------ | ------- | ---------------------- | ------------ | ----- | ------- | ------- |
| Cifar10 | simple-trigger | 10 | DM |MLP | 36.41 | 77.03 |
| Cifar10 | simple-trigger | 50 | DM |MLP | 36.88 | 76.79 |
| Cifar10 | relax-trigger | 10 | DM |MLP  | 36.31 | 76.04 |
| Cifar10 | relax-trigger | 50 | DM |MLP  | 36.81 | 76.21 |

Overall, we believe we made two major contributions on extending the frontier of backdoor analysis for dataset distillation, 
1. Deep theoretical understanding. 
2. The conclusion hold empirically on sota dataset distillation, such as FRePo and DM.

Reviewer comment (Theory):
Regarding Equation 9, why is the objective of the KIP-based backdoor attack to minimize the empirical loss of $f_{\mathcal{S}^*}$ on either $\mathcal{D}_A$ or $\mathcal{D}_B$, rather than simultaneously reducing the empirical loss on both $\mathcal{D}_A$ and $\mathcal{D}_B$ as your statement about "Backdoor Attack" (The next to the last paragraph above Equation 7)? While you attempt to address this in Equation 10 by introducing $\tilde{D}=D_A \cup D_B$ to establish an upper bound on the loss of $f_{\mathcal{S}^*}$ with respect to $D$, this formulation seems somewhat unreasonable to me.

Author response:
The objective of the KIP-based backdoor attack is to minimize the projection loss and conflict loss, which is based on Equation 10. Although $D$ can be $D_A$ or $D_B$, the inequality (Equation (10) in the paper) holds for both $D=D_A$ and $D=D_B$. Hence, if we can ensure the projection loss and conflict loss are sufficiently low, we can simultaneously ensure the empirical loss (empirical risk) of $\mathcal{D}_A$ and the empirical loss (empirical risk) of $\mathcal{D}_B$ are also sufficiently low. In Equation (10), we introduce $\tilde{D} = D_A \cup D_B$ in order to derive the common upper bound. Assume $\ell(f, (x, y)) = \|f(x)-y\|^2_2$. Equation (10) can be derived by following process, 

$$
\mathbb{E} _{(x, y) \sim D} \ell(f _{\mathcal{S}^*}, (x, y))= \frac{1}{N} \sum _{(x, y) \sim D} \ell(f _{\mathcal{S}^*}, (x, y))
$$

$$
\leq\frac{1}{N} \sum _{(x, y) \sim \tilde{D}} \ell(f _{\mathcal{S}^*}, (x, y))
$$

$$
 = \frac{N_A + N_B}{N}[\frac{1}{N_A + N_B}\sum _{(x, y)\sim \tilde{D}} \ell(f _{\mathcal{S}^*}, (x, y))]
$$

$$
=\frac{N_A + N_B}{N} [\min _{\mathcal{S}} \mathbb{E } _{(x,y)\sim\tilde{D}} \ell(f _{\mathcal{S}}, (x, y))]
$$

$$
 =\frac{N_A + N_B}{N}
[\min _{\mathcal{S}} \mathbb{E} _{(x,y) \sim \tilde{D}}
\|f _{\mathcal{S}}(x) - y\|_2^2]
$$

$$
\leq
\frac{N_A + N_B}{N} [\min _{\mathcal{S}}
 \mathbb{E} _{(x,y)\sim\tilde{D}}
 \|f _{\mathcal{S}}(x) - f _{\tilde{D}}(x) + f _{\tilde{D}}(x) - y\| _2^2]
$$

$$
\leq\frac{N_A + N_B}{N}
[\min _{\mathcal{S}}
\{\mathbb{E} _{(x,y)\sim\tilde{D}} \|f _{\mathcal{S}}(x) - f _{\tilde{D}}(x)\|_2^2 
+
\mathbb{E} _{(x,y)\sim\tilde{D}}
 \| f _{\tilde{D}}(x) - y\|_2^2\}]
$$

Notice that the above inequality holds for both $D = D_A$ and $D=D_B$. Hence, the right hand side of Equation 10 is a common upper bound for empirical risk of $\mathcal{D}_A$ ($\mathbb{E} _{(x, y)\sim D_A} \ell(f _{\mathcal{S}^*}, (x, y))$) and empirical risk of $\mathcal{D}_B$ ($\mathbb{E} _{(x, y)\sim D_B} \ell(f _{\mathcal{S}^*}, (x, y))$).

Reviewer comment (Theory):
It appears that the introduced projection loss can be directly optimized with respect to the trigger $T$. What's the rationale behind setting an upper bound and optimizing the projection loss through this bound? Does this approach offer computational benefits?

Author response:
Recall that the projection loss is defined as
 
$$
\min _{\mathcal{S}} \mathbb{E} _{(x, y) \sim \tilde{D}} \ell(f _{\mathcal{S}}, (x, f _{\tilde{D}}(x)))
$$

which measures the information loss when we compress the information from larger dataset $\tilde{D}=D_A \cup D_B$ to smaller dataset $D _{\mathcal{S}}$. In our proposed method (relax-trigger), we constrain that $\mathcal{S} = \mathcal{S}_A$. We can get

$$
\min _{\mathcal{S}} \mathbb{E} _{(x, y)\sim\tilde{D}}
\ell(f _{\mathcal{S}}, (x, f _{\tilde{D}}(x)))
\leq\mathbb{E} _{(x, y) \sim \tilde{D}}\ell(f _{\mathcal{S}_A}, (x, f _{\tilde{D}}(x))).
$$

However, we think that directly optimize the right hand side of above inequality has some drawbacks. The optimal trigger T can induce a model $f _{\tilde{D}}$ such that $f _{\tilde{D}}(x) \approx f _{\mathcal{S}_A}(x)$ for all $(x, y) \in \tilde{D}$. Nevertheless, the $\|f _{\tilde{D}} - f _{\mathcal{S}_A}\| _{\mathcal{H} _k}$ may still very large. In other words, $f _{\tilde{D}}$ may behave very differently with $f _{\mathcal{S} _A}$ when the input is out of the distribution of $\tilde{D}$. Hence, we think that such a dataset $\tilde{D}$ may compress very limited information into $\mathcal{S}_A$. To deal with this issue, we introduce the projection lemma to derive the upper bound of projection loss. By the projection lemma, we first perform orthogonal decomposition on $f _{\tilde{D}}$ as 

$$
f _{\tilde{D}} = f _{1} + f _{2},
$$

where $f _{1}\in\mathcal{H} _{\mathcal{S} _A} = span [\{ k( \cdot, x) | (x, y) \in \mathcal{S}_A \} ]$ and  $f _{2} \in \mathcal{H} _{\mathcal{S} _A}^{\perp}$. $f_1$ is considered as the model that which can be well-expressed by $\mathcal{S} _A$, while $f _2$ is considered as the model irrelevant with $\mathcal{S} _A$. Hence, we utilize the $f _2$ to construct the upper bound of projection loss. We hope that $f _2$ will tends to $0$ as we optimize the trigger $T$. We believe that the trigger $T$ derived by optimizing our upper bound of projection loss can make $f _{\tilde{D}}$ well compressed into $\mathcal{S} _A$. This is the rationale behind the upper bound of projection lemma. 

As for the computational benefits, we do not observe apparent computational benefits or drawbacks when we utilize the upper bound of projection loss to derive the trigger. However, we believe that our theory-driven approach can induce a better trigger as we mentioned above.

Reviewer comment (Experiments):
Based on my W3, could you share additional experimental evidence to validate the efficacy of your proposed triggers when applied to models with alternative architectures trained on the synthesized datasets?

Author response:
In our intuition, as we enlarge the size of synthetic dataset, the cross model ability should be relieved. Besides, some existing works (Neural tangent generalization attacks. ICML, 2021) have shown that NTK possesses certain cross model ability. So, we think that the issue of cross model ability can be controlled by tuning the size of the synthetic dataset and choosing suitable NTK. The rigorous behavior of cross model ability related to the size of synthetic dataset is expected to be discussed in the future. The experimental results show that both CTA and ASR go up as we increase the IPC (Image Per Class), which suggests that the cross model issue may be relieved as the IPC is large enough.

The experiment for cross model ability is presented as follows. We train the distilled dataset poinsoned by simple-trigger and relax-trigger on 3-layers MLP and 3-layers ConvNet.

| Dataset   | Trigger-type | IPC (Image Per Class) | Cross_model | CTA (%)      | ASR(%)    |
| -------------- | ------- | --- | ----------- | ------------ | ------------|
| CIFAR-10| simple-trigger  | 10  | MLP         | 11.58 (2.10) | 40.00 (48.98)  |   
| CIFAR-10| simple-trigger  | 10  | CNN         | 47.37 (7.44) | 40.00 (48.98) |     
| CIFAR-10| simple-trigger  | 10  | NTK (baseline)        | 41.70 (0.25)  | 100.00 (0.00)   |     
|                |         |     |             |              |             |    
| CIFAR-10| simple-trigger  | 50  | MLP         | 48.08 (4.72)  | 40.00 (48.98)    |     |
| CIFAR-10| simple-trigger | 50  | CNN         | 95.96 (1.10)  | 100.00 (0.00) | 
| CIFAR-10| simple-trigger | 50  | NTK (baseline)         | 43.66 (0.40)  | 100.00 (0.00)   |   
|                |         |     |             |              |             |
| CIFAR-10| relax-trigger | 10  | MLP         | 10.52 (7.44)  | 19.40 (38.80) | 
| CIFAR-10| relax-trigger | 10  | CNN         | 64.21 (6.98)  | 81.80 (11.44) |    
| CIFAR-10 | relax-trigger | 10  | NTK (baseline)         | 41.66 (0.74)  | 100.00 (0.00) |  
|                |         |     |             |              |             |  
| CIFAR-10 | relax-trigger | 50  | MLP         | 44.24 (4.49)  | 78.28 (24.57)|  
| CIFAR-10| relax-trigger | 50  | CNN         | 93.13 (2.24)  | 82.80 (6.53)  |
| CIFAR-10| relax-trigger | 50  | NTK (baseline)         | 43.64 (0.40)  | 100.00 (0.00) |

Reviewer comment (Writing):
Some sections could benefit from more straightforward explanations to make the paper more accessible to readers not deeply familiar with the subject matter.

Author response:
We will use more straightforward examples to explain the concept of some terms, like conflict loss and projection loss.

1. For conflict loss, the conflict loss is defined as 
$$
 \mathbb{E} _{(x, y)\sim\tilde{D}} \ell(f _{\tilde{D}}, (x, y)),
$$
    where $\tilde{D} = D_A \cup D_B$.
    The conflict measures the information conflict between $D_A$ and $D_B$. For example, we consider a dog/cat picture classification problem. In the dataset $D_A$, we label the dog pictures with $0$ and label the cat pictures with $1$. However, in the dataset $D_B$, we label the dog pictures with $1$ and label the cat pictures with $0$. It is clear that the model trained on $\tilde{D}$ must perform terribly on the dataset either $D_A$ or $D_B$. In this case, the information between $D_A$ and $D_B$ have strong conflict and the conflict loss would be large.

2. For projection loss, the projection loss is defined as 
$$
\min _{\mathcal{S}} \mathbb{E } _{(x, y)\sim
\tilde{D}} \ell(f _{\mathcal{S}}, (x, f _{\tilde{D}}(x)))\nonumber.
$$
    The projection loss reflects the natural information loss when we compress a large dataset into a small dataset. Take writing an abstract for example. If we want to write a 100 word abstract to  describe a 10000 words article, the abstract may suffer some lack of semantics to some degree. Such a phenomena also happens for dataset distillation. When the information of a large dataset is complex enough, the information loss for dataset distillation will be significant; When the information of a large dataset is very simple, it is possible that there is only very limited information loss. We introduce the projection loss defined above to measure this phenomenon.

Reviewer comment (Experiments):
The datasets evaluated in the paper appear to be limited in scope. Typically, researchers conduct experiments on more comprehensive datasets like ImageNet, or other comparable datasets, to convincingly demonstrate the effectiveness of a proposed attack method.

Author response:
Following the reviewer's suggestion, we additionally perform our KIP-based backdoor attack on ImageNet. The ASR in our experiments can achieve 100%, which suggests our simple-trigger and relax-trigger are both effective on ImageNet. Here, we randomly choose ten sub-classes to perform our experiment. We also resize each image in the ImageNet into 128x128.

| Trigger-type   | Dataset  |Model |IPC (Image Per Class) | CTA (%) | ASR (%) |
| -------------- | -------- |--- |--------------------- | ------- | --- |
| simple-trigger | ImageNet |NTK|10                    | 15.00   | 100.00    |
| simple-trigger | ImageNet |NTK|50                    | 16.60   | 100.00   |
| relax-trigger  | ImageNet |NTK|10                    | 16.40   | 100.00   |
| relax-trigger  | ImageNet |NTK|50                    | 17.00    | 100.00  |

Reviewer comment (Experiments):
Are there some potential defense methods during the dataset distillation process to mitigate backdoor attacks?

Author response:
We think there are two scenarios for KIP-based backdoor attack:

1. **The distillation process is totally controlled by the attacker.**
Under this scenario, the victim does not have any chance to perform defense during the dataset distillation.

2. **The distillation process is controlled by the attacker, but the victim can update the dataset during the distillation process.** 
Under this scenario, the victim may send his/her dataset class by class to the attacker during the distillation process. By this approach, we can turn the dataset distillation process into a time-varying version. A research result (On the Permanence of Backdoors in Evolving Models. arXiv, 2023, https://arxiv.org/abs/2206.04677) has shown that the backdoor can be removed in the time-varying model. We think that this could be a potential defense. Nevertheless, whether we can extend this concept to distillation-based backdoor attack is still an open problem.

Reviewer comment (Experiments):
Given the variety of dataset distillation methods available, could the choice of distillation method potentially impact the conclusions drawn about the efficacy of the proposed attack method?

Author response:
The choice of distillation method can influence the efficacy of our attack method to some degree. However, in our paper, we can find that KIP faithfully obeys the optimization problem of performance-matching dataset distillations. So, KIP should be a good surrogate of all performance-matching distillation methods. Hence, our KIP-based backdoor attack should remain valid and demonstrate similar behaviors for all performance-matching dataset distillations. 

For other types of dataset distillations, it depends on whether there exists a kernel which can well approximate the framework of other types of distillations. If we can find such a kernel, then KIP can be a surrogate. Nonetheless, how to construct such a kernel is still an open problem. To the best of our knowledge, there is no literature about this research direction that provides kernel approx for other types of distillation.


We perform an experiment for this problem. We first utilize our triggers to poison the datasets. Then, we distill these datasets with different distillation methods, FRePo (Dataset Distillation using Neural Feature Regression. NeurIPS, 2022) and DM (Dataset condensation with distribution matching. WACV, 2023). The experimental results show that our triggers can successfully transfer to the FrePo and DM.


| Trigger-type | Dataset | IPC (Image Per Class)} | Distillation | Model | CTA (%) | ASR (%) |
| ------------ | ------- | ---------------------- | ------------ | ----- | ------- | ------- |
| Cifar10 | simple-trigger | 10 | FRePO | ConvNet | 60.32 | 83.10 |
| Cifar10 | relax-trigger | 50 | FRePO | ConvNet | 68.34 | 81.61 |


| Trigger-type | Dataset | IPC (Image Per Class)} | Distillation | Model | CTA (%) | ASR (%) |
| ------------ | ------- | ---------------------- | ------------ | ----- | ------- | ------- |
| Cifar10 | simple-trigger | 10 | DM |MLP | 36.41 | 77.03 |
| Cifar10 | simple-trigger | 50 | DM |MLP | 36.88 | 76.79 |
| Cifar10 | relax-trigger | 10 | DM |MLP  | 36.31 | 76.04 |
| Cifar10 | relax-trigger | 50 | DM |MLP  | 36.81 | 76.21 |

Reviewer comment (Experiments):
Can the proposed attacks evade other data distillation techniques (e.g., gradient matching based methods and distribution matching based methods)? It would further strengthen the experimental evaluation by examining the transferability of the proposed attacks.

Author response:
This is a very insightful question! Our proposed attacks can evade other data distillation techniques. In particular, we perform experiments to examine the transferability of our theory-induced triggers. We first use our simple-trigger and relax-trigger to poison the dataset. Then, we distill dataest with a different distillation method, FRePo (Dataset Distillation using Neural Feature Regression. NeurIPS, 2022) and DM (Dataset condensation with distribution matching. WACV, 2023). The experimental results shows that our triggers can successfully transfer to the FrePo and DM.

The rationale is that KIP faithfully obeys the optimization problem of performance-matching dataset distillations except for our assumption that the models lie in the Reproducing Kernel Hilbert Space (RKHS). So, KIP can be a surrogate for all performance-matching dataset distillations. We believe that our KIP-based backdoor attack should evade all performance-matching dataset distillations, as suggested by our theoretical analysis.  As for other types of dataset distillations, it depends on whether there exists some kernel-like approximation or interpretation to the considered dataset distillation method. If we can find such a kernel-like approximation, then KIP should be a valid surrogate. However, how to find or construct the kernel is still an open question. To the best of our knowledge, there is no literature about this research direction that provides kernel approximation for other types of distillation. 

| Trigger-type | Dataset | IPC (Image Per Class)} | Distillation | Model | CTA (%) | ASR (%) |
| ------------ | ------- | ---------------------- | ------------ | ----- | ------- | ------- |
| Cifar10 | simple-trigger | 10 | FRePO | ConvNet | 60.32 | 83.10 |
| Cifar10 | relax-trigger | 50 | FRePO | ConvNet | 68.34 | 81.61 |


| Trigger-type | Dataset | IPC (Image Per Class)} | Distillation | Model | CTA (%) | ASR (%) |
| ------------ | ------- | ---------------------- | ------------ | ----- | ------- | ------- |
| Cifar10 | simple-trigger | 10 | DM |MLP | 36.41 | 77.03 |
| Cifar10 | simple-trigger | 50 | DM |MLP | 36.88 | 76.79 |
| Cifar10 | relax-trigger | 10 | DM |MLP  | 36.31 | 76.04 |
| Cifar10 | relax-trigger | 50 | DM |MLP  | 36.81 | 76.21 |

Reviewer comment (Theory):
In Equation (9), why the second term is called the generalization gap?

Author response:
Given a model $f_{\mathcal{S}^*}: \mathcal{X}\rightarrow\mathcal{Y}$, a dataset $D\sim\mathcal{D}^N$, the generalization gap is defined as 
$$
        \mathbb{E} _{(x,y)\sim\mathcal{D}} \ell(f _{\mathcal{S}^*}, (x, y))
        - \mathbb{E} _{(x, y) \sim
        - D} \ell(f _{\mathcal{S}^*}, (x, y)),
$$
where $\ell(\cdot, \cdot)$ is some loss function, $\mathbb{E} _{(x, y)\sim\mathcal{D}} \ell(f _{\mathcal{S}^*}, (x, y))$ is called risk and $\mathbb{E} _{(x, y) \sim D} \ell(f _{\mathcal{S}^*}, (x, y))$ is called empirical risk. This mathematical formulation suggests the gap between the performances of $f _{\mathcal{S}^*}$ applied on $D$ and $\mathcal{D}$, respectively. The above equation (Equation (9) in the paper) also indicates whether the performance of $f _{\mathcal{S}^*}$ on $D$ can be generalized to $\mathcal{D}$. Hence, we call the second term in Equation (9) as generalization gap.