Paper: Sparse-Guard: Sparse Coding-Based Defense against Model Inversion Attacks (ICLR-2024)

Reviewer comment (Experiments):
The evaluation is rather limited since it only conducts experiments on MNIST and Fashion MNIST, both datasets which are easy to fit by a network due to the overall low sample variance. Finding meaningful shared features in the sparse code layers is rather easy for the model. Also, the samples contain no private information at all. The evaluation should also contain more complex dataset evaluations, e.g., the common CelebA dataset, to prove that the approach is also usable within more complex tasks. Also, repeating the experiments with different seeds to provide a standard deviation of the results would make the evaluation more reliable.

Author response:
***Re: limited experiments due to only 2 datasets (MNIST and Fashion MNIST) in datasets:***

- We have now **doubled the datasets to 4 including the high-res CIFAR-10 dataset** and a dataset of medical images that represents a realistic attack setting. We also add a new attack and multiple new defenses as described below. In sum, we have quadrupled the number of experiments. ***Across all (quadrupled) experiments, Sparse-Guard now outperforms benchmarks by a factor of up to 704. It also has the best PSNR (the most important metric) across ~every single experiment~.*** In one single experiment, it loses to the Wang et al. defense in terms of SSIM by a factor of 0.001, while significantly outperforming it in terms of PSNR and FID.

Reviewer comment (Experiments):
The overall evaluation setting seems a bit strange. I understand the split-network setting and that reconstructing inputs given only the intermediate activations indeed can be a privacy breach. But why should the adversary have access to the activations of the training samples? I think a more realistic evaluation should consider unseen (test) samples and then try to reconstruct those given the intermediate activations.

Author response:
***Re: evaluation setting is strange because we allow the adversary access to training samples to reconstruct instead of unseen test samples:***

-  This aspect of our setup is identical to the standard setups in the benchmarks we consider (Titcomb et al. 2021, Gong et al., 2023). Per Section 2, our goal is to consider settings that capture ‘worst-case’ black-box attacks with a powerful attacker. Specifically, We suppose that the attacker builds the inverted model using a held-out set of training data. Then, we suppose that the attacker has access to the *training data*'s intermediate network outputs, which he can submit to his inverted model to reconstruct training examples. Of course, you're right that this setting supposes that the attacker has obtained some leaked data. The fact that Sparse-Guard is robust in this setting suggests that it is a good defense even in the `worst-case' with a powerful attacker. In real-world settings, an attacker may have to approximate the training data's intermediate outputs, but that would make an attack more difficult (and also allow for more subjective decisions in terms of how to set this up fairly for experiments).

Reviewer comment (Experiments):
Moreover, I think the approach should also be evaluated on common MIAs that utilize GANs to reconstruct training samples based only on the network's weights, e.g., [1, 2, 3]. Otherwise, the defense mechanisms should be positioned only for split-network (and federated learning) settings. Also, the approach should be compared to related information bottleneck defense approaches [4,5].

Author response:
***Re: evaluating on new MIA's that use GAN's, and evaluating on new defenses:***

- Thank you. We **added the new Plug and Play attack** that you and the other reviewers suggested (the newest of the attacks you suggest), and we also added **two new defenses**: the Wang et al. defense you suggest and the ***new differential-privacy guaranteeing DP-SGD defense of Hayes et al.** that is currently under development at Google DeepMind and Meta AI, and that is currently the only defense with provable guarantees for model inversion attacks. Appendix tables 8-13 show all of these new results. 
- **We run all experiments and all defenses on all 4 datasets (instead of 2) and then re-run them under the Plug-and-play attack.** This new set of experiments on *[4 datasets, 6 benchmarks, 2 attacks, 3 settings]* is now **significantly more comprehensive than other recent papers** including the Plug and Play paper (which evaluates 3 datasets and 3 benchmarks).
- ***Across all (quadrupled) experiments, Sparse-Guard now outperforms benchmarks by a factor of up to 704. It also has the best PSNR (the most important metric) across ~every single experiment~.*** In one single experiment, it loses to the Wang et al. defense in terms of SSIM by a factor of 0.001, while significantly outperforming it in terms of PSNR and FID.

Reviewer comment (Novelty):
I think the overall technical contribution is rather low since the approach seems to be simply re-using the sparse coding layer framework of Rozell et al. (2008) and demonstrating that such networks can also act as a defense against MIAs. I still think the direction of the paper is interesting but the technical novelty seems limited.

Author response:
***Re: whether the technical contribution is rather low since the approach seems to be simply re-using the sparse coding layer framework of Rozell et al. (2008):***

- We are using the new (2022) extension of Rozell sparse coding to convolutional networks, which modifies the the Rozell  2008 update rule (see [1]) to extend it to the convolutional setting. However, Rozell and subsequent sparse-coding architectures including [1] consider a single sparse coded layer. **To our knowledge, our paper is the first time an architecture with intermittent sparse layers has been proposed.** We compare the 'standard' new convolutional architecture of [1] with our intermittent convolutional architecture (the standard one is called `Sparse-Standard' in all of our tables), and we find that our intermittent architecture significantly outperforms it. We thank you for raising this issue and we have revised our paper to clarify (1) the fact that we are using the new convolution sparse-coding extension rather than the 2008 approach as well as (2) the novelty of our architecture compared to a standard sparse-coding architecture.
     
        [1] Teti et al., Lateral Competition Improves Robustness Against Corruption and Attack (ICML, 2022)

Reviewer comment (Presentation):
The related work is comprehensive but mixes up different model inversion settings and approaches. For example, the [1] proposes MIAs that try to reconstruct features from specific classes by optimizing the latent vectors of a GAN. It uses the target model for guidance (and there exist much more works in this line of research, e.g., [2,3]). This is a completely different setting from the one investigated by the paper, which uses the intermediate activations of training samples to train a decoder-like model. I think a clearer separation between different types of MIAs would make the related work part stronger. Also, mixing works investigating the memorization of training samples in LLMs with vision-based inversion attacks might be confusing to the reader.

Author response:
***Re: the related work is comprehensive but...." [we should clarify settings/approaches and aspects like images vs. LLM's:***

- Thank you for this suggestion -- we have revised this section.

Reviewer comment (Presentation):
Table captions should be above the table (Table 1).

Author response:
***Re: space after Table 2 should be increased:***

- Thank you -- we have fixed this. In general, we noticed many more spacing issues with this year's ICLR template compared to previous years (and also many more than ICML & NeurIPS templates). We are not sure what is causing this, but have looked through the paper to make sure there are no more compressed whitespaces.

Reviewer comment (Presentation):
The space after Table 2 should be increased (and manipulating the spaces might even run counter to the official guideline!).

Author response:
***Re: space after Table 2 should be increased:***

- Thank you -- we have fixed this. In general, we noticed many more spacing issues with this year's ICLR template compared to previous years (and also many more than ICML & NeurIPS templates). We are not sure what is causing this, but have looked through the paper to make sure there are no more compressed whitespaces.

Reviewer comment (Experiments):
How much longer does it take to train a network using the Sparse-Guard architecture compared to a model without it?

Author response:
***Re: How much longer does it take to train a network using the Sparse-Guard architecture compared to a model without it?:***

- Sparse-Guard is equal or faster than benchmarks on all experiments except the case of MNIST under the Plug-and-Play attack, so we include a table of the timings for this 'worst-case' experiment in Appendix table 14. In that 'worst-case' experiment, Sparse-Guard offers a significantly better defense than all benchmarks, and it is still faster than Gong et al.'s GAN-based defense, but its 11% slower than Wang et al.

Reviewer comment (Evaluation):
Why is the FID metric valid to evaluate the privacy leakage? Generally, we are interested in how well a single sample can be reconstructed and less about recovering the overall feature distribution.

Author response:
***Re: Why is the FID metric valid?***

- FID is one of the standard metrics for the evaluation of inversion attacks, and it is used in (for example) the Plug and Play paper you suggest. It computes whether the reconstructed images are 'overall-distributionally-similar' to the originals by computing their distributional distance, rather than e.g. pixel-by-pixel distance. It also captures an element of how a reconstructed image 'generally looks like' a training image: in other settings, FID is also often used to evaluate whether fake images generated by GAN's 'look like' they could be real.

Reviewer comment (Experiments):
The attacks used in this study do not represent state-of-the-art techniques [1, 2, 3], and the baseline defense methods employed also fall short of the current state-of-the-art [4].

Author response:
**Thank you for your constructive review---we believe we address your remaining reservations and would appreciate it if you would improve your score in light of our response.**

The main concern and reason for your score of 3 seems to be whether Sparse-Guard's performance advantage holds against additional defense baselines, additional new attacks, and additional datasets. We now show that it does:

**Appendix Tables 10, 11, & 13 add two additional state-of-the-art benchmark *defenses*:**

- We added the ***Information Regularization defense of Wang et al.*** suggested by Reviewer ETXH;
- We also added ***very recent DP-SGD defense of Hayes et al.*** just released by Deepmind and Meta AI, which is currently the only defense with provable guarantees for model inversion attacks;

**Appendix Tables 12 and 13 re-run all experiments using the *Plug-and-Play* attack you suggested**. **Sparse-Guard outperforms benchmarks by a factor of up to 6.7 under this novel attack. In particular, it offers better PSNR (the most important metric in the literature) across all combinations of datasets and defenses.

**We also doubled the number of datasets in our experiments.** Specifically, **[Appendix tables 8-13]** rerun **all** defenses, attacks, and experiment settings with the original 2 datasets as well as:

- The **hi-res CIFAR-10** dataset ;
- An additional dataset of **medical images**. 

In sum, we have **quadrupled our experiments**, and we now show that Sparse-Guard outperforms alternatives across multipple times more combinations of attacks, and defenses, and datasets compared to recent papers such as the Plug and Play paper you suggested.

**We also thank you for your comments that our sparse-coding approach is novel**, as well as your positive comments about our public model inversion defense PyTorch codebase.

Reviewer comment (Experiments):
The study relies solely on synthetic datasets like MNIST and FMNIST, lacking the inclusion of real-world datasets, such as facial recognition data, which could enhance the practical relevance of the findings.

Author response:
***We strongly feel that a new defense that is both practically effective on so many (now 5!) different datasets (including faces) and that also introduces a novel research direction in the privacy attack setting is of immediate interest both to other researchers and also to companies like Google and Facebook, which are actively developing public products for sensitive application domains using available model inversion defenses.***

Reviewer comment (Writing):
The paper does not do a good job at the exposition of how sparse coding is implemented. This is especially important as the implementation here seems to be *convolutional* sparse coding and differs from traditional sparse coding where matrix multiplication rather than a convolution is applied. e.g. (Bristow, Hilton, Anders Eriksson, and Simon Lucey. "Fast convolutional sparse coding." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013.)

Author response:
**Thank you for this constructive feedback---We have greatly revised *Section 3: Sparse-Guard Architecture* where we introduce sparse coding and the LCA algorithm to provide a much more clearer picture of our methods.** In particular, the update rule provided in equation 3 describes the Rozell update for the convolutional case, which was used in previous work [1,2] and validated against other non-Rozell-based convolutional sparse coding solvers (https://sporco.readthedocs.io/en/latest/examples/csc/index.html). Although the original Rozell sparse coding algorithm was not introduced in the convolutional setting, it can readily be adapted to convolutional networks because it is based on the general principle of feature-similarity-based competition between neurons, which is agnostic to different neural network architectures. As a result, the original Rozell formulation can be readily adapted to the convolutional setting by modifying just two terms in the update rule. The first term, which is represented as $\Psi(t)$ in Equation 2 of our revised paper, is computed via a matrix multiplication between the transposed dictionary matrix and an input vector in the original Rozell formulation. In the convolutional case, the matrix multiplication becomes a convolution between the input and the convolutional dictionary. The second term that needs to be adapted for the convolutional setting is where the similarity between each feature and every other feature is computed, given by $\Omega * \Omega$ in our revised version. In the original Rozell formulation, this was done with a matrix multiplication between the transposed dictionary and the non-transposed dictionary. In the convolutional setting, it can be done with a convolution between the dictionary and itself. We also agree with the reviewer that the general 'spatiotemporal' description we referred to was potentially confusing in our setting so we have replaced it with a more precise description. **We thank the reviewer for pointing out the opportunity to improve our description of our sparse-coding approach.**
     
    -   [1] Teti et al., Lateral Competition Improves Robustness Against Corruption and Attack
    -   [2] Kim et al., Modeling Biological Immunity to Adversarial Examples

Reviewer comment (Theory):
Rozell et al. 2008 were cited for the update rule. However, in that paper, the update rule was not given for convolutional sparse coding. Either a more explicit derivation for the update rule can be given or a different citation would be relevant.

Author response:
**Thank you for this constructive feedback---We have greatly revised *Section 3: Sparse-Guard Architecture* where we introduce sparse coding and the LCA algorithm to provide a much more clearer picture of our methods.** In particular, the update rule provided in equation 3 describes the Rozell update for the convolutional case, which was used in previous work [1,2] and validated against other non-Rozell-based convolutional sparse coding solvers (https://sporco.readthedocs.io/en/latest/examples/csc/index.html). Although the original Rozell sparse coding algorithm was not introduced in the convolutional setting, it can readily be adapted to convolutional networks because it is based on the general principle of feature-similarity-based competition between neurons, which is agnostic to different neural network architectures. As a result, the original Rozell formulation can be readily adapted to the convolutional setting by modifying just two terms in the update rule. The first term, which is represented as $\Psi(t)$ in Equation 2 of our revised paper, is computed via a matrix multiplication between the transposed dictionary matrix and an input vector in the original Rozell formulation. In the convolutional case, the matrix multiplication becomes a convolution between the input and the convolutional dictionary. The second term that needs to be adapted for the convolutional setting is where the similarity between each feature and every other feature is computed, given by $\Omega * \Omega$ in our revised version. In the original Rozell formulation, this was done with a matrix multiplication between the transposed dictionary and the non-transposed dictionary. In the convolutional setting, it can be done with a convolution between the dictionary and itself. We also agree with the reviewer that the general 'spatiotemporal' description we referred to was potentially confusing in our setting so we have replaced it with a more precise description. **We thank the reviewer for pointing out the opportunity to improve our description of our sparse-coding approach.**
     
    -   [1] Teti et al., Lateral Competition Improves Robustness Against Corruption and Attack
    -   [2] Kim et al., Modeling Biological Immunity to Adversarial Examples

Reviewer comment (Writing):
The sentence "The learned spatiotemporal representation closest to input image X is represented by this sparse presentation R_X" is confusing. Why is it a spatiotemporal representation? Where is the temporal element, all of the inputs are static images. Should 'sparse presentation' also be sparse representation?

Author response:
**Thank you for this constructive feedback---We have greatly revised *Section 3: Sparse-Guard Architecture* where we introduce sparse coding and the LCA algorithm to provide a much more clearer picture of our methods.** In particular, the update rule provided in equation 3 describes the Rozell update for the convolutional case, which was used in previous work [1,2] and validated against other non-Rozell-based convolutional sparse coding solvers (https://sporco.readthedocs.io/en/latest/examples/csc/index.html). Although the original Rozell sparse coding algorithm was not introduced in the convolutional setting, it can readily be adapted to convolutional networks because it is based on the general principle of feature-similarity-based competition between neurons, which is agnostic to different neural network architectures. As a result, the original Rozell formulation can be readily adapted to the convolutional setting by modifying just two terms in the update rule. The first term, which is represented as $\Psi(t)$ in Equation 2 of our revised paper, is computed via a matrix multiplication between the transposed dictionary matrix and an input vector in the original Rozell formulation. In the convolutional case, the matrix multiplication becomes a convolution between the input and the convolutional dictionary. The second term that needs to be adapted for the convolutional setting is where the similarity between each feature and every other feature is computed, given by $\Omega * \Omega$ in our revised version. In the original Rozell formulation, this was done with a matrix multiplication between the transposed dictionary and the non-transposed dictionary. In the convolutional setting, it can be done with a convolution between the dictionary and itself. We also agree with the reviewer that the general 'spatiotemporal' description we referred to was potentially confusing in our setting so we have replaced it with a more precise description. **We thank the reviewer for pointing out the opportunity to improve our description of our sparse-coding approach.**
     
    -   [1] Teti et al., Lateral Competition Improves Robustness Against Corruption and Attack
    -   [2] Kim et al., Modeling Biological Immunity to Adversarial Examples

Reviewer comment (Evaluation):
Multiple claims in the paper are made about sparse coding “removing unnecessary private information”. This claim is not really supported by any study. In fact, the empirical study concluded that the effect of sparse coding layers is an "unclustering effect". How the conclusion of jettisoning unnecessary information is drawn is unclear. What is considered unnecessary information in the first place? In fact, it would be interesting to see if any other algorithm that produces the same unclustering effect will provide a similar effectiveness in defense against model inversion attacks.

Author response:
We mean to say that the representations that are produced via sparse coding only contain information that is relevant for reconstructing a denoised (a.k.a. less detailed) version of the input, and these representations are much sparser and less detailed than those in standard models. As a result, we were attempting to point out that the attacker has much less information to use when performing model inversion attacks against Sparse-Guard compared to standard models. Put differently, a 'perfect attacker' could only ever hope to reconstruct a sparse representation of an input image under our defense *in the worse case*, meaning that even a perfect attacker could *not hope to* reconstruct many input image details that are dropped by sparse-coding. By 'unnecessary private information', we mean input image details that we may desire to keep private but that are not necessary for the task of training a high-accuracy classifier. Specifically, sparse-coding networks are known to sparsify inputs and network layers without losing significant classification accuracy.

    The unclustering effect we observe in our empirical analysis is the direct result of this approach. Specifically, a standard (non-sparse) network trained on dense/noisy/detailed input images results in many partly-redundant network features that are `noisy versions of each other'. Networks with many noisy/partly redundant features are easier to attack, because an adversary can 'home in on' clusters of such features in order to 'home in on' an input image it wants to reconstruct. Our sparse-coding approach's de-noising effect tends to prevent the network from learning many partly-redundant features, resulting in the de-clustering effect we observe in Fig. 4. The resulting de-clustered features are intuitively less susceptible to gradient-based inversion attacks.

Reviewer comment (Experiments):
The major drawback is that the experiments are only conducted on simple, low-resolution datasets.

Author response:
**Thank you for your constructive review. We believe we address your remaining reservations here and would appreciate it if you would improve your score in light of our response.**

The main concern and reason for your score of 3 seems to be your statement that "The major drawback is that the experiments are only conducted on simple, low-resolution datasets." We've taken your advice and:

- **Re-run all experiments on CIFAR-10 as well as an additional dataset of medical images** [Appendix Tables 8-13]. **Sparse-Guard outperforms benchmarks by a factor of up to 173.9 on CIFAR-10**.
- Re-run all experiments on all 4 datasets using the **Plug-and-Play** attack you suggested [Appendix Tables 12-13]. **Sparse-Guard outperforms benchmarks by a factor of up to 6.6 under this attack.**
- We also added 2 more state-of-the-art defense benchmarks including the new differential-privacy guaranteeing DP-SGD defense from Google DeepMind and Meta AI [Appendix Tables 10, 11, & 13].

***Across all (quadrupled) experiments, Sparse-Guard now outperforms benchmarks by a factor of up to 704. It also has the best PSNR (the most important metric) across every single experiment.*** 

We absolutely agree with your conclusion that "an effective defense would be significant" on CIFAR-10 and similar hi-res data, and we believe we have addressed this concern.

We also thank you for your comments that the paper is "well motivated and easy to follow" and your positive comments about our PyTorch codebase.

Reviewer comment (Novelty):
There exist many model inversion attacks capable of extracting high-resolution data, from CIFAR-10, CelebA, to ImageNet.

Author response:
**Thank you for your constructive review. We believe we address your remaining reservations here and would appreciate it if you would improve your score in light of our response.**

The main concern and reason for your score of 3 seems to be your statement that "The major drawback is that the experiments are only conducted on simple, low-resolution datasets." We've taken your advice and:

- **Re-run all experiments on CIFAR-10 as well as an additional dataset of medical images** [Appendix Tables 8-13]. **Sparse-Guard outperforms benchmarks by a factor of up to 173.9 on CIFAR-10**.
- Re-run all experiments on all 4 datasets using the **Plug-and-Play** attack you suggested [Appendix Tables 12-13]. **Sparse-Guard outperforms benchmarks by a factor of up to 6.6 under this attack.**
- We also added 2 more state-of-the-art defense benchmarks including the new differential-privacy guaranteeing DP-SGD defense from Google DeepMind and Meta AI [Appendix Tables 10, 11, & 13].

***Across all (quadrupled) experiments, Sparse-Guard now outperforms benchmarks by a factor of up to 704. It also has the best PSNR (the most important metric) across every single experiment.*** 

We absolutely agree with your conclusion that "an effective defense would be significant" on CIFAR-10 and similar hi-res data, and we believe we have addressed this concern.

We also thank you for your comments that the paper is "well motivated and easy to follow" and your positive comments about our PyTorch codebase.