Paper: Targeted Model Inversion: Distilling Style Encoded in Predictions (ICLR-2024)

Reviewer comment (Experiments):
Dependence on a surrogate generator (and dataset).

Author response:
**W1+Q1. How would the attacker know which dataset is similar?**

For a target ML service for users, it is trivial to know what kind of image they receive as input (face/medical/scenery/animal/...), the meaning of their output (similarity to celebrities/medical diagnosis/geolocation/...), and the characteristics of the output prediction (number of classes, meaning of each class) with little effort. Based on this information, the adversary can choose a pre-trained model or a dataset to use among numerous pre-trained models and datasets readily available on the Internet (e.g., Kaggle alone provides 268,605 datasets across 165 official subjects, including people, health, genetics, and business.). For instance, assume a scenario where the target service implements  geolocation estimation based on photos of scenery [10]. The service can predict which country or continent you are in based on the input photo. Accordingly, the attacker can decide to search for datasets containing landscape images (such as the Google landmarks dataset or InstaCities1M) or related pretrained generators for MI attacks.

We emphasize that our adversary does not require additional knowledge compared to other prior MI attacks; our adversary leverages an auxiliary dataset that the adversaries that previous black- and white-box MI attacks [1-7] assumed.

Reviewer comment (Theory):
Increased space-complexity.

Author response:
**W2+Q2. Imposes an additional space constraint, in terms of the large stylegan that is to be trained**

We agree that training a StyleGAN network is a computational hurdle. However, building this StyleGAN network is a one-time procedure. After this preparation phase, MI attacks are completely offline while enabling high-fidelity input reconstruction for a large number of queries.

Reviewer comment (Theory):
Sloppy notations and incomplete math.

Author response:
**Q5+Q6. Incomplete math.**

We appreciate the suggestion and will replace our math formulations in Eq2 and Eq3 with the formal equations as follows: 
- Eq2. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (w-m'\_\\theta(\\hat{y}))^2\\right]$
- Eq3. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (\\hat{y}-f'\_\\theta(x))^2\\right]$

Reviewer comment (Experiments):
Adding to the above point, the proposed method imposes an additional space constraint, in terms of the large stylegan that is to be trained.

Author response:
**W2+Q2. Imposes an additional space constraint, in terms of the large stylegan that is to be trained**

We agree that training a StyleGAN network is a computational hurdle. However, building this StyleGAN network is a one-time procedure. After this preparation phase, MI attacks are completely offline while enabling high-fidelity input reconstruction for a large number of queries.

Reviewer comment (Experiments):
Given the above two points, I do not see the comparisons to be fair as the proposed method has the luxury of using an additional full-blow generator network which the previous methods don't have. Therefore, I recommend that the Authors have to try modifying the other methods while they have access to a generator, albeit on a surrogate dataset.

Author response:
**Q3. Unfair comparison; proposed method has the luxury of using additional full-blow generator network which the previous methods don't have**

We note that all existing MI attacks [3-9] that utilize an image prior assume the same setting. Among the methods selected for baseline comparison, AMI is the only method that does not involve using a generator network, but still, it requires an auxiliary dataset to train the inverse mapping of the target model. One noticeable difference is that the auxiliary dataset is no longer needed for our TMI attacks that utilize an image prior, if there is a available pre-trained generator.  However, AMI does not support exploiting the pre-trained model.

Reviewer comment (Experiments):
The method seems very stylegan specific. Can a different GAN architecture be used? I am asking specifically because, in my experience, the latent spaces of other GANs are not as versatile as that of StyleGAN.

Author response:
**Q4. Can a different GAN architecture be used?**

| | **Acc@1 ↑** | **Acc@5 ↑** | **F-dist↓** | **Cover ↑** |
|-|:-:|:-:|:-:|:-:|
| TMI | .3804 | .6255 | .2950 | .2067 |
| TMI with UNet-GAN | .2201 | .4811 | .3254 | .2063 |
| MIRROR-b | .2026 | .4533 | .3564 | .0613 |

Yes, it is possible to utilize different GAN architectures since every GAN incorporates a latent space and a discriminator network. We conducted an additional experiment with UNet-GAN [11] pretrained on FFHQ and compared the results to the base experiment in the table above. For the surrogate model $f’$, we used the left-half of the UNet-GAN’s discriminator. Since UNet-GAN does not incorporate a mapping network, a custom mapping layer was trained from scratch. Observe that while the overall performance is slightly decreased compared to TMI with StyleGAN, it is still more effective than MIRROR-b. The slight drop in performance is largely due to the entangled latent space of UNet-GAN, where the latent space $\\mathcal{Z}$ is directly used without mapping it to an intermediate disentangled latent space $\\mathcal{W}$ in advance. We conclude that while StyleGAN is still the most effective image prior to be utilized, other GAN-based methods can generally benefit from our suggested approach of leveraging a new mapping layer and distilling the discriminator for a surrogate model. Therefore, the contribution of TMI is not only limited to the packaged attack method as a whole, but also the individual components that can be applied independently. The techniques from TMI might also be able to augment state-of-the-art diffusion models for MI attacks. For example, the mapping network can serve as a reliable encoder for latent diffusion models. We believe that designing an MI attack upon diffusion models is plausible, however, out of scope in this paper.

Reviewer comment (Writing):
The description of the method has to be more formal. For instance, before Eq. 2, it is said that there is a loss function that is getting minimized. The optimization problem has to be stated neatly (using formal math).

Author response:
**Q5+Q6. Incomplete math.**

We appreciate the suggestion and will replace our math formulations in Eq2 and Eq3 with the formal equations as follows: 
- Eq2. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (w-m'\_\\theta(\\hat{y}))^2\\right]$
- Eq3. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (\\hat{y}-f'\_\\theta(x))^2\\right]$

Reviewer comment (Theory):
A lot of mathematical details are missing - What is the expectation over, in Eq. 2 and 3? As I understand, they both are over two different distributions but are not mentioned.

Author response:
**Q5+Q6. Incomplete math.**

We appreciate the suggestion and will replace our math formulations in Eq2 and Eq3 with the formal equations as follows: 
- Eq2. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (w-m'\_\\theta(\\hat{y}))^2\\right]$
- Eq3. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (\\hat{y}-f'\_\\theta(x))^2\\right]$

Reviewer comment (Experiments):
While the Authors show a few images when there is a distributional shift between the surrogate model and the predictor, it is very minor and insignificant in my opinion. Both the datasets are facial images. What happens if you take a stylegan trained on cars dataset and use it for a predictor trained on human faces?

Author response:
**Q7. What happens if you take a stylegan trained on cars dataset and use it for a predictor trained on human faces?**

Using an unrelated dataset will render TMI less effective, since its reconstruction domain is limited to the image manifold of the prior modeled in the generator. However, please note that as mentioned in **W1+Q1**, to the attacker’s perspective, it is probable to find a dataset or pretrained StyleGAN specific enough for TMI, given the abundance of public datasets and repositories. In addition, prior works on MI often use subsets from the target network’s trainset to train their inversion model, whereas our base evaluation for TMI utilize relevant, but different datasets. We believe that this adversary setting is an acceptable norm in the model inversion literature.

Reviewer comment (Experiments):
What is an example scenario where we do have access to the full prediction vector (e.g., the probabilities and not just the most likely class) but we (a) do not have the input and (b) we have limited queries to the model?

Author response:
**W1. Example scenarios**

Consider an age classifier that gets a face-image query and classifies it into one of five classes that correspond to age groups, the adversary can populate a list of arbitrary prediction vectors that represent each class and then conduct TMI attacks to reconstruct the facial images by conducting TMI attacks. When this age classifier is hosted on an MLaaS cloud instance, the owner of this cloud instance should limit an excessive number of queries due to their computation and financial constraints. We explored this scenario in Section 5.3 (see Figure 4).

We also described several example scenarios in Section 3 in which the adversary obtains prediction vectors. These scenarios include users posting prediction results on social media simply for entertainment (e.g. celebrity look-alike apps such as StarByFace show the look-alike percentage to celebrities), medical professionals sharing diagnosis predictions for educational or consultative purposes, split inference settings where the inference result is disclosed to different parties [1] or an untrusted MLaaS server. Prior studies have addressed countermeasures to protect inference privacy by obfuscating [2,3] or encrypting the input [4]. However, the prediction vector is left in plaintext format.

Reviewer comment (Evaluation):
How many queries do you have to perform to the surrogate model? I'd imagine that this can be just as expensive as querying the original model (if compute is the issue, I'm not sure "query budget" is the right metric to optimize).

Author response:
**W2. How long does this method take + Q1. How many queries to the surrogate model?**

| | **Query** | **Time (minutes)** |
|-|-|-|
| TMI | 5k (offline) | 2.38 |
| AMI | 0 | $\\sim$0 |
| MIRROR-b | 10k | 0.18 |
| RLB-MI | 80k | 67 |
| LO-MI | 25k | 1.91 |
| P&P | 34k | 2.69 |
| MIRROR-w | 160k | 72.67 |

We compared the number of queries and the computation time per each attack in the table above. Please note that the required time is highly dependent to the baseline implementations and their own GAN prior. By default, TMI queries the surrogate model the same number of times to the update steps. As stated in A.1.2, we used $n=5000$ for our experiments. For other baselines, we assigned a default query budget that the respective attack assumed in their paper. We note that TMI with 5K updates exhibited the superior performance over all black-box baselines, as shown in Table 2. In the revised paper, we will include these experimental results. We emphasize that leveraging a surrogate model contributes to substantially decreasing the number of black-box queries directed at a target model (please refer to Section 5.3 for more details). Please note that the adversary can conduct MI attacks in an *offline* fashion by abusing this surrogate model, thus circumventing potential constraints associated with online queries. Consequently, the target service provider may remain unaware of whether they are under attack or not.

Reviewer comment (Experiments):
How important is the surrogate model? If you used the original model instead (ignoring the query budget), how much better are your images?

Author response:
**Q2. Replacing the surrogate model with the original (ignoring the query budget)**
| | **Performance gain** |
|:-:|:-:|
| Acc@1 ↑ | .3408 → .8264 |
| Acc@5 ↑ | .6255 → .9534 |
| F-dist ↓ | .2950 → .1975 |
| Cover ↑ | .2067 → .2619 |

We appreciate this valuable suggestion. We conducted an additional experiment using the original model instead of the surrogate model, and the performance gains are listed in the table above. As expected, the performance is improved by an average of 63.7%, outperforming even the white-box attacks across all four metrics. This also indicates that retraining a mapping network can be applied for conducting  white-box TMI attacks to improve their reconstruction performance. We will include this result in A.2.1. Effect of the Surrogate Model.

Reviewer comment (Experiments):
Is this method specific to GANs (could you adapt one of the current off-the-shelf diffusion models for example).

Author response:
**Q3. Is this method specific to GANs?**

Although our paper is focused on StyleGAN, the contribution of TMI is not only limited to the packaged attack method as a whole, but also the individual components that can be applied independently. For example, the mapping network might also serve as a reliable encoder for latent diffusion models. We believe that designing an MI method using a diffusion models is also promising. However, we consider exploring this direction out of scope for this paper.

Reviewer comment (Experiments):
**Image Prior** An unacknowledged weakness of the paper is the generality of the approach to a broader set of application domains. By choosing a StyleGAN prior, the applicability of the current method becomes restricted to the domains on which (or domains related to) to the StyleGAN's training distributions. The evaluations, and experiments -- while impressive are of less impactful in my opinion.

Author response:
**W1. Applicability of the current method restricted to domains related to the StyleGAN's training distributions**

We acknowledge that TMI necessitates a StyleGAN prior model. However, all previous MI studies [1-9] have assumed access to an auxiliary dataset of which underlying distribution is either the same or slightly different from the distribution of the target dataset. Therefore, we proposed a new MI method that harnesses this auxiliary dataset in a black-box manner, achieving superior MI performance over the SoTA methods.

In addition, we note that our TMI evaluation spans two distinct application domains, namely chest X-ray and facial datasets. This suggests that, given an appropriate StyleGAN prior, TMI is applicable to diverse classification tasks for MI. Given the abundance of pre-trained models and datasets readily available on the internet (e.g., Kaggle alone provides 17,678 image datasets across 164 official subjects, including people, healthcare, genetics, and business.), we believe that the adversary is capable of selecting their preferred StyleGAN prior for their needs. Even when there is no available dataset or pretrained StyleGAN, the adversary still has an option to collect their own images and train a StyleGAN specific to their need. Image search or sharing services (such as Google Images, Flickr, and Pinterest) allow attackers to access a large volume of image data.

Reviewer comment (Experiments):
Two potential mitigation strategies come to mind -- (a) in order to work with more SoTA foundation models like StyleGAN-XL, the current approach will require to work with conditional generative models, which can make it much more potent and realistic; or (b) Leverage stronger inversion techniques on _unconditional_ StyleGAN that are able to invert arbitrary OOD images using pre-trained styleGAN as well -- for example 
	* GAN inversion for out-of-range images with geometric transformations, CVPR'21
	* Image2stylegan++: How to edit the embedded images?, CVPR'21
	* Improved StyleGAN-v2 based Inversion for Out-of-Distribution Images, ICML'22

Author response:
**W2. Explore leveraging advanced GAN techniques**

We will discuss potential improvements by integrating SoTA models and the connection between GAN inversion to MI attacks. GAN inversion typically requires an original image to find a suitable latent point, whereas our TMI attacks do not assume white-box access to the target model or access to the original image. Leveraging a prediction output without the original image, TMI performs the best for identifying the optimal latent point that contributes to reconstructing the input image among the SoTA MI methods.

Reviewer comment (Experiments):
The examples of images that are "significant" deviation from the original dataset, unfortunately are not that OOD. It's well known that most encoders (including pSp or e4e ) do a reasonable job of inverting these paintings and other closely related domains. So i think a more accurate test of TMI will be to use a much more generic dataset, which will likely fail.

Author response:
**W3. more accurate test of TMI will be to use a much more generic dataset, which will likely fail**

Using a much more generic dataset will render TMI less effective, since its reconstruction domain depends on the image manifold of the prior modeled in the generator. However, please note that as mentioned in **W1**, to the attacker’s perspective, it is probable to find a dataset or pretrained StyleGAN specific enough for TMI, given the abundance of public datasets and repositories.
In addition, prior works on MI often use subsets from the target network’s trainset to train their inversion model, whereas our base evaluation for TMI utilize relevant, but different datasets. We believe that this adversary setting is an acceptable norm in the model inversion literatures. Also note that TMI assumes a weaker black-box adversary with a limited number of input queries.

Reviewer comment (Evaluation):
The feature distance measure appears to be poorly correlated to image quality -- i think the metric maybe misleading since an approach like AMI, which arguably fails on most of the evaluations conducted, has a reasonable F-dist score comparable or better than some of the other baselines which are clearly better.

Author response:
**W4. Feature distance poorly correlated to image quality**

We appreciate this valuable insight and agree that feature distance alone is insufficient to demonstrate reconstruction performance. Therefore, we used four metrics to comprehensively measure the MI performance of TMI and the other baselines.  We will discuss the downsides of using feature distance alone in measuring MI performance.

Reviewer comment (Experiments):
Can the ablations on query budget be done for one of the baselines as well to see how well (or badly) they behave as the budget decreases?

Author response:
**Q1. Ablations on query budget**

|  | **Query budget** | **Acc@1↑** | **Acc@5↑** | **F-dist↓** | **Cover↑** |
|-|-:|:-:|:-:|:-:|:-:|
| **P&P** | 5k    | .101 | .284 | .393 | .041 |
|  | 10k    | .104 | .299 | .390 | .046 |
|  | 50k    | .156 | .364 | .373 | .060 |
|  | 100k    | .153 | .388 | .365 | .048 |
|  |  |  |  |  |  |
| **MIRROR-w** | 5k    | 0 | .013 | .485 | .020 |
|  | 10k    | .026 | .071 | .460 | .031 |
|  | 50k    | .259 | .478 | .365 | .059 |
|  | 100k    | .354 | .609 | .341 | .062 |
|  |  |  |  |  |  |
| **TMI** | 5k    | .232 | .448 | .340 | .121 |
|  | 10k    | .274 | .540 | .318 | .144 |
|  | 50k    | .358 | .646 | .286 | .182 |
|  | 100k    | .341 | .626 | .295 | .207 |

We conducted additional experiments to assess the performance of the baseline methods across varying query budgets (5, 10, 50, 100K). Please note that the query budget denotes the total number of queries allowed to attack every label (i.e., 530 attacks in case of FaceScrub). We included the experimental results of two best-performing baseline methods (i.e., P&P and MIRROR-w) and our TMI attack in the table above. The graphical representations of the performance of all baseline methods can be found ***[here](https://drive.google.com/file/d/1q_2EXRYds6srSQQ2gETbRreVYIjDJo1n)***. As the table above shows, the overall MI performance increases as the number of queries increases. TMI consistently outperforms the baseline attacks across different query budgets. We also note that TMI holds a distinctive advantage of requiring zero queries to a target model after the preparation phase, which significantly facilitates the reconstruction of inputs for a large number of prediction vectors.

During the experiments, we identified an error in the query count of LO-MI in Table 1. For each attack attempt, LO-MI requires 25k queries, instead of 2k. This will be corrected in the final version. We confirm that this erroneous typo does not impact the overall conclusion of our study since we overstated the performance of the baseline attack (i.e., LO-MI). Correcting this value highlights the superior performance of TMI attacks.

Reviewer comment (Writing):
A (non-technical) question -- is the use of gendered language for describing the attacker common? 

>"..where the adversary _(Eve)_ is able to.."  
>".. *She* uses $D_{aux}$ to train their *her* StyleGAN network".. 

I found that a bit odd but i am not closely familiar with how this is done in security and safety research so I will defer to the authors.

Author response:
**Q3. Use of gendered language for describing the attacker**

We will replace those words with gender-neutral terms. We note that prior studies in the security and privacy domain have often employed gender-specific terms when referring to the adversary.