Paper: Ensembler: Combating model inversion attacks using model ensemble during collaborative inference (ICLR-2024)

Reviewer comment (Theory):
The threat model in this paper is not compelling. The authors claim that "it has auxiliary information on the architecture of the entire DNN, as well as a dataset in the same distribution as the private training dataset used to train the DNN". In real world, it is unclear how to obtain the information for the attackers. Also, the authors assumed the server has reasonably large computation resources but no detailed discussion or analysis on it. Thus, it is unknown how practical it is for the threat model. Without valid threat model, it is unclear the usefulness of this paper.

Author response:
Dear reviewer,

We want to thank you for your valuable suggestions, and we try to address your concerns here.

The reviewer questions on the threat model and practicability of the proposed framework.

For threat models, we are not able to comment on how to get those auxiliary information. However, when considering the possible attacks and the related defenses, it is always good to consider the extreme cases. This is why we assume that 1. Server somehow guesses the dataset and model architecture correctly. 2. The client only holds 1 layer for head and 1 layer for tail. Actually, the first assumption is not as hard as it seems. For the model architecture, since the client has very limited computing resources, the server could just fill the top layers with convolutional layers, and last layers with dense layers. It has a “high chance” of getting it correctly. For the dataset, the server could just use the standard open-sourced training datasets. Since it has large computing resources, it could afford the price to train on a few of them, which could match the one used by the client. Of course, if the server guesses incorrectly, then this is to the advantage of the client as the reconstruction quality will be much worse, as shown by [1].

[1] Zecheng He, Tianwei Zhang, and Ruby B. Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC ’19, pp. 148–162, New York, NY, USA, 2019.

Reviewer comment (Experiments):
The practicality of implementing "Ensembler" in a real-world scenario is somewhat unclear, given the inherent complexity and computational overhead of maintaining multiple models and the distillation process. Also, this paper mainly target on small-scale image dataset such as CIFAR-10. It is unclear whether their findings or method can be generalized to large-scale settings. Thus, the practicality is damaged.

Author response:
Dear reviewer,

We want to thank you for your valuable suggestions, and we try to address your concerns here.

The reviewer questions on the threat model and practicability of the proposed framework.

For threat models, we are not able to comment on how to get those auxiliary information. However, when considering the possible attacks and the related defenses, it is always good to consider the extreme cases. This is why we assume that 1. Server somehow guesses the dataset and model architecture correctly. 2. The client only holds 1 layer for head and 1 layer for tail. Actually, the first assumption is not as hard as it seems. For the model architecture, since the client has very limited computing resources, the server could just fill the top layers with convolutional layers, and last layers with dense layers. It has a “high chance” of getting it correctly. For the dataset, the server could just use the standard open-sourced training datasets. Since it has large computing resources, it could afford the price to train on a few of them, which could match the one used by the client. Of course, if the server guesses incorrectly, then this is to the advantage of the client as the reconstruction quality will be much worse, as shown by [1].

We have added some experiments on larger scale datasets. Please see the response to all reviewers for details.


[1] Zecheng He, Tianwei Zhang, and Ruby B. Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC ’19, pp. 148–162, New York, NY, USA, 2019.

Reviewer comment (Theory):
Need to improve or provide justification on the threat model and practicality.

Author response:
Dear reviewer,

We want to thank you for your valuable suggestions, and we try to address your concerns here.

The reviewer questions on the threat model and practicability of the proposed framework.

For threat models, we are not able to comment on how to get those auxiliary information. However, when considering the possible attacks and the related defenses, it is always good to consider the extreme cases. This is why we assume that 1. Server somehow guesses the dataset and model architecture correctly. 2. The client only holds 1 layer for head and 1 layer for tail. Actually, the first assumption is not as hard as it seems. For the model architecture, since the client has very limited computing resources, the server could just fill the top layers with convolutional layers, and last layers with dense layers. It has a “high chance” of getting it correctly. For the dataset, the server could just use the standard open-sourced training datasets. Since it has large computing resources, it could afford the price to train on a few of them, which could match the one used by the client. Of course, if the server guesses incorrectly, then this is to the advantage of the client as the reconstruction quality will be much worse, as shown by [1].

We have added some experiments on larger scale datasets. Please see the response to all reviewers for details.


[1] Zecheng He, Tianwei Zhang, and Ruby B. Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC ’19, pp. 148–162, New York, NY, USA, 2019.

Reviewer comment (Theory):
My biggest issue is contrived threat model in this work. The server is assumed to be potentially malicious (which is reasonable) but at the same time, the proposed defense requires the server to train multiple models. If the server is indeed malicious, why would it want to do so? Even if it does, how can the client know if the server is malicious in its part of the models as well. For instance, it could shuffle or modify the outputs of its server nets such that only some of them are useful for the client model.

Author response:
The reviewer’s major concern seems to be on the threat model. We want to apologize for any confusions, but also to point out that our threat model targets specifically on collaborative inference. We made it clear in Section 2.1 that we assume the training phase is secure. This is a reasonable setting because training and deployment have different pipelines. For training of ML models, individual and small companies could buy a single GPU and train on their own machine. This is a one-time cost, and it is usually not as expensive. But deployment of ML service is continuous, which often requires a cloud server (which could be malicious). Concretely, if I want to build an image-classification app with privacy features, I could buy one GPU and train my model locally. Then, I would rent a cloud server that offers computation, and connects the app to the cloud. In this case, the app serves as the client, and the cloud is the malicious server. 

The reviewer also asks about the server shuffling and modifying the server nets. We want to strengthen that we consider a semi-honest adversary attacker. The definition of semi-honest attacker is that the semi-honest party has to follow the predefined protocols. It can curiously seek extra information, but it cannot change the pipeline.

Reviewer comment (Theory):
If the server is indeed malicious, a very simple attack can be: server sends N queries to the client posing as a user, observes intermediate outputs of client's model (sent to it normally), and then also observes the final output posing as the user. This can make reconstruction attacks (perhaps via shadow models) much more potent.

Author response:
Finally, the reviewer asks about the server sending queries to the client. This is called a “black-box” attack. We have provided discussion on works related to this area. We also want to point out that our work focuses on “query-free” attacks, which explicitly prohibits the server querying the client to get the direct relationship between input and intermediate output. Concretely, using the example I gave above, unless the server is a member of the app users, it will not have access to querying the service. How to protect the adversary server becoming a member of the app is out of the concern of this paper, but a possible suggestion is to take extra care when renting the cloud service so that the server is not able to trace back to the app.

Reviewer comment (Experiments):
**The experiments are really insufficient** in a sense that the evaluated scenario is limited (e.g. only evaluating on CIFAR-10 with ResNet-18). In section 3, when studying how model splitting strategies affect the effectiveness of model inversion attacks, only a single qualitative result is shown in each case (a total of 3 samples, one for each of VGG16, ResNet18 and English to German translator). In section 5, the proposed Ensembler defense is evaluated only on CIFAR-10 with ResNet-18 architecture.

Author response:
Weakness1: The experiments are really insufficient

The reviewer wants to see more experiments on both Section 3 and 5. Due to time and resource limits, we are not able to expand Section 3. However, Section 3 serves mainly as a complementary component to our proposed framework. All we want to say in Section 3 is that the conclusion drawn from previous papers is correct. “The deeper the first split point is, the worse the reconstructed image is.” On the other hand, the introduction of a second split point has no benefits to the defense. Section 3 will help us eliminate some of the concerns on the usage of two split points in our framework. While this is a necessity for Ensembler, it is seldom used before, making some people wonder if the increased protection should be attributed to this second split point.

As discussed in the response to all reviewers, we have included more experiments in Section 5. We want to thank the reviewer for giving very detailed suggestions on using ImageNet or Tiny-ImageNet. We tried to use Tiny-ImageNet, but due to resource constraints, we anticipate that we will not be able to finish the experiments by the deadline. Thus, we consider the two datasets. CIFAR100 has equivalent intermediate feature size, using a modified network, with Tiny-ImageNet; and CelebA-HQ has similar intermediate feature size with ImageNet. Due to resource limits, we could not consider more architectures, but we do believe that the proposed model is agnostic to architecture choices.

Reviewer comment (Evaluation):
**The proposed defense is neither well-explained nor well-investigated.** The motivation behind the proposed design is not clear: Why/How would this increase the difficulty of model inversion attacks? (1) Intuitively, each of the N group of middle layers obtained by the server are intentionally biased during training (through adding a fixed perturbation to its input). Since it is assumed that the attacker has access to a dataset with similar distribution, is it possible that such bias can be reversed by adding another learnable perturbation to the input of these middle layers? (2) Since the actual model uses only P out of N models, could the attacker also learn to select a subset of models that maximize the reconstruction quality? **It is important to have an adaptive attack** by incorporating both (1) and (2) in evaluating the proposed defense (unless one formally proves its effectiveness). Based on current supports from the paper, the proposed defense offers only complexity but not security.

Author response:
Weakness2: The proposed defense is neither well-explained nor well-investigated.

We address the intuition of the proposed framework in the response to all reviewers. The reviewer asks two specific questions, and we will address them here:
Is it possible that such bias can be reversed by adding another learnable perturbation to the input of these middle layers?
Yes. In fact, the Gaussian noise could easily be approximated by a few convolutional layers. This is what we did for the shadow network. For Gaussian noise, we use 2 convolution layers with kernel size of 3 and 64 filters to approximate it. From single/Dr-single, we see that it works well.
However, the whole purpose of the Gaussian kernel in the first stage of training is to ensure that each network, especially the server nets, has distinctive weights, yet remaining good accuracy. The key of the proposed framework is not about Gaussian noise. Rather, it is an algorithm that generates 2^N potential correct answers with N times complexity.
Could the attacker also learn to select a subset of models that maximize the reconstruction quality?
We address this in the response to all reviewers. Each possible combination of server nets would result in a different $M_{c,h}$, which gives good classification accuracy. However, only one of them is the “correct” one that maximizes the reconstruction quality. The shadow model is not able to learn the reconstruction quality because it does not know which one is the “correct” one.

Reviewer comment (Presentation):
The paper spends too many pages on background introduction. Only until page 6, the authors start to introduce their own approaches. Also, the paper particularly mentions the privacy of NLP in Section 3.2, but this seems to be irrelevant to the paper. The evaluation is also performed on Cifar-10 with ResNet-18, which can confuse the audience a lot.

Author response:
Thank you very much for your suggestion. We have moved the NLP part to the Appendix. We still want to include them as NLP is an increasingly important area, and privacy in this area should also be researched.

Reviewer comment (Experiments):
The evaluation is weak. Only ResNet-18 and Cifar10 are considered. It is also unknown how the effectiveness of the proposed approaches varies with different choices of N and P.

Author response:
We apologize for the weak evaluation presented in the paper. We have provided two additional experiments on datasets with larger intermediate feature size. Regarding the second part of the concern, P does not affect the model’s performance in accuracy. We have experiments with P=3,4,5 for three datasets, and the proposed framework performs as good as the unprotected network. N is associated with the time complexity for inference and reconstruction. As discussed in the response to all reviewers, the proposed framework has a lower-bound inference complexity of N, with hypothesized reconstruction complexity of 2^N.

Reviewer comment (Evaluation):
Since multiple networks are used in the approaches, the redundancy of computation and the efficiency cost would be a concern. The authors should carefully discuss it.

Author response:
We have now included the discussion of time complexity in the main text.