Paper: The Fundamental Limits of Least-Privilege Learning (ICLR-2024)

Reviewer comment (Experiments):
The study of data attribute leakage is an active research field. Some work attempts to mitigate attribute leakage by designing sophisticated algorithms. This paper does not mention and compare cutting-edge defense schemes in the empirical experiment part.

Author response:
**Comparison to other learning techniques:** The review rightly notes that the literature suggests a wide range of learning techniques to address attribute leakage in deep network models. As we show in _Section 3.2, Theorem 2,_ however, the trade-off between a representation’s utility and achieving the LPP applies “regardless of the way the feature representations are obtained” (_Section 3.2, page 6_). In Section 4, we provide an empirical evaluation that supports our theoretical results: Regardless of a model's learning task and even under censoring, the LPP cannot be fulfilled while simultaneously providing high utility for the prediction task.

We furthermore note that to the best of our knowledge none of the existing techniques in the literature can actually achieve the desired goal of LPL. To do so, the features learned by the model would need to simultaneously restrict leakage for any sensitive attribute $S$. Techniques to defend against attribute inference are mostly limited to one or two fixed sensitive attributes and hence cannot achieve this goal.

Reviewer comment (Experiments):
More extensive experiments should be used to evaluate the proposed theory. At present, this work is only conducted on a dataset and a neural network model.

Author response:
We have uploaded a revised version of our paper that contains additional experiment results that confirm that, as predicted by our main theoretical result, the least-privilege and utility trade-off holds for any feature representation. In particular:
- In Appendix C.2, we show that the trade-off holds regardless of the learning technique used to obtain the feature map $f_E(X) = Z$. We show results for two more techniques that aim to hide sensitive information about their input data: differentially private training and maximum entropy adversarial representations learning (MAX-ENT). We chose to evaluate the trade-off under MAX-ENT as Zhao et al., 2020 report hat together with gradient reversal (our first choice for previous experiments), MAX-ENT provides the best trade-off between hiding a particular sensitive attribute and task accuracy.
 - In Appendix C.1, we show that the trade-off holds regardless of a model's architecture.
- In Appendix C.3, we show that the trade-off equally applies to a very different type of data, i.e., on a tabular dataset.

We hope that these revisions address your comments in a satisfactory way.

[Zhao et al., 2020: https://proceedings.neurips.cc/paper/2020/file/6b8b8e3bd6ad94b985c1b1f1b7a94cb2-Paper.pdf]

Reviewer comment (Novelty):
Advanced defense strategies should be discussed. If advanced defense strategies can solve the problem of data attribute leakage, then the value of this work will be limited. Therefore, could the author provide a more adequate overview of the research area discussed? It’s not just data attribute leaks, it should also include cutting-edge defense methods.

Author response:
**Comparison to other learning techniques:** The review rightly notes that the literature suggests a wide range of learning techniques to address attribute leakage in deep network models. As we show in _Section 3.2, Theorem 2,_ however, the trade-off between a representation’s utility and achieving the LPP applies “regardless of the way the feature representations are obtained” (_Section 3.2, page 6_). In Section 4, we provide an empirical evaluation that supports our theoretical results: Regardless of a model's learning task and even under censoring, the LPP cannot be fulfilled while simultaneously providing high utility for the prediction task.

We furthermore note that to the best of our knowledge none of the existing techniques in the literature can actually achieve the desired goal of LPL. To do so, the features learned by the model would need to simultaneously restrict leakage for any sensitive attribute $S$. Techniques to defend against attribute inference are mostly limited to one or two fixed sensitive attributes and hence cannot achieve this goal.

Reviewer comment (Experiments):
Could the authors conduct empirical experiments on a wider range of datasets and models? For example, conduct experiments on some NLP tasks. More extensive experimental results can more fully verify the proposed theory.

Author response:
**Additional dataset:** In the supplementary material, we provide the results of an additional experiment which demonstrates the strict trade-off between model utility and the LPP on a very different type of data and model. As for tabular data, together with image data, sharing feature encodings instead of raw data is often suggested as a solution to limit harmful inferences, we chose the Texas Hospital dataset and the TabNet model architecture (Arik & Pfister, 2021) for these experiments.

The results show that, as predicted by Theorem 2 (Section 3.2), for any learning task there always exists a sensitive attribute that violates the LPP.

Reviewer comment (Experiments):
Does the complexity of the feature extractor affect the extent of data attribute leakage?

Author response:
**Complexity of feature extractor:** Our theoretical results imply that the trade-off applies not only regardless of the learning technique, but also regardless of a model’s architecture. We have run additional experiments using a different model architecture (ResNet-18) on the same image dataset to confirm that, as predicted by Theorem 2, the trade-off applies regardless of the exact architecture choices. We have uploaded these results as supplementary material, and we them to the Appendix.

Reviewer comment (Experiments):
The observations are only shown on a single dataset.

Author response:
**Limits of empirical evaluation:** In the Supplementary Material, we provide the results of an additional experiment which demonstrates that the strict trade-off between model utility and the LPP also holds on a very different type of dataset and model. As for tabular data, together with image data, sharing feature encodings instead of raw data is often suggested as a solution to limit harmful inferences, we chose the Texas Hospital dataset and the TabNet model architecture (Arik & Pfister, 2021) for these experiments.

The results show that, as predicted by Theorem 2, Section 3.2, for any learning task there always exists a sensitive attribute that violates the LPL.

We limit the empirical evaluation to single attribute inference attacks as, despite their simplicity, they are sufficient to support our theoretical result: As soon as the adversary’s inference gain on a single attribute exceeds the features’ utility for the learning task, the features cannot be claimed to fulfil the LPL claim. We note, however, that Theorem 2 covers any type of inference attack, incl. data reconstruction attacks for the case where $S=X$.

Reviewer comment (Novelty):
While the paper uses least-privilege principle to formalize information leakage, I do not know how this largely differentiate other works that use the similar idea (though those works assume a fixed sensitive attribute), e.g., Zhao et al., 2020; Brown et al., 2022 and Salamatian et al., Privacy-Utility Tradeoff and Privacy Funnel. What is the key technical challenge when we do not assume a fixed sensitive attribute, but assume it is a superset of the input?

Author response:
**Novelty**: Our proofs leverage known mathematical arguments and prior information theoretic results; and we believe that this is clearly expressed in the manuscript via appropriate references.  
However, our main result (Theorem 2, Section 3.2) **is a new formal result about a novel problem** that has not been stated in any prior work nor can be derived as a corollary from any of the related information-theoretic problems.

Reference [b] characterises the fundamental limits of the Privacy Funnel (PF) problem. In PF, motivated by the problem of privacy-preserving data publishing or anonymisation, the goal is to publish a representation $Z$ of the original data $X$ that preserves at least $t$ bits of information about $X$, i.e., $I(Z,X) > t$ while minimising the information about a sensitive attribute $S$ with $I(Z, S)$. A crucial difference between the PF problem and the least-privilege problem that we study is that in PF there exists no definition of the *intended learning task* critical to the least-privilege concept  (represented in our work through the task label $Y$ and reflected in the utility definition $I(Z ,Y)$). Other differences are that PF only considers _a fixed set of sensitive attributes_ instead of all attributes other than the task and uses a weaker notion of leakage (mutual information compared to maximal leakage).

As we discuss in Section 3.2 (page 5 and 6), the least-privilege problem that we study is most closely related to information bottleneck problems, such as CEB. However, there are crucial differences to any of the previously studied problems. For instance, to properly formalise the least-privilege principle – to learn nothing else other than the intended task – we need to consider a stronger notion of leakage than mutual information as in CEB. 

In our work, we show that these differences between the least-privilege and other problems (PF or CEB) lead to a very different trade-off, **which was not known prior to our work**.

Reviewer comment (Experiments):
Theoretically, the paper shows the fundamental leakage and observes this on a dataset. I am curious how common such observation is in more datasets.

Author response:
**Limits of empirical evaluation:** In the Supplementary Material, we provide the results of an additional experiment which demonstrates that the strict trade-off between model utility and the LPP also holds on a very different type of dataset and model. As for tabular data, together with image data, sharing feature encodings instead of raw data is often suggested as a solution to limit harmful inferences, we chose the Texas Hospital dataset and the TabNet model architecture (Arik & Pfister, 2021) for these experiments.

The results show that, as predicted by Theorem 2, Section 3.2, for any learning task there always exists a sensitive attribute that violates the LPL.

We limit the empirical evaluation to single attribute inference attacks as, despite their simplicity, they are sufficient to support our theoretical result: As soon as the adversary’s inference gain on a single attribute exceeds the features’ utility for the learning task, the features cannot be claimed to fulfil the LPL claim. We note, however, that Theorem 2 covers any type of inference attack, incl. data reconstruction attacks for the case where $S=X$.

Reviewer comment (Novelty):
The contribution is limited.

Author response:
Thank you for the review. As the review has pointed out in the summary, our paper provides the first formalisation and a generic characterization of the trade-offs involved in least-privilege learning, which was proposed in the prior work (Melis et al., 2019; Brown et al., 2022). Our characterization applies to any method of obtaining feature representations, covering the model partitioning and collaborative learning settings. Please let us know if there are any questions we can answer to change the assessment of the contribution significance, or to clarify any concrete issues with the presentation.

Reviewer comment (Theory):
Arguments for why sharing label information is a practical worst-case assumption.

Author response:
**Strong assumption on sharing label information:** The assumption that the service provider has access to the correct label information of a particular target record is primarily motivated by the observation that it is impossible to achieve the strict LPP definition given in Section 3.1, Definition 1. To allow for any meaningful learning, we have to at least allow for the fundamental leakage through the task label (see _Section 3.2, page 5_), which means we have to assume that the label is known to the adversary. We note that a weaker assumption on the fundamental leakage adversary would result in an even worse trade-off between unintended feature leakage and a model’s utility.

In some preliminary experiments, we did assess the success of an adversary that only has access to a record’s predicted task label instead of the ground truth. We observed that, as expected, the adversary’s success in predicting a sensitive attribute from the predicted task label increased with increasing model utility. In our analysis and final experiments, we decided to only consider the worst-case adversary that has access to the correct task label as this gives us a lower bound on the adversary’s multiplicative gain and hence the best method to analyse the trade-off.

Reviewer comment (Theory):
The model splitting between server and clients analyzed in this paper is counter-intuitive. Namely, the authors assume that clients use only the representation layers of the model, while the server uses only a classification head of the model. Since the classification heads are usually small and easy to train, I do not see the incentive for clients to share input data representations to the server in this setting (for prediction tasks). For example, the clients may download the classification head weights or tune a classification head on local data and compute the labels locally at inference time. This also seems different from Melis et al. 2019, where the clients only compute one embedding layer (rather than a large part of the whole model).

Author response:
**Regarding the model partitioning scheme:** To clarify this point, we would like to distinguish between two questions  (1) In what form are data representations made available to the service provider/adversary? (2) What is the architecture of the model and how is it split into an encoding $f_E$ and classification part $f_C$?
 
On (1), as we describe at the start of Section 2, some works evaluate unintended feature leakage in a collaborative learning setting (Melis et al., 2019), others in  a model partitioning setting (Brown et al., 2020, Song & Shmatikov, 2020, Zhao et al., 2020 or Osia et al., 2018).

Melis et al., 2019 assume that the adversary/service provider has access to _the full set of gradient updates across all layers of the model_. They state that these "gradient updates can [...] be used to infer feature values, which are in turn based on [...] private training data". In our notation, the data representations $Z = f_E(X)$ are hence the (noisy) feature values of a (batch of) training examples across _all layers of the model_.
Works such as Brown et al., 2020, Song & Shmatikov, 2020 or Zhao et al., 2020 assume that the service provider is given direct access to a record's representation at a _particular output layer of the model_. 

While we have empirically evaluated the model partitioning setting, we highlight that **our theoretical results equally apply to both settings.**

On (2), this question only becomes relevant in the model partitioning setting. Here, we decide at which layer we split a model into its encoding and classification parts. This decision determines the output of the encoder $Z = f_E(X)$, which in turn determines what information is made available to the service provider/adversary to make a prediction about $Y = f_C(Z)$ and to makes a guess about $S = g(Z)$. 

The main result of our work (Theorem 2) shows that the trade-off between limiting the inference about $S$ and preserving information about $Y$ holds for _any feature mapping $f_E$_; and equally for any classifier $f_C(Z)$. That is, **these results hold independently of the size of the classification part on the server side**.

To provide evidence of this fact, in Appendix C.2.2, we added an experiment in which we used the code by Zhao et al., 2020 to reproduce our results under their model architecture and training method (maximum entropy adversarial representation learning). In Figure 8, we show that the trade-off in Theorem 2 also applies to this scenario, which has a more extensive classification part than our original experiments.

Reviewer comment (Theory):
The definitions and notations lack clarity at times. Most importantly, the information leakage is defined by successful inference of *any* attribute. This may be overly strong as many attributes are less sensitive. I'm wondering whether the proved trade-off in this paper is largely a result of this overly strong definition of information leakage.

Author response:
**Trade-off as a result of strong assumptions:** As we discuss in Section 3.2, the strict trade-off between utility and the LPP exists because the LPP restricts the adversary’s inference gain in terms of maximal leakage. In the standard CEB problem (Fischer, 2020), which considers a weaker notion of unintended leakage, the mutual information between the representations $Z$ and the sensitive raw data $X$, there is hence no such trade-off (see _Section 3.2, page 6_). Such a weak notion of leakage, however, is not adequate to formalise and assess the original claim from many papers that it is possible to learn feature representations that protect against any harmful inferences and fulfil the least-privilege principle.

Reviewer comment (Experiments):
Could the authors discuss more about the effect of the splitting method, i.e., what parts are deemed as representations, on the theoretical and empirical conclusions? For example, when the client only has access to a small part of the model, would the leakage still be unbalanced on different attributes?

Author response:
**Effect of the splitting method on theoretical and empirical conclusions:** In _Section 3.2, Theorem 2,_ we show that the trade-off between a representation’s utility and achieving the LPP applies to any feature representation regardless of the exact splitting method, model architecture or “the way the feature representations are obtained” (_Section 3.2, page 6_).

In our experiments, we evaluate the trade-off between utility and unintended inferences on the last layer feature representations of a CNN model. We choose the last layer because it is where a record’s feature activations are expected to be most learning task specific (Melis et al., 2019 and Mo et al. 2021) and hence the claim that these representations meet the LPP is the strongest. Our results demonstrate that, even at the last layer and under censoring of particular attributes, the least-privilege claim does not hold true.

We have run additional experiments: (1) using a different model architecture (ResNet-18) on the same image dataset, and (2) using a different model (TabNet) on an additional tabular dataset (Texas Hospital dataset) to confirm that, as predicted by Theorem 2, the trade-off applies regardless of the exact model architecture choices. We have uploaded these empirical results as supplementary material, and will add add them to the Appendix.

Reviewer comment (Theory):
Could the authors discuss how the theoretical and empirical trade-off (between information leakage and model utility) might change if we only consider a smaller set of attributes (rather than the whole input feature space) as sensitive?

Author response:
**Worst-case assumption on attributes considered sensitive and how it affects the trade-off:** As mentioned in Section 3 ("Attribute Inference"), we want to model an adversary that captures the _least-privilege_ principle which implies that the representations shared with the service provider leak _nothing else_ other than the fundamental leakage implied by the learning task. To adequately capture this principle, we have to consider any attribute, other than the learning task, to be sensitive.

If we were to restrict the set of attributes considered sensitive, the problem would become similar to a version of the Privacy Funnel (PF) problem (Asoodeh & Calmon’20). PF is different from the least-privilege learning problem (a form of generalized Conditional Entropy Bottleneck (CEB) problem; see next) that we consider in this work. As Asoodeh & Calmon point out, PF and bottleneck-type problems like CEB  are "duals" of each other.

Reviewer comment (Theory):
What are the transition orders between random variables in the Markov chain $Y - X - W$ defined in Section 2?

Author response:
- The transition kernels between Y - X - W can be arbitrary so long as $Y \bot W \mid X$.

Reviewer comment (Evaluation):
In Figure 4, bottom right plot, what is the meaning of color? Why is the third row from the top colored blue despite its negative value? What does a negative model utility $\tilde{I}_{\infty}(Y, Z)$ mean?

Author response:
- The bar in the third row should indeed be coloured red. We apologise for this data plotting error. A negative value indicates that the performance of the classifier has dropped below the majority class baseline guess, i.e., that the features learned by the model do not provide any utility for the prediction task. This is a common side effect of censoring techniques which often lead to a degradation in model performance (see, for instance, Song & Shmatikov, 2019 and Raff & Sylvester, 2018).

Reviewer comment (Theory):
Assumption A requires a positive posterior but does not require any lower bound for the posterior density. Does it mean the posterior could also be arbitrarily close to a point distribution?

Author response:
- The positive posterior requirement implies both upper and lower bound as the notation is a shorthand for $P(Y = y \mid X = x) > 0$ for all $y \in \mathbb{Y}$. Indeed, the assumption allows distributions that are arbitrarily close to a point distribution, but not exactly point distributions.