Paper: Towards Poisoning Fair Representations (ICLR-2024)

Reviewer comment (Theory):
**On the necessity of working at representation level**. After reading the paper, it is still unclear to me why do we need to consider I(Z; a) rather than I(Y(X); a). Here Y(X) is the prediction of the model. In fact, considering I(Y(X); a) has several benefits: first, its maximization still yields an unfair model; second, its estimation is much easier due to the low-dimensionality of Y(X) and a (for example, we can estimate it easily by the well-known KSG estimator [1] which typically works quite well in low-dimensional cases). Importantly, maximizing I(Y(X); a) also do not require an access to Y. Can the authors justify the reason behind considering I(Z; a) instead?

Author response:
>**On the necessity of working at representation level.**

Attacking mutual information at representation level aligns better with fair representation learning (FRL) and is more universal than attacking $I(Y(X), a)$. For example, a bank may want to obtain a representation for each user that can be used to determine their eligibility for **existing** and **upcoming** financial products such as credit cards without concerning about fairness again [1]. 
Here each financial product is associated with a (unique) label, and determining the eligibility entails a classification task.
In this case, there are two challenges for delivering a $Y(X)$-based attack. 
First, one has to determine and justify 
which classifier $Y(\cdot)$ to use and why consider the fairness of this specific classification task. 
Second, for any **upcoming** financial product, its label $Y$ does not exist and one cannot obtain classifier $Y(X)$ (one needs $Y$ to train such a classifier), 
let along attacking it. 
In contrast, a representation-level attack can overcome the two challenges in a single shot. 
As discussed in section 2.2,
by maximizing $I(z, a)$, any $I(Y(X), a)$ **will be** maximized so long as the fairness concern exists. 
This implies launching attack on all labels $Y$ simultaneously, including the ones where classifies $Y(X)$ cannot be trained.

Reviewer comment (Theory):
**Issues in the discussion related to other MI proxies**: some comments in remark 2.1 do not seem completely sensible to me. For example, the author mention that other analytical proxies like (K)CCA and sliced mutual information suffer from differentiation difficulties. This may not be true in my view (for example, in CCA, you can first solve the optimal weights analytically, then substitute it back to the formula of CCA. The resultant formula has a structure very similar to eq2 in your paper). In addition, I think the authors miss the possibility of using non-parametric dependence measure e.g. distance correlation (dCorr) [2]. This is also fully analytical and may potentially be applicable in your scenario. Ultimately, I think the real advantage of your FLD-based method is it provably optimises a variational lower bound of MI, whereas other proxies (KCCA, slice MI, dCorr) may not. This seems to be a better justification of the use of your method.

Author response:
>**Issues in the discussion related to other MI proxies.**

We appreciate the detailed explanation and valuable feedback from the reviewer. 
We've revised the remark 2.2 in the paper as follows: 

>*Maximizing I(z,a) is a general framework to poison FRL and admits other proxies such as sliced mutual information (Chen et al., 2022), kernel canonical correlation analysis (Akaho, 2007), and non-parametric dependence measures (Szekely et al., 2007). In this work, we use FLD because of its conceptual simplicity, interpretability, and good empirical performance. As one may recognize, when $p(z | a = 1)$ and $p(z | a = 0)$ are Gaussian with equal variance, FLD is optimal (Hamsici \& Martinez, 2008) whose BCE loss attains the tight lower bound of $I(z, a)$ up to constant $H(a)$. In this case, our method provably optimize the lower bound of I(z, a) whereas other proxies 
whereas other proxies may not due to the lack of direct connections to mutual information. While the Gaussianity may not hold in general, FLD score as a measure of data separability is still valid (Fisher, 1936), and we verify its efficacy for our goal in Appendix D.2, where we show that FLD score is highly informative for the empirical minimal BCE loss of a logistic regression.*

Reviewer comment (Experiments):
**Slightly limited evaluation**: most of the evaluations in this work are conducted on tabular data where the network size is small. Whether the method developed in this work will scale to larger networks e.g. those in computer vision and NLP remain unclear to me. However, given the nature of many existing literature in fairness research, which also only focus on tabular data, this should not be a main criticism for the paper.

Author response:
>**Slightly limited evaluation.**

As pointed out by the reviewer, we focus on simple tabular data because many fair representation learning methods focus on these data [1, 2, 3, 4]. Moreover, previous work on attacking classical fair machine learning also only focused on tabular data [5, 6, 7], and we follow this convention. 

To better evaluate the effectiveness of our method, we added experiments on two more benchmark datasets COMPAS and Drug Consumption. 
Results have been updated in Appendix D.8. 
We reported decrease of BCE losses in Figure 15, increase of DP violations in Figure 16, and accuracy of predicting $y$ from $z$ in Figure 17.
Again, on two new datasets our attacks successfully outperformed AA baselines to a large extent. 
Observing these good performance, we will study the vulnerability of bias mitigation methods for CV and NLP tasks as our future work. 

[1] Zhao et al. Conditional learning of fair representations, 2019.

[2] Moyer et al. Invariant representations without adversarial training, 2018.

[3] Creager et al. Flexibly Fair Representation Learning by Disentanglement, 2018.

[4] Reddy et al. Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics, 2021.

[5] Solans et al. Poisoning attacks on algorithmic fairness, 2020.

[6] Mehrabi et al. Exacerbating algorithmic bias through fairness attacks, 2021.

[7] Chang et al. On Adversarial Bias and the Robustness of Fair Machine Learning, 2020.

Reviewer comment (Evaluation):
Authors use their own defined vanilla metric, and lack related fairness-aware metrics like Equality odds (EO)

Author response:
In section 2.2 we analyzed how our attack can deteriorate the well-known fairness-aware metric demographic parity (DP).
In experiments we verified that **our attack successfully exacerbated DP violations of predicting label $y$ from  representation $z$ learned by four victims on four datasets** (Adult, German, and new COMPAS, Drug Consumption). Due to page length these results were deferred to Appendix D.3 (Adult and German datasets) and Appendix D.8 (COMPAS and Drug Consumption datasets) respectively.

Reviewer comment (Experiments):
Authors are encouraged to conduct more experiments on more datasets like COMPAS and Drug Comsumptionm, please kindly follow this AAAI paper which authors have cited: Exacerbating Algorithmic Bias through Fairness Attacks.

Author response:
Thanks for your suggestions, we conducted experiments on COMPAS and Drug Consumption datasets and reported results in Appendix D.8 of our updated paper. 
We reported decrease of BCE losses in Figure 15, increase of DP violations in Figure 16, and accuracy of predicting $y$ from $z$ in Figure 17. Again, on two new datasets our attacks successfully outperformed AA baselines to a large extent.

Reviewer comment (Experiments):
Personally, I reckon authors are encouraged to conduct experiments on deeper NN (I think simple MLP is not that DEEP to be called "DNN"), though the datasets are relatively simple. I'm curious about these experiments to investigate ENG. Authors are encouraged to conduct more analysis on the further version of this work, which is good for community: )

Author response:
To see how our attacks perform on deeper and larger NNs, 
we tested how fair representations learned by CFAIR and CFAIR-EO can be attacked when the NNs are larger. Experiments were conducted on the largest Adult dataset. 
We increased depths of encoder to 3 hidden layer and adversarial classifier to 5 hidden layer. We increased the dimension of representations to 128 and set all hidden sizes to 60. Due to time constraint, we did not tune the model architectures to obtain the best clean performance. Training epochs were increase from 50 to 100, and all other hyper-parameters such as batch size and learning rates were unchanged. As shown in the table below, **our attacks succeeded in all settings while three baselines suffered from several failures.**

Finally, we would like to mention that the choices of model architecture and training settings can have crucial influence on the performance with and without attack, so these results are just for illustration. Moreover, motivated by the promising results, we plan to extend our attack to fair machine learning on large-scale image and text datasets.

| Victim | Attack | 5\% | 10\% | 15\% |
| ------ | ------------- | ----- | ----- | ----- |
 CFAIR | NRAA-a  |  0.0047     | 0.0204 | -0.4015 
|  | NRAA-y  |  0.0163 | -0.278 | -0.3552 
|  | RAA-a   |  0.0086     | -0.0138 | -0.3312 
|  | RAA-y   |  0.011      | -0.0226 | -0.8761 
|  | ENG-EUC |  0.003      | 0.0058 | 0.01 
|  | ENG-FLD |  0.0019     | 0.0055 | 0.0105 
|  | ENG-sFLD|  0.0033     | 0.0052 | 0.0037 
| CFAIR-EO | NRAA-a  |  0.0171 | 0.0279 | -0.3513 
|  | NRAA-y  |  0.0194 | -0.3161 | -0.2261 
|  | RAA-a   |  -0.1668 | -0.2093 | -0.0896 
|  | RAA-y   |  -0.0585 | -0.5564 | -0.3719 
|  | ENG-EUC |  0.0021 | 0.007 | 0.0046 
|  | ENG-FLD |  0.0075 | 0.0128 | 0.0205 
|  | ENG-sFLD|  0.0073 | 0.0056 | 0.0213

Reviewer comment (Experiments):
One of the problem of introducing elastic penalty is how to choose properly the two penalty parameters $\lambda_{1}$ and $\lambda_2$. Though it can be chosen empirically, it can be dataset-dependent.

Author response:
We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

>**One of the problem of introducing elastic penalty is how to choose properly the two penalty parameters...Would it make significantly difference if we simply choose the L1 norm penalty instead?**

We chose the penalty parameters following the convention in previous work [1] and the choices were briefly selected from a relatively large range, therefore, we believe that the attack performance can benefit from a fine-grained search for better parameter values. 

Regarding the advantage of using elastic-net penalty than using $L_1$ norm penalty only, 
conceptually, elastic-net penalty is able to choose a group of correlated features whereas using $L_1$ or $L_2$ penalty only cannot [2]. 
Empirically, to check the effect of using $L_1$ norm penalty only, we tried the same choices of $\lambda_1$ as in the main experiment 
and compare the corresponding attack performance and perturbation magnitude. 
Due to time constraint we only conducted experiments on Adult dataset and reported results in Appendix D.5 of the updated paper. 

In short, we note that **comparing with elastic-net penalty, $L_1$ norm penalty usually resulted in perturbations with larger $L_1$ and $L_2$ norms but the attack performance was hardly improved, especially when small- to intermediate- level penalty parameter values were used.**

[1] Chen et al. Ead: elastic-net attacks to deep neural networks via adversarial examples, 2018.

[2] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net, 2005.

Reviewer comment (Experiments):
Would it make significantly difference if we simply choose the L1 norm penalty instead?

Author response:
We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

>**One of the problem of introducing elastic penalty is how to choose properly the two penalty parameters...Would it make significantly difference if we simply choose the L1 norm penalty instead?**

We chose the penalty parameters following the convention in previous work [1] and the choices were briefly selected from a relatively large range, therefore, we believe that the attack performance can benefit from a fine-grained search for better parameter values. 

Regarding the advantage of using elastic-net penalty than using $L_1$ norm penalty only, 
conceptually, elastic-net penalty is able to choose a group of correlated features whereas using $L_1$ or $L_2$ penalty only cannot [2]. 
Empirically, to check the effect of using $L_1$ norm penalty only, we tried the same choices of $\lambda_1$ as in the main experiment 
and compare the corresponding attack performance and perturbation magnitude. 
Due to time constraint we only conducted experiments on Adult dataset and reported results in Appendix D.5 of the updated paper. 

In short, we note that **comparing with elastic-net penalty, $L_1$ norm penalty usually resulted in perturbations with larger $L_1$ and $L_2$ norms but the attack performance was hardly improved, especially when small- to intermediate- level penalty parameter values were used.**

[1] Chen et al. Ead: elastic-net attacks to deep neural networks via adversarial examples, 2018.

[2] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net, 2005.

Reviewer comment (Experiments):
Discussion over the choice of the two penalty parameters $\lambda_{1}$ and $\lambda_2$ in the elastic penalty.

Author response:
We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

>**One of the problem of introducing elastic penalty is how to choose properly the two penalty parameters...Would it make significantly difference if we simply choose the L1 norm penalty instead?**

We chose the penalty parameters following the convention in previous work [1] and the choices were briefly selected from a relatively large range, therefore, we believe that the attack performance can benefit from a fine-grained search for better parameter values. 

Regarding the advantage of using elastic-net penalty than using $L_1$ norm penalty only, 
conceptually, elastic-net penalty is able to choose a group of correlated features whereas using $L_1$ or $L_2$ penalty only cannot [2]. 
Empirically, to check the effect of using $L_1$ norm penalty only, we tried the same choices of $\lambda_1$ as in the main experiment 
and compare the corresponding attack performance and perturbation magnitude. 
Due to time constraint we only conducted experiments on Adult dataset and reported results in Appendix D.5 of the updated paper. 

In short, we note that **comparing with elastic-net penalty, $L_1$ norm penalty usually resulted in perturbations with larger $L_1$ and $L_2$ norms but the attack performance was hardly improved, especially when small- to intermediate- level penalty parameter values were used.**

[1] Chen et al. Ead: elastic-net attacks to deep neural networks via adversarial examples, 2018.

[2] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net, 2005.

Reviewer comment (Theory):
The assumption of the threat model is strong. The proposed attack is under the assumption of a white-box threat model, where the attacker has full access to and control over the victim's trained model. This implies that the attack is primarily effective in scenarios where the victim has already trained a model and relies on the attacker's data for subsequent fine-tuning. Such a specific condition might limit the general applicability of the attack in diverse real-world scenarios.

Author response:
>**Could the authors clarify their threat model more clearly?**

As the first work towards conducting poisoning attack against fair representation learning (FRL) methods, we followed previous works on attacking classical fair machine learning such as [1, 2, 3] and assumed a strong attacker that has full knowledge to the victim model: including its architecture, training details, and parameters. 
This type of attack is known as white-box attack and has been widely used to benchmark the vulnerability (robustness) of a model [1, 2, 3, 4] and is necessary to develop a black-box attack in general (in the sense that an optimization-based attack is often conducted by attacking a collection of white-box surrogated victim models) [5, 6]. Therefore, we believe that this requirement does not undermine our contribution on understanding how vulnerable FRL methods are. Nonetheless, we agree with the reviewer that a black-box attack would be more practical and we will delve into developing stronger and more transferable attack towards FRL methods in our future work. 

[1] Solans et al. Poisoning attacks on algorithmic fairness, 2020.

[2] Mehrabi et al. Exacerbating algorithmic bias through fairness attacks, 2021.

[3] Chang et al. On Adversarial Bias and the Robustness of Fair Machine Learning, 2020.

[4] Koh et al. Stronger data poisoning attacks break data sanitization defense, 2022. 

[5] Huang et al. MetaPoison: Practical General-purpose Clean-label Data Poisoning, 2021. 

[6] Geiping et al. Witches’ brew: Industrial scale data poisoning via gradient matching, 2021.

Reviewer comment (Theory):
Lack the reason why the method can attack FRL. The foundational principle of Fair Representation Learning (FRL) is to ensure fairness by removing sensitive features from the intermediate representation. The proposed attack, on the other hand, seeks to amplify the presence of sensitive information within these representations. The paper does not adequately elucidate why FRL techniques, designed to minimize sensitivity, are unable to counteract or mitigate the effects of the proposed attack. This leaves a gap in understanding the inherent vulnerabilities of FRL against the described attack strategy.

Author response:
>**Could the authors give more insight to explain why their attack cannot be mitigated by fair representation learning (FRL)?**

Thanks for your question. We plan to explore this direction in our next step towards understanding the vulnerability of existing fair representation learning (FRL) method. Here we give two possible causes. 

First, fair machine learning methods often assume that the underlying data distribution is unchanged.
When a distribution shift appears, many of these methods showed degraded performance [1, 2].
We presume that FRL methods suffer from a similar vulnerability as well, i.e., their training is not robust enough against distribution shift. 
Consequently, a data poisoning attack, which injects carefully-crafted poisoning samples to the training data, can be seen as imposing an adversarial distribution shift to the empirical training distribution. This may deteriorate the fairness performance of FRL methods on the validation (targeted) data that follows the original distribution.

Second, as mutual information is difficult to estimate and optimize, existing FRL methods also rely on approximate solutions that require training of some auxiliary models such as a sensitive feature predictor to conduct adversarial training or a deep neural network-based mutual information estimator.
In practice, these auxiliary models may also encounter unstable training and result in sub-optimal fairness results, and the performance degradation can be exacerbated by the data poisoning attack. 


[1] An et al. Transferring Fairness under Distribution Shifts via Fair Consistency Regularization, 2022.

[2] Jiang et al. Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach, 2023.

Reviewer comment (Experiments):
Similar to weakness 2, could the authors give more insight to explain why their attack cannot be mitigated by fair representation learning (FRL)?

Author response:
>**Could the authors give more insight to explain why their attack cannot be mitigated by fair representation learning (FRL)?**

Thanks for your question. We plan to explore this direction in our next step towards understanding the vulnerability of existing fair representation learning (FRL) method. Here we give two possible causes. 

First, fair machine learning methods often assume that the underlying data distribution is unchanged.
When a distribution shift appears, many of these methods showed degraded performance [1, 2].
We presume that FRL methods suffer from a similar vulnerability as well, i.e., their training is not robust enough against distribution shift. 
Consequently, a data poisoning attack, which injects carefully-crafted poisoning samples to the training data, can be seen as imposing an adversarial distribution shift to the empirical training distribution. This may deteriorate the fairness performance of FRL methods on the validation (targeted) data that follows the original distribution.

Second, as mutual information is difficult to estimate and optimize, existing FRL methods also rely on approximate solutions that require training of some auxiliary models such as a sensitive feature predictor to conduct adversarial training or a deep neural network-based mutual information estimator.
In practice, these auxiliary models may also encounter unstable training and result in sub-optimal fairness results, and the performance degradation can be exacerbated by the data poisoning attack. 


[1] An et al. Transferring Fairness under Distribution Shifts via Fair Consistency Regularization, 2022.

[2] Jiang et al. Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach, 2023.

Reviewer comment (Theory):
The paper lacks a clear threat model. For example, it is unclear whether the attacker’s objective is useful for compromising algorithms with and without mechanisms for mitigating the fairness gap. On the other side, it is unclear what is the attacker’s objective and the relation of the attack strategy with respect to the model’s performance. Other works in the research literature, like Chang et al. or Van et al. (“Poisoning Attacks in Fair Machine Learning”) have already considered the trade-off between targeting performance and the fairness gap.

Author response:
>**The paper lacks a clear threat model: it is unclear whether the attacker’s objective is useful for compromising algorithms with and without mechanisms for mitigating the fairness gap. On the other side, it is unclear what is the attacker’s objective and the relation of the attack strategy with respect to the model’s performance.**

In this work we proposed a new objective to attack fair representation learning (FRL) models by maximizing the mutual information (MI) between high-dimensional representations $z$ learned by the victim and the sensitive attribute $a$.
This attacking objective aligns well with the well-known MI-based fairness goals used in FRL as detailed in section 2.1.
Moreover, this objective jeopardizes the well-known fairness metric demographic parity (DP) of any classifiers predicting label $y$ from representation $z$ as discussed in section 2.2, and is empirically verified in our experiments covering four FRL models trained on four well-studied benchmark datasets.
We further added accuracy of predicting label $y$ from representation $z$, and found that **our attacks had minimal impact on the accuracy.**

Reviewer comment (Novelty):
In the end the attack proposed by the authors relies on the maximization of the FLD score in the outer optimization objective. This relies on strong assumptions on the distribution of the data and its representation and may not hold for many practical scenarios. In this sense, it is unclear why this strategy is better compared to other attacks already proposed in the research literature, like Solans et al., Mehrabi et al., Chang et al., or Van et al. (“Poisoning Attacks in Fair Machine Learning”), which have strong connections to this work. On the other side, there is no mention to “Subpopulation Poisoning Attacks” by Jagielski et al., which are also very relevant to this work and propose more scalable alternatives for crafting poisoning attacks targeting fairness.

Author response:
>**It is unclear why this strategy is better compared to other attacks already proposed in the research literature**

Existing works, including the listed one, focused on attacking fairness in classification tasks, i.e., the fairness is directly defined and attacked based on scalar predictions. 
We instead seek to degrade the *fairness* of *high-dimensional* representations directly, which is a much more challenging task and requires a new formulation as discussed in section 1.  
Notably, 
FRL methods do not always need access to label $y$ during the training time, see ICVAE-US for an example. For such victims, all existing poisoning attack formulation fall short to apply while our formulation is able to handle them.

>**There is not mention to “Subpopulation Poisoning Attacks” by Jagielski et al., which are also very relevant to this work and proposes more scalable alternatives for crafting poisoning attacks targeting fairness. & The authors used a gradient matching strategy for approximating the solution of the bilevel optimization problem. However, Jagielski et al. (“Subpopulation Poisoning Attacks”) shows that other strategies can be more efficient for this. I think this aspect requires further analysis.**

Thanks for bringing to us this work.
After reading through it, we agree that subpopulation attack is a direction that deserves more exploration in the future study on deriving stronger attack against FRL methods, and we have updated the paper to discuss this accordingly. 
At the same time, we found subpopulation attack less relevant to the current main focus of our paper. 
First, a subpopulation attack, which identifies and attacks the most vulnerable subgroups of data, fills the gap between the availability and targeted attack. While in this work, we focus on an availability-like attack, where the attacker seeks to degrade fairness guarantee for as many samples as possible (on the whole validation set). 
Second, as mentioned by the reviewer, subpopulation attack provides more scalable alternatives to craft poisoning samples, 
while the main focus of this paper is to formulate an attack goal towards fairness of high-dimensional representations and to propose a tractable proxy that can be used as an objective for poisoning samples crafting.

Reviewer comment (Novelty):
Given the existing works in the research literature, I believe it is too bold that the authors claim that “We propose the first data poisoning attack on FRL as outlined in Figure 1.” I think that the authors should clarify this and position better their paper and contributions with respect to other existing works.

Author response:
>**I think that the authors should clarify this and position better their paper and contributions with respect to other existing works.**

The main contributions of our work lie in two-fold. 
First, we provide a new MI-based attack goal to degrade fairness of *high-dimensional* representations $z$. Specifically, we maximize $I(z, a)$ where $a$ is the sensitive feature. 
In contrast, previous attacks mostly focused on *scalar* predictions on $\hat{y} = h(z)$ and seeks to maximize $I(h(z), a)$ (loosely speaking) which is much easier to evaluate than $I(z, a)$ as $z$ is high-dimensional. 
As discussed in section 2.2, the benefit of using this new MI-based attack goal is that by maximizing $I(z, a)$, for any classifier $h$, $I(h(z), a)$ is free to grow (so DP violation increases [1]) so long as the fairness concern exists. Notably, this condition holds even if ground truth $y$ is unobserved so that previous attack goals cannot be defined.

Second, we provide a theoretical analysis on the required poisoning samples to launch a gradient-matching based attack.
This analysis also sheds light on defending against the proposed attack method and can be of independent interest to other gradient-matching based attackers. 

To avoid possible confusion, we've updated this statement as "*We propose the first data poisoning attack that directly targets on fairness of high-dimensional representations as shown in Figure 1.*" 

[1] Zemel et al. Learning Fair Representations, 2013.

Reviewer comment (Evaluation):
The experimental evaluation is not convincing: In the experiments the authors just reported results evaluating the BCE loss but did not consider any of the existing metrics for measuring the fairness gap. On the other hand, there is no mention of how the attack affects the accuracy at all. Apart from that, it is necessary to have a more comprehensive comparison with other methods in the research literature, e.g., Solans et al., Chang et al., Van et al. (“Poisoning Attacks in Fair Machine Learning”), Jagielski et al. (“Subpopulation Poisoning Attacks”). For some of these attacks, the authors can use the same outer objective and use the same approximation for solving the bilevel optimization problem.

Author response:
>**The experimental evaluation is not convincing: In the experiments the authors just reported results evaluating the BCE loss but did not consider any of the existing metrics for measuring the fairness gap.**

We reported violation of demographic parity (DP), one of the most widely-used fairness notion, from the four victim models trained on Adult and German datasets in Appendix D.3.
In the revised paper we further added corresponding results on COMPAS and Drug Consumption datasets in Appendix D.8 of updated paper. 
**In all cases, our attacks successfully increased DP violations, clearly showing its effectiveness.**

>**For some of these attacks, the authors can use the same outer objective and use the same approximation for solving the bilevel optimization problem.**

As analyzed in the responses above, we believe that previous outer objectives may not be good choices for our purpose and they indeed fail to apply in certain cases. 
For example, in [1] the outer objective is a weighted average of classification accuracy and relaxed group fairness gap. 
However, ICVAE-US does not involve any classification in its training, therefore, these two terms are undefined.

[1] Van et al. Poisoning Attacks on Fair Machine Learning, 2021.

Reviewer comment (Experiments):
The authors used a gradient matching strategy for approximating the solution of the bilevel optimization problem. However, Jagielski et al. (“Subpopulation Poisoning Attacks”) shows that other strategies can be more efficient for this. I think this aspect requires further analysis.

Author response:
>**There is not mention to “Subpopulation Poisoning Attacks” by Jagielski et al., which are also very relevant to this work and proposes more scalable alternatives for crafting poisoning attacks targeting fairness. & The authors used a gradient matching strategy for approximating the solution of the bilevel optimization problem. However, Jagielski et al. (“Subpopulation Poisoning Attacks”) shows that other strategies can be more efficient for this. I think this aspect requires further analysis.**

Thanks for bringing to us this work.
After reading through it, we agree that subpopulation attack is a direction that deserves more exploration in the future study on deriving stronger attack against FRL methods, and we have updated the paper to discuss this accordingly. 
At the same time, we found subpopulation attack less relevant to the current main focus of our paper. 
First, a subpopulation attack, which identifies and attacks the most vulnerable subgroups of data, fills the gap between the availability and targeted attack. While in this work, we focus on an availability-like attack, where the attacker seeks to degrade fairness guarantee for as many samples as possible (on the whole validation set). 
Second, as mentioned by the reviewer, subpopulation attack provides more scalable alternatives to craft poisoning samples, 
while the main focus of this paper is to formulate an attack goal towards fairness of high-dimensional representations and to propose a tractable proxy that can be used as an objective for poisoning samples crafting.

Reviewer comment (Novelty):
The authors say: “Heuristics such as label flipping (Mehrabi et al., 2021) lack a direct connection to the attack goal, thereby having no success guarantee and often performing unsatisfactorily.” I think this is not true. Although more limited on the attacker’s capabilities, smart manipulation of the labels can lead to successful attacks. See for example “Subpopulation Poisoning Attacks” by Jagielski et al.

Author response:
>**The authors say: “Heuristics such as label flipping (Mehrabi et al., 2021) lack a direct connection to the attack goal, thereby having no success guarantee and often performing unsatisfactorily.” I think this is not true. Although more limited on the attacker’s capabilities, smart manipulation of the labels can lead to successful attacks.**

Thanks for your thought. We meant to express that label flipping-based attacks do not directly optimize the attack goal, and the goodness of these manipulations can be hard to quantify. As a result, one may not provide theoretical analysis on the minimal number of poisoning samples as we did for the gradient-matching based attack. However, we agree that smart label flipping attack can lead to successful attacks in practice, and we have updated this statement as "*Heuristics such as label flipping (Mehrabi et al., 2021) do not directly optimize the attack goal, thereby often requiring great effort to design good manipulations to success.*"

Reviewer comment (Theory):
Equation (2) relies on strong assumptions about the distribution (Gaussian distribution and continuous variables) of the different subpopulations. How is this a good proxy for approximating the original problem? How does this compare with existing attacks (as the ones mentioned before)?

Author response:
>**Equation (2) relies on strong assumptions about the distribution (Gaussian distribution and continuous variables) of the different subpopulations. How is this a good proxy for approximating the original problem? How does this compare with existing attacks (as the ones mentioned before)?** 

Fair representations are often real-valued [1, 2, 3, 4] and thereof we believe that the "continuous variable" assumption naturally holds in general fair representation learning scenarios.

In terms of the Gaussian assumption, as discussed in remark 2.2, when this assumption holds, Eq (2) is not only a proxy, but also an optimal solution. When this assumption does not hold, using FLD (Eq (2)) as a data separability measure is still valid [5] and we believe that it is straightforward to use data separability as a measure of how difficult it is to classify them and as a proxy for the optimal BCE loss one can achieve on this classification task (which also depicts the classification difficulty). Notably, this BCE loss is always a lower bound of the mutual information in the original problem. In comparison, to our best knowledge, there is no analysis on how existing attack goals (as the ones mentioned before) relate to the mutual information between high-dimensional representations $z$ and sensitive feature $a$.

Empirically, Figure 4 in Appendix D.2 verified that FLD score acted pretty well in approximating the optimal BCE loss to classify different classes of fair representations learned by four different fair representation learning methods. In short, in this work, we use FLD because of its conceptual simplicity, interpretability, and good empirical performance.

[1] Zhao et al. Conditional learning of fair representations, 2019.

[2] Moyer et al. Invariant representations without adversarial training, 2018.

[3] Creager et al. Flexibly Fair Representation Learning by Disentanglement, 2018.

[4] Reddy et al. Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics, 2021.

[5] Fisher. The use of multiple measurements in taxonomic problems, 1936.

Reviewer comment (Theory):
How assumption 2 works in the current threat model for the attack? The authors say: “Before the attack, the victim is well trained.” What does this mean?

Author response:
>**How assumption 2 works in the current threat model for the attack? The authors say: “Before the attack, the victim is well trained.” What does this mean?**

A gradient matching-based attack relies on a *trained* victim model [1]. 
To simplify the theoretical analysis, assumption 2 states that this trained model is converged to a local stationary point of the lower-level objective before being attacked. 
This assumption is reasonable because the lower-level objective is the one that the model is trained on. 
Moreover, it allows us to bound the influence of clean gradient from clean training data, and to derive a lower bound for the minimal number of poisoning samples. This bound is valid in the sense that when the assumption is violated, typically a larger amount of poisoning samples is needed. 

The rational behind is as follows. 
When training the victim on poisoned data, each mini-batch consists of two types of samples: poisoning samples will push the victim model to optimize the upper-level objective (attack goal), whereas clean sample will counteract with this goal and instead force the victim to optimize the lower-level objective. By assuming that the victim has converged with respect to the lower-level objective, how clean samples counteracts with poisoning samples will be minimized: they only add randomness to the poisoning gradients. This allows us to derive the bound in a similar way to stochastic gradient descent. 

[1] Geiping et al. Witches’ brew: Industrial scale data poisoning via gradient matching, 2020.

Reviewer comment (Theory):
Also, for the theoretical analysis: Why do the authors think that assumption 3 is reasonable?

Author response:
>**Also, for the theoretical analysis: Why do the authors think that assumption 3 is reasonable?**

Similar to assumption 2, 
assumption 3 allows us to simplify the derivation of the lower bound. 
This assumption assumes that all poisoning samples in hand are well-crafted. 
This assumption can be achieved by conducing the gradient matching on poisoning samples one by one.
As gradient matching is a standard optimization problem, modern optimizers can solved it readily. 
In addition, assumption 3 does not require all poisoning samples to match the upper-level gradient exactly.
Instead, each of them only needs to form an unbiased estimation. 
Notably, when the assumption is violated, typically a larger amount of poisoning samples is needed, and the derived lower bound is still valid.