Paper: Adversarial Latent Feature Augmentation for Fairness (ICLR-2024)

Reviewer comment (Experiments):
Adversarial perturbations suffer from robust overfitting [1] when used as data augmentation in adversarial training, when used as data augmentation in fairness, is there exits some overfitting in promoting fairness?

Author response:
### Overfitting problem in data augmentation with adversarial training
In the proposed framework, overfitting is not a matter, because unlikely typical adversarial training, we use different objective functions in the attacking step and training step. Moreover, we can interpret the 'unfair region' described in Figure 1(a) as an overfitted or underfitted area. We directly recover the unfair region making the misclassification rates for each subgroup fair, and alleviate the overfitting and underfitting of original prediction.

Reviewer comment (Experiments):
The rotation of the decision boundary is because of the exists of adversarial pertuabations in the proposed method, will different forms of perturbations such as L2 perturbation or L-infinity effect the rotation of decision boundary?

Author response:
### About the L2 perturbation and hyperparameters
In fact, our framework is not limited to $L_\infty$ perturbation. $L2$ perturbation is also applicable because the perturbation vector $\delta$ is trained by the fairness attack objective function Eq.(5). Especially, the Sinkhorn distance regularizes the perturbated dataset has a similar distribution with the original dataset.

The reason we adapt $L_\infty$ is the hyperparameter $\epsilon$ is not straightforward if we use $L2$ perturbation. However, we may be able to omit the hyperparameter $\epsilon$ by computing the proper $\epsilon$ given dataset and decision boundary. In detail, we can measure the Euclidean distance between each subgroup and the pre-traind decision boundary. And we can set the mean distance as $\epsilon$ for the $L2$ perturbation. 

For example, if we use $L2$ perturbation by setting $\epsilon$ as the measured distance for COMPAS dataset and MLP with $\alpha=1.0$ and $\lambda = 0.5$, we obtain the range of $\delta\in[-0.2722,0.2552]$. Following the measured perturbation, we obtain a similar result with $L_\infty$ by setting $\epsilon=0.27$. Therefore, we can conclude that the choice of $L2$ and $L_\infty$ doesn't affect the performance. If a practitioner wants to control the hyperparameter $\epsilon$, $L_\infty$ is suitable. On the other hand, if one wants to simplify the setting, the $\epsilon$ measurement with $L2$ perturbation could be a wise choice. We will add this ablation study in the revision.


| COMPAS | magnitude of $\delta$| |Accuracy |  | $\Delta DP$ |  | $\Delta EOd$ |  |
| --- | --- | --- | --- | --- | --- | --- |--- | --- |
| Method |min $\delta$ | max $\delta$ | mean | std. | mean | std. | mean | std. |
| $L2$-perturbation |-0.2722|0.2552| 0.6664 | 0.0029 | 0.0108 | 0.0056 | 0.0663 | 0.0215 |
| $L_\infty$-perturbation |-0.2700|0.2700| 0.6613| 0.0033| 0.0126| 0.0065| 0.0686| 0.0212|

Reviewer comment (Experiments):
The proposed method has a relatively large number of hyperparameters, will this affect its efficiency when applied to more complex datasets?

Author response:
### About the L2 perturbation and hyperparameters
In fact, our framework is not limited to $L_\infty$ perturbation. $L2$ perturbation is also applicable because the perturbation vector $\delta$ is trained by the fairness attack objective function Eq.(5). Especially, the Sinkhorn distance regularizes the perturbated dataset has a similar distribution with the original dataset.

The reason we adapt $L_\infty$ is the hyperparameter $\epsilon$ is not straightforward if we use $L2$ perturbation. However, we may be able to omit the hyperparameter $\epsilon$ by computing the proper $\epsilon$ given dataset and decision boundary. In detail, we can measure the Euclidean distance between each subgroup and the pre-traind decision boundary. And we can set the mean distance as $\epsilon$ for the $L2$ perturbation. 

For example, if we use $L2$ perturbation by setting $\epsilon$ as the measured distance for COMPAS dataset and MLP with $\alpha=1.0$ and $\lambda = 0.5$, we obtain the range of $\delta\in[-0.2722,0.2552]$. Following the measured perturbation, we obtain a similar result with $L_\infty$ by setting $\epsilon=0.27$. Therefore, we can conclude that the choice of $L2$ and $L_\infty$ doesn't affect the performance. If a practitioner wants to control the hyperparameter $\epsilon$, $L_\infty$ is suitable. On the other hand, if one wants to simplify the setting, the $\epsilon$ measurement with $L2$ perturbation could be a wise choice. We will add this ablation study in the revision.


| COMPAS | magnitude of $\delta$| |Accuracy |  | $\Delta DP$ |  | $\Delta EOd$ |  |
| --- | --- | --- | --- | --- | --- | --- |--- | --- |
| Method |min $\delta$ | max $\delta$ | mean | std. | mean | std. | mean | std. |
| $L2$-perturbation |-0.2722|0.2552| 0.6664 | 0.0029 | 0.0108 | 0.0056 | 0.0663 | 0.0215 |
| $L_\infty$-perturbation |-0.2700|0.2700| 0.6613| 0.0033| 0.0126| 0.0065| 0.0686| 0.0212|

When it comes to $\lambda$, the selection of $\lambda$ is significant to the accuracy-fairness trade-off since it works as the weight between the original and perturbed dataset. Therefore, we vary $\lambda$ and report the 'line' of the performances in the Pareto Frontier as well as varying $\alpha$ in Eq.(5). We suggest using $\lambda=0.5$ since it empirically shows the best accuracy-fairness trade-off for all datasets. Still, the hyperparameter tuning is needed for $\alpha$. Here we report the training time for experiments for each combination of hyperparameters. Each fine-tuning is conducted in 50 epochs. We measure the training time for a single run for baseline, and the average of 10 runs for fine-tuning. 

| Dataset | Baseline | Fine-tuning|
| --- | --- | --- |
| Adult | 40.046s| 37.018s |
| COMPAS |9.118s| 6.166s|
| German |3.934s |1.177s| 
| Drug| 4.748s | 1.838s|

Therefore, we can omit the hyperparameter $\epsilon$ and $\lambda$ by using $L2$ perturbation taking the mean Euclidean distance between the latent feature and the decision boundary as $\epsilon$ and setting $\lambda=0.5$ to make the original dataset and perturbed dataset equally contribute. However, even though only one hyperparameter, $\alpha$, is need to be tuned, the hyperparameter tunig itself takes less time compared to the original ERM because fine-tuning trains only the last layer of the model.

Reviewer comment (Writing):
Why "GAN-based perturbation might yield unsuitable generated features not aligning on the sensitive hyperplane" for tabular datasets? The intuition is unclear to me. It would be better to explain using concrete examples and provide citations.

Author response:
### The statement about GAN in tabular dataset
The statement in the paper reviewer mentioned is based on our empirical observation of the existing work, FAAP [1] which is applicable only to the image dataset. As shown in Figures 1 and 2 in the paper demonstrating the experimental result on tabular datasets, FAAP shows poor performance in tabular datasets and we suspect it is because the GAN-based perturbation might not be converged well.

Reviewer comment (Experiments):
Can the proposed method be applied to non-tabular datasets, such as vision datasets (i.e., images and videos)? If not, the scope of the method is limited. If so, why does W1 matter?

Author response:
### Application to vision dataset
We reported the experimental results for the image dataset, CelebA as shown in Figure 4. In this case, FAAP shows comparable results to ours, and W1 is not a matter in this case. Therefore, it shows that FAAP works only for image datasets, while ALFA (ours) works for both tabular and image datasets. Moreover, we conduct additional experiments applying ALFA to the NLP dataset, as shown in the global response in this rebuttal.

Reviewer comment (Writing):
In Eq.(1), the perturbation \delta should be bolded. What is the \bar{a} in Eq.(1)? Is this the mean of all a_i? The notation of \bar{a} is not defined before using.

Author response:
### Missing notations
Thanks for pointing out the typos. We will revise them and screen again the entire notations thoroughly.

Reviewer comment (Novelty):
Adversal fairness learning is not novel. The earliest work dates back to 2018 [1]. I doubt the novelty of the proposed method in 3.3.

Author response:
### The novelty of adversarial fairness
ALFA and [2] have different perspectives on adversarial training. [2] uses an adversary to predict sensitive attributes, while the classifier predicts the label. The objective function for the classifier is designed to minimize the binary cross-entropy (BCE) loss on the label while maximizing BCE loss on the sensitive attribute, which makes the classifier not use the demographic information during the prediction. However, no mathematical derivation is introduced as to how maximizing BCE on the sensitive attribute makes the classifier fair. Moreover, [2] is designed only for the COMPAS dataset. 

We contain recent adversarial training methods in the paper such as [1] and [3], and describe how they are different from ours, in Section 1 and Section 2, respectively.

Reviewer comment (Experiments):
According to the setting and empirical studies, the proposed method can be applied only when there is only one sensitive attribute with binary values.

Author response:
### Beyond binary labels and sensitive attributes
Please see the global response, addressing more general scenarios beyond binary classification with binary-sensitive attributes.

Reviewer comment (Experiments):
Storage cost increase due to dataset augmentation. Not necessarily a flaw.

Author response:
### Storage cost
ALFA is effective in terms of storage or memory cost. In detail, we can think about two types of augmentation, offline augmentation (storing augmented data in the drive) and online augmentation (augmentation on the data loader, stored in the memory temporarily).

For the offline augmentation, more storage cost may be required compared to the baseline ALFA. However, it requires less storage cost compared to the data augmentation on the input space. For example, for $N$ samples, the storage cost for $N$ images might be huge, while ours requires a single $N \times d$ matrix where $d$ is the dimension of the latent feature.

For the online augmentation, the data augmentation on the input space may require more training time and memory space. However, as ours is a fine-tuning method using a latent feature from original images and a perturbed latent feature, we only require two $N \times d$ matrices in the memory space removing the dataloader from the input images in the memory.

Reviewer comment (Experiments):
Limitation: Paper only deals with binary classification and two protected classes.

Author response:
### Beyond binary $Y$ and $A$
Please see the global response, addressing more general scenarios beyond binary classification with binary-sensitive attributes.

Reviewer comment (Theory):
The current solution of adjusting rotation of the linear classifier would only work when the two classes are separated in their latent space. For more complicated cases like lesser data, higher dimensional data etc. the latent space might not be separated and intuition would not hold. However this would be future direction to look into.

Author response:
### Linear separability assumption
We agree with the reviewer's point that the linear separability assumption is needed in our framework. However, linear separability is a widespread assumption [1] even though the size of the dataset is small or the number of features is large even in the input space. Moreover, deep neural networks transform the input data into well-separated latent features, making the classification easier.

[1] Moshkovitz, Michal, Yao-Yuan Yang, and Kamalika Chaudhuri. "Connecting interpretability and robustness in decision trees through separation." International Conference on Machine Learning. PMLR, 2021.

## Linear separability
If the demographic groups in the latent are perfectly inter-mixed, it indicates a fair representation in itself. Naturally, we hypothesize they have separation. In fact, literature [1,2,3] requires the linear separability of sensitive groups since they use an auxiliary classification model predicting sensitive attributes. 
However, linear separability is not strictly required in our framework. Even if the demographic groups are partially inter-mixed, we can still define the unfair regions and the decision boundary can be rotated as long as the latent feature distribution is linearly separable in terms of labels. 

[1] Zhibo Wang, Xiaowei Dong, Henry Xue, Zhifei Zhang, Weifeng Chiu, Tao Wei, and Kui Ren. Fairness-aware adversarial perturbation towards bias mitigation for deployed deep models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10379-10388, 2022b

[2] Christina Wadsworth, Francesca Vera, Chris Piech. Achieving Fairness through Adversarial Learning: an Application to Recidivism Prediction. FAT/ML Workshop, 2018

[3] Ramaswamy, Vikram V., Sunnie SY Kim, and Olga Russakovsky. "Fair attribute classification through latent space de-biasing." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

Reviewer comment (Theory):
In theorem 3.2, $N_p = 4\times \max(n_{i,j})$. How is this maintained? Does this mean some samples would be repeated before adversarial attack?

Author response:
### Sample repetition
Yes, the data sampling is repeated to maintain the size of each subgroup to be the same. Although this repetition is not a mandatory condition empirically, we make the subgroup size equal to ensure the efficiency of the fairness attack as shown in Theorem 3.2.

Reviewer comment (Reproducibility):
The final loss function is a convex combination of true and perturbed dataset. I wanted to understand the weight attributed to the perturbed dataset. What values of $\lambda$ are used in different cases?

Author response:
### Contribution of $\lambda$
The selection of $\lambda$ is significant to the accuracy-fairness trade-off since it works as the weight between the original and perturbed dataset. Therefore, we vary $\lambda$ and report the 'line' of the performances in the Pareto Frontier in Figures 2 and 3 in the paper, as well as varying $\alpha$ in Eq.(5).

## Contribution of $\lambda$
We visualize the effect of $\lambda$ for three datasets, COMPAS, German, and Drug with Logistic Regression varying $\lambda=0.1,0.3,0.5,0.7,0.9$ and the set $\lambda=0$ as a baseline in Figure 11, Appendix M. Compared to the baseline($\lambda=0$), any $\lambda$ improve fairness. Intuitively, relying heavily on either original samples or perturbed samples might not be a good strategy. We suggest $\lambda=0.5$ as the default setting making the original and augmented features have an equal contribution.

Reviewer comment (Theory):
How can linear decision boundary handle the case of multiple classes in protected attribute? A rotation would not be able to fix the bias in the dataset.

Author response:
## Experimental results for multi-class classification
Here we report a new experimental result with $EOd_{\text{multi}}$. As $EOd_{\text{multi}}$ is not strictly defined in literature, we follows the definition of Predictive Equality (PE) and Equal Opportunity (EO) notion in [4] for each class, and take maximum of the summation of PE and EO similar to $DP_{\text{multi}}$. We define $EOd_{\text{multi}}$ as 
\begin{align}
EOd_{\text{multi}} =  \max_{k\in[K]} &\Bigl( \bigl\vert P(\hat{Y}=k \vert a=1, Y=k) -P(\hat{Y}=k \vert a=0, Y=k)\bigr\vert  \\\\
&+ \bigl\vert P(\hat{Y}\neq k \vert a=1, Y\neq k) -P(\hat{Y}\neq k \vert a=0, Y\neq k) \bigr \vert \Bigr).
\end{align}


Here we reveal the technical changes in the rebuttal period. In the binary classification, the fine-tuning learning rate of 0.01 and 0.001 didn't affect the experimental result, so we used 0.01 for convenience (including the previous result in the rebuttal). However, we found that ALFA's performance on multi-class classification is sensitive to hyperparameters including learning rate. While we fix $\lambda=0.5$, we tune other hyperparameters and obtain improved results with $\alpha=0.01$, $\epsilon=1$, and $lr=0.001$.


| Drug-multi| Accuracy |  | $\Delta DP_{\text{multi}}$ |  |$\Delta EOd_{\text{multi}}$ |  |
| --- | --- | --- | --- | --- |  --- | --- | 
| Model | mean | std. | mean | std. |mean | std. | 
| MLP | 0.5207|0.0024|0.1917|0.0062|0.3010|0.0158|
| MLP + ALFA (lr=0.01) | 0.4851|0.0231|0.1669|0.0224|0.3185|0.0551|
| MLP + ALFA (lr=0.001)| 0.5501|0.0013|0.0374|0.0323|0.1934|0.0302|

We grid search the hyperpameter by fixing $\lambda=0.5$, set the searching range as $\epsilon=[0.1, 0.2,0.5,1]$, $\alpha=[0.01,0.1,1,10]$, and $lr=[0.01, 0.001]$. Each fine-tuning is conducted in 30 epochs. We measure the training time for a single run for baseline, and the average of 10 runs for fine-tuning. 

| Dataset | Baseline | Fine-tuning|
| --- | --- | --- |
| Drug (Multi)| 9.580s | 6.929s|

[4] Rouzot, Julien, Julien Ferry, and Marie-José Huguet. "Learning Optimal Fair Scoring Systems for Multi-Class Classification." 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2022.

Reviewer comment (Evaluation):
It is hard to infer the experimental conclusions from the many plots which are represented - is there any reason for not observing a non decreasing or non increasing trend in most of the experiments?

Author response:
Thanks for pointing out. We will revise the explanation for the figure. In detail, the Pareto Frontier shows the trade-off between accuracy and fairness metric, while the black solid line shows the optimal trade-off. The closer a point to the black line, the better the accuracy-fairness trade-off is. In most cases, ours is located in the upper-right of the figure, which means it can achieve fairness while minimizing the compromising of accuracy. 

In fact, most of the existing frameworks shown in the figure are designed to improve fairness while maintaining accuracy. And only a few cases the fairness frameworks can improve accuracy too. It is hard to observe either decreasing or increasing accuracy in the experimental result. Moreover, we exclude some experimental results if they do not converge well, or are worse than the base rate.

Reviewer comment (Experiments):
There seems to be no comparison with baselines such as reweighting and using only ERM features (no data augmentation) - how does the performance compare?

Author response:
We mainly focus on augmentation-based methods or some methods conducted in the latent space. However, as the reviewer asked, we can further compare the baseline and reweighting methods. In particular, we implement the state-of-the-art reweighting methods [2]. The revised Pareto Frontier, Figures 2 and 3 in revised paper's PDF file shows that ALFA outperforms the baseline and reweighting method. Moreover, the improvement in [2] is not consistent, whereas ALFA shows promising improvement.

Reviewer comment (Writing):
Overall, the conciseness and coherence of the paper could be improved - e.g. :During the adversarial training, we ensure that the semantic essence of the perturbed features is preserved “ -> semantic essence could be replaced with a more technical term w.r.t the true invariant / sensitive attributes.

Author response:
Thanks for pointing out. We will thoroughly revise paragraphs to be more readable and understandable, especially for the points suggested such as 'semantic essence', the readability of plots, and 'hyperplane rotation'.

Reviewer comment (Theory):
Is there any underlying assumption on the structure of the space? Perhaps, it would be helpful to have a causal model explaining the structure of the input space (e.g. to depict the spurious correlation between Y and A).

Author response:
Although we don't explicitly use the causal structure, we assume the causal structure like this figure, the sensitive attribute ($A$) implicitly affects other features ($X$), and the final classifier learns the latent feature $Z$ produced by an encoder utilizing from $A$ and $X$. But ALFA mitigates bias in $Z$, producing $Z^\prime$, which is independent of A. **We present the causal structure depicted in Appendix L of the revised paper's PDF file.**

Reviewer comment (Experiments):
Could you compare your intuitions with the following adversarial group robustness technique? Paranjape, Bhargavi, et al. "AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization." arXiv preprint arXiv:2212.00921 (2022)

Author response:
The rough ideas of 'finding unfair regions' in our method and 'identifying error-prone groups' in AGRO [6] are similar. However, the main differences between them are 1) computational cost, and 2) the objective of adversarial training. 
- First, AGRO suggested an end-to-end framework to mitigate spurious correlation, while ours is a fine-tuning method, which doesn't require the encoder doesn't have to be trained. As only the final classifier is trained in ALFA, our method is more effective. 
- Second, adversarial training in AGRO focuses on discovering the error-prone group which is not accurate though the group information is unknown. After the inference of the 'worst-group', the classifier aims to minimize the error of the worst group. In a word, AGRO is an improved version of G-DRO [7] which requires group information. In terms of the usage of group information, AGRO is a prominent methodology that can automatically detect the error-prone group. 

    However, our proposed method considers both the underestimated and overestimated groups, where one of them could be the worst group in AGRO. For example, if a privileged group with a negative label ($y=0,a=1$) has the highest misclassification rate, it is an overestimated group and an error-prone (worst) group too. On the other hand, if an unprivileged group with a positive label ($y=1,a=0$) has a high misclassification rate but not as much as in the worst-group, it is an underestimated group but not defined as an error-prone group in AGRO. To sum up, we address more diverse cases. As shown in Figure 2 in the rebuttal PDF files, both overestimated and underestimated groups are considered in ALFA. In detail, given the demographic information, ALFA detects 'unfair regions' which implies either disproportionately overestimated or underestimated, and directly recovers the region using a fairness attack as shown in Figure 1 in the manuscript. 
     
    Although ours considers both overestimated and underestimated groups while AGRO considers underestimated groups only, the idea of AGRO seeking the error-prone group without the group information is great. It could be a future direction of our fairness attack method operating without the sensitive attribute.
    
- Third, the objectives of AGRO and ours are different. As AGRO addresses the spurious correlation of the model, it is interested in the worst-group accuracy only. In contrast, as our objective is to make a fair classifier, we are interested in the balanced true and false positive rates across the demographic group. Although they share the interest in mitigating bias in the model, the difference in the objective and evaluation metric provokes different thoughts about the subgroup we take care of.

Reviewer comment (Writing):
"Hyperplane rotation" may be hard to understand in the first read, could we replaced with a more common term like decision boundary.

Author response:
Thanks for pointing out. We will thoroughly revise paragraphs to be more readable and understandable, especially for the points suggested such as 'semantic essence', the readability of plots, and 'hyperplane rotation'.

Reviewer comment (Reproducibility):
One of the key aspects of ALFA's design is its ability to generate biased feature perturbations during training. This is achieved by introducing a novel loss function that encourages the model to focus on features that are particularly sensitive to small perturbations in the input space. While this approach can effectively rotate the decision boundary to cover previously unfavorable regions, it may also introduce computational complexity and training time challenges. ALFA's computational requirements are not explicitly stated in the article, but it's essential to consider them in order to evaluate its practicality for large-scale datasets and complex models.

Author response:
When it comes to computational complexity, ALFA requires the same memory space as the original training during the attacking step. In the fine-tuning step, as it stores perturbed latent features as a matrix form and is directly applied to the final classifier, only the final classifier needs the additional computational cost for the single latent feature matrix $Z^\prime \in \mathbb{R}^{N \times d}$ where $N$ is the number of samples and $d$ is the dimension of the latent feature. 
For example, let's assume we use $L$ layers of MLP, and each layer has the same dimension $d$ for simplicity. Then, the computational complexity is $\mathcal{O}(pdN + (L-1) d^2N + dN) )$ where $p$ is the number of features of the input. During the attacking step, we freeze the encoder and classifier, only computing the covariance and the perturbation update, where each of them takes $\mathcal{O}(dN)$ complexity. Moreover, during the fine-tuning step, it also takes complexity of $\mathcal{O}(dN)$. Therefore, the additional computational complexity for an iteration is $\mathcal{O}(3dN)$ including covariance calculation, perturbation update, and fine-tuning, which is significantly light compared to the ERM.

Reviewer comment (Experiments):
While ALFA may improve fairness measures for certain groups, it's important to consider its overall impact on model stability and robustness. Adversarial loss can make training process vulnerable to small perturbations in the input space, potentially leading to increased numerical instability. This issue is not addressed in the article, and further research is needed to investigate these potential trade-offs when employing ALFA in practice.

Author response:
In typical adversarial training, the min-max optimization or bi-level (alternative) optimization may not be stable. However, it doesn't happen in ALFA since it requires a one-time attacking step to maximize the fairness constraint, and the augmented data is used in the training to minimize binary cross-entropy loss. Therefore, our framework has the same stability as traditional empirical risk minimization (ERM).

Reviewer comment (Experiments):
The paper considers fine-tuning only the classifier part of the network (i.e., the last layer) and freezes the encoder part of the network. While this method works for simple tabular-like datasets, I have concerns about how this method performs in the case of datasets with a large label space, like ImageNet.

Author response:
### Extension to diverse data types and scenarios
We adopt the fine-tuning strategy of freezing the encoder part, our experimental results show that this strategy effectively achieves fairness even for the image dataset (CelebA) and text dataset (Wiki), as shown in the manuscript and the global response. 

We also further research the usage of ALFA in the multi-class classification, as it can be conceptualized as multiple One-Vs-All binary classifications as presented in the global response. Though ALFA can address multi-class classification, the improvement is not significant, and the dataset used in the experiment is not large enough in terms of its size and label space.

We summarize our future works here. First, we will find or construct a large dataset having fairness concerns since the existing benchmark datasets for fairness research are relatively small in terms of size and label space. As we have verified the effectiveness of finding and directly recovering unfair regions, we expect that ALFA can be applied to large label space. Second, we are interested in the long-tail classification scenario, where the label space is large and extremely imbalanced. Although this topic is not directly related to the fairness problem, we motivate this scenario because it might suffer from highly overestimated or underestimated classes, which our framework can address.

Reviewer comment (Novelty):
In general, the idea is not novel as the perturbation in the latent space is already explored by the earlier methods.

Author response:
### Comparison to existing perturbation methods in latent space
We have investigated the literature about latent perturbation methods for fairness and identified two existing works [1,2]. Both use GAN for the perturbation, and the direction of the perturbation is regarding the auxiliary decision boundary, produced by the sensitive attribute. In detail, [1] generates a perturbed sample having opposite sensitive attributes but the same target labels locating it in a symmetric point. Therefore, the role of perturbation is to make the number of samples and distribution similar across the subgroups. However, [1] is based on the strong assumption that each subgroup's data distribution should be symmetric about the decision boundaries. [2] also uses the sensitive attribute decision boundary and generates the perturbation towards that decision boundary. We show the conceptual image showing that this perturbation doesn't necessarily lead to the fair decision boundary, as shown in Figure 5(b) in the paper. Moreover, both [1] and [2] are designed only for the image dataset. In our paper, we empirically show that FAAP [2] produces poor results in tabular datasets as shown in Figure 2 and 3.

**The novelty of our method can be summarized:** 
1) the adaptiveness to diverse types of data, 
2) defining unfair regions where a demographic group is disproportionately overestimated (high false positive rate) and underestimated (high false negative rate), 
3) directly recovering the unfair regions using fairness attack which is counter-intuitive usage of fairness constraint, generating biased dataset to mitigate bias.

Although we have identified only two existing methods, we are open to further discussion about the perturbation method in latent space for fairness. In particular, we will study further how to build interpretable latent perturbation for fairness.

Reviewer comment (Experiments):
What is the performance of the adversarial training by perturbing the input space instead of latent space with similar attack objective?

Author response:
### Perturbation in input space
In fact, as the logistic regression doesn't have an encoder, the fairness attack is conducted on the input space in this case. For other models, we observed that the same objective function replacing latent features to input features works comparably but slightly worse than the current version. Because our strategy, decision boundary rotation, is based on the assumption that the latent features are linearly separable, the perturbation on the input space doesn't effectively contribute to the rotation of the decision boundary due to the neural network's non-linear transformation.

Reviewer comment (Evaluation):
In section 4.4.1, authors could quantitatively discuss the improvements in terms of performance and fairness metrics. It is not clear to me from the graphs about the comparison of different methods.

Author response:
### Explanation of figure and quantitative discussion
We will revise the explanation for the figure. In detail, the Pareto Frontier shows the trade-off between accuracy and fairness metric, while the black solid line shows the optimal trade-off. The closer a point to the black line, the better the accuracy-fairness trade-off is. In most cases, ours is located in the upper-right of the figure, which means it can achieve fairness while minimizing the compromising of accuracy. 

For the fair quantitative comparison, we pick an experimental result when the accuracy is the highest among the points in the Pareto Frontier for each method. We report the table in Appendix G. The experimental results show that the fairness improvement is outstanding in most cases, achieving the best or second-best fairness score.