Paper: Unsupervised Lifelong Learning with Sustained Representation Fairness (ICLR-2024)

Reviewer comment (Experiments):
The datasets used for evaluation are quite simple once’s - tabular data only. It’s shown that AdaBoost is why more accurate at tabular [1] data so in general why should one use DNN here?

Author response:
We use an elastic DNN in this work because its depth can serve as an indirect metric to measure the distance between two tasks. We appreciate your suggestion and conduct experiments on the vision datasets. Please find out the answer in A2 of Reviewer 8gZZ due to character limitations.

Reviewer comment (Experiments):
Are there any kind of loopbacks studied? Like the algorithm does a decision which is used for new data which is used to further train the algorithm etc.?

Author response:
Our design will not incur loopback and was widely used in the literature, like [3]. This is because not all the predicted pseudo labels are integrated in the retained dataset; rather, only those predicted with high confidence and more likely to be correct labels are integrated. To bolster the confidence of these samples, we implemented a margin mechanism. More specifically, a classifier has been designed to determine the membership of each sample, distinguishing whether it originates from the learned retained dataset or a new task. Only those samples from new tasks that exhibit a probability exceeding 80 percent (a margin commonly used across most datasets) of belonging to the retained dataset, as determined by the classifier's sigmoid layer prediction, are actually added to the retained dataset

[3] Sohn, Kihyuk, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A. Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." *Advances in neural information processing systems* 33 (2020): 596-608.

Reviewer comment (Writing):
„in-domain bias and fail to generalize well to new tasks with data distributional shifts (Barrett et al.).“: Missing year in the reference

Author response:
We will correct this typo in camera ready.

Reviewer comment (Experiments):
4.1. in the table you write „tasks“ what exactly are the tasks in the dataset? This is not really well explained in the text.

Author response:
Each task represents a distinct, non-overlapping subset of the original dataset. For instance, in the case of the adult dataset, individuals (instances) are categorized into 12 separate groups (tasks) based on their occupations, with examples including Tech-support, Sales, and Farming-fishing, among others. The rationale behind selecting occupation, as inspired by [4], is to ensure a varying distribution of P(Y|P) across different tasks. This approach leads to a dynamic proportion of males and females classified as having high incomes within various occupational categories. Once a model has effectively learned from a specific group of people (task), it is crucial to sustain its performance across other groups (tasks).

[4] Le Quy, Tai, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi. "A survey on datasets for fairness‐aware machine learning." *Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery* 12, no. 3 (2022): e1452.

Reviewer comment (Novelty):
There is a resent paper which uses stochastic quantized representations [2], this can ensures fair representation also after a distribution shift. However, I agree that the method proposed here is a more general way to address the problem of new incoming data.

Author response:
Thanks for bringing this paper into our radar. We scrutinized the paper and found that it focused on a different learning problem from ours. Specifically, the paper [5] proposed an alternative de-biasing function through the random (stochastic) projection of discriminative feature representations. However, there is no evidence to show that such a de-biasing function can hold invariant across different tasks, each of which has a distinct demographic distribution. Thus, their proposed method cannot be directly adapted into our lifelong learning context. We will add this discussion in our camera ready. 

[5] Cerrato, Mattia, Marius Köppel, Roberto Esposito, and Stefan Kramer. "Invariant representations with stochastically quantized neural networks." In *Proceedings of the AAAI Conference on Artificial Intelligence*, vol. 37, no. 6, pp. 6962-6970. 2023.

Reviewer comment (Experiments):
The paper needs more justification to support/clarify why their approach works so well. The theoretical section provides some insights into it but would be interesting to see how these theoretical results compare with a simple baseline, e.g. an adversarial network.

Author response:
Theorem 1 indicates our algorithm's performance is similar to an optimal-depth adversarial network, which is infeasible in practice, with comparable cumulative loss bounded by $O(\frac{\ln(L)}{1-\beta})$.
FaDL serves as the baseline and our results show that FaDL fails to generalize fair representations across different tasks, underperforming our algorithm by 7.6% in accuracy on average.

Reviewer comment (Experiments):
The claim of being unsupervised is not well supported as the method uses task labels for the first task. Moreover, since the focus is on learning fair representations the protected label should ideally be considered as the primary supervision. On a similar note, comparison with some of the baselines may be slightly unfair because they do not rely on task labels.

Author response:
We agree that coining our regime as purely “unsupervised” is not rigor. However, given its reliance on labeling a single task in a multi task context, it is less burdensome than semi-supervised approaches requiring labels for all tasks, 
distinguishing our approach from those like FaDL and FaIRL, which necessitate full label information.

Reviewer comment (Writing):
Introduction, paragraph 3: The sentence starting with "This strategy..." is quite hard to understand.

Author response:
Divergent demographic distributions means that P (Y | P) varies across different tasks at various extends. We display this for the Adult dataset only in the table below due to the character limitations.
"Incrementing fair representation" extracts latent features for learned data $R^{(i-1)}$ and new task $\mathcal{T}_i$. This process is efficient due to a similar demographic distribution between $R^{(i-1)}$ and $\mathcal{T}_i$, achieved by starting with similar tasks and leaving distinct tasks for later. Gradually, $R^{(i-1)}$ incorporates various aspects of prior tasks, aligning with diverse task distributions. The feature extractor's parameters are reused, enhancing the incremental nature of the representation process.

Distribution of P(Y=1|P=1) / P(Y=1|P=0) across all tasks in Adult.
|Dataset|T_1|T_2|T_3|T_4|T_5|T_6|T_7|T_8|T_9|T_10|T_11|T_12|
|-------|---|---|---|---|---|---|---|---|---|----|----|----|
|Adult|0.624/0.376|0.842/0.158|0.971/0.029|0.849/0.151|0.982/0.018|0.923/0.077|0.933/0.067|0.609/0.391|0.975/0.025|0.785/0.215|0.948/0.052|0.907/0.093|

Reviewer comment (Writing):
In the same paragraph of Section 2.2: Several terms like "negative model use" and "independent classifier groups" are unclear and need to be defined earlier in the paper.

Author response:
The term is first introduced on Page 2 (Introduction) with citing papers. It's akin to "negative transfer" [3], which means a large difference between learned and new tasks makes reusing the trained model be less effective than starting with a new model for the new task.

[3]https://openaccess.thecvf.com/content_CVPR_2019/html/Wang_Characterizing_and_Avoiding_Negative_Transfer_CVPR_2019_paper.html

Reviewer comment (Experiments):
The setup is unclear. If FaRULi performs a reordering of the learning tasks after it receives different tasks, when exactly are the predictions of the task recorded? It is also unclear if this is consistent with the baseline approaches.

Author response:
In each round, $\mathcal{T}_i$ is chosen for prediction based on similarity to $R^{(i-1)}$. The process stops when adversarial training converges, marked by reaching a maximum number of epochs. The task with the lowest Q value is then chosen.
All competing models follow the same setup. Tasks are split into training and testing subsets where testing subsets are used to calculate accuracy and statistical parity after learning on training subsets. A high Q suggests either uniform weight distribution across layers or significant weights in deeper layers, implying a deep network. Conversely, a low Q indicates that shallow layers are more important.

Reviewer comment (Writing):
In Section 2.3: It is unclear how Q measures the distance between different tasks.

Author response:
In each round, $\mathcal{T}_i$ is chosen for prediction based on similarity to $R^{(i-1)}$. The process stops when adversarial training converges, marked by reaching a maximum number of epochs. The task with the lowest Q value is then chosen.
All competing models follow the same setup. Tasks are split into training and testing subsets where testing subsets are used to calculate accuracy and statistical parity after learning on training subsets. A high Q suggests either uniform weight distribution across layers or significant weights in deeper layers, implying a deep network. Conversely, a low Q indicates that shallow layers are more important.

Reviewer comment (Writing):
In Theorem 1: It is unclear where the loss L_FaRULi is defined.

Author response:
$\mathcal{L}_{FaRULi}$ is the cumulative loss of all layers over T rounds. Details can be found in Appendix 1.1.

Reviewer comment (Evaluation):
The information in Table 1 is presented in a manner that makes it challenging to interpret. To facilitate comparisons, it is advisable to emphasize the highest accuracy and the lowest fairness violation. In light of the results, it's apparent that FaRULi only manages to achieve a somewhat modest level of performance concerning accuracy and fairness. This raises the question of how this outcome supports the claim that "On average, our model surpasses three leading competitors by 12.7% in prediction accuracy and by a substantial 42.8% in terms of ensuring statistical parity."

Author response:
We respectfully point out that only FaDL and FaIRL are our rival models, representing the state of the art in the FRL context. Compared to FaDL and FaIRL, our proposed approach demonstrated significant performance improvement in terms of prediction accuracy by 20.4% and statistical parity by 44.4%. The other methods including UnFaIRL, FaMTL, and ULLC are not direct competitors. Specifically, UnFaIRL without optimizing the learning order is the variant of our proposed approach, crafted for ablation study. FaMTL and ULLC represent the performance upper bound in our context. Namely, FaMTL learns all tasks with all labels at once, where the fairness constraint is enforced in such offline multi-task learning setting, thus it always presents the best accuracy performance. While ULLC can learn tasks in sequence with only one labeled task (like our algorithm does), it does not impose fairness constraints. We thus can observe that ULLC slightly outperforms our algorithm in accuracy, but is inferior to ours in the fairness metric by substantial margins.

Reviewer comment (Experiments):
The experiment exclusively utilizes tabular data. However, it raises the question of how this proposed approach can effectively address more intricate tasks, such as those in the domains of computer vision or natural language processing.

Author response:
Yes, we followed [1] to construct an MNIST image dataset, with the protected feature being the background color of each digit type. Specifically, MNIST is divided into five tasks, each containing ten digits from 0 to 9. These tasks differ in two aspects. First, the class distribution P(Y) varies across tasks, with the largest number of images representing a certain digit type differing from one task to the others. Second, we designated red and green colors as the protected features, and the conditional P(Y|P) also varies. For instance, Task 1 may have more digit "1" in red and digit "9" in green, while the other tasks may exhibit reversed patterns. Here, we intentionally designated Task 3 ($\mathcal{T}_3$) as the most distant task, in which its conditional P(Y|P) differs significantly from all the other four tasks. Intuitively, a fair classifier should not establish a superficial correlation between color and digit, thus accurately predicting an input image regardless of its background color. We compared our FaRULi model with FaDL, and the results are presented in the table below. $\mathcal{T}_1$ is selected as the initial task with full labels. From the results, we observed that our model outperforms FaDL by 3.4% and 0.148 in Accuracy and Equalized odds (EO) on average, respectively. Moreover, FaDL performs better in the first two tasks $\mathcal{T}_1$ and $\mathcal{T}_2$ (average accuracy = 92.5%, average EO = 0.242) but cannot generalize to other tasks. We extrapolate that the function to debias color from digit type learned in $\mathcal{T}_1$ and $\mathcal{T}_2$ by FaDL cannot be adapted to other tasks, even if $\mathcal{T}_3$ has slightly different P(Y|P). In contrast, our model maintains good performance on $\mathcal{T}_1$, $\mathcal{T}_2$, $\mathcal{T}_4$, and $\mathcal{T}_5$, all of which have different conditional P(Y|P). On the most distant $\mathcal{T}_3$, our model achieves comparable accuracy performance as FaDL. This aligns with our assumption that the model cannot generalize to a highly dissimilar task, and our model resolves the negative transfer issue by prioritizing $\mathcal{T}_2$, $\mathcal{T}_4$, and $\mathcal{T}_5$ and achieving significantly better EO performance on them compared to FaDL.

| MNIST           | Task-1 | Task-2 | Task-3 | Task-4 | Task-5 |
|-------------------|------------|--------|--------|--------|--------|
| FaDL Accuracy     | 0.939      | 0.911  | 0.855  | 0.851  | 0.886  |        
| FaDL EO           | 0.267      | 0.217  | 0.396  | 0.379  | 0.334  |        
| Our Model Accuracy| 0.952      | 0.929  | 0.887  | 0.946  | 0.933  |        
| Our Model EO      | 0.093      | 0.164  | 0.218  | 0.099  | 0.130  |        

[1] Bahng, Hyojin, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. "Learning de-biased representations with biased representations." In *International Conference on Machine Learning*, pp. 528-539. PMLR, 2020.

Reviewer comment (Evaluation):
The authors choose demographic/statistical parity as their fairness criterion of choice, which asks that subjects from group A receive positive predictions with equal likelihood as those from group B regardless of the two groups' baseline likelihoods of receiving a positive outcome. For example, in the diabetes task, this would require that even if subjects from group A have a much higher prevalence of diabetes, the model would still have to predict diabetes at equal rates for both groups, essentially forcing the model to make false predictions. There are scenarios where this may be considered a desirable outcome, but they seem rather rare; for this reason, equal opportunity or equalized odds are much more widely used criteria in the algorithmic fairness literature. In their discussion of fairness definitions, the authors completely gloss over these two absolute standard criteria.

Author response:
Q1: Can the fairness constraint ..., such as equalized odds (EO)? 

A1: Yes, our proposed framework is general, allowing for various fairness metrics. Based on your suggestion, we've implemented EO, presenting parts of results as follows:

| Dataset  | FaDL  | FaIRL | FaRULi |
|-----|-----|------|-------|
| Diabetes | 0.068 | 0.051 | 0.031  |
| Law      | 0.234 | 0.091 | 0.087  |

We can observe that the empirical superiority of our proposed FaRULi algorithm is invariant to the fairness metric, and our algorithm outperforms its two competitors FaDL and FaIRL in both datasets with 0.031 and 0.087, respectively.

Reviewer comment (Evaluation):
Enforcing statistical parity in the face of baseline/prevalence differences forces model misclassifications. This is likely to have strongly affected the results obtained by the authors in the different tasks, as they may simply be varying in their level of baseline differences. Moreover, the degree to which this will affect model performance/accuracy will differ based on how strongly statistical parity is enforced / the choice of the regularization constant, which seems likely to differ between the compared methods. These two factors may have strongly influenced the results obtained by the authors, maybe even more so than any intrinsic differences between the datasets/tasks and modeling/training approaches. For example, FaDL may obtain worse accuracy but better statistical parity on the Dutch dataset because it was more successful at enforcing statistical parity, which is fundamentally incompatible with high accuracy in datasets with strong baserate differences.

Author response:
A5: Yes, our experiments on the Law School dataset show that increasing $\lambda_1$ decreased accuracy from 91.6% to 4.7%, demonstrating that strict fairness constraints can lower accuracy and affect fairness. Both statistical parity (SP) and EO values fall to zero, with the learner classifying all instances as Y=0 to avoid unfair outcomes. 

| $\lambda_1$ = |0.01|0.03|0.05|0.1|0.5|1|
|------------|-------|-------|-------|-------|-------|-------|
| Acc| 0.916 |0.939|0.899|0.941|0.047|0.047|
| SP|0.162|0.002 |0.054 |0.012 |0|0|
| EO|0.269|0.087|0.107|0.050|0|0|

Reviewer comment (Experiments):
The author suggests several possible remedies regarding fairness constraint enforcement:  
- Implementing at least one other kind of fairness constraint, preferably equal opportunity or equalized odds, as this should be feasible given the generality of the authors’ approach and would likely reduce the fairness-performance trade-off.  
- Reporting the baseline incidences \(P(Y \mid P)\) for all protected groups in all datasets.  
- Adding single-task supervised unconstrained and fairness-constrained baselines for each dataset to better gauge upper performance bounds.  
- Assessing the impact of the fairness regularization constants, or tuning them such that all approaches achieve comparable levels of fairness/unfairness.

Author response:
Q1: Can the fairness constraint ..., such as equalized odds (EO)? 

A1: Yes, our proposed framework is general, allowing for various fairness metrics. Based on your suggestion, we've implemented EO, presenting parts of results as follows:

| Dataset  | FaDL  | FaIRL | FaRULi |
|-----|-----|------|-------|
| Diabetes | 0.068 | 0.051 | 0.031  |
| Law      | 0.234 | 0.091 | 0.087  |

We can observe that the empirical superiority of our proposed FaRULi algorithm is invariant to the fairness metric, and our algorithm outperforms its two competitors FaDL and FaIRL in both datasets with 0.031 and 0.087, respectively.

Q2: Reporting P(Y|P). Please clarify what "more divergent ..." and "incrementing ..." mean?

A2: Due to character limitations,  please find our answers in A4 of Reviewer kvtG. 

Q4: Supervised with or w/o fairness-constrained baselines.

A4: Results of Adult are shown as below.

|Constraint|T_1|T_2|T_3|T_4|T_5|T_6|T_7|T_8|T_9|T_10|T_11|T_12|
|----------|---|---|---|---|---|---|---|---|---|----|----|----|
|w/o Acc|0.872|0.77|0.79|0.76|0.859|0.909|0.844|0.951|0.755|0.751|0.726|0.821|
|w/o EO|0.101|0.173|0.167|0.138|0.643|0.688|0.286|0.044|0.216|0.221|0.278|0.195|
|with Acc|0.817|0.665|0.775|0.566|0.857|0.856|0.842|0.932|0.761|0.589|0.687|0.715|
|with EO|0.082|0.152|0.049|0.219|0.039|0.167|0.071|0.24|0.055|0.088|0.47|0.141|

Q5: Fairness regularization constants

A5: Yes, our experiments on the Law School dataset show that increasing $\lambda_1$ decreased accuracy from 91.6% to 4.7%, demonstrating that strict fairness constraints can lower accuracy and affect fairness. Both statistical parity (SP) and EO values fall to zero, with the learner classifying all instances as Y=0 to avoid unfair outcomes. 

| $\lambda_1$ = |0.01|0.03|0.05|0.1|0.5|1|
|------------|-------|-------|-------|-------|-------|-------|
| Acc| 0.916 |0.939|0.899|0.941|0.047|0.047|
| SP|0.162|0.002 |0.054 |0.012 |0|0|
| EO|0.269|0.087|0.107|0.050|0|0|

Reviewer comment (Theory):
In the last paragraph on page 4, and a few other places, the authors present a long list of conjectures about the behavior of their method (“deep layers will outperform... weight parameters of deep layers [will] start to exceed that of shallow layers... the parameters of shallow layers [will] initially increase sharply... weights of the deepest layers [will] increase last and remain relatively small...”). These appear to be unproven conjectures without experimental or theoretical validation. The authors should either make clear that these are hypotheses or provide experimental demonstrations.

Author response:
Q6: The dynamics of ... be conjectures. 

A6: We respectfully disagree, referencing studies in traditional BP [2-3] and adversarial learning [4]. These show that while deeper neural networks offer more representation capacity, they're also more sensitive to initializations, potentially losing correlation with shallow features. Our backbone architecture is built upon [4], and our EFRL network design can enjoy the same merit as the elastic network in [4] does. We will further add experimental results as [3] did to support the intuition behind our design.

[2] https://arxiv.org/abs/1605.07648

[3] https://link.springer.com/chapter/10.1007/978-3-319-46493-0_39

[4] https://dl.acm.org/doi/abs/10.1145/3442381.3449839

Reviewer comment (Experiments):
The proposed task selection mechanism is just one method to assess task similarity/distributional differences between successive tasks. The authors could either compare it to one or two standard distributional similarity metrics for choosing which task to incorporate next, or provide an argument for why existing similarity metrics are not applicable here.

Author response:
Q8: Could the authors either compare ..., or provide some ...here?

A8: We argue that traditional distance metrics are unsuitable due to two reasons: they struggle with high-dimensional data [5] and require predefined thresholds to distinguish "close" from "faraway" tasks, which aren't feasible in lifelong learning. Our experiments on high-dimensional image datasets are detailed in A2 of Reviewer 8gZZ.

[5] https://kops.uni-konstanz.de/entities/publication/7cac4320-01b6-4403-b38a-ba87d550bc0a

Reviewer comment (Evaluation):
On p.3 towards the bottom, the claim that “..., a comparable performance on Ti can be achieved” seems overly optimistic.

Author response:
Q9: "..., a comparable performance ..." which seems like...

A9: We'll revise our statement: If a shared subspace between $\mathcal{T}_i$ and $R^{(0)}$ is found, generalization from fully labeled $R^{(0)}$ to unlabeled $\mathcal{T}_i$ is possible, proved by many UDA studies [6-7]. However, such $\mathcal{T}_i$ may not always be feasible due to significant task differences or starting with an outlier task. Therefore, we will adjust our wording to "comparable performance on $\mathcal{T}_i$ is more likely" to cover these possibilities.

[6] https://proceedings.mlr.press/v37/ganin15.html

[7] https://openaccess.thecvf.com/content_cvpr_2018/html/Pinheiro_Unsupervised_Domain_Adaptation_CVPR_2018_paper.html

Reviewer comment (Theory):
On p.9, the statement “Bias in ML models can be attributed to the information carried by protected features” is a highly simplistic characterization of bias. There are many forms of bias with different causes.

Author response:
Q3: Types of bias in ML models,

A3: Yes, we're aware of the survey and have cited it. Our focus is on "fair representation learning (FRL)," not machine learning bias. Existing FRL studies, like [1], mainly address deriving fair data representations from unfair data. We'll emphasize our sub-domain and differentiate it from other biases in camera-ready.

[1] https://dl.acm.org/doi/abs/10.1145/3278721.3278779