Paper: Balancing Act: Constraining Disparate Impact in Sparse Models (ICLR-2024)

Reviewer comment (Experiments):
Have the authors tried applying this technique to other pruning methods? For instance, how well would this fine-tuning method work if structured pruning was used instead of unstructured pruning? How well would the method work if other unstructured pruning methods were used instead of IMP (i.e. SynFlow [1] or SNIP [2])?

Author response:
> Have the authors tried applying this technique to other pruning methods? For instance, how well would this fine-tuning method work if structured pruning was used instead of unstructured pruning? How well would the method work if other unstructured pruning methods were used instead of IMP (i.e. SynFlow [1] or SNIP [2])?

As the reviewer acknowledged, our proposed method is agnostic to the choice of pruning method. We expect it to behave well under popular pruning techniques like those mentioned by the reviewer.

In particular, structured sparsity is practically relevant since it enables fast inference on existing hardware. Following the reviewer's suggestion, **we conducted additional experiments** demonstrating the behavior of our proposed technique (and all other baselines) in this setting.

**Experiment set-up**
- UTKFace dataset with target=race and protected attribute=race.
- Same experimental setting as other UTKFace experiments in the main paper.
- Metrics averaged over 5 seeds.
- Layerwise structured output-channel sparsity at 85%.

**Results**
*Please read the clarifications below the table to adequately interpret the behavior of $\Psi_{\text{PW}}$ in test for EL+RB and CEAG!*

| Sparsity |  Method  |            |        Train         |                 |                  |           |        Test        |                 |
|----------|----------|------------|----------------------|-----------------|------------------|-----------|--------------------|-----------------|
|          |          |  Accuracy  | $\Psi_{\text{PW}}$   | $\max_g \psi_g$ | Tol ($\epsilon$) | Accuracy  | $\Psi_{\text{PW}}$ | $\max_g \psi_g$ |
|85        | NFT      | 89.1±0.39  |      50.6±1.08       |    45.8±0.82    |        –         | 80.7±0.27 |       4.1±1.7      |    2.0±0.61     |
|85        | NFT + ES | 87.7±0.71  |      58.3±3.08       |    52.7±2.91    |        –         | 81.5±0.11 |       4.7±2.06     |    3.2±1.52     |
|85        | EL + RB  | 86.9±0.47  |      3.7±1.38        |    1.2±0.53     |        –         | 77.9±0.44 |       27.3±2.58    |    3.2±0.65     |
|85        | CEAG     | 85.6±0.47  |      2.7±0.71        |    1.0±0.36     |        ≤ 1%      | 78.2±0.39 |       30.1±2.19    |    3.7±0.28     |




***Key-points:***
* CEAG and EL+RB successfully mitigate pruning-induced disparate impact in the structured sparsity setting.
* The results show that our method CEAG achieves the lowest disparate impact in train.
* Our method has a high $\Psi_{\text{PW}}$ in test because the under-represented group *improved* its test accuracy.

***Clarifications:***
We note an unusual pattern in the train-vs-test performance between mitigation methods like EL+RB and CEAG, and the NFT baselines: mitigation methods have very low disparity ($\Psi_{\text{PW}}$) in training but very high disparity in test (and vice versa for NFT). 

The reason behind this behavior is the performance of the different models on the under-represented group *"Others"*, corresponding to ~3% of the dataset. The relevant metrics (averaged over 5 seeds) are summarized in the table below.

* The pre-trained model had overfitted to the training samples of group *"Others"*.
* In the case of NFT, the training objective allows the models to sacrifice performance on under-represented classes in order to favor more numerous ones. In particular, we observed that NFT caused a hyper-degradation (EAG) in group *"Others"* of 45.77%.
* On the other hand, mitigation techniques like EL+RB and CEAG do **not** yield models that obliviously sacrifice the performance on group *"Others"*. For example, for CEAG the loss in performance in group *"Others"* (13.4%) in training is similar to the overall training accuracy reduction (14.14%). 


In other words, **the mitigation methods successfully prevent the appearance of excessive disparity due to pruning** by making sure no class (under-represented or not) is degraded more than $\epsilon$ beyond the average model degradation.


| Metric (avg. over 5 seeds) | Dense | NFT | EL+RB | CEAG |
| -------- | -------- | -------- | -------- | -------- |
| Overall Acc. Train | 99.74 | 89.1  |86.9  | 85.6 |
| Overall Acc. Test  | 83.82 | 80.7 | 77.9 | 78.2 |
| Acc. Others Train | 99.37 |42.96  | 88.37 | 85.97 |
| Acc. Others Test | 29.23 | 26.31 | 47.38 | 49.97 |
| Overall Acc. Gap Train ($\Delta$) | 0 | 10.64 | 12.84 | 14.14 | 
| Overall Acc. Gap Test ($\Delta$) |  0   | 3.12 | 5.92 | 5.62 |
| Acc. Gap Others Train ($\Delta_{\text{Others}}$) | 0 | 56.41 | 11.0 | 13.4 |
| Acc. Gap Others Test ($\Delta_{\text{Others}}$) | 0 | 2.92 | -18.15 | -20.74 |
| EAG Others Train ($\psi_{\text{Others}}$) | 0 | 45.77 ⚠️ | -1.84 | -0.74 |
| EAG Others Test ($\psi_{\text{Others}}$) | 0 | -0.2 | -24.07 | -26.36 |

Reviewer comment (Theory):
Is there a sparsity level at which the method fails to achieve models with the desired worst-case groupwise accuracy loss? Put another way, is there a sparsity level at which the constrained optimization problem described in eq. (5) become infeasible? Can the authors comment on how this might play out in the case of structured pruning?

Author response:
> Is there a sparsity level at which the method fails to achieve models with the desired worst-case groupwise accuracy loss? Put another way, is there a sparsity level at which the constrained optimization problem described in eq. (5) become infeasible? Can the authors comment on how this might play out in the case of structured pruning?

It has been previously documented [A, B, C] that deep neural networks are robust to high levels of sparsity, but their training dynamics become challenging. Our constrained formulation suffers from similar challenges, and we would expect that these difficulties become more pronounced in the constrained setting—as per the reviewer's intuition.

For our experiments, we chose commonly used sparsity levels for the different architectures and we were able to find feasible solutions that achieved disparity below that of the (unconstrained) NFT baseline. Empirically, the "breaking point" for the tolerance level did not increase as a result of adding the constraints. 

Note that when increasing the sparsity level, we sometimes had to increase the constraint level (e.g. Table 25 in revision [UTKFace > Intersectional]). However, this is mostly due to the loss of model capacity, as is observed in the increase in $\max_g \psi_g$ experienced by the *unconstrained* NFT baseline. 


[A] Song Han, Jeff Pool, John Tran, William J. Dally. Learning Both Weights and Connections for Efficient Neural Network. In NeurIPS, 2015.

[B] Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks. arXiv:1902.09574, 2019.

[C] Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The Difficulty of Training Sparse Neural Networks." arXiv:1906.10732, 2019.

Reviewer comment (Theory):
Are there any formal results that can be provided for solving the CEAG (eq (5))? For instance, is Algorithm 1 guaranteed to find a solution provided the feasible set is nonempty?

Author response:
>Are there any formal results that can be provided for solving the CEAG (eq (5))? For instance, is Algorithm 1 guaranteed to find a solution provided the feasible set is nonempty?

No, even if the feasible set is non-empty, there is no guarantee that Algorithm 1 will find a solution. The possibility for analysis is limited as we consider a non-convex, non-differentiable constrained optimization problem. Viewed from the min-max angle, the use of proxy-constraints results in a non-zero-sum game. Providing convergence guarantees for these optimization problems is a challenging, open area of research, beyond the scope of our work.

We invite the reviewer to read our response to **Reviewer Cvd6**, including some pointers to local convergence results under idealized problem conditions.

Reviewer comment (Experiments):
Have the authors considered ways by which the test-case performance can be improved, say by dataset splits?

Author response:
> Have the authors considered ways by which the test-case performance can be improved, say by dataset splits?

This is an interesting idea. In the development of the paper, we considered an early-stopping approach based on constraint and objective estimates on a validation set. Due to the reasons detailed below, we decided not to proceed with this approach in our main experiments. However, exploring techniques to improve test-case performance for disparity-mitigation methods is an important research direction.

The early-stopping approach mentioned above required creating validation splits, which were unfortunately not available for the datasets we considered. In order to make fair comparisons with existing methods, we decided to retain the same choice of model architecture and train/test split. However, in some tasks, we make use of models that have been pre-trained on the entire training set (containing our self-made validation datapoints). Thus the constraint and objective estimates on the validation set would be unreliable due to data leakage. Since these challenges would have prevented us from executing a proper ablation study on the benefits of this technique, we decided not to pursue this.

Reviewer comment (Experiments):
Is the CEAG method affected by dataset imbalance? Suppose certain groups have comparatively fewer samples in the dataset. How, if at all, would this affect the efficacy of the method?

Author response:
> Is the CEAG method affected by dataset imbalance? Suppose certain groups have comparatively fewer samples in the dataset. How, if at all, would this affect the efficacy of the method?

This is an important question and of great importance in fairness. Our proposed CEAG method is designed specifically with this issue in mind.

Data imbalance affects training of fairness-aware methods in two main ways:
* Under-represented groups might not have enough samples to sway the model's updates toward satisfying their constraints.
* Estimating the constraints associated with under-represented groups may be challenging due to the low number of samples in the stochastic estimates.

CEAG directly addressed both of these challenges by (1) introducing explicit per-group constraints and (2) using Replay Buffers (RB, Section 4.2) to reduce estimation noise.

* If a constraint for any group (under-represented or not) is violated persistently, its associated Lagrange multiplier will increase, biasing the model gradient towards achieving the satisfaction of the constraint.
* On the other hand, RBs allow our method to scale to a large number of constraints (even in the data imbalance setting) by constructing less noisy estimates of the constraint violations. Moreover, RBs also improve the stability of the training dynamics under an imbalanced dataset with few protected groups. 

We would like to highlight that the use of RBs was one of the key components for improving the scalability and stability of Equalized Loss, originally proposed by Tran et al. [A].

Experimentally, our method excels in tasks with high data imbalance (such as intersectional fairness) compared to all other techniques—see Table 2 at 92.5% sparsity.

[A] Cuong Tran, Ferdinando Fioretto, Jung-Eun Kim, and Rakshit Naidu. Pruning has a disparate impact on model accuracy. In NeurIPS, 2022.

Reviewer comment (Experiments):
Can the paper deals with other pruning techniques to demonstrate the effectiveness of constraint to subsets?

Author response:
Yes, during the rebuttal phase, we have carried out additional experiments under the _structured_ sparsity setting. At a high level, we observed similar trends as in the unstructured pruning setting: our method reliably mitigates the disparate impact of pruning in training, while preserving the overall performance of the model. We invite the reviewer to see our response to **Reviewer KSs2** containing the details and results for these experiments. 

The additional results on structured sparsity demonstrate that our technique is agnostic and robust to the choice of pruning method. Therefore, we expect it to perform well under other commonly used sparsity techniques.

Reviewer comment (Theory):
Can we give some formal results with Equation(6) like the analytical solution or convergenece guarantee?

Author response:
In general, the min-max problem in Eq(6) does not have an analytical solution, as is the case for many optimization problems considered in the deep learning field. 

Under idealized circumstances, it is possible to develop convergence guarantees of gradient-based optimization methods for min-max problems. We have already provided pointers to relevant literature on the related work section and Appendix A. 

Unfortunately, the existing literature focuses on proving local convergence of gradient-based methods. This analysis relies on the assumption of an initialization close enough to a solution to the min-max problem, which is unrealistic in our specific problem.

Despite the lack of theoretical guarantees, our experimental results clearly demonstrate that we are able to reliably find feasible solutions (i.e. low disparate impact) that are approximately optimal (i.e. model performance close to the dense baseline).

For the reviewer's convenience, we list the mentioned references here:

[A] Tianyi Lin, Chi Jin, and Michael Jordan. On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems. In ICML, 2020.

[B] Guodong Zhang, Yuanhao Wang, Laurent Lessard, and Roger B Grosse. Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimization. In AISTATS, 2022.