Paper: Coupling Fairness and Pruning in a Single Run: a Bi-level Optimization Perspective (ICLR-2024)

Reviewer comment (Experiments):
My main concern is the random initiation of the mask. It is not clear what kind of randomization is used in the experiment, and whether different randomization strategies will lead to different performances.

Author response:
In our experiments, we utilized two methods for initialization. The first method sets the initial values to the pre-trained weights. The second method employs Kaiming Initialization [1]. We observed that both methods yield similar performance, indicating that different initialization approaches do not significantly impact the final results. We plan to include a more detailed discussion regarding this observation in the experimental section of our paper.

Reviewer comment (Theory):
The paper presents a bi-level pruning method (i.e., optimizing one variable while fixing another one). How do you guarantee the convergence theoretically. Again, since the mask is randomly initialized, will different randomization lead to different convergences? Noted that different randomization could also impact the objective function (leading to different landscapes), thus, both theoretical and experimental studies regarding the issue of convergence should be conducted.

Author response:
This work highlights a critical social issue in pruning and offers an efficient and effective solution. The theoretical proof will be the focus of our future research efforts.

Reviewer comment (Experiments):
The adopted datasets and models for evaluation seem to be toy to me, perhaps more realistic dataset (e.g., healthcare) and more advanced model should be evaluated.

Author response:
In our study, we deliberately selected smaller models to distinctly showcase the effectiveness and equity of the proposed pruning method. This approach results in a pruned model with reduced parameters while maintaining performance levels that are comparable to larger models. Regarding datasets, we adhered to the standard configurations commonly employed in related research, e.g., [1].

Reviewer comment (Theory):
The motivation of using conves relaxation in 4.3 is not clear to me. Based on the submission, it seems the advantage of using convex relaxation is to let the fairness constraint convex and differentiable, but noted that our deep neural network is non-convex in nature (e.g., ReLU, maxpooling, etc.), the argument above is not convincing to me.

Author response:
While it is indeed true that deep neural networks are inherently non-convex, the application of convex relaxation in our context serves a specific and strategic purpose. The convex relaxation simplifies these constraints, making them more tractable within the optimization process. In addition, convex relaxation effectively reduces estimation errors between the actual non-differentiable functions and their relaxed counterparts [2]. This approach simplifies the tuning process and facilitates a balanced trade-off between performance and fairness. We will add experiments with non-convex surrogates in the next version.

Reviewer comment (Novelty):
The idea of using bi-level optimization is not new to pruning. [a] uses it in a very similar fashion to that in the paper. While [a] is cited in the manuscript, it is not explicitly called out that they also use a very similar bi-level optimization framework - which looks misleading.

Author response:
This work primarily concentrates on **fairness in network pruning**, and our key contribution is an efficient and equitable pruning strategy. While study [a] utilizes first-order derivatives to calculate the Implicit Gradient (IG) for bi-level optimization, thus laying a solid foundation for network pruning, our research explores a less examined societal dimension of this domain. We introduce a constrained bi-level optimization problem and propose an innovative approach to network pruning that integrates fairness considerations, distinguishing our work from existing methodologies.

Reviewer comment (Presentation):
Definition 2 (Compressed Model Performance Degradation) and Definition 3 (Performance Degradation Fairness) seem unnecessary, since the authors essentially use only the fairness performance metric (definition 1) in practice when pruning the model.

Author response:
We use Definition 2 in the experiment 5.3. Although we do not use Definition 3 in the evaluation section, it is useful to examine bias in terms of model pruning in other cases. We will integrate more evaluation perspectives in the next version.

Reviewer comment (Experiments):
Experiments are limited across dimensions such as datasets and models. More datasets (including larger ones) should be included such as in [b]. Larger models such as ResNets should also be included in the study.

Author response:
In our study, we deliberately selected smaller models to distinctly showcase the effectiveness and equity of the proposed pruning method. This approach results in a pruned model with reduced parameters while maintaining performance levels that are comparable to larger models. Regarding datasets, we adhered to the standard configurations commonly employed in related research, e.g., [1]. We will continuously seek larger datasets for fairness and computer vision.

Reviewer comment (Experiments):
Stronger baselines need to be used. At the moment, none of the baselines used optimize for fairness. However, there are other works such as [b] that optimize for fairness and should be used as a baseline to show the benefit of bi-level optimization over existing pruning approaches.

Author response:
In Section 5.3, we outline the design of our baseline methods, integrating a two-stage fair training and pruning strategy. This approach sequentially combines model pruning and fair training, with a distinct contrast to our proposed single-run method. Whereas FairGRAPE is focused on minimizing the variance in performance disparities, our method is specifically aimed at reducing the relative disparities among various subgroups. Thus, we didn't include FairGRAPE in the comparison.

Reviewer comment (Experiments):
The experiment suffers from samll scale. Only small models (ResNet10 and Mobilenetv2) are used as an uncompressed model. The used datasets (CelebA and LFW) are also small.

Author response:
In our study, we deliberately selected smaller models to distinctly showcase the effectiveness and equity of the proposed pruning method. This approach results in a pruned model with reduced parameters while maintaining performance levels that are comparable to larger models. Regarding datasets, we adhered to the standard configurations commonly employed in related research, e.g., [1]

Reviewer comment (Novelty):
Althogh the propsed method is reasonable, it seems trivial. Eq 5 defines the joint optimization framework. It simply combines two constrains. This paper is short of novel techniques.

Author response:
As highlighted in section 4.2, the complexity of bi-level optimization problems increases significantly with additional constraints. This complexity is particularly evident in our methods, where constraints encompass both upper and lower-level variables, making the problem more challenging to solve. Addressing the theoretical proof of our approach remains an area for future investigation. We plan to concentrate our future research efforts on this theoretical aspect. In this paper, we have brought attention to an important ethical issue in pruning, and have proposed a potential solution for it.