Original: Reproducibility report: Deep Fair Clustering for Visual Learning
Revised: [Re] Deep Fair Clustering for Visual Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Reproducibility Summary

Original: Scope of Reproducibility
Revised: Scope of Reproducibility

Original: The authors propose a novel method for Deep Fair Clustering (DFC), combining existing frameworks for fair clustering—which typically have difﬁculty with high-dimensional large-scale data—with previous work on deep clustering—which typically has difﬁculty with fairness.
Revised: The authors propose a novel method for Deep Fair Clustering (DFC), combining existing frameworks for fair clustering—which typically have difﬁculty with high-dimensional large-scale data—with previous work on deep clustering—which typically has difﬁculty with fairness.

Original: Our reproducibility work targets the central claim that DFC learns fairrepresentations with minimal utility loss and obtains superior results on both fairness and accuracy
Revised: Our reproducibility work targets the central claim that DFC learns fairrepresentations with minimal utility loss and obtains superior results on both fairness and accuracy.

Original: Methodology
Revised: Methodology

Original: We used the code repository made available by the authors and extended it to include support for pretraining, differentdatasets and comparative methods.
Revised: We used the code repository made available by the authors and extended it to include support for pretraining, differentdatasets and comparative methods.

Original: We compare the DFC method against Deep Embedded Clustering (DEC) (11),—which implements a comparable deep clustering method without fairness constraints—on the same four datasets(obtained from MNIST (7), USPS (6), MTFL (13) and Ofﬁce-31 (9)) and fairness metrics that were used in the paper.
Revised: We compare the DFC method against Deep Embedded Clustering (DEC) (11) whichimplements a comparable deep clustering method without fairness constraints, on the same four datasets (obtained from MNIST (7) USPS (6), MTFL (13) and Ofﬁce-31 (9)) and fairness metrics that were used in the paper.

Original: We select one dataset for additional experiments aimed at validating the contribution of individual components ofthe DFC towards fairness.
Revised: We select onedataset (MNIST-UPS) for additional experiments aimed at validating the contribution of individual components ofthe DFC towards fairness.

Original: All experiments were run on a GeForce 1080Ti GPU
Revised: All experiments were run on a GeForce 1080Ti GPU.

Original: Hyperparameter optimization wasperformed using the Weights & Biases Sweeps feature (1).
Revised: Hyperparameter optimization wasperformed using the Weights & Biases Sweeps feature (1).

Original: Results
Revised: Results

Original: On the selected dataset, we reproduced accuracy to within 2% of reported value, normalized mutual information (NMI)and Entropy to within 1%, and balance to within 5%.
Revised: On the selected dataset, we reproduced accuracy to within 2% of reported value, normalized mutual information (NMI)and entropy to within 1%, and balance to within 5%.

Original: Our DFC method outperformed our DEC method on all accuracyand fairness metrics.
Revised: Our DFC method outperformed our DEC method on all accuracyand fairness metrics.

Original: We reproduced the accuracy of the non-digit datasets to within 1% (Ofﬁce-31) and 5% (MTFL)but failed to obtain similar results for balance.
Revised: We reproduced the accuracy of the non-digit datasets to within 1% (Ofﬁce-31) and 7% (MTFL)but failed to obtain similar results for balance.

Original: What was easy
Revised: What was easy

Original: We found no major challenges reproducing the provided code in as far as we used their provided pretrained models andthe selected dataset  used in the code.
Revised: We found no major challenges reproducing the provided code in as far as we used the author’s provided pretrainedmodels and the selected dataset (MNIST-USPS) used in the code.

Original: What was difﬁcult
Revised: What was difﬁcult

Original: Extending the code for the non-digit datasets was a challenge, as some hyperparameter settings and architecture details were difﬁcult to infer from the paper.
Revised: Extending the code for the non-digit datasets was a challenge, as some hyperparameter settings and architecture details were difﬁcult to infer from the paper.

Original: We found that performance was sensitive to small changes in implementation and training of the encoders, causing us to run into time and resource constraints when trying to reproduce all results for these datasets due to the large number of models that require pretraining
Revised: We found that performance was sensitive to small changes in the implementationand training of the encoders. Consequently, we ran into time and resource constraints when trying to reproduce allresults for these datasets, due to the large number of models that required pretraining.

Original: Communication with original authors
Revised: Communication with original authors

Original: We had helpful one-off contact with the authors to verify hyperparameter settings.

Original:  Introduction
Revised:  Introduction

Original: Machine learning (ML) is increasingly used in high-stake decision-making where the data might contain sensitiveattributes, such as gender, race or socioeconomic background.
Revised: Machine learning (ML) is increasingly used in high-stake decision-making where the data contains sensitive attributes,such as gender, race or socioeconomic background.

Original: Examples include college admission, loan approval andbail/parole judgements.
Revised: Examples include college admission, loan approval and bail/parolejudgements.

Original: Such ML algorithms are vulnerable to bias and unfairness (4), and extensive literature hasbrought attention to the various challenges and inherent trade-offs in this ﬁeld (5)(14).
Revised: Such ML algorithms are vulnerable to bias and unfairness (4), and extensive literature has brought attentionto the various challenges and inherent trade-offs in this ﬁeld (5)(14).

Original: Clustering is an important aspect of many such  applications.
Revised: Clustering is an important aspect of many such ML applications.

Original: To illustrate what it might mean for a clustering assignmentto be unfair, we can consider its use in feature engineering, e.g., to label data in an unsupervised setting to increase itsexpressive power (2).
Revised: To illustrate what it might mean for a clusteringassignment to be unfair, we can consider its use in feature engineering, e.g., to label data in an unsupervised setting toincrease its expressive power (2).

Original: In some cases, if the inherent structure of the training data differs between subgroups, standardclustering is likely to lead to partition assignments that correlate signiﬁcantly with sensitive attributes.
Revised: In some cases, if the inherent structure of the training data differs between subgroups,standard clustering likely leads to partition assignments that correlate signiﬁcantly with sensitive attributes.

Revised: Often, theseare controversial correlations that do not reﬂect true causal mechanisms.

Original: This couldfacilitate subsequent discrimination based on sensitive attributes, either knowingly or unknowingly.
Revised: This could facilitate subsequent discriminationindirectly based on sensitive attributes, either knowingly or unknowingly.

Original: In short,  clustering is an active and complex ﬁeld that still faces many challenges.
Revised: In short, fair clustering is an ongoing, important, and complex ﬁeld that still faces many challenges.

Original: The paper ’Deep Fair Clustering for
Revised: The paper ’ Deep

Original:  Visual Learning’ addresses various such challenges.
Revised: Fair Clustering for Visual Learning’ addresses various such challenges.

Original: Moreover, their method for fair clustering outperforms competitivefair clustering methods as well as competitive deep clustering methods, showing that their method is able to imposefairness constraints without signiﬁcantly compromising clustering accuracy.
Revised: Moreover, their method for fair clusteringoutperforms competitive fair clustering methods as well as competitive deep clustering methods. This shows that, iftheoretically sound, their method is able to impose fairness constraints without signiﬁcantly compromising clusteringaccuracy.

Original:  Scope of reproducibility
Revised:  Scope of reproducibility

Original: The paper aims to tackle the problem of fair clustering of high-dimensional data by introducing a Deep Fair Clustering(DFC) method suitable for image data.
Revised: The paper aims to tackle the problem of fair clustering of high-dimensional data by introducing a Deep Fair Clustering(DFC) method suitable for image data.

Original: Their notion of fairness extends the demographic equipartition criterion andstates that the clustering assignment should be independent of the protected subgroup membership.
Revised: Their notion of fairness extends the demographic equipartition criterionand requires that the clustering assignment is independent of the protected subgroup membership.

Original: To achieve thisindependence, they leverage the feature representation encoding from deep methods and train the model to ﬁlter outsensitive attributes from the representations.
Revised: To achieve thisindependence, they leverage the feature representation encoding from deep methods and train a model to ﬁlter outsensitive attributes from the representations.

Original: Simultaneously, DFC enforces fairness constraints and optimizes clusteringperformance, notably with  minimal trade-off between the two.
Revised: Simultaneously, DFC enforces fairness constraints and optimizes clusteringperformance, notably with a minimal trade-off between the two.

Original: The central claims are summed up as follows:
Revised: The central claims are summed up as follows:

Original: • The ﬁrst claim states that DFC achieves fair clustering of large-scale and high-dimensional visual data bylearning fair representations of the input.
Revised: • The ﬁrst claim states that DFC achieves fair clustering of large-scale and high-dimensional visual data bylearning fair representations of the input.

Original: To support this claim, we train DFC and DEC on the MNIST-USPS dataset and compare the learned representations in ﬁgure 2.
Revised: To support this claim, we train DFC and DEC on the MNIST-USPSdataset and compare the learned representations in ﬁgure 3.

Original: The ﬁrst row of table 1 shows the correspondingaccuracy and fairness metrics of the ﬁnal performance of both methods.
Revised: The ﬁrst row of table 1 shows the correspondingaccuracy and fairness metrics of the ﬁnal performance of both methods.

Original: • The second claim regards the validity and effectiveness of the proposed minimax optimization formulation.In
Revised: • The second claim regards the validity and effectiveness of the proposed minimax optimization formulation.

Original: particular, we support this claim by recognizing DEC as DFC without the minimax optimization formulation(in particular, without the fairness components 4, 5 and the separate clustering of subgroups, see ﬁgure 1).To our best knowledge, this corresponds to the implementations in (11) and (8).
Revised: Inparticular, we support this claim by recognizing DEC as DFC without the minimax optimization formulation(in particular, without the fairness components 4, 5 and the separate clustering of subgroups, see ﬁgure 1).To our best knowledge, this corresponds to the implementations in (11) and (8).

Original: The claim is supported bycomparing DFC performance with DEC performance on accuracy and fairness metrics in table 1.
Revised: The claim is supported by comparing DFC performance with DEC performance on accuracy and fairness metrics in table 1.

Revised: In addition,ﬁgure 4 shows the performance of the discriminator during DFC, which provides some insight in the extend towhich DFC achieves ’masked’ representations during trainig.

Original: • The third claim states that DFC shows superior performance on four real-world visual datasets (see 3.1).
Revised: • The third claim states that DFC shows superior performance on four real-world visual datasets (see 3.1).

Original: We
Revised: We

Original: aim to show this by running experiments on the same four datasets with both DEC and DFC and compareperformance in table 1.
Revised: aim to show this by running experiments on the same four datasets with both DEC and DFC and compareperformance in table 1.

Original:  Methodology
Revised:  Methodology

Original: Each claim requires us to implement the DFC method and the DEC method and train it on the MNIST-USPS dataset.
Revised: Each claim requires us to implement the DFC method and the DEC method and train it on the MNIST-USPS dataset.

Original: From here, we address each claim separately in three sets of experiments.
Revised: From here, we address each claim separately in three sets of experiments.

Original: For the ﬁrst, we visualize the representationsusing dimensionality reduction methods and compare the obtained clusters.
Revised: For the ﬁrst, we visualize the representationsusing dimensionality reduction methods and compare the obtained clusters.

Original: For the second claim, we compare the DECand DFC on accuracy, NMI, Balance and Entropy metrics.
Revised: For the second, we compare the DEC and DFC on accuracy, NMI, Balance and Entropy metrics.

Revised: Moreover, we monitor discriminator accuracy for DFC.

Original: For the third claim, we run our experiments with the threeother datasets that are used in the paper and compare results on the above mentioned four metrics.
Revised: For thethird, we run our experiments with the three other datasets that are used in the paper and compare results on the abovementioned four metrics.

Original: Before moving on to the details we brieﬂy explain the DFC method as we implemented it based on the authorsdescription.
Revised: Before discussing the details, we brieﬂy explain the DFC method as we implemented it based on the authors’ description.

Original: The method has several components that are trained simultaneously in a minimax optimization scheme.
Revised: The method has several components that are trained simultaneously in a minimax optimization scheme.

Original: Afull overview of the method is shown in ﬁgure 1.
Revised: An overview ofthe method is shown in ﬁgure 1.

Original: The task is to cluster datapoints X with sensitive categorical attribute G belonging to protected subgroup G ( X ) ∈ [ M ] into K clusters.
Revised: The task is to cluster datapoints X with sensitive categorical attribute G belonging toprotected subgroup G ( X ) ∈ [ M ] into K clusters.

Original: A feature encoder F ( X ) transforms the data Xinto representations Z .
Revised: A feature encoder F ( X ) transforms the data X into representations Z .

Original: As the ﬁgure shows, each subgroup is clustered separately and requires a separate, pretrainedencoder.
Revised: As the ﬁgure shows, each subgroup is clustered separately and requires a separate, pretrained encoder.

Original: The clustering assignment P = A ( Z ) , taken from the Deep Embedded Clustering method (DEC), createsa soft assignment Preﬂecting the probability of datapoints to be assigned to each cluster, once for each protectedsubgroup, and once for the full dataset.
Revised: The clusteringassignment P = A ( Z ) , taken from the Deep Embedded Clustering method (DEC), creates a soft assignment P

Original: A discriminator D aims to reconstruct the protected subgroup membership based on the soft assignment P
Revised: A discriminator D aims to reconstruct the protected subgroup membership based on the soft assignment P

Original:  Encoder
Revised:  Encoder

Original: Central to the DFC approach is the encoding of fair representations.
Revised: Central to the DFC approach is the encoding of fair representations.

Original: Transforming the input data usingfeature encoding is important, in the ﬁrst place, because (deep) clustering at pixel level does not often yield good results.
Revised: Transforming the input data usingfeature encoding is important, in the ﬁrst place, because (deep) clustering at pixel level does not often yield good results.

Original: However, in DFC the encoded representations are at the same time an important step towards fairness.
Revised: In DFC, the encoded representations are at the same time an important step towards fairness.

Original: The authorsnote that the representations can be ﬁltered from the sensitive attributes, noting that "fair representations are achievedwhen the discriminator cannot distinguish between representations from different protected subgroups.
Revised: The authors note that"fair representations are achieved when the discriminator cannot distinguish between representations from differentprotected subgroups.

Original: " To this end, theencoder is ﬁnetuned simultaneously with the rest of the clustering algorithm.
Revised: " To this end, the encoder is ﬁnetuned simultaneously with the rest of the clustering algorithm.

Original:  Discriminator
Revised:  Discriminator

Original: A discriminator is implemented to provide a measure of fairness. It is trained to discriminate protectedsubgroup membership based on the soft assignment P .
Revised: A discriminator is constructed to monitor and guide the fairness of representations Z .

Original:  Deep Embedded Clustering
Revised:  Deep Embedded Clustering

Original: For the clustering assignment, the authors closely follow the DEC method.
Revised: For the clustering assignment, the authors closely follow the DEC method.

Original: As in(11), the optimal cluster assignment is learned using Stochastic Gradient Descent (SGD),repeating the following twoprocesses:
Revised: As in (11),the cluster assignment is learned using Stochastic Gradient Descent (SGD), repeating the following two processes:

Original: A soft assignement of the datapoints to cluster centroid using the Student’s t-distribution as a kernel to measurethe similarity between embedded point z and centroid c k :
Revised: A soft assignment of the datapoints to cluster centroids, using the Student’s t-distribution as a kernel to measure the similarity between embedded point z and centroid c k :

Original: [Equation]
Revised: [Equation]

Original: Where p k is the probability of a datapoint belonging to cluster center c k and α the degrees of freedom of the
Revised: where p k is the probability of a datapoint belonging to cluster center c k and α the degrees of freedom of the

Original: Student’s t-distribution.
Revised: Student’s t-distribution.

Original: The obtained distribution P of soft assignments is matched to an auxiliary distribution Q:
Revised: The obtained distribution P of soft assignments is matched to an auxiliary distribution Q:

Original: [Equation]
Revised: [Equation]

Original:  which is calculated for each protected subgroup separately.
Revised: (2) which is calculated for each protected subgroup separately.

Original: Finally, the clustering is optimized by minimizingthe KL-divergence loss:
Revised: Finally, the clustering is optimized by minimizingthe KL-divergence loss:

Original: [Equation]
Revised: [Equation]

Original: Which ensures  the soft assignment matches the target distribution as closely as possible.
Revised: This ensures that the soft assignment matches the target distribution as closely as possible.

Original: Note that p k , z and c k are optimized simultaneously, such that both the P and Q distribution change dynamically.

Original: Deep Fair clustering
Revised: Deep Fair Clustering

Original: DFC implements DEC as above, but adds two loss function to impose the fairness constraintwhile promoting cluster utility.
Revised: DFC implements DEC as above, but adds two loss functions to impose the fairness constraintwhile promoting cluster utility.

Original: To encourage fair partitions according to the demographic parity, the authors introducean fairness adversarial loss :
Revised: To encourage fair partitions according to the demographic parity, the authors introducea fairness adversarial loss :

Original: [Equation]
Revised: [Equation]

Original: Where L is the cross-entropy loss function and ◦ denotes function composition, in particular the result of the encoder
Revised: Where L is the cross-entropy loss function and ◦ denotes function composition, in particular the result of the encoder

Original: F ( X ) , the clustering assignment A and the discriminator D applied to the data in sequence.
Revised: F ( X ) , the clustering assignment A and the discriminator D applied to the data in sequence.

Original: The discriminator D is implemented as a Generative Adversarial Network (GAN) (see ﬁgure 1) with the exact inverse architecture of theused encoder.
Revised: The discriminator D is implemented as a Generative Adversarial Network (GAN) (see ﬁgure 1) with the exact inverse architecture of the usedencoder.

Original: It monitors how well the DFC is able to reconstruct the protected subgroup based on the predictedcluster probabilities of a datapoint.
Revised: It monitors how well the DFC is able to reconstruct the protected subgroup based on the predicted clusterprobabilities of a datapoint.

Original: Effectively, with respect to the fairness adversarial loss , D is maximized with theprobability of assigning the correct membership label for each sample.
Revised: With respect to the fairness adversarial loss , D is maximized with the probability ofassigning the correct membership label for each sample.

Original: Simultaneously, clustering A and representations F ( X ) are trained to maximally confuse D .
Revised: Simultaneously, clustering A and representations F ( X ) aretrained to maximally confuse D .

Original: To increase the clustering performance under the fairness constraint, DFC adds a structure preservation loss :
Revised: To increase the clustering performance under the fairness constraint, DFC adds a structure preservation loss :

Original: [Equation]
Revised: [Equation]

Original: Where ˆ P g is the soft assignment for a datapoint in protected subgroup g , and P g is the result for subgroup g when theclustering was done over the complete dataset.
Revised: where ˆ P g is the soft assignment for a datapoint in protected subgroup g , and P g is the result for subgroup g when theclustering was performed over the complete dataset.

Original: In other words, they expect, given a subgroup, that the clustering resultis similar when trained only on the subgroup data and when trained on all subgroups and looking only at the samples ofone subgroup, and encourage the distributions to be similar.
Revised: In other words, based on theoretical considerations as described inthe original paper (8), DFC expects local and global subgroup clustering partition to be similar.

Original: The ﬁnal learning objective is as follows:
Revised: The ﬁnal learning objective is as follows:

Original: [Equation]
Revised: [Equation]

Original: Note how the fairness adversarial loss (4) is simultaneously maximized w.r.t.
Revised: Note how the fairness adversarial loss (4) is simultaneously maximized w.r.t.

Original: F and A (6) to minimize performance ofthe discriminator as a result of clustering, and minimized w.r.t D (7) to optimize its parameters and obtain a saddle-pointsolution.
Revised: F and A (6) to minimize performance ofthe discriminator as a result of clustering, and minimized w.r.t D (7) to optimize its parameters and obtain a saddle-pointsolution.

Original: The values for α are 1 by default.
Revised: The values for α are 1 by default.

Original: Following the paper, for the ofﬁce-31 dataset α s is recalculated as(512 / 128) 2 · (31 / 10) to adjust for difference in batch size and cluster number.
Revised: Following the paper, for the ofﬁce-31 dataset α s is recalculated as(512 / 128) 2 · (31 / 10) to adjust for difference in batch size and cluster number.

Original: 3.1 Datasets
Revised: 3.1 Datasets

Original: Following the original paper, the following four datasets were used in our experiments:
Revised: We perform our experiments on the same four datasets as in the original paper.

Original:  MNIST-USPS is obtained from combining the MNIST dataset (7)1 and the USPS dataset (6) 2 .
Revised: To test claim 1 and 2, we restrictourselves to the MNIST-USPS dataset, obtained from combining the MNIST dataset (7) and the USPS dataset (6).

Original: From both, only the trainimages are taken (67,291 images in total, 60,000 and 7,261 respectively).
Revised: From both, only the train images are taken, 60,000 and 7,261 respectively.

Original: Both datasets consist of images of handwrittendigits, with a dimensionality of 1024 (32x32 pixels).

Original: The number of clusters is 10, corresponding to the number ofsingle digits.

Original: The sensitive attribute is the source of an image (MNSIT or USPS).

Original: The clusters are well balanced in bothdatasets (2.301 and 2.270 for MNIST and USPS respectively)

Original: but as USPS is much smaller, the maximum balance is0.120.

Original:  Color Reverse MNIST is obtained by reversing the black and white pixels in the MNIST dataset and combining thenormal and the reversed version (120,000 images in total).
Revised: The Color Reverse MNIST is obtained byreversing the black and white pixels in the MNIST dataset and combining the normal and the reversed version.

Original: Here, the sensitive attribute is whether the image comesfrom the normal or the reversed version. The[-entropy is 2 . 303 for both subgroups and the balance is 1 . 
-]
Revised: The[-entropy is 2 . 303 for both subgroups and the balance is 1 . 
-]

Original: TheThe Multi-task Facial Landamark (MTFL) dataset (13) constist of 12,995 face images annotated with gender,smiling, with or without glasses and the degree of the pose.
Revised: The Multi-task Facial Landamark (MTFL) dataset (13) consist of facial recognition pictures, from which 1000 imageswith and 1000 without glasses are sampled randomly.

Original: The authors set the glasses as the sensitive attribute andcluster for gender.

Original: For balanced subgroups, 1000 images with and 1000 without glasses are sampled randomly, yieldinga perfect balance of 1.00.

Original: The (binary) entropy is 0.693. The Ofﬁce-31 dataset (9) consists of images from 31 different categories collected from two distinct domains: Amazonand Webcam. Each domain contains all the categories.
Revised: Finally, the The Ofﬁce-31 dataset (9) consists of images from 31different categories collected from two distinct domains: Amazon and Webcam. For important statistics, see ﬁgure 2.

Original: Amazon and Webcam are selected for the experiments, becausethese two have the largest domain divergence, and the domain source is set as the protected attribute. As the Amazondomain contains more data samples, the maximum balance is 0.273. The maximum entropy is 3.434.
Revised: [Figure]

Original: 3.1.1 Hyperparameters
Revised: 3.1.1 Hyperparameters

Original:  The DFC model is dependent on 5 pretrained models; a VAE and a DEC for each subgroup and a VAE for the wholedataset.
Revised: [Figure] The DFC model is dependent on 5 pretrained models; a VAE and a DEC for each subgroup and a VAE for the wholedataset.

Original: Since the optimal hyperparameters of the DFC are likely to depend on the hyperparameter choices of thepretrained models, a complete search would bevery expensive.
Revised: Since the optimal hyperparameters of the DFC are likely to depend on the hyperparameter choices of thepretrained models, a complete search would be expensive.

Original: To make hyperparameter search manageable  theassumptionis made that the optimal hyperparameters are independent of the input data and the pretrained models ituses.
Revised: To make hyperparameter search manageable we make theassumption that the optimal hyperparameters are independent of the input data and the pretrained models it uses.

Original: This assumption enables to search the hyperparameter space of the DFC, DEC and VAE independently.
Revised: Thisassumption enables to search the hyperparameter space of the DFC, DEC and VAE independently.

Original: Note that the VAE only needs to be pretrained and optimized for the MNIST-USPS and Reverse MNIST datasets, the otherdatasets use a pretrained ResNET50 encoder.
Revised: Note that the VAEonly needs to be pretrained and optimized for the MNIST-USPS and Reverse MNIST datasets, the other datasets use apretrained ResNET50 encoder.

Original: The hyperparameter search for the DEC and the VAEwere focused on ﬁnding the best combination of the learningrate, batch size and the number of epochs.
Revised: The hyperparameter search for the DEC and the VAE focused on ﬁnding the best combination of the learning rate,batch size and the number of epochs.

Original: These searches were performed with Bayesian optimization and containedrespectively 32 and 18 experiments on MNIST.
Revised: These searches were performed with Bayesian optimization and containedrespectively 32 and 18 experiments on MNIST.

Original: The VAE Bayesian search was set to maximize the accuracy of the
Revised: The VAE Bayesian search was set to maximize the accuracy of the

Original: KMeans clustering afterwards, which is also used in the training of the DFC as initial centroids.
Revised: KMeans clustering afterwards, which is also used in the training of the DFC as initial centroids.

Original: The goal of the DECwas to optimize its clustering accuracy.
Revised: The goal of the DECwas to optimize its clustering accuracy.

Original: The DEC and VAE were pretrained using the optimal parameters on MNIST and USPS to facilitate the hyperparametersearch of the DFC.
Revised: The DEC and VAE were pretrained using the optimal parameters on MNIST and USPS to facilitate the hyperparametersearch of the DFC.

Original: The focus was on ﬁnding the optimal learning rate, batch size, and number of epochs.
Revised: The focus was on ﬁnding the optimal learning rate, batch size, and number of epochs.

Original: A randomgrid search was used since it is not clear which metric should be optimized.
Revised: A randomgrid search was used because it is not clear which metric should be optimized.

Original: The best performing set of parameters ischosen based on the Accuracy, NMI, entropy and balance.
Revised: The best performing set of parameterswas chosen based on the Accuracy, NMI, entropy and balance.

Original: The range of the hyperparameter space was inspired by hyperparameters mentioned by the authors in the paper, emailcontact and a DEC MNIST example 3 .
Revised: The range of the hyperparameter space was inspired by hyperparameters mentioned by the authors in the paper, emailcontact and a DEC MNIST example 1 .

Original: The hyperparameters of the KMeans clustering were set to the same values the authors used.
Revised: The hyperparameters of the KMeans clustering were set to the same values the authors used.

Original: TSNE hyperparameterswere set to defaults as can be seen in Appendix C, which contains a complete list of all hyperparameters and overviewsof the hyperparameter searches.
Revised: TSNE hyperparameterswere set to defaults as can be seen in Appendix C, which contains a complete list of all hyperparameters and overviewsof the hyperparameter searches.

Original:  Experimental setup and implementation
Revised:  Experimental setup and implementation

Original: All experiments were run on cuda enabled computers with seeds set to 2019 before training each model, unless otherwisespeciﬁed.
Revised: All experiments were run on cuda enabled computers with seeds set to 2019 before training each model, unless otherwisespeciﬁed.

Original: Our code containing the experiments is available at our GitHub repo 4 .
Revised: Our code containing the experiments is available at our GitHub repo 2 .

Original: Our code is strongly based on theimplementation provided by the authors 5 .
Revised: Our code is strongly based on theimplementation provided by the authors 3 .

Original: We have found that their code is a clear reﬂection of the paper.
Revised: We have found that their code is a clear reﬂection of the paper.

Original: However, itwas necessary to extend their implementation to provide support for pretraining, other datasets and other methods.
Revised: However, itwas necessary to extend their implementation to provide support for pretraining, other datasets and other methods.

Original: The DFC architectures described in the paper relies on several models joint together.
Revised: The DFC architectures described in the paper relies on several models joint together.

Original: Roughly, it includes an encoder, aclustering assignment and a discriminator.
Revised: Roughly, it includes an encoder, aclustering assignment and a discriminator.

Original: Some experiments require sightly different model combinations, which isalways made explicit.
Revised: Some experiments require slightly different model combinations, which isalways made explicit.

Original: 4.1 Pretrained Encoder
Revised: 4.1 Pretrained Encoder

Original: The type of encoder can be chosen independently from the DFC.
Revised: The type of encoder can be chosen independently from the DFC.

Original: The authors use different architectures based on thecomplexity of the dataset; for the digit based datasets a standard Variational Autoencoder is used, for the other datasetsa pretrained ResNet50.
Revised: The authors use different architectures based on thecomplexity of the dataset; for the digit based datasets a standard Variational Autoencoder is used, for the other datasetsa pretrained ResNet50.

Original: The standard VAE is based on four blocks of a 2D convolutional layer, batch normalization and a ReLU activation layer.
Revised: The standard VAE is based on four blocks of a 2D convolutional layer, batch normalization and a ReLU activation layer.

Original: This is scaled down in 2 steps with fully connected layers to the 64 dimensional hidden space.
Revised: This is scaled down in 2 steps with fully connected layers to the 64 dimensional hidden space.

Original: Since the decoder wasmissing from the author’s implementation, some guesses had to be made about the exact implementation, we tried tofollow conventions as much as possible.
Revised: Since the decoder wasmissing from the author’s implementation, some guesses had to be made about the exact implementation, we tried tofollow conventions as much as possible.

Original: A standard KL-Divergence and MSE loss were used for pretraining.
Revised: A standard KL-Divergence and MSE loss were used for pretraining.

Original: The exactarchitecture can be found in Appendix A.
Revised: The exactarchitecture can be found in Appendix A.

Original: For MTFL and Ofﬁce-31, considering their complex and high resolution images, the authors use ImageNet-pretrained
Revised: For MTFL and Ofﬁce-31, considering their complex and high resolution images, the authors use ImageNet-pretrained

Original: ResNet50 (3).
Revised: ResNet50 (3).

Original: The authors did not mention how the ResNet is ﬁnetuned exactly.
Revised: The authors did not mention how the ResNet is ﬁnetuned exactly.

Original: Due to the relatively small size of thedatasets used for the experiments, it was decided to only ﬁnetune the last layer during training the DFC.
Revised: Due to the relatively small size of thedatasets used for the experiments, it was decided to only ﬁnetune the last layer during training the DFC.

Original: 4.2 Pretrained DEC
Revised: 4.2 Pretrained DEC

Original: The structural preservation loss, discussed before 5, is calculated using two separate DEC models.
Revised: The structural preservation loss, discussed before 5, is calculated using two separate DEC models.

Original: These models couldbe seen as the golden standard for each subgroup.
Revised: These models couldbe seen as the golden standard for each subgroup.

Original: Since they are equally weighted, this ensures that the protectedsubgroup is underrepresented in the loss.
Revised: Since they are equally weighted, this ensures that the protectedsubgroup is underrepresented in the loss.

Original: These DECs also require an encoder to transform the images.
Revised: These DECs also require an encoder to transform the images.

Original: The sameencoder training methods and architectures are used for all encoders.
Revised: The sameencoder training methods and architectures are used for all encoders.

Original: The DECs were implemented as a stripped down
Revised: The DECs were implemented as a stripped down

Original: DFC with three modiﬁcations.
Revised: DFC with three modiﬁcations.

Original: The fairness and the structural preservation loss were removed.
Revised: The fairness and the structural preservation loss were removed.

Original: Lastly, the clustering lossis calculated over the entire batch at once, instead of individually per subgroup.
Revised: Lastly, the clustering lossis calculated over the entire batch at once, instead of individually per subgroup.

Original: This results in an exact implementationof the original DEC paper (11).[Table]
Revised: This results in an exact implementationof the original DEC paper (11).

Original: 4.3 Centroid Initialization
Revised: 4.3 Centroid Initialization

Original: The deep clustering methods DEC and DFC both beneﬁt from properly initializing the Student’s t-dsitribution clusterdeﬁnitions.
Revised: The deep clustering methods DEC and DFC both beneﬁt from properly initializing the Student’s t-distribution clusterdeﬁnitions.

Original: This is done with a simple KMeans clustering approach.
Revised: This is done with a simple KMeans clustering approach.

Original: KMeans is trained on the entire dataset that is usedby the clustering method afterwards.
Revised: KMeans is trained on the entire dataset that is usedby the clustering method afterwards.

Original: The ﬁnal centroids learned are copied to be the initial cluster deﬁnitions.
Revised: The ﬁnal centroids learned are copied to be the initial cluster deﬁnitions.

Original: 4.4 DFC
Revised: 4.4 DFC

Original: The DFC consists of a pretrained encoder, a discrminator module and cluster assignment module.
Revised: The DFC consists of a pretrained encoder, a discriminator module and cluster assignment module.

Original: The weights of thesemodules are updated with the Adam optimizer, using an inverse exponential learning rate scheduler.
Revised: The weights of thesemodules are updated with the Adam optimizer, using an inverse exponential learning rate scheduler.

Original: The discriminatorhas a standard architecture, details are attached in Appendix B.
Revised: The discriminatorhas a standard architecture, details are attached in Appendix B.

Original: As described by the authors, the learning rate of thediscriminator uses a multiplication factor of 10 .
Revised: As described by the authors, the learning rate of thediscriminator uses a multiplication factor of 10 .

Original: 4.4.1 Clustering Assignment
Revised: 4.4.1 Clustering Assignment

Original: The fairness constraints and learning objectives of DFC rely on clustering being performed on the subgroups separately.
Revised: The fairness constraints and learning objectives of DFC rely on clustering being performed on the subgroups separately.

Original: Since these subgroups are homogeneous with respect to sensitive attributes, they can be clustered irrespective offairness-considerations.
Revised: Since these subgroups are homogeneous with respect to sensitive attributes, they can be clustered irrespective offairness-considerations.

Original: Therefore, the same model can be use as in the Deep Embedding clustering.
Revised: Therefore, the same model can be used as in the Deep Embedding clustering.

Original: Accordingly, theauthors use the DEC clustering for the subgroups separately, which consists of a method similar to iterative k-meansclustering, where a Student’s t-distribution is used to measure similarity between an embedded representation and acluster centroid, followed by KL-divergence minimization.
Revised: Accordingly, theauthors use the DEC clustering for the subgroups separately, as described in section 3.

Original: For full details see  (11).
Revised: For full details see also (11).

Original: 4.5 Competitive methods
Revised: 4.5 Competitive methods

Original: Due to time constraints the decision was made to only implement one competitive method.
Revised: Due to time constraints, we chose to only implement one competitive method.

Original: DEC waschosen since thiscan be seen as the direct ancestor model of the DFC. As explained before, this method was implemented as a strippeddown version of the DFC which enables the evaluation on individual subgroups while training on the complete set.
Revised: DEC was a compelling candidate becauseit allows to compare fair deep clustering with ordinary deep clustering.

Original: 4.6 Metrics
Revised: 4.6 Metrics

Original: Four metrics are used to evaluate performance; accuracy, normalized mutual information, balance and entropy.
Revised: Four metrics are used to evaluate performance; accuracy, normalized mutual information, balance and entropy.

Original: Theseare the same metrics as in the original paper, which includes detailed explanations.
Revised: Theseare the same metrics as in the original paper, which includes detailed explanations.

Original: Furthermore, for evaluating thefairness two more metrics are implemented.
Revised: Furthermore, for evaluating thefairness two more metrics are implemented.

Original: The accuracy of the discriminator is computed to function as a proxy forthe fairness of the cluster assignments.
Revised: The accuracy of the discriminator is computed to function as a proxy for thefairness of the cluster assignments.

Original: The probabilities outputted by the discriminator are converted to subgroupclasses by thresholding at 0 .
Revised: The probabilities outputted by the discriminator are converted to subgroup classes bythresholding at 0 .

Original: An accuracy of 50% would indicate that the discriminator cannot guess the subgroupclass of a sample by evaluating the cluster assignment, and therefore fair cluster assignments.
Revised: An accuracy of 50% indicates fully masked representations, and therefore fair cluster assignments.

Original: Secondly, TSNE (10) isimplemented to visualize the latent space learned by the encoder.
Revised: Secondly, TSNE (10) is implemented to visualize the latent space learned by the encoder.

Original: In an ideal fair representation, the representation of thetwo subgroups should be indistinguishable.
Revised: In an ideal fair representation,the representation of the two subgroups should be indistinguishable.

Original: For example, the digit 5 of MNIST and the digit 5 of USPS both encodedas the character ’5’ would be ideal, since the representations are indistinguishable.
Revised: For example, the digit 5 of MNIST and the digit 5of USPS both encoded as the character ’5’ would be ideal, since the representations are indistinguishable.

Original: The goal of a TSNE mapping isthat distant samples in the high dimensional space are also far away from each other in the lower dimension, and viceversa.
Revised: The goal of a TSNE mapping is that distant samples in the high dimensional space are also far away from each other in the lowerdimension, and vice versa.

Original: Therefore, in a visualization of a fair latent space, it should be hard to distinguish clusters of protected subgroups.
Revised: Therefore, in a visualization of a fair latent space, it should be hard to distinguish clusters ofprotected subgroups.

Original: All results are calculated over 5 different seeds: 0, 1, 42, 2019 and 2020.
Revised: All results are calculated over 5 different seeds: 0, 1, 42, 2019 and 2020.

Original: However, it was not possible to provide thevariance for the results that require a ResNet encoder due to time and resource limitations.
Revised: However, it was notpossible to provide the variance for the results that require a ResNet encoder due to time and resource limitations.

Original: 4.6.1 Computational requirements
Revised: 4.6.1 Computational requirements

Original: We trained all models on an 8.25MB GeForce 1080Ti GPU.
Revised: We trained all models on an 8.25MB GeForce 1080Ti GPU.

Original: Pretraining VAE encoders for each dataset took (5 × ) ±45 min, pretraining the DEC models (7 × ) ± 1 hr.
Revised: Pretraining VAE encoders for each dataset took (5 × ) ±45 min, pretraining the DEC models (7 × ) ± 1 hr.

Original: The sweep for optimal parameter search took (20 × ) 1hr and runningthe DFC and DEC models for all four datasets on 5 random seeds took (8 × ) ± 1 hr GPU time.
Revised: The sweep for optimal parameter search took (20 × ) 1hr and runningthe DFC and DEC models for all four datasets on 5 random seeds took (8 × ) ± 1 hr GPU time.

Original: The total experimentsrequired a total of ± 40 hr GPU time.
Revised: The total experimentsrequired a total of ± 40 hr GPU time.

Original:  Results
Revised:  Results

Original: Our results support the ﬁrst claim that DFC learns fair representations that do not cause much loss in utility and thesecond claim that the minimax optimization formulation contributes to clustering that is both fair and accurate.
Revised: Our results support the ﬁrst claim that DFC learns fair representations that do not cause much loss in utility and thesecond claim that the minimax optimization formulation contributes to clustering that is both fair and accurate.

Original: Wereproduced accuracy on the MNIST-USPS dataset to within 2% of reported value, normalized mutual information(NMI) and Entropy to within 1%, and balance to within 5%.
Revised: Wereproduced accuracy on the MNIST-USPS dataset to within 2% of reported value, normalized mutual information(NMI) and Entropy to within 1%, and balance to within 5%.

Original: Our DFC method outperformed our DEC method onall accuracy and fairness metrics.
Revised: Our DFC method outperformed our DEC method onall accuracy and fairness metrics.

Original: We reproduced the accuracy of the Ofﬁce-31 datasets to within 1% and the MTFLdataset to within 5% , but failed to obtain similarly good results for the fairness metrics.[Figure]
Revised: We reproduced the accuracy of the Ofﬁce-31 datasets to within 1% and the MTFLdataset to within 7% , but failed to obtain similarly good results for the fairness metrics.

Original: 5.1 Claim 1: Learning fair representations for the MNIST-USPS dataset
Revised: 5.1 Claim 1: Learning fair representations for the MNIST-USPS dataset

Original: Figure 2 shows the learned representation of the DFC method (top) and the DEC method (bottom).
Revised: Figure 3 shows the learned representation of the DFC method (top) and the DEC method (bottom).

Original: Protected subgroupmembership is indicated by color for MNIST (red) and USPS (blue).
Revised: Protected subgroupmembership is indicated by color for MNIST (red) and USPS (blue).

Original: Monitoring the DFC discriminator performanceshows that the accuracy for recovering the protected subgroup drops from 60% at the ﬁrst 1000 runs to 50% after 2000iterations.
Revised: Monitoring the DFC discriminator performanceshows that the accuracy for recovering the protected subgroup drops from 60% at the ﬁrst 1000 runs to 50% after 2000iterations.

Original: The corresponding accuracy and fairness metrics are shown in table 1.
Revised: The corresponding accuracy and fairness metrics are shown in table 1.

Original: We obtain an accuracy within 2% ofthe accuracy reported in the paper.
Revised: We obtain an accuracy within 2% ofthe accuracy reported in the paper.

Original: NMI and Entropy scores fell within 1% of the original ﬁndings, and balance was5 .
Revised: NMI and Entropy scores fell within 1% of the original ﬁndings, and balance was

Original: 5% lower in our experiments.
Revised: [Figure]

Original: 5.2 Claim 2: Validity of minimax optimization formalization
Revised: 5.2 Claim 2: Validity of minimax optimization formalization

Revised: [Figure]

Revised: An accuracy of 0.5 indicates that representations cannot be discriminated for subgroup membership.

Original: The ﬁrst two rows of table 1 show performance on all four metrics for the DFC and DEC methods.
Revised: The ﬁrst two rows of table 1 show performance on allfour metrics for the DFC and DEC methods.

Original: Results indicate that DFC outperforms DEC both in accuracy and in fairness methods.
Revised: Resultsindicate that DFC outperforms DEC both in accuracyand in fairness methods.

Original: Moreover, the table shows that our implementation ofthe DEC method achieves a 17% higher accuracy and performs better than the original implementation of the DEC onall other metrics except balance.
Revised: Moreover, the table shows thatour implementation of the DEC method achieves a 17%higher accuracy and performs better than the originalimplementation of the DEC on all other metrics exceptbalance.

Revised: To further investigate how subcomponents of DFC contribute towards performance and fairness, we plot thedecoder accuracy in ﬁgure 4.

Revised: Interestingly, the plot showsthat DFC initially achieves perfectly masked representation up to around 8 .

Revised: 5 k training steps, where F ( X ) and

Revised: A no longer succeed in fully confusing D , presumablyin favour of cluster favourable representations.

Revised: This indicates that some trade-off is still present.

Original: 5.3 Claim 3: Different datasets
Revised: 5.3 Claim 3: Different datasets

Original: The results for all datasets are can be found in table 1.
Revised: The results for all datasets are can be found in table 1.

Original: Our implementation of DFC outperformed DEC on the threeother datasets and showed similar accuracy as reported in the original paper.
Revised: Our implementation of DFC outperformed DEC on the threeother datasets and showed similar accuracy as reported in the original paper.

Original: Results for entropy were also consistentlyhigher than the DEC implementations and similar to the entropy reported in the original paper.
Revised: Results for entropy were also consistentlyhigher than the DEC implementations and similar to the entropy reported in the original paper.

Original: The balance, however,was much lower than in the original paper, and did not differ signiﬁcantly from the DEC results.
Revised: The balance, however,was much lower than in the original paper, and did not differ signiﬁcantly from the DEC results.

Original:  Discussion
Revised:  Discussion

Original: Our results show that the reported performance of DFC can be reproduced on the MNIST-USPS digit dataset.
Revised: For the MNIST-USPS digit datasets, the results indicate that the method is able to learn fair representations and clusters.

Revised: The entropy results indicate equal fair partitions, the TSNE visualizations show good clusters and the discriminator isnot able to distinguish between subgroups in the cluster assignments.

Original: We werepartly unable to reproduce its fairness performance on the three other datasets but can think of various limitations of ourapproach that could provide an explanation for this.
Revised: We were partly unable to reproduce its fairness performance on the three other datasets but can think of variouslimitations of our approach that might provide an explanation for this.

Original: First, we put notable effort into adjusting and training pretrainedmodels for the MNIST-USPS experiments, and performed a hyperparameter search to ﬁnd  optimal settings.
Revised: First, we put notable effort into adjustingand training pretrained models for the MNIST-USPS experiments, and performed a hyperparameter search to ﬁnd [Table] optimal settings.

Original: Due totime and resource constraints, we were not able to repeat this process for all datasets, which may have led to differentresults.
Revised: Due to time and resource constraints, we were not able to repeat this process for all datasets, whichmay have led to different results.

Original: Secondly, our use of the ResNet50 encoder differed slightly from that in the paper.
Revised: Secondly, our use of the ResNet50 encoder differed slightly from that in the paper.

Original: Correspondence with theauthors had conﬁrmed that their implementation ﬁnetuned all parameters of ResNet50.
Revised: Correspondence with the authors had conﬁrmed that their implementation ﬁnetuned all parameters of ResNet50.

Original: However, for us GPU memoryconstraints made it impossible to store all gradients that would be required.
Revised: However, for us GPU memory constraints made it impossible to store all gradients that would be required.

Original: Therefore, we resolved to only ﬁnetune theﬁnal layer, based on transfer learning literature (12) and the relatively small sizes of the datasets.
Revised: Therefore,we resolved to only ﬁnetune the ﬁnal layer, based on transfer learning literature (12) and the relatively small sizes of thedatasets.

Original: This could partiallyexplain our different results.[Table]
Revised: This could partially explain the difference in results.

Original: The results indicate that the method is able to learn fair representations and clusters.

Original: The entropy results indicate equalfair partitions, the TSNE visualizations show good clusters and the discriminator is not able to distinguish betweensubgroups in the cluster assignments.

Original: However, during evaluation, questions about the added fairness constraints andthe experiment setup of this method arise.
Revised: However, evaluating our full results, an additional question about the added fairness constraints and the experimentsetup of this method arose.

Original: The optimal clustering solutions for the datasets exactly match optimalfairness.
Revised: In the case of MNIST-USPS, optimal clustering solutions for the datasets closely matchoptimal fairness according to the constraints.

Original: Since the  datasets are fairly balanced  the equal partition deﬁnition of fairness is in line with the optimalsolution. Furthermore, fair clustering would perform equally good on all subgroups, which also in line with the optimalsolution.
Revised: Since the digit datasets are quite nicely balanced in the sense that theyhave similar partitions across subgroups, the equal partition deﬁnition of fairness is in line with the optimal solution,and we suspect that the fairness constraints here do not pose as much of a challenge as they might in more imbalanceddatasets.

Original:  Further research needs to be done to investigate this hypothesis.
Revised: For this reason, we were particularly interested in the results for the non-digit datasets. Further research mightbe needed to investigate the theoretical framework in the context of datasets with different inherent fairness challenges.

Original: 6.1 What was easy and/or difﬁcult
Revised: 6.1 What was easy and/or difﬁcult

Original: Based on the implementation made  by the authors it was straightforward to evaluate the method on MNIST-USPS,using the provided pretrained encoders, DECs and initial centroids.
Revised: Based on the implementation made availabe by the authors it was straightforward to evaluate the method on MNISTUSPS, using the provided pretrained encoders, DECs and initial centroids.

Original: These results matched the results achieved in theoriginal paper.
Revised: These results matched the results achievedin the original paper.

Original: However, essential code and speciﬁcations were missing to extend these results to other datasets andmethods.
Revised: However, essential code and speciﬁcations were missing to extend these results to other datasetsand methods.

Original: Examples include the hyperparameters for pretraining all the models, hyperparameters for running on otherdatasets, missing speciﬁcations for the Decoder of the VAE and the ﬁnetuning of ResNet50.
Revised: Examples include the hyperparameters for pretraining all the models, hyperparameters for running onother datasets, missing speciﬁcations for the Decoder of the VAE and the ﬁnetuning of ResNet50.

Original: This made it hard toimplement the methodto work for other datasets.
Revised: This made it hard toreplicate the method for other datasets.

Original: Despite the missing parts, the codMoste wasvery helpful for understanding the method in detail.
Revised: Despite the missing parts, the code was helpful for understanding the method in detail.

Original: Most importantly, itcleared up the workings of the structural preservation loss which was not immediately clear from the paper.
Revised: Most importantly, it cleared upthe workings of the structural preservation loss which was not immediately clear from the paper.

Original: The  number of different pretrained models needed was another factor that caused problems.
Revised: The large number of different pretrained models needed was another difﬁculty.

Original: First of all, the DFCfor MNIST-USPS depends on 8 other models: 3 encoders, 2 DECs and 3 KMeans cluster initializations.
Revised: The DFC for MNIST-USPS depends on8 other models: 3 encoders, 2 DECs and 3 KMeans cluster initializations.

Original: This incombination with the fact that the DFC is sensitive to the quality of the pretrained models, made it hard to debugperformance issues that could be caused by the model itself or any of the previous methods.
Revised: This, in combination with the fact that the DFC is sensitive to the quality of the pretrained models, made it hard to debug performance issues that could be causedby the model itself or any of the previous methods.

Original: And practically, itrequires excellent experiment management to keep track of which models are trained with which pretrained models andparameters.
Revised: In addition, it required excellent experiment management to keeptrack of which models are trained with which pretrained models and parameters.

Original: 6.2 Communication with original authors
Revised: 6.2 Communication with original authors

Original: We had one-off email contact with the authors regarding questions about implementation.

Original: Speciﬁcally, we consultedthem about hyperparameters for pretraining the encoders and which layers of ResNet50 to ﬁnetune.
Revised: We consulted the authors about hyperparameters for pretraining the encoders and which layers of ResNet50 to ﬁnetune.

Original: We received ahelpful response in which our questions were adequately answered.
Revised: We received a helpful response in which our questions were adequately answered.

Original: A Variational Autoencoder Architecture
Revised: A Variational Autoencoder Architecture

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C Hyperparameters
Revised: C Hyperparameters

Original: [Table] [Table]
Revised: [Table] [Table]