Original: Improving Fairness in Image Classification via Sketching
Revised: Improving Fairness in Image Classification via Sketching

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Fairness is a fundamental requirement for trustworthy and human-centered Artificial Intelligence (AI) system.
Revised: Fairness is a fundamental requirement for trustworthy and human-centered Artificial Intelligence (AI) system.

Original: However, deep neural networks (DNNs) tendto make unfair predictions when the training data are collected from differentsub-populations with different attributes ( i.e. color, sex, age), leading to biased
Revised: However, deep neural networks (DNNs) tend to make unfair predictions when the training data are collected from different sub-populations with different attributes ( i.e. color, sex, age), leading to biased

Original: DNN predictions.
Revised: DNN predictions.

Original: We notice that such a troubling phenomenon is often caused bydata itself, which means that bias information is encoded to the DNN along withthe useful information ( i.e. class information, semantic information).
Revised: We notice that such a troubling phenomenon is often caused by data itself, which means that bias information is encoded to the DNN along with the useful information ( i.e. class information, semantic information).

Original: Therefore,we propose to use sketching to handle this phenomenon.
Revised: Therefore, we propose to use sketching to handle this phenomenon.

Original: Without losing the utilityof data, we explore the image-to-sketching methods that can maintain the usefulsemantic information for the targeted classification while filtering out the uselessbias information.
Revised: Without losing the utility of data, we explore the image-to-sketching methods that can maintain useful semantic information for the target classification while filtering out the useless bias information.

Original: In addition, we design a fair loss for further improving the modelfairness.
Revised: In addition, we design a fair loss to further improve the model fairness.

Original: We evaluate our method on extensive experiments on both general scenedataset and medical scene dataset.
Revised: We evaluate our method through extensive experiments on both general scene dataset and medical scene dataset.

Original: Our results show that the desired image-tosketching method improves model fairness and achieves satisfactory results amongstate-of-the-art (SOTA).
Revised: Our results show that the desired image-tosketching method improves model fairness and achieves satisfactory results among state-of-the-art (SOTA).

Original: Our code would be released based on acceptance.

Original:  Introduction
Revised:  Introduction

Original: Neural networks make many great achievements in different computer vision scenarios includinggeneral scenes and medical scenes, bringing much wellness tothe society.
Revised: Neural networks make many great achievements in different computer vision scenarios including general scenes and medical scenes, which bring much wellness to society.

Original: However, serious challengealso ensues.
Revised: However, serious challenge also ensues.

Original: Standard vision model tends to make unfair predictions on skin color, age, sex, etc.[1, 2, 10, 12, 14, 18, 22, 27, 28].
Revised: Standard vision model tends to make unfair predictions on skin color, age, sex, etc.

Original: One key reason comes from the bias in the input data.
Revised: One key reason comes from the bias in the input data.

Original: Whenthe images are input to network, the bias information is also encoded thus leading model’s unfairprediction.
Revised: When the images are fed into the network, the bias information is also encoded thus leading model’s unfair prediction.

Original: Taking advantage of the advances in  latest image-to-sketch research [4, 7, 8, 13, 24, 26], we exploitthe sketch to relieve unfairness while keeping the class or semantic information.
Revised: Taking advantage of the advances in the latest image-to-sketch research [4, 7, 8, 13, 24, 26], we exploit the sketch to relieve unfairness while keeping the class or semantic information.

Original: Since sketchesconvey much semantic information from the original images, neural networks could also use them tomake predict without sacrificing much accuracy, especially where class discrimination relies littleon the color information ( i.e. skin cancer).
Revised: Since sketches convey most semantic information from the original images, neural networks could also use them to make predictions without sacrificing much accuracy, especially where class discrimination relies little on the color information ( i.e. skin cancer).

Original: Meanwhile, sketches throw the bias factors ( i.e. color,texture) out with the color and texture information.
Revised: Meanwhile, sketches throw the bias factors ( i.e. color, texture) out with the color and texture information.

Original: Previous methods often handle the unfairness problem into three ways: Pre-processing approach, In-processing approach, and Post-processing approach.
Revised: Previous methods often handle the unfairness problem in three ways: Pre-processing, In-processing, and Post-processing.

Original: (a).
Revised: (a).

Original: The pre-processing work deals with thebias in the databaseto train a fairer model [19, 21, 27, 31], which means the approaches to reducethe bias contained in the training set, such as transforming the data representation [19], generatingpaired training data to balance protected attributes [21], and generating adversarial examples tobalance the number of protected attributes in the training set [31].
Revised: The pre-processing work deals with the bias in the database [19, 21, 27 , 31], which tries to reduce the bias contained in the training set, such as transforming the data representation [19], generating paired training data to balance protected attributes [21], and generating adversarial examples to balance the number of protected attributes in the training set [31].

Original: In this work, the way to processthe input images into sketches is also pre-processing approach.
Revised: In this work, the way to process the input images into sketches is also pre-processing approach.

Original: (b).
Revised: (b).

Original: In-processing method is tointroduce a fairness mechanism in the training model [3, 14, 29, 30], such as adding fairness loss to constrain [29] and using adversarial training to force the model to produce a fair output [3, 14, 30],these methods aim to obtain fairer models by explicitly changing the training procedure.
Revised: Inprocessing method aims to introduce a fairness mechanism in the training model [3, 14, 29, 30], such as adding fairness loss to constrain [29] and using adversarial training to force the model to produce a fair output [3, 14, 30]. The purpose is to obtain fairer models by explicitly changing the training procedure.

Original: (c).Postprocessing approaches adjusts model’s predictions according to some fairness criteria [11, 16].
Revised: (c). Post-processing approach adjusts model’s predictions according to some fairness criteria [11, 16].

Original: Oneway is to adjust the model predictions by editing the protected attributes accordingly [16], howeverthis method is difficult to implement in practice [27], another work is to post-process a pre-traineddeep learning model and create a new classifier with the same accuracy for people with differentprotected attributes [11].
Revised: One way is to adjust the model predictions by editing the protected attributes accordingly [16], however this method is difficult to implement in practice [27]. Another work is to post-process a pre-trained deep learning model and create a new classifier with the same accuracy for people with different protected attributes [11].

Original: Without manipulating model in training stage or adjusting prediction results. We use image sketchas an intuitive solution to complete the pre-process step in the data side.
Revised: Without manipulating model in training stage or adjusting prediction results, we use image sketching as an intuitive solution to complete the pre-process step on the data side.

Original: Our contribution could besummarized as follow: (1).
Revised: Our contribution could be summarized as follow: (1).

Original: We make the first attempt to exploit sketching for improving computervision fairness.
Revised: We make the first attempt to exploit sketching for improving computer vision fairness.

Original: This simple solution could effectively abandon the bias of input data for fair prediction.(2)
Revised: This simple solution could effectively abandon the bias of input data for fair prediction.

Original: We introduce a fairness loss to further improve model’s fairness results.
Revised: We introduce a fairness loss to further improve model’s fairness results.

Original: Our method achievessatisfactory performance among state-of-the-arts.
Revised: Our method achieves satisfactory performance among state-of-the-art.

Original:  Methods 
Revised:  Methods 2.1 Overview

Original: 2.1 Overview

Original: An overview of our method has been shown in Fig.
Revised: An overview of our method has been shown in Fig.

Original: 1, based on standard vision classification paradigm,our pipeline mainly involves two modifications.
Revised: Based on standard vision classification paradigm, our pipeline mainly involves two modifications.

Original: First we convert the input images into their sketchesand feed them to  following classification model for prediction.
Revised: First, we convert the input images into their sketches and feed them to the following classification model for prediction.

Original: Towards further fairness improvement,in second step we introduce a fairness loss function to mitigate the bias in the model.
Revised: Towards further fairness improvement, in the second step we introduce a fairness loss function to mitigate the bias in the model.

Original: Sufficientexperiments and detailed explanations are shown in Sec. 3.
Revised: Sufficient experiments and detailed explanations are shown in Sec. 3.

Original: For each image index i , x i denotes the original input images, y i denotes the image label and z idenotes the bias sensitive attribute.
Revised: For each image index i , x i denotes the original input images, y i denotes the image label, and z i denotes the bias sensitive attribute.

Original: In our method, the input image x i would first  sent in to sketchgeneration model S to generate x i ’s corresponding sketch S ( x i ) , then the sketch image S ( x i ) wouldbe sent into following classifier M to generate classification results ˆ y i
Revised: In our method, the input images x i would first be sent into sketch generation model S to generate x i ’s corresponding sketch S ( x i ) . Then the sketch image S ( x i ) would be sent into following classifier M to generate classification results ˆ y i

Original: =
Revised: =

Original: M ( S ( x i )) 
Revised: M ( S ( x i )) .

Original: , classificationloss L cls would by calculating the cross-entropy (CE) loss between ˆ y i and y i , additionally we designa fairness loss L fair (Sec. 2.3) for further fairness improvement.
Revised: Classification loss L cls would by calculating the cross-entropy (CE) loss between ˆ y i and y i , additionally we design a fairness loss L fair (see Sec. 2.3) for further fairness improvement.

Original: Overall, the classfication model Mwould be optimise by the total loss L ( λ is a non-negative number to balance, which we setit to 1.0in our experiments):
Revised: Overall, the classification model M would be optimized by the total loss L ( λ is a non-negative number to balance, which we set to 1.0 in our experiments):

Original: [Equation]
Revised: [Equation]

Original: Target label is smiling and sensitive attributes are gender, skin color, and hair color respectively.

Original: As to evaluation metrics, based on previous works [14, 27], we apply Statistical Parity Difference(SPD) , Equal Opportunity Difference (EOD) , Difference in Equalized Odds (DEO) , and Average
Revised: As to evaluation metrics, based on previous works [14, 27], we apply Statistical Parity Difference ( SPD ), Equal Opportunity Difference ( EOD ), Difference in Equalized Odds ( DEO ), and Average

Original: Odds Difference (AOD) to measure and evaluate model’s fairness.
Revised: Odds Difference ( AOD ) to measure and evaluate model’s fairness.

Original: For all four evaluation metrics, thesmaller the value of their indicators, the higher the fairness of the model.
Revised: For all four evaluation metrics, the smaller the value of their indicators, the higher the fairness of the model.

Original: 2.2 Sketch Generation
Revised: 2.2 Sketch Generation

Original: For sketch generation model S , we employ  latest image-to-sketching method [4] to convert inputimage x
Revised: For sketch generation model S , we employ the latest image-to-sketching method [4] to convert input image x

Original: i into its corresponding sketch S ( x i ) .
Revised: i into its corresponding sketch S ( x i ) .

Original: The sketching method [4] is an unsupervised model trained on unpaired data between photographs(domain A ) and sketchings (domain B ).
Revised: The sketching method [4] is an unsupervised model trained on unpaired data between photographs (domain A ) and sketches (domain B ).

Original: It setups an adversarial training network with generators
Revised: It setups an adversarial training network with generators

Original: G A , G B and discriminators D A , D B, respectively for domain A and B .
Revised: G A , G B and discriminators D A , D B respectively for domain A and B .

Original: Different from previousimage-to-sketching methods, this model not only focuses on image appearance but also explicitlyinstills geometric and semantic information into drawings.
Revised: Different from previous image-to-sketching methods, this model not only focuses on image appearance but also explicitly instills geometric and semantic information into drawings.

Original: Specifically, the model is forced to drawlines in important geometric areas under a geometric loss which predicts depth information.
Revised: Specifically, the model is forced to draw lines in important geometric areas under a geometric loss which predicts depth information.

Original: Inaddition, a CLIP [20] model guidance is adopted to constraint the semantic meaning of sketchescloser to  original images [4, 20].
Revised: In addition, a CLIP [20] model guidance is adopted to constraint the semantic meaning of sketches close to that of in original images [4, 20].

Original: In the training stage, geometric loss L Geom is implemented to maximize the depth information tocapture as much depth information as possible.
Revised: In the training stage, geometric loss L Geom is implemented to maximize the depth information and capture as much depth information as possible.

Original: Using a state-of-the-art depth estimation network[17] F , a network G Geom is designed to predict the depth map of sketching during training.
Revised: Using a SOTA depth estimation network F [17], a network G Geom is designed to predict the depth map of sketches during training.

Original: Given inputa , G A is the sketching generation network and I ( G ( a )) are the extracted ImageNet [6] features.
Revised: Given input a , G A is the sketch generation network and I ( G ( a )) are the extracted ImageNet [6] features.

Original: Thedepth maps of photographs can be denoted as F ( a ) and depth maps of sketches is G Geom ( I ( G ( a ))) .
Revised: The depth maps of photographs can be denoted as F ( a ) and depth maps of sketches is G Geom ( I ( G ( a ))) .

Original: The geometric loss is the absolute difference between these two predicted depth maps.
Revised: The geometric loss is the absolute difference between these two predicted depth maps.

Original: L Geom =| G Geom ( I ( G ( a 
Revised: L Geom = | G Geom ( I ( G ( a )))

Original: [Equation]
Revised: − F ( a ) |

Original: The semantic loss encourages minimizing the semantic meaning gap between input image and linedrawings.
Revised: The semantic loss encourages minimizing the semantic meaning gap between input images and line drawings.

Original: The smaller the semantic distance, the more accurate line drawings are translated into.
Revised: The smaller the semantic distance, the more accurate line drawings are translated into.

Original: The semantic loss is defined as the absolute CLIP difference between input image and sketches.
Revised: The semantic loss is defined as the absolute CLIP difference between input images and sketches.

Original: [Equation]
Revised: [Equation]

Original: Moreover, appearance loss L cycle and adversarial loss L GAN are implemented to encode inputappearance and generate images to belong to their domains.
Revised: Moreover, appearance loss L cycle and adversarial loss L GAN are implemented to encode input appearance and generate images belonging to their domains.

Original: The overall loss function is the weightedsummation of these four losses:
Revised: The overall loss function is the weighted summation of these four losses:

Original: [Equation]
Revised: [Equation]

Original: Chan et al.
Revised: Chan et al.

Original: [4] set λ CLIP = 10 , λ Geom = 10 , λ GAN = 1 , and λ cycle = 0 .
Revised: [4] set λ CLIP = 10 , λ Geom = 10 , λ GAN = 1 , and λ cycle = 0 .

Original: 1 in practice.
Revised: 1 in practice.

Original: Takingadvantage of sketch, we are able to reduce the bias.
Revised: Taking advantage of sketching, we are able to reduce the bias.

Original: As shown in Figure 2, the sketching images arenot inferior in accuracy and has an obvious advantage in the fairness indicator DEO when comparedwith the original images and the grayscale images.
Revised: As shown in Fig. 2, the sketch images are not inferior in accuracy and have an obvious advantage in the fairness indicator DEO when compared with the original images and the grayscale images.

Original: 2.3 Fairness Constraint
Revised: 2.3 Fairness Constraint

Original: Since the initial purpose of image-to-sketching method [4, 8, 13, 24, 26] is not designed for fairnessimprovement, we introduce an additional fairness loss function to further mitigate bias of the modelduring training.
Revised: Since the initial purpose of image-to-sketching method [4, 8, 13, 24, 26] is not designed for fairness improvement, we introduce an additional fairness loss function to further mitigate bias of the model during training.

Original: The fairness loss function L fair is designed as a mean square error ( MSE ) loss of
Revised: The fairness loss function L fair is designed as a mean square error ( MSE ) loss of

Original: Statistical Parity Difference (SPD) for every mini-batch.
Revised: Statistical Parity Difference (SPD) for every mini-batch.

Original: During model training, the fairness loss
Revised: During model training, the fairness loss

Original: L fair would combine with classification loss L cls to optimise model parameters (see Eq. 1). 
Revised: L fair would combine with classification loss L cls to optimize model parameters (see Eq. 1). [Equation]

Original: [Equation]
Revised: [Equation]

Original: Specifically, SPD measures the difference of probability in positive predicted label ( ˆ y = 1 ) betweenprotected ( z = 1 ) and unprotected ( z = 0 ) attribute groups.
Revised: Specifically, SPD measures the difference of probability in positive predicted label ( ˆ y = 1 ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups.

Original: [Equation]
Revised: [Equation]

Original:  Experiments 
Revised:  Experiments 3.1 Datasets and Experimental Setup

Original: 3.1 Datasets and Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We adopted two datasets for our evaluation, CelebA [15] and Skin ISIC 2018 [5, 25], 
Revised: We adopt two datasets for our evaluation, CelebA [15] and Skin ISIC 2018 [5, 25], CelebA

Original: CelebA for face recognition classification and Skin ISIC 2018 for skin cancer classification.•

Original:  CelebA
Revised: • CelebA

Original: The CelebA dataset consists of 202,599 human face images with 40 features per image [15].
Revised: The CelebA dataset consists of 202,599 human face images with 40 features per image [15].

Revised: Following Wang et al.

Original: Following [27], we choose smiling and attractive as the predicted label for binary classclassification.
Revised: [27], we choose smiling and attractive as the predicted label for binary class classification.

Original: For each predicted label, we choose gender (male and female), hair color(black hair and others), and skin color (pain skin and others) as the bias sensitive attribute.
Revised: For each predicted label, we choose gender (male and female), hair color (black hair and others), and skin color (painful skin and others) as the bias sensitive attributes.

Original: We select 10,000 images for every experiment, of which 70% for training, 15% for validation,and 15% for testing.
Revised: We select 10,000 images for every experiment, of which 70% for training, 15% for validation, and 15% for testing.

Original: For each experiment, the number of images in protected attribute andunprotected attribute are equal, which means we adopt a fair dataset.
Revised: For each experiment, the number of images in protected attribute and unprotected attribute are equal, which means we adopt a fair dataset.

Original: • Skin  ISIC {+
+}
Revised: • Skin ISICThe Skin ISIC {+
+}

Original: The Skin ISIC 2018 dataset consists of 10,015 skin images for 7-class skin cancer classification [5, 25]. 
Revised: dataset consists of 10,015 skin images for 7-class skin cancer classification [5, 25]. Following Li et al.

Original: Following [14], we choose gender (male and female), age ( ≥ 60 and < 60), andskin color (dark and light skin) as the bias sensitive attribute.
Revised: [14], we choose gender (male and female), age ( ≥ 60 and < 60), and skin color (dark and light skin) as the bias sensitive attributes.

Original: We randomly split the datasetto training set (80%) and testing set (20%) similar to [14].
Revised: We randomly split the dataset into training set (80%) and testing set (20%).

Original: Training and Evaluation.
Revised: Training and Evaluation.

Original: We use original images and sketches to train the classification modelseparately.
Revised: We use original images and sketch images to train the classification model separately.

Original: Furthermore, we compare with gray-scale images whose color information is removed.
Revised: Furthermore, we compare with grayscale images whose color information is removed.

Original: This allows us to evaluate the impact between changing only the color of the image and modifyingboth the color and texture.
Revised: This allows us to evaluate the impact between changing only the color of the image and modifying both the color and texture.

Original: In addition, the fairness loss function is added to cross-entropy loss whentraining on sketched images.
Revised: In addition, the fairness loss function is added to cross-entropy loss when training on sketch images (see Eq. 1).

Original: We use ResNet-18 [9] to train CelebA dataset and VGG11 [23] to train ISIC dataset follow previoussetting [14, 27].
Revised: We use ResNet-18 [9] to train CelebA dataset and VGG11 [23] to train ISIC dataset following previous setting [14, 27].

Original: Batch size is set to 64 in CelebA and 32 in ISIC.
Revised: Batch size is set to 64 in CelebA and 32 in ISIC.

Original: Learning rate is set to 1e-3.
Revised: Learning rate is set to 1e-3.

Original: Forevaluation metrics, we evaluate the results CelebA dataset with SPD and DEO .
Revised: For evaluation metrics, we evaluate the results CelebA dataset with SPD and DEO .

Original: And we evaluate theresults of ISIC dataset with SPD , EOD , and AOD .
Revised: And we evaluate the results of ISIC dataset with SPD , EOD , and AOD .

Original: 3.2 Experimental Results
Revised: 3.2 Experimental Results

Original: On CelebA dataset, tables from 1 to 6 show that classification accuracy does not drop too much onother styles of images compared to the one onthe original image.
Revised: On CelebA dataset [Table] tables from 1 to 6 show that classification accuracy does not drop too much on other styles of images compared to the one on original images.

Original: Particularly, sketches either withor without fairness loss, exhibits the minimum values of fairness metrics, SPD and DEO , showinghighest fairness compared to the original and grayscale images.
Revised: Particularly, sketches either with or without fairness loss exhibit the minimum values of fairness metrics, SPD and DEO , showing the highest fairness compared to the original and grayscale images.

Original: The result of Sketchs with L fairdemonstrates that the fairness loss could further improve fairness in some cases. 
Revised: The result of sketches with L fair demonstrates that the fairness loss could further improve fairness in some cases. [Table] [Table] [Table]

Original:  We also compare a SOTA fairness method, fair training + FAAP [27].
Revised: [Table] [Table] [Table] We also compare a SOTA fairness method, fair training + FAAP [27].

Original: As shown in table 1, our modeloutperforms [27] in terms of SPD on gender attribute.
Revised: As shown in table [Table] our model outperforms [27] in terms of SPD on gender attribute.

Original: For accuracy and DEO , our model still deliverscompetitive performance.
Revised: For accuracy and DEO , our model still delivers competitive performance.

Original: While  [27] only considers gender as sensitive attribute, we also assess other sensitive attributes thatare likely to cause bias such as skin color and hair color.
Revised: While Wang et al. [27] only consider gender as sensitive attribute, we also assess other sensitive attributes that are likely to cause bias such as skin color and hair color.

Original: Table 3 andtable 4 reports the result on skincolor as sensitive attribute.
Revised: Table 3 and 4 report the result on skin color as sensitive attribute.

Original: The accuracy on sketches doesn’t drop much but the fairness indices arefar below that of in original photographs, significantly reducing the bias in orginal images.
Revised: The accuracy on sketches doesn’t drop much but the fairness indices are far below that of in original photographs, significantly reducing the bias in original images.

Original: Table 5and table 6 also show similar patterns.
Revised: Table 5 and 6 also show similar patterns.

Original: On both skin color and hair color, the use of sketching imagesare demonstrated to effectively improve fairness without much sacrafice in classification accuracy. 
Revised: On both skin color and hair color, the use of sketch images is demonstrated to effectively improve fairness without much sacrifice in classification accuracy. the sensitive attribute is gender

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Tables 7 to 9 are the experimental results of ISIC dataset.
Revised: Tables 7 to 9 are the experimental results of ISIC dataset.

Original: In table 7, the sensitive attribute is gender.
Revised: In table 7, the sensitive attribute is gender.

Original: We find that although using sketches does not show more fairness in terms of SPD , the gap betweenthe original images and the sketch images is very small.
Revised: We find that although using sketches does not show more fairness in terms of SPD , the gap between the original images and the sketch images is very small.

Original: In terms of EOD and AOD , using sketchimages can still effectively reduce bias.
Revised: In terms of EOD and AOD , using sketch images can still effectively reduce bias.

Original: In addition, the decline  is more significant when fairness lossis added.
Revised: In addition, the decline of bias attributes is more significant when fairness loss is added.

Original: In table 8 and 9, we can find similar pattern as table 7.
Revised: In table 8 and 9, we can find similar patterns as table 7.

Original: After adding fairness loss to themodel, EOD and AOD , significantly decreases. [Table] [Table] [Table]
Revised: After adding fairness loss to the model, EOD and AOD decrease significantly.

Original:  Conclusion and Limitation
Revised:  Conclusion and Limitation

Original: In this paper [Table] we propose to use image sketching methods to improve model fairness among severalbias types ( i.e. color, sex, age) based on the intuition that  suitable sketching method can filter out thebias information while keeping semantic information for classification.
Revised: In this paper, we propose to use image sketching methods to improve model fairness among several bias types ( i.e. color, sex, age) based on the intuition that a suitable sketching method can filter out the bias information while keeping semantic information for classification.

Original: We employ the latest imagesketching method [4] with CLIP’s [20] language encoder guidance to ensure that the images retaincategory and semantic information.
Revised: We employ the latest image sketching method [4] with CLIP’s [20] language encoder guidance to ensure that the images retain category and semantic information.

Original: Moreover, we also introduce an additional fairness loss to furtherimprove model’s fairness.
Revised: Moreover, we also introduce an additional fairness loss to improve model’s fairness further.

Original: Wed have been made to evaluate our method, which shows that imagesketching is a simple but effective way for fairness improvement.
Revised: We have evaluated our method, showing that image sketching is a simple but effective way to improve fairness.

Original: We note that our method may cause accuracy drop, especially when we add the fairness loss L fairas an additional supervision.
Revised: We note that our method may cause accuracy drop, especially when we add the fairness loss L fair as additional supervision.

Original: This may be because that bluntly adding fairness makes the modelmore difficult to converge.
Revised: This may be because that bluntly adding fairness makes the model more difficult to converge.

Original: In our further study, we’re going to improve optimization strategies andcombine this improvementfor fairness with some sketch adjustment methods ( i e. style transfer),which may tackle the accuracy drop issue and even effectively improve the model fairness among allbias settings. 
Revised: In the future study, we’re going to improve optimization strategies and combine this improvement with some sketch adjustment methods ( i.e. style transfer), which may tackle the accuracy drop issue and even effectively improve the model fairness among all bias settings. Acknowledgments

Revised: This work is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) DGECR-2022-00430, the University of British Columbia’s Health Innovation Funding

Revised: Investment (HIFI) Awards, JST Moonshot R&D Grant Number JPMJMS2011, and JST ACT-X Grant

Revised: Number JPMJAX190D, Japan.

Original: A Fairness evaluation metrics
Revised: A Fairness evaluation metrics

Original: Let x i , y i , z i as the original input images, label, and bias sensitive attribute for every image i in thedataset.
Revised: Let x i , y i , z i as the original input images, label, and bias sensitive attribute for every image i in the dataset.

Original: S ( x i ) can be represented as sketch image and M ( S ( x i )) is the predicted label ˆ y i .
Revised: S ( x i ) can be represented as sketch image and M ( S ( x i )) is the predicted label ˆ y i .

Original: The truepositive rate ( TPR ) and false positive rate ( FPR ) are:
Revised: The true positive rate ( TPR ) and false positive rate ( FPR ) are:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Based on [14, 27], Statistical Parity Difference (SPD) , Equal Opportunity Difference (EOD) , Difference in Equalized Odds (DEO) , and Average Odds Difference (AOD) are applied to measure andevaluate the fairness.
Revised: Based on [14, 27], Statistical Parity Difference (SPD) , Equal Opportunity Difference (EOD) , Difference in Equalized Odds (DEO) , and Average Odds Difference (AOD) are applied to measure and evaluate the fairness.

Original: The smaller the value of these indicators, the higher the fairness of the model.
Revised: The smaller the value of these indicators, the higher the fairness of the model.

Original: • Statistical Parity Difference (SPD) measures the difference of probability in positive pre- dicted label ( ˆ y = 1 ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups.
Revised: • Statistical Parity Difference (SPD) measures the difference of probability in positive predicted label ( ˆ y = 1 ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups.

Original: [Equation]
Revised: [Equation]

Original: • Equal Opportunity Difference (EOD) measures the difference of probability in positivepredicted label ( ˆ y = 1 ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groupsgiven positive target labels ( y = 1 ).
Revised: • Equal Opportunity Difference (EOD) measures the difference of probability in positive predicted label ( ˆ y = 1 ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups given positive target labels ( y = 1 ).

Original: It can also be calculated as the difference in true positive rate between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups
Revised: It can also be calculated as the difference in true positive rate between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups.

Original: [Equation]
Revised: [Equation]

Original: • Difference in Equalized Odds (DEO) measures the difference of probability in positivepredicted label ( ˆ y = 1 ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groupsgiven target labels ( y ∈ { 0 , 1 } ).
Revised: • Difference in Equalized Odds (DEO) measures the difference of probability in positive predicted label ( ˆ y = 1 ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups given target labels ( y ∈ { 0 , 1 } ).

Original: [Equation]
Revised: [Equation]

Revised: • Average Odds Difference (AOD) measures the average of difference in true positive rate ( TPR ) and false negative rate ( FPR ) between protected ( z = 1 ) and unprotected ( z = 0 ) attribute groups.

Original: [Equation]
Revised: [Equation]