A N E THICAL M ODULE FOR B IAS M ITIGATION OF PRE - TRAINED F ACE R ECOGNITION M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
In spite of the high performance and reliability of deep learning algorithms in broad range everyday applications, many investigations tend to show that a lot of models exhibit biases, discriminating some subgroups of the population.
This urges the practitioner to develop fair systems whose performances are uniform among individuals.
In this work, we introduce a post-processing method designed to mitigate bias of state-of-the-art models.
It consists in learning a shallow neural network, called the Ethical Module , which transforms the deep embeddings of a pre-trained model to give more representation power to the discriminated subgroups.
Its training is supervised by the von Mises-Fisher loss, whose hyperparameters allow to control the space allocated to each subgroup in the latent space.
Besides being very simple, the resulting methodology is more stable and faster than most current methods of bias mitigation.
In order to illustrate our idea in a concrete use case, we focus here on gender bias in facial recognition and conduct extensive numerical experiments on standard datasets.
I NTRODUCTION
In the past few years, Face Recognition (FR) systems have reached extremely high levels of performance, paving the way to a broader range of applications, where the reliability levels were previously prohibitive to consider automation.
This is mainly due to the adoption of deep learning techniques in computer vision since the famous breakthrough of Krizhevsky et al.
The increasing use of deep FR systems has however raised concerns as any technological ﬂaw could have strong societal impact.
Besides recent punctual events 1 that received signiﬁcant media coverage, the academic community has studied bias of FR systems since many years (dating back at least to Phillips et al.
(2003) who investigated racial bias of non-deep FR algorithms).
Abdurrahim et al.
(2018) identify three sources of biases: race (understood as biological attributes such as skin color), age Srinivas et al.
and gender Albiero et al.
The National Institute of Standards and Technology
Grother et al.
(2019) conducted a thorough analysis of the performances of several FR algorithms in function of these attributes and revealed high disparities.
For instance, some of the top stateof-the-art algorithms in absolute performances have more than seven times false acceptances for females than for males.
In this paper, we introduce a methodology to mitigate gender bias for FR.
Though focusing on a single source of bias has obvious limitations regarding intersectional effects
Buolamwini & Gebru (2018), it is a ﬁrst step to gain insights into the mechanisms at work, before turning to more complex situations.
Actually, the method promoted in this paper, much more general than the application considered here, could possibly alleviate many other types of bias.
This will be the subject of a future work.
The topic corresponding to the study of different types of bias and to the elaboration of methods to alleviate them is referred to as fairness in machine learning, which has received increasing attention in recent years Mehrabi et al.
(2019), Caton & Haas (2020), Du et al.
Roughly speaking, achieving fairness means learning an algorithm that does not mistreat some predeﬁned subgroups, while still exhibiting a good predictive performance on the overall population: in general, a trade-off has to be found between fair treatment and pure accuracy 2 .
In this regard, one needs to carefully deﬁne what will be the relevant fairness metric .
From a theoretical viewpoint, several ones have been introduced Castelnovo et al.
, Garg et al.
(2020) depending on how the concept of equity is understood.
In practice, these very reﬁned notions can be inadequate, as they ignore speciﬁc use case issues, and one thus needs to adapt them carefully.
This is particularly the case in FR, where high security standards cannot be negotiated.
One of the contributions of our work is to introduce two new metrics that incorporate the needs for both security and fairness (see section 2.2).
Once the metric has been chosen, different strategies can be considered to alleviate bias of algorithms which can be roughly grouped in three categories: pre-, in- and post-processsing methods Caton & Haas (2020), depending on whether the practitioner “fairness” intervention occurs before, during or after the training phase.
In this work, we introduce a novel post-processing method allowing to correct gender bias of FR pre-trained models.
It is based on the idea that state-of-the-art models Wang et al.
, Deng et al.
(2019a),
Huang et al.
(2020) offer already very good deep embeddings of face images, as witnessed by their high performances on large scaled evaluation datasets ( e.g. IJB-C Maze et al.
(2018)), although they could induce various bias.
This indicates that the deep face embeddings obtained by these models are quite relevant and can be considered as initialization points to be enhanced.
As a result, our methodology consists in learning a shallow Multi-Layers Perceptron (MLP) in order to transform the deep embeddings of the pre-trained model and balance the representation space used by the discriminated groups.
The training of this shallow network is supervised by the von Mises-Fisher loss which is particularly well-suited for our purpose.
Indeed, it incorporates an hyperparameter for females (resp. males), whose variation is directly linked with the area they cover in the latent space.
To emphasize on both the post-processing aspect and the sought fairness, we call our methodology the Ethical Module .
[Figure]
• Simplicity and interpretability.
Though it is of great simplicity, the method exhibits good performance.
In addition, the von Mises-Fisher loss has a nice parametric interpretation which eases the understanding of the debiasing mechanisms at work.
• Computational complexity and stability.
The training of the shallow MLP is fast (few hours) and avoids the re-training of a full large-scale model, which can be costly and timeconsuming.
Moreover, the training process is stable and supports the reliability of the method.
This is in contrast with adversarial methods whose training losses tend to oscillate.
Finally, our method is ﬂexible and we conjecture it could use small database to already debias representation since the shallow MLP has only few parameters.
• Taking advantage of foundation models.
In the recent survey Bommasani et al.
(2021), the authors judiciously point out a change of paradigm in deep learning: very efﬁcient pre-trained models with billions of parameters they call foundation models are at our disposal such as BERT Devlin et al.
(2018) in NLP or ArcFace Deng et al. (2019a) in FR.
Many works rely on these powerful models and ﬁne tune them, inheriting from both their strengths and weaknesses such as their biases.
Hence the need to focus on methods to investigate the fairness of foundation models: our method is in line with this approach.
• Compliance with the fairness through unawareness principle.
If the Ethical Module requires access to the sensitive label during its training phase, the latter only appears in the von Mises-Fisher Loss.
As a result, face images are the only argument of the learned embedding function.
Organization of the paper.
Section 2.1 presents the widely spread usage of FR and its main challenges.
It is followed by section 2.2 where we discuss different fairness metrics that arise in FR and introduce two new ones we think are more relevant with regards to operational use cases.
In section 3, we present the von Mises-Fisher loss that is used for the training of the Ethical Module and discuss its beneﬁts.
Finally, in section 4, we present at length our numerical experiments, which consist in learning an Ethical Module on the ArcFace model, pre-trained on the MS1MV3 dataset (Deng et al.
(2019b)).
Our results show that, remarkably, some speciﬁc choices of hyperparameters provide high performance and low fairness metrics both at the same time.
Related works.
The correction of bias in FR has been the subject of several recent papers.
Liu et al.
and Wang & Deng (2020) use reinforcement learning to learn fair decision rules but despite their mathematical relevance, such methods are computationally prohibitive.
Another line of research followed by Yin et al.
(2019), Wang et al. (2019a) and Huang et al.
(2019) assume that bias come from the unbalanced nature of FR datasets and builds on imbalanced and transfer learning methods.
Unfortunately, these methods do dot completely remove bias and it has been recently pointed out that balanced dataset are actually not enough to mitigate bias Wang et al. (2019b).
Gong et al.
(2019), Alasadi et al.
(2019) and Dhar et al.
(2020; 2021) rely on adversarial methods that can reduce bias but are also known to be unstable and computationally expensive.
All of the previously mentioned methods try to learn fair representations.
In contrasts, some other works do not affect the latent space but modify the decision rule instead:
Terh ¨ orst et al.
(2020) act on the score function whereas Salvador et al.
(2021) rely on calibration methods.
Despite encouraging results, these approaches do not solve the source of the problem which is the bias incurred by the embeddings used.
O N F AIRNESS IN THE C ONTEXT OF D EEP F ACE R ECOGNITION
In this section, we ﬁrst brieﬂy recall the main principles of deep face recognition and introduce some notations.
The interested reader may consult Masi et al.
(2018) or Wang & Deng (2018) for a detailed exposition.
Then, we present the fairness metrics we adopt and argue of their relevance in our framework.
2. A Q UICK O VERVIEW OF F ACE R ECOGNITION
A typical FR dataset consists of N images ( x i ) 1 ≤ i ≤ N of faces that have been pre-processed and are all of size ( h, w, c ) .
It is assumed that there are K identities among the images and we denote by y i ∈ { 1 , . . .
K } the identity of x i for i = 1 , . . .
, N .
The goal of a FR algorithm is to learn a proper d -dimensional representation of the face images, by means of a function f : R h × w × c → R d , in order to minimize the intra-identity distances and maximize the inter-identity distances.
We denote by z i = f
( x i ) the face embedding of x i .
Since the advent of deep learning, the function f is usually a deep Convolutional Neural Network (CNN) whose parameters are learned on a large FR dataset.
Test phase.
There are generally two FR use cases: identiﬁcation , which consists in ﬁnding the speciﬁc identity of a probe face among several previously enrolled identities, and veriﬁcation (which we focus on throughout this article), which aims at deciding whether two face images correspond to the same identity or not.
To do so, the closeness between two embeddings is usually quantiﬁed with the cosine similarity measure s ( z i , z j ) :
= z (cid:124)
i
z j / ( || z i ||·|| z
j || ) , where ||·|| stands for the usual
Euclidean norm (the Euclidean metric ||
z i − z
j || is also used in some early works e.g. Schroff et al.
Therefore, an operating point t ∈ [ − 1 , 1] has to be chosen to classify a pair ( z i , z j ) as genuine (same identity) if s ≥ t and impostor (distinct identities) otherwise.
Training.
For the training phase only, a fully-connected layer is added on top of the deep embeddings so that the output is a K -dimensional vector, predicting the identity of each image within the training set.
The full model (CNN + fully-connected layer) is trained as an identity classiﬁcation task.
Until 2018, most of the popular FR loss functions were of the form:
[Equation]
where the µ k ’s are the fully-connected layer’s parameters, κ > 0 is the inverse temperature of the softmax function used in brackets and n is the batch size.
Early works (Taigman et al.
(2014); Sun et al.
(2014)) took κ = 1 and used a bias term in the fully-connected layer but Wang et al.
(2017) showed that the bias term degrades the performance of the model.
It was thus quickly discarded in later work.
Since the canonical similarity measure at the test stage is the cosine similarity, the decision rule only depends on the angle between two embeddings, whereas it could depend on the norms of µ k and z i during training.
This has led Wang et al.
(2017) and Hasnat et al.
(2017) to add a normalization step during training and take µ k , z
i ∈ S
d − 1 := { z ∈ R d : ||
z ||
= 1 } as well as introducing the re-scaling parameter κ in Eq.
these ideas signiﬁcantly improved upon former models and are now widely adopted.
Denoting by θ i the angle between µ y i and z i , the major advance over the loss of Eq.
(with normalization of µ k , z i ) in recent years was to consider largemargin losses which replace µ (cid:124)
y i z i
=
cos( θ i ) by a function that reduces intra-class angle variation, such as the cos( mθ i ) of Liu et al.
(2017) or the cos( θ i ) − m of Wang et al.
The most efﬁcient choice is cos( θ i + m ) and is due to Deng et al. (2019a) who called their model ArcFace, on which we build our methodology.
A ﬁne training should result in the alignment of each embedding z
i with the vector µ y i .
The aim is to bring together embeddings with the same identity.
Indeed, during the test phase, the learned algorithm will have to decide whether two face images are related to the same, potentially unseen, individual (one refers to an open set framework).
Evaluation metrics.
Denoting by G the set of genuine pairs and by I the set of impostor pairs in a given test set, we introduce the False Acceptance and False Rejection Rates, deﬁned as follows:
[Equation]
These quantities are crucial to evaluate a given algorithm in our context: face recognition is intrinsically linked to biometric applications, where the usual accuracy evaluation metric is not sufﬁcient to assess the quality of a learned decision rule.
For instance, security automation in an airport requires a very low FAR while keeping a reasonable FRR to ensure a pleasant user experience.
As a result, the most widely used metric consists in ﬁrst ﬁxing a threshold t so that the FAR is equal to a pre-deﬁned value α ∈ [0 , 1] , and then computing the FRR at this threshold.
We use the canonical
FR notation to denote the resulting quantity:
[Equation]
The FAR level α determines the operational point of the FR system and corresponds to the security risk one is ready to take.
According to the use case, it is typically set to 10 − i with i ∈ { 1 , . . .
2. F AIRNESS M ETRICS
While the FRR@FAR metric is the standard choice for measuring the performance of a FR algorithm, it does not take into account its variability among different subgroups of the population.
In order to assess and correct for potential discriminatory biases, the practitioner must rely on suitable fairness metrics.
The ﬁrst step is to deﬁne a ﬁnite set A of sensitive attributes against which we wish to assess fairness.
Since we focus here on gender bias, we take A = { 0 , 1 } where 0 stands for “male” and 1 for “female”.
We extend the notions of FAR and FRR within the male (resp. female) subgroups, considering same-gender pairs only, and denote by FAR 0 , FAR 1 and FRR 0 , FRR 1 the resulting quantities.
Before specifying our choice for the fairness metric used here, let us review some existing ones
Cavazos et al.
(2020) that derive from fairness in the context of binary classiﬁcation (here, one classiﬁes pairs in two groups: genuines or impostors).
The Demographic Parity criterion requires the prediction to be independent of the sensitive attribute, which amounts to equalizing the likelihood of being genuine conditional to a = 0 and a = 1 .
Besides heavily depending on the number and quality of impostors and genuines pairs among subgroups, this criterion does not take into account the FAR s and FRR s, which are instrumental in FR as previously mentioned.
An attempt to incorporate those criteria could be to compare the intra-group performances: FRR 0 @(FAR 0 = α ) v.s. FRR 1 @(FAR 1 = α ) .
However, the operational points t 0 and t 1 satisfying FAR 0 (t 0 ) =
α and
FAR 1 (t 1 )
=
α generically differ as pointed out by Krishnapriya et al.
To fairly asses the equity of an algorithm, one needs to compare intra-groups FAR s and FRR s at a same threshold.
Two such criteria exist in the fairness literature: the Equal Opportunity fairness criterion which requires
FRR 0 ( t ) =
FRR 1 ( t ) and the Equalized Odds criterion which additionally requires FAR 0 ( t ) =
FAR 1 ( t ) .
Nevertheless, working at an arbitrary threshold does not really make sense since, as previously mentioned, FR systems typically choose an operational point achieving a predeﬁned FAR level so as to limit security breaches.
This is why most current papers consider a ﬁxed operational point t such that the population False Acceptance Rate equals a ﬁxed value α .
For instance, Dhar et al.
(2020) computes
[Equation]
However, we think the choice of a threshold achieving a global FAR is not entirely relevant for it depends on the relative proportions of females and males of the considered dataset together with the relative proportion of intra-groups impostors and genuines.
For instance, at ﬁxed images quality, if females represent a small proportion of the dataset, the threshold t of Eq. 3 is close to the male threshold t 0 satisfying FAR 0 ( t 0 )
=
α and away from the female threshold t 1 satisfying FAR 1 ( t 1 )
=
α .
Such a variability among datasets could lead to incorrect conclusions.
In this paper, we go one step further and work at a threshold achieving max a FAR a =
α instead of
FAR = α .
This alleviates the previously proportions dependence.
Besides, this allows to monitor the risk one is willing to take among each subgroup: for a pre-deﬁnite rate α deemed acceptable, one typically would like to compare the performance among subgroups for a threshold where each subgroup satisﬁes FAR a ≤ α .
Our two resulting metrics are thus:
[Equation]
such that
[Equation]
and
[Equation]
One can read the above acronyms “Bias in FRR / FAR ”.
In addition to being more security demanding than previous metrics, BFRR and BFAR are more amenable to interpretation: the ratios of
FRR s or FAR s corresponds to the number of times the algorithm makes more mistakes on the discriminated subgroup.
Those metrics generalizes well for more than 2 distinct values of the sensitive attribute.
T HE V ON M ISES -F ISHER L OSS FOR B IAS M ITIGATION
We now turn to a detailed description of the von Mises-Fisher (vMF in abbreviated form) loss which supervises the training of our Ethical Module’s MLP.
It stems from the vMF probability measure on the hypersphere and was ﬁrst introduced in Hasnat et al.
(2017) as a new powerful way to discriminate different identities, before the advent of large-margin losses.
The von Mises-Fisher distribution.
The vMF distribution in dimension d with mean direction µ ∈ S
d − 1 and concentration parameter κ > 0 is a probability measure deﬁned on the hypersphere S d −by
the following density:
[Equation]
with C d ( κ ) = κ d 2 −(
[Equation]
where I ν stands for the modiﬁed Bessel function of the ﬁrst kind at order ν , whose logarithm can be computed with high precision (Kim (2021)).
Figure 2 illustrates the inﬂuence of the concentration parameter κ on the vMF distribution.
The vMF distribution corresponds to a Gaussian distribution in dimension d with mean µ and variance-covariance matrix (1 /κ )
I d , conditioned to live on the hypersphere.
It makes it a very natural directional law : for instance, it has maximal entropy among all probability measures on the hypersphere with ﬁxed variance.
A Fair von Mises-Fisher loss.
The general form of the vMF Loss on the face embeddings is
[Equation]
where a scalar κ k is affected to each identity k ∈ { 1 , . .
, K } .
Notice that there are two ways of minimizing L vMF : either by aligning ground-truth µ y i with associated normalized face embeddings
1    =   1    =   1    =   1    = x i or by pushing back wrong µ k (with k (cid:54) =
y i ) from x i .
The vMF Loss was ﬁrst introduced in the context of FR by Hasnat et al.
(2017), who took a unique value κ .
In that case, the vMF loss reduces to the classical loss L of Eq.
1 (when z i and µ i are normalized).
This makes L vMF a natural generalization of cosine similarity-based losses which are at the basis of reﬁned state-of-the-art models that also incorporate margins.
Building on L vMF , we consider gender-based hyperparameters.
If a k ∈ { 0 , 1 } denotes the gender of identity k ∈ { 1 , . .
, K } , the fair version of the vMF-loss we consider to train the Ethical Module is:
[Equation]
It depends on two hyper-parameters κ 0 , κ 1 > 0 , respectively corresponding to the concentration parameters of male and female.
By tuning κ 0 and κ 1 , one can modify the surfaces covered by the male / female and therefore plays on their representative powers.
A maximum likelihood interpretation of the von Mises-Fisher loss.
It is common knowledge that, besides its high performances in many contexts, the Cross-Entropy (CE) loss is also attractive because of its interpretation in terms of maximum likelihood estimation.
Indeed, minimizing the CE amounts to maximizing the likelihood with respects to the network parameters.
[Equation]
Interestingly, the vMF loss also enjoys this property if one places a proper model on the face embeddings.
Let us describe this model, which is a mixture of vMF distributions.
Recall that K ≥ 1 is the number of identities in the dataset.
The model consists in assuming that the dataset consists of i.i.d.
realizations of a probability law with density:
[Equation]
Let us discuss its parameters Θ vMF :=
{ ( π i , µ i , κ i ) } 1 ≤ i ≤ K .
Each identity k ∈ { 1 , . . .
, K } is represented by a vMF distribution with mean µ k and concentration κ k .
In this vision, the mean µ k is an abstraction of the identity and κ k takes into account its variability (caused by lightening conditions, aging...).
We also denote by π k the proportion of identity k in the overall population.
Figure 3 illustrates this vMF mixture in the case K
= 4 (the detail of the parameters is given in the supplementary material).
Under the vMF mixture model assumption, the probability p ij that a face embedding z
i belongs to identity k is given by
[Equation]
Therefore, the negative log-likelihood of the model is given by
[Equation]
At least at a heuristical level, both playing on the parameters π k s and κ k s affects the surfaces covered by the different classes.
It is thus natural to restrict the scope to the case where the π k s are all equal and only adjust the concentration parameters.
In that case, the above NLL is in fact the vMF Loss
L vMF previously introduced.
N UMERICAL E XPERIMENTS
Pre-trained model.
We use the trained model ArcFace 3 whose CNN architecture is a ResNet (Han et al.
As emphasized before, it achieves state-of-the-art performances in FR.
It has been trained on the MS1M-RetinaFace dataset (also called MS1MV3), introduced by Deng et al.
(2019b) in the ICCV 2019 Lightweight Face Recognition Challenge.
MS1MV3 is a cleaned version of the MS-Celeb1M dataset (Guo et al.
; all its face images have been pre-processed by the Retina-Face detector of Deng et al. (2019c) and are of size 112 × 112 pixels.
It contains 5 .
1 M images of 93 k identities.
We also consider other pre-trained models 4 (AdaCos Zhang et al.
(2019), CosFace Wang et al.
CurricularFace Huang et al.
whose backbone is a MobileFaceNet Chen et al.
and trained on the MS-Celeb-1M-v1c-r dataset 5 .
This dataset is another cleaned version of the MS-Celeb1M dataset and it contains 3 . 28 M images of 73 k identities. The images are also pre-processed by the Retina-Face detector and are of size 112 × 112 pixels. Gender labels.
For a fair comparison, we train our Ethical Module on the training set used to train the pre-trained models (MS1MV3 for ArcFace, MS-Celeb-1M-v1c-r for the models with MobileFaceNet backbone).
However, ground-truth gender labels for MS1MV3/MS-Celeb-1M-v1c-r are not available.
As the training of our Ethical Module needs the gender label of each face image within the training set, we use a private gender classiﬁer to get those gender labels.
Current gender classiﬁers achieve around 95 % prediction accuracy on standard evaluation datasets and are widely used in FR to get gender annotations (Acien et al.
; Gong et al.
Since some images from the same identity might be assigned different gender predictions, it is common practice to use a majority vote to decide the correct gender for each identity.
We follow Albiero et al.
and only keep in our training sets the identities for which at least 75 % of the same-identity face images are assigned the same gender.
Doing so, we discard 25 k images and 835 identities for MS1MV3, 10 k images and 500 identities for MS-Celeb-1M-v1c-r.
Ethical Module.
The face embeddings output by the pre-trained models are of dimension 512 .
Thus, the MLP within our Ethical Module has an input layer of 512 units.
To emphasize the fact that our gender bias mitigation solution is much less costly than current solutions such as Wang & Deng (2020) and Dhar et al.
(2020) , in terms of both training time and computation power (see supplementary material A.1), we choose a shallow MLP of size ( 512 , 1024 , 512 ), the output dimension being the same than for Arcface.
This MLP is trained with the fair version L FvMF of the vMF loss introduced in Eq.
For each experiment, we train the Ethical Module during 50 epoch with the Adam optimizer (Kingma & Ba (2014)).
The batch size is set to 1024 and the learning rate to 0 .
The training is efﬁcient as we ﬁrst compute the face embeddings of the pre-trained models (on MS1MV3 for ArcFace, on MS-Celeb-1M-v1c-r for the models with MobileFaceNet backbone), store them, and then train a shallow MLP on those embeddings. Using one single GPU (NVIDIA RTX 3090), the computation of the embeddings takes 4 hours and each training takes 8 hours.
Reproducibility.
We plan to release the code used to conduct our experiments.
4. G RID -S EARCH ON IJB-C AND FAIRNESS EVALUATION
In order to select relevant pairs of gender-hyperparameters ( κ 0 , κ 1 ) , we perform a grid-search and keep track of the canonical performance metric FRR@(FAR = 10 − 3 ) together with our two fairness
See metrics BFRR(10 − 3 ) and BFAR(10 − 3 ) introduced in Eq. 4 and 5.
To obtain reliable results, we need to compute the latter metrics on a sufﬁciently large dataset containing gender labels.
We choose IJB-C (Maze et al.
(2018)), which contains about 3,5k identities for a total number of about 31 k images and 117 k unconstrained video frames.
The 1:1 veriﬁcation protocol 6 is performed on 19 k genuine pairs and 15 M impostor pairs.
[Figure]
A heatmap representation of the grid-search is provided in the supplementary material A.2.
Several interesting trends emerge from the ablation study of Figure 4, suggesting an underlying regularity of the model with respect to the hyperparameters space.
More precisely:
• when κ 0 is ﬁxed and κ 1 increases, BFAR tends to decrease, BFRR ﬁrst increases and then decreases and FRR@FAR tends to increase, • when κ 1 is ﬁxed and κ 0 increases, BFAR ﬁrst increases and then decreases, BFRR tends to decrease and and FRR@FAR increases.
A strength of our approach is that it lends itself well to a geometric interpretation. The latter allows to derive two heuristic views on the problem, whose details are provided in the supplementary material A.3.
In a nutshell, we are able to (i) identify a pair of hyperparameters with a priori good fairness properties, without relying on a grid-search study, and (ii) provide explanations for the trends of Figure 4.
A very promising route for future work would be to investigate a two-step strategy for ﬁnding the best hyperparameters by selecting an initial point ( κ 0 , κ 1 ) with (i) and then navigating the hyperparameter space with (ii) .
Many ( κ 0 , κ 1 ) pairs could be considered as relevant and instead of deﬁning an objective criterion, we select three of them in order to illustrate the trade-off one needs to make between fairness metrics and pure performance.
The point ( κ 0 = 25 , κ 1 = 20) is quite relevant regarding BFRR and performance ( FRR@FAR ), but is not adapted for BFAR .
On the other hand, the point (15 , 20) is good when it comes to BFAR and performance.
Finally, the point (45 , 30) is interesting when considering BFRR and BFAR .
In Table 1, we summarize the different metrics evaluated for these three points on the LFW dataset (Huang et al. (2008), see A.4) and compare them with the ArcFace pre-trained baseline, at two FAR levels.
We also conduct these experiments on different pre-trained models with MobileFaceNet backbone. Additional results (using the IJB-C dataset for evaluation, considering several types of ResNet backbones for ArcFace) can be found in the supplementary material A.5.
[Table]
4. A CCURACY AND VERIFICATION EVALUATION
[Table]
We ﬁnally investigate the FRR@FAR metric (Table 2) of the three selected points ( κ 0 , κ 1 ) on IJB-B (Whitelam et al.
In the veriﬁcation setting, this dataset contains 10 k genuine pairs and 8 M impostor pairs.
Notice that we do not lose too much in performance with respect to the original model.
C ONCLUSION
In this paper, we introduce a novel method, the Ethical Module , to mitigate gender bias of Face
Recognition state-of-the-art models.
It consists in learning a shallow MLP on top of a frozen pretrained model, so as to correct the biases that could exist in the embedding space.
To achieve fairness, we rely on a fair version of the von Mises-Fisher loss that incorporates an hyperparameter per gender, the variation of which allowing to monitor the space covered by males and females in the latent space.
Measuring the fairness of Face Recognition systems is a very challenging task and we introduce two new metrics that both respond to the need for security and equity.
Besides being very simple, the resulting methodology is more stable and faster than most current methods of bias mitigation.
It both leverages the strong accuracy of pre-trained models while correcting their bias.
We illustrate the soundness of our methodology on several pre-trained models, and strongly believe it could also be used to alleviate other types of bias.
Our work opens several lines of research: for instance, it would be interesting to extend our ideas to the context of multiclass sensitive attributes and of continuous sensitive attributes such as age.
Another idea would be to somehow incorporate our fairness criteria during the training of the Ethical Module.
Finally, we think that incorporating large-margin constraints into the loss used to train the Ethical Module would be a promising attempt to go beyond the trade-off between fairness and performance.