I MPROVING F EDERATED L EARNING F ACE R ECOGNITION VIA P RIVACY -A GNOSTIC C LUSTERS
Anonymous authors
Paper under double-blind review
A BSTRACT
The growing public concerns on data privacy in face recognition can be greatly addressed by the federated learning (FL) paradigm.
However, conventional FL methods perform poorly due to the uniqueness of the task: broadcasting class centers among clients is crucial for recognition performances but leads to privacy leakage.
To resolve the privacy-utility paradox, this work proposes PrivacyFace, a framework largely improves the federated learning face recognition via communicating auxiliary and privacy-agnostic information among clients.
PrivacyFace mainly consists of two components: First, a practical Differentially Private Local
Clustering (DPLC) mechanism is proposed to distill sanitized clusters from local class centers.
Second, a consensus-aware recognition loss subsequently encourages global consensuses among clients, which ergo results in more discriminative features.
The proposed framework is mathematically proved to be differentially private, introducing a lightweight overhead as well as yielding prominent performance boosts ( e.g. , +9.63% and +10.26% for TAR@FAR=1e-4 on IJB-B and
IJB-C respectively).
Extensive experiments and ablation studies on a large-scale dataset have demonstrated the efﬁcacy and practicability of our method.
I NTRODUCTION
Face recognition technique offers great beneﬁts when used in right context, such as public safety, personal security and convenience.
However, misuse of this technique is a concern as it involves unique and irrevocable biometric data.
The rapid commercial applications based on face recognition and facial analysis techniques have stimulated a global conversation on AI ethics, and have resulted in various actors from different countries issuing governance initiatives and guidelines.
EU’s General Data Protection Regulation (GDPR) (Voigt & Von dem Bussche, 2017), California Consumer
Privacy Act (CCP) and Illinois Personal Information Protection Act (IPI) enforces data protection “by design and by default” in the development of any new framework.
On the nationwide “ show” of year 2021, the China Central Television (CCTV) called out several well-known brands for illegal face collection without explicit user consent.
As researchers, it is also our duty to prevent the leakage of sensitive information contained in public datasets widely used by the research community.
Therefore, faces in ImageNet (Deng et al., 2009) were recently all obfuscated (Yang et al., 2021) and a large face dataset called MS-Celeb-1M (Guo et al., 2016) was pulled of the Internet.
In the wake of growing social consensus on data privacy, the ﬁeld of face recognition calls for a fundamental redesign about model training while preserving privacy.
A potential solution is the paradigm called Federated Learning (FL) (McMahan et al., 2017a).
Given C clients with local datasets {D 1 , D 2 , · · · , D C } as shown in Fig.
1a, FL decentralizes the training process by combining local models ﬁne-tuned on each client’s private data and thus hinders privacy breaches.
Typical examples of these clients include personal devices containing photo collections of a few family members, or open-world scenarios such as tourist attractions visited by tens of thousands of people.
In most circumstances, we can safely assume very few classes would co-exist in two or more clients.
Despite the numerous FL-based applications in various domains (Kairouz et al., 2019) ranging from health to NLP, there are very little progress (Aggarwal et al., 2021; Bai et al., 2021) in training face recognition models with FL schemes.
Unlike other tasks, parameters of the last classiﬁer for a face recognition model are crucial for recognition performance but strongly associated with privacy.
These parameters can be regarded as mean embeddings of identities (Wang et al., 2018;
Meng et al.,
[Figure] [Figure] [Figure]
(a) (b) /",'.(13server 3… /",'.(31/",'.(1server 3/",'.(
[Figure]
2021a
; Shen et al., 2020) (also called as class centers), from where individual privacy could be spied out as studied by plenty of works (Kumar Jindal et al., 2018; Boddeti, 2018; Mai et al., 2020;
Dusmanu et al., 2021).
That prevents the FL approach from broadcasting the whole model among clients and the central server, and consequently leads to conﬂicts in the aggregation of local updates in the overlapped regions.
As depicted in the global feature distribution of Fig.
1a, both clients try to spread out their own classes in the same area (pointed by the arrow) of the normalized feature space.
Thus, the training loss could oscillate to achieve consensus given the sub-optimal solutions from multiple clients.
On the other hand, a large batch with sufﬁcient negative classes is necessary to learn a discriminative embedding space for advanced face recognition algorithms.
During the conventional FL updates, each class is only aware of local negative classes while those from other clients are untouchable.
This further limits performances of FL approaches in face recognition.
The privacy-utility paradox motivates us to introduce PrivacyFace, a framework improves federated learning face recognition by broadcasting sanitized information of local class globally.
In the framework, a novel algorithm called Differentially Private Local Clustering (DPLC)
ﬁrst generates privacy-agnostic clusters of class centers while any speciﬁc individual in the cluster cannot be learned, irrespective of attacker’s prior knowledge, information source and other holds.
Recall that the privacy cost of a differential privacy scheme is propotional to the l 2 -sensitivity while inversely propotional to the query number.
Our DPLC reaches a low l 2 -sensitivity by restricting the cluster size as well as covering sufﬁcient class centers.
In addition, the number of necessary centers to communicate in DPLC is irrelevant to the number of classes in the training data.
These characteristics jointly equip DPLC with much smaller privacy cost than the naive alternative, which sanitizes each class center individually by Gaussian noise.
In our experiments, DPLC’s privacy cost is only 1.7eof that of the naive approach.
That persuasively reveals the high security level of our approach.
The second part of PrivacyFace is the consensus-aware face recognition loss.
Following principles of Federated Averaging (FedAvg) (McMahan et al., 2017a), a server iteratively gathers feature extractors and privacy-agnostic clusters from clients, averages parameters of feature extractors and distributes them to clients.
Accordingly, the consesus-aware loss notiﬁes each client not to embed samples in the inappropriate zone (differential private clusters marked by DP) of the feature space during the local optimization, as shown in Fig.
1b.
This process aids each client to train more discriminative features as well as align all consensuses.
Compared to the conventional approach, our
PrivacyFace boosts performances by +9.63% and +10.26% for TAR@FAR=1e-4 on IJB-B and IJBC respectively with only single-digit privacy cost.
Moreover, the additional computational cost as well as communication cost are negligible ( e.g. , the extra clusters to broadcast only occupy 16K storage while the backbone already takes 212M).
In a word, PrivacyFace is an efﬁcient algorithm which improves conventional federated learning face recognition by a large margin on performances, while requires little privacy cost as well as involves lightweight computational/communication overheads.
P RELIMINARIES 2. D EEP F ACE R ECOGNITION
Most of early works in deep face recognition rely on metric-learning based loss, including contrastive loss (Chopra et al., 2005), triplet loss (Schroff et al., 2015) and N-pair loss (Sohn, 2016).
These methods are usually inefﬁcient in training on large-scale datasets.
One possible reason is that their embedding spaces at each iteration are constructed only by a positive sample and limited negative ones.
Therefore, the main body of research (Ranjan et al., 2017; Liu et al., 2017; Wang et al., 2018; Deng et al., 2019; Meng et al., 2021b) has focused on devising more effective classiﬁcationbased loss and achieved leading performances on a number of benchmarks.
Suppose that we are given a training dataset D with N face samples { x i , y i } Ni =1 of n identities, where each x i is a face image and y i ∈ { 1 , · · · , n } denotes its associated class label.
Considering a feature extractor φ generating the embedding f i = φ ( x i ) and a classiﬁer layer with weights W = [ w 1 , · · · , w n ] , a series of face recognition losses can be summarized as
[Equation]
The speciﬁc choices of the similarity functions u ( w , f ) , v ( w , f ) yield different variants, e.g.:
CosFace (Wang et al., 2018) : u ( w , f ) = s · (cos θ − m ) , v ( w , f ) = s · cos θ, (2)
ArcFace (Deng et al., 2019) : u ( w , f ) = s · cos( θ −
[Equation]
(3) where s, m are hyper-parameters and θ = arccos( w T f / (cid:107) w T f (cid:107) ) is the angle between w and f .
2. D IFFERENTIAL P RIVACY
Differential Privacy (DP) (Dwork et al., 2006) is a well-established framework under which very little about any speciﬁc individual can be learned in the process irrespective of the attacker’s prior knowledge, information source and other holds (Feldman et al., 2017; Nissim et al., 2016; Stemmer & Kaplan, 2018).
We state the relevant deﬁnitions and theories (Dwork & Roth, 2014) below.
Deﬁnition 1 (Differential Privacy) .
A randomized algorithm M : X → Y is ( (cid:15), δ ) -DP
if for every pair of neighboring datasets X, X (cid:48)
∈ X (i.e., X and X (cid:48) differ in one row), and every possible output T ∈ Y the following equality holds:
Pr [ M ( X ) ∈ T ] ≤ e (cid:15)
Pr [ M ( X (cid:48) )
∈ T ] + δ .
Here (cid:15), δ ≥ 0 are privacy loss parameters, which we consider a privacy guarantee meaningful if δ = o ( 1 n ) , where n is the size of dataset.
Deﬁnition 2 ( l 2 -sensitivity) .
The l 2 -sensitivity of a function f : X → R d is
[Equation]
Deﬁnition 3 (Gaussian Mechanism) .
Suppose a function f : X → R d and I d is the ddimensional identity matrix.
The mechanism M ( X ) = f
( X ) + N (0 , σ 2 I d ) is ( (cid:15), δ ) -DP
if
[Equation]
(cid:113) 2 ln( 1 . 25 δ )
Deﬁnition 4 (Composition of Differentially Private Algorithms (Dwork & Roth, 2014; Dwork &
Lei, 2009)) .
Suppose M =
( M 1 , M 2 , · · · , M k ) is a sequence of algorithms, where M i is ( (cid:15) i , δ i ) - differentially private, and the M i ’s are potentially chosen sequentially and adaptively.
Then M is ( (cid:80) ki
=1 (cid:15), (cid:80) ki =1 δ k ) -differentially private.
P RIVACY F ACE
This section details the PrivacyFace framework as illustrated in Fig.
At its core is a novel clustering algorithm that extracts non-sensitive yet informative knowledge about the local class distribution (Fig. 2a).
After the central server broadcasting local DP-guaranteed outputs (Fig. 2b), each client optimizes over a consensus-aware objective that takes into account both the local data and the privacy-agnostic clusters to learn a discriminative embedding space (Fig. 2c) for face recognition.
xxx 𝑝 𝜌 ̂𝑝 𝑣 ∼ 𝒩(0,Ι⋅𝜎 B ) … … local classes clusters from other clients ̂𝑝 ̂𝑝 ̂𝑝 p local class centers local class centers 𝑝 B 𝑝(a)
Local clustering.
xxx 𝑝 𝜌 ̂𝑝 𝑣 ∼ 𝒩(0,Ι⋅𝜎 B ) … … local classes clusters from other clients ̂𝑝 ̂𝑝 ̂𝑝 p local class centers local class centers 𝑝 B 𝑝(b) Gaussian mechanism.
xxx 𝑝 𝜌 ̂𝑝 𝑣 ∼ 𝒩(0,Ι⋅𝜎 B ) … … local classes clusters from other clients ̂𝑝 ̂𝑝 ̂𝑝 p
[Figure]
class centers local class centers 𝑝 B 𝑝
[Figure]
3. D IFFERENTIALLY P RIVATE L OCAL C LUSTERING
Our target is to improve performances of federated learning face recognition by communicating auxiliary and privacy-agnostic information among clients.
To distill useful information from each client and resolve the privacy-utility paradox, we propose a specialized algorithm called Differential
Private Local Clustering (DPLC) with rigorous theoretical guarantees.
Problem Design .
To facilitate the discussion, let us ﬁrst introduce the margin parameter ρ ∈ [0 , π ] which deﬁnes the boundary of a cluster centered at p by saying that any w in the cluster satisﬁesarccos( w T p ) ≤ ρ .
We design the local clustering problem with two principles.
First, the clustering results are expected to carry information only about population rather than individual from local dataset.
Supported by Theorem 2 proved later, this principle ensures PrivacyFace to gain insight about the underlying distribution, while preserving the privacy of individual record.
Second, the spaces conﬁned by the clusters should be tight because they represent the inappropriate zones to escape from in other clients’ perspectives.
A cluster with large margin ρ would occupy a huge proportion of the sphere, leaving limited room for other embeddings ( e.g., half of sphere would be occupied if ρ = π/ 2 ).
More speciﬁcally, we quantify the occupancy ratio by the following theorem:
Theorem 1.
Assume there is a unit and d-dimensional sphere S d and an embedding point p ∈ S d .
Embeddings with angles less than ρ to p (i.e., { f : arccos( f T p ) ≤ ρ } ) occupy 12 I sin 2 ( ρ ) (cid:0)
d − 12 , 12 (cid:1) of the surface area of S d .
Here I x
( a, b ) is the regularized incomplete beta function.
[Figure]
Proved in Sec.
A.1.3 of the appendix, Theorem 1 indicates that the occupancy ratio increases monotonically with respect to the margin ρ given a feature dimension d .
Fig.
3 plots a series of functions between cluster margin ρ and occupancy ratio under different dimensions.
Take the rightmost curve corresponding to a typical setting d = 512 in face recognition as an example.
The occupancy ratios are 0 .
055 , 5 · 10 − 5 , 4 · 10 −when ρ = 1 . 5 , 1 . 4 , 1 . 3 respectively.
In another word, the whole feature space would be fully occupied if we sample 20 , 000 clusters over the original face classes when ρ = 1 . 4 .
It is obvious that the more space left for optimization, the higher likelihood to achieve better performance.
Therefore, we seek for a margin-constrained clustering of local data with ﬁxed ρ ≤ 1 . 4 when d = 512 for better privacy-utility trade-offs.
A counter example to the latter principle is the classical k -means algorithm, which have been extended to differentially private versions in a number of works (Nissim et al., 2016; Feldman et al., 2017; Stemmer & Kaplan, 2018).
However, k -means naturally has no control about the scopes of the generated clusters, not to mention the difﬁcult hyper-parameter k to pick.
That motivates us to propose a speciﬁed clustering algorithm for our task.
An Approximated Solution .
Given the margin ρ , the problem of ﬁnding clusters to cover all class centers is a generalization of the well-studied 2D disk partial covering problem which is unfortunately NP-complete (Xiao et al., 2004).
We take a greedy approach by iteratively ﬁnding a cluster centered at p ∗ with margin ρ to cover the most class centers w i and weeding out those covered before the next round.
We cast the problem of identifying such a cluster at each iteration as:
Deﬁnition 5 (Spherical Cap Majority Covering Problem) .
Denote 1 ( · ) as an indicator function.
Assuming a unit d-dimensional sphere S d and class centers { w 1 , w 2 , · · · , w n } ∈ S d , the target is to ﬁnd a cluster center p ∗ ∈ S d where p ∗ = arg max
p ∈ S
d (cid:80) ni
=1 1 (arccos( w Ti p ) ≤ ρ ) .
Finding the optimal p ∗ is also NP-complete.
To address this, we ﬁrst sort out the densest area S in terms of the neighbor count for each w .
An efﬁcient approximation of p ∗ is then the average of class centers in the area, p = 1 |S| (cid:80) i ∈S w i .
Lines 3-5 of Algorithm 1 summarize the detailed procedure, by which the approximate cluster centers p i can be quickly enumerated as depicted in Fig.
2a.
In line 8, the class centers would be further normalized in accordance with the DP mechanism.
Algorithm
Data : Class center embeddings W = { w 1 , w 2 , · · · , w n } encoded in the classiﬁer layer.
Parameters : Margin ρ ; Minimum cluster size T ; Maximum #queries Q ; Privacy budget (cid:15), δ .
Result : Privacy-agnostic cluster centers ˆ p ’s.
Let I = { 1 , 2 , · · · , n } and calculate θ i,j = arccos( w Ti w j ) for i, j ∈ { 1 , 2 , · · · , n } ;for q = 1 , 2 , · · · Q do
For each { w i , i ∈ I} , ﬁnd the indexes of its neighbors by S i =
{ j : θ i,j ≤ ρ, j ∈ I} ;Find a set with the most elements, i.e. , S = arg max |S i | ;p ← 1 |S| (cid:80) i ∈S w i ;
approximate solution for problem
if |S| ≥ T then
Sample a v from distribution N (0 , σ 2 I d ) where σ = 2 |S| (cid:15) (cid:113) (1 − cos(2 ρ ))
ln( 1 . 25 δ )
;ˆ p q ← p + v (cid:107) p + v (cid:107) , I ← I / { arccos(
w Ti p (cid:107) p (cid:107) 2 ) ≤ ρ, i ∈ I} ;elseBreak out of the loop ;
Differential Privacy Endorsement .
Although the centers p reveal the population-level property of the local training set, they are still outcomes of a deterministic algorithm, thereby vulnerable to adversary attack during the FL updating.
We prove below that p can be perturbed to achieve DP:
Theorem 2.
Deﬁne a function p (cid:44) f
( S )
= 1 |S| (cid:80) i ∈S w i .
Then the Gaussian Mechanism ˆ p (cid:44)
M ( S ) =
f ( S ) + N (0 , σ 2 I d ) is ( (cid:15), δ ) -DP
if σ ≥ 2 |S|· (cid:15) (cid:113) (1 − cos(2 ρ ))
ln( 1 . 25 δ ) .
Proof.
Let S be the set of indexes of class centers which have cosine similarities larger than cos ρ with respect to a center w o .
Assuming that S , S (cid:48) are neighbors differed at w and w (cid:48) , where vectors are normalized, i.e., (cid:107) w (cid:107)
= (cid:107) w (cid:48)
(cid:107) =
(cid:107) w o (cid:107) = 1 , then we have w To w ≥ cos ρ and w To w (cid:48) ≥ cos ρ .
Lemma 2 in appendix states that
[Equation]
) for all { w , w (cid:48) , w o } .
Therefore, the lower bound of w T w (cid:48) is
[Equation]
Following this inequality, the l 2 -sensitivity of f ( S ) is 1 |S| (cid:112) 2 − 2 cos(2 ρ ) as (cid:107) f ( S )
− f ( S (cid:48) )
(cid:107) 2 = 1 |S| (cid:107) w − w (cid:48)
(cid:107) 2 = 1 |S| √ 2 − 2 w T w (cid:48) ≤ 1 |S| (cid:112) 2 − 2 cos(2 ρ ) .
By Deﬁnition 3, we easily conclude that
M ( S ) =
1 |S| (cid:80) i ∈S w i + N (0 , σ 2 I d ) is ( (cid:15), δ ) -DP if σ ≥ 2 |S|·
[Table]
Additional proofs can be found in Sec.
A.1.2 of appendix.
Theorem 2 outlines the setting of noise variance σ given the privacy budget (cid:15), δ and the cluster margin ρ .
Basically, the lower bound of σ is proportional to 1 |S| .
That implies when the size of cluster |S| is large enough, adding small noise is sufﬁcient to achieve promising privacy level.
We found setting minimum cluster size to
T (cid:44) min |S| = 512 yields empirically stable performance.
It is worth emphasizing that the number of original classes n does not directly inﬂuence the introduced noise σ .
This advantage ensures that
DPLC maintains a high utility when dealing with large-scale face recognition problems.
Full algorithm .
Algorithm 1 presents the full DPLC and we highlight several key properties below.
Properties 1, 2 state that DPLC is efﬁcient and differentially private in theory.
Besides the efﬁcency and privacy, another major concern of the DP algorithm is the utility.
Speciﬁcally, as ˆ p is a combination of the source cluster center p and Gaussian noise v , we should prevent the noise v to have overwhelming effects to the output, which is further veriﬁed by our Property 3 and Property 4.
Property 1.
The complexity of the DPLC algorithm is O ( Q · n 2 ) .
Proof.
The computational complexity of calculating θ i,j is O ( n 2 ) .
In each loop, we also compare the value θ i,j , ∀ i, j ∈ S with ρ , whose worst time complexity is O ( n 2 ) .
Because remaining steps are of linear complexity, the total complexity is O ( n 2 + Q · n 2 )
=
O ( Q · n 2 ) .
Property 2.
DPLC is a ( Q · (cid:15), Q · δ ) -differentially private algorithm.
Proof.
The mechanism f ( S ) + N (0 , σ 2 I d ) is ( (cid:15), δ ) -differentially private according to Lemma 2.
As DPLC queries at most Q results, it’s easy to conclude that our DPLC algorithm is ( Q · (cid:15), Q · δ ) - differentially private based on Deﬁnition 4.
Property 3.
Denote Φ as the cumulative distribution function of a standard Gaussian distribution.
For vector magnitudes of p , v , we have (cid:107) p (cid:107) 2 ∈ (cos ρ, 1] and P ( (cid:107) v (cid:107)
[Equation]
r ) (cid:39)
[Equation]
Proof.
For p , the upper bound is 1 as
[Equation]
p
[Equation]
2 =
[Equation]
1 |S|
[Equation]
i ∈S w i
[Equation]
2 ≤ 1 |S|
[Equation]
i ∈S
[Equation]
w i
[Equation]
2 = 1 .
As S denotes the set of indexes of class centers which has cosine similarities larger than cos ρ with a center w o , the lower bound of (cid:107) p (cid:107) 2 can be calculated by
[Equation]
Therefore, we have (cid:107) p (cid:107) 2 ∈ (cos ρ, 1] .
For v ∼ N (0 , σ 2 I d ) , let v =
[ v 1 , v 2 , · · · , v d ] , we have v i σ following the standard normal distribution for all i .
Thus, the sum of square of v i σ follows a χ 2 -distribution with d − 1 degrees of freedom, i.e. , 1 σ 2 (cid:107) v (cid:107)
= (cid:80) di
=1 (cid:0) v i σ (cid:1)
∼ χ 2 d − 1 .
The mean and variance of (cid:107) v (cid:107) 22 are E ( (cid:107) v (cid:107) 22 ) =
σ 2 ( d − 1) and
V ar ( (cid:107) v (cid:107) 22 ) =
2 σ 4 ( d − 1) , based on properties of the χ 2 -distribution.
By the central limit theorem, (cid:107) v (cid:107) 22 converges to N ( σ 2 ( d − 1) , 2 σ 4 ( d − 1)) when d is large enough.
In practice, for d ≥ 50 , (cid:107) v (cid:107)is sufﬁciently close to N ( σ 2 ( d − 1) , 2 σ 4 ( d − 1))
(Box et al., 1978).
Therefore, we have
[Equation]
Note that in our work, σ can be a small value with little privacy cost.
Thus, with high probability, the norm of v is close to zero.
Property 4.
If (cid:107) p (cid:107) 2 ≥ (cid:107) v (cid:107) 2 , the cosine similarity between p and ˆ p is always greater than (cid:112) 1 − (cid:107) v (cid:107) 22 / | p (cid:107) 22 .
Proof.
Denote magnitudes of p , v as l p , l v and cosine similarity between p and v as x ( i.e. , x = p · v (cid:107) p (cid:107)(cid:107) v (cid:107) ∈ [ − 1 , 1] ).
Let a =
l r l p < 1 and s be the cosine similarity between p + v and p , then s =
( p +
[Equation]
) T p (cid:107) p +
[Equation]
(cid:107) 2 (cid:107) p (cid:107) 2 = l 2 p + l p l v x l p √ l 2 p + l 2 v +2 l p l v x = 1+ ax √ 1+ a 2 +2 ax .
The ﬁrst order derivative of s with respect to x is dsdx = a 2 ( a + x ) (1+ a 2 +2 ax ) 32 .
It’s easy to conclude that s takes the smallest value when x = − a , which can be reached as a is always smaller than 1.
In the end, we have s ≥ (cid:113) 1 − l 2 v /l 2 p .
In DPLC, we reasonably require large local clusters S at line 6 while preserving privacy in terms of (cid:15), δ .
That constraint ensures the synthesized noise v to have a small magnitude (property 3), which thereby results in slight offsets of the sanitized vector ˆ p to the source p (property 4).
With these theoretical guarantees, DPLC can obtain good end-to-end utility with low privacy cost.
O PTIMIZATION WITH G LOBAL C ONSENSUS
PrivacyFace employs a Federated Learning (FL) paradigm for optimization.
Suppose there are C clients, each of which c = 1 , · · · , C owns a training dataset D c with N c images from n c identities.
For client c , its backbone is parameterized by φ c and the last classiﬁer layer encodes class centers in W c =
[ w c 1 , · · · , w cn c ] .
Without loss of generality, we assume that each class center w c is normalized.
Central to PrivacyFace is the DPLC algorithm, which generates for each client, Q c clusters deﬁned by the centers P c
= { ˆ p c 1 , · · · , ˆ p cQ c } with margin ρ .
As presented in Algorithm 2, the training alternates the following updates in a sufﬁcient number of rounds M .
Client-side .
In line 3 of Algorithm 2, the server broadcasts current feature extractor φ t − 1 to each client.
Then each client updates its local cluster centers P c that can be safely shared to others in line 5.
After receiving P = {P 1 , · · · , P C } , each client executes one-round optimization over the consensus-aware face recognition loss on local dataset D c in line 7:
[Equation]
(5) where f ci is the feature extracted by φ c on i -th instance in client c .
µ (ˆ p , f , ρ ) = s · cos(max( θ − ρ, 0)) computes the similarity between f and the cluster centered at ˆ p with margin ρ .
As illustrated in Fig.
2c, PrivacyFace aims to learn a globally consistent embedding that can not only classify local classes, but also achieve consensuses with other clients on the incompatible clusters ˆ p .
Server-side .
Similar to other FL approaches, a central server orchestrates the training process and receives the contributions of all clients to the new feature extractor at line 8.
The well-known FedAvg (McMahan et al., 2017a) is then utilized to compute an average of all local models.
Compared to the conventional FL method as described in Fig.
1a, PrivacyFace empirically achieves better convergence thanks to the consensus-aware loss that implicitly takes the data distribution of other clients into account.
Built on the client-wise DPLC algorithm, the framework is potentially compatible with other optimizer, e.g., FedSGD (Shokri & Shmatikov, 2015) as shown in Sec.
A.2.1 of appendix.
Algorithm 2 : The PrivacyFace Training Scheme. Data
: D c s exclusively owned by each of the C
clients;
A pre -trained extractor φ 0 . Parameters : DPLC-related ρ, T, Q ; Privacy budget (cid:15), δ ; Maximum #communications M .
Result :
A general recognition model φ M .
for t = 1 , · · · , M do for each client c = 1 , · · · , C do
Synchronize
the local extractor φ ct − 1 =
φ t − 1 as the up-to-date one in server;
Generate privacy-agnostic P c = { ˆ p c 1 , ˆ p c 2 , · · · , ˆ p cq c } via Algorithm 1 ;The server gathers P c and distributes P = {P 1 , P 2 , · · · , P C } to all clients;for each client c = 1 , · · · , C doUpdate local model { φ ct − 1 , W ct − 1 } to { φ ct , W ct } by optimizing the loss (Eq. 5) ;Communicate the feature extractor φ ct to the server while keeping W ct locally;The server updates the model by φ t = 1 C (cid:80)(cid:80)(cid:80) Cc =1 φ ct ;
E XPERIMENTS
This section together with the appendix describes extensive experiments on challenging benchmarks to illustrate the superiority of PrivacyFace in training face recognition with privacy guarantee.
Datasets.
CASIA-WebFace (Yi et al., 2014) contains 0.5M images from 10K celebrities and serves as the dataset for pre-training.
BUPT-Balancedface (Wang & Deng, 2020), which comprises four
[Figure]
sub-datasets categorized by racial labels (including African, Asian, Caucasian and Indian) and each sub-dataset contains 7K classes and 0.3M images, is used to simulate the federated setting.
We adopt RFW (Wang et al., 2019), IJB-B (Whitelam et al., 2017) and IJB-C (Maze et al., 2018) for evaluation.
RFW is proposed to study racial bias and shares the same racial taxonomy as BUPT-Balancedface.
IJB-B and IJB-C are challenging ones, containing 1.8K and 3.5K subjects respectively from largevolume in-the-wild images/videos.
All images are aligned to 112 × 112 based on ﬁve landmarks.
Training.
We assign one client for each of the four sub-datasets and use the perfect federated setting ( i.e. , no client goes ofﬂine during training).
For the pre-trained model φ 0 , we adopt an opensource one 2 trained on CASIA-WebFace, which builds on ResNet18 and extracts 512 -d features.
We ﬁnetune φ 0 by SGD for M = 10 communication rounds on BUPT-Balancedface, with learning rate 0.001, batch size 512 and weight decay 5 e - 4 .
For reproducibility and fair comparison, all models are trained on 8 1080Ti GPUs with a ﬁxed seed.
To alleviate the large domain gaps across subdatasets, we build a lightweight public dataset with the ﬁrst 100 classes from CASIA-WebFace to ﬁnetune local models before gathered by the server.
Unless stated otherwise, the parameters for
PrivacyFace are default to T = 512 , Q = 1 , ρ = 1 . 3 and (cid:15)
= 1 .
As the parameter δ is required to be O ( 1 | D | ) (Dwork & Roth, 2014), we set δ = 1 | D |
1 ≈
e - in BUPT Balancedface
Baselines.
In the absence of related methods, we compare PrivacyFace with or without Gaussian noise added, and against a conventional FL method as indicated in Fig.
1a.
We denote these methods as φ + ˆ p , φ + p and φ respectively, based on communicated elements among the server and clients.
Besides, we implement centralized training on the global version of BUPT-Balancedface and denote the trained model as “global training”.
The model is also ﬁnetuned from φ 0 for 10 epochs with learning rate of 0.001 and serves as an upper bound in all experiments.
Privacy-Utility Trade-Offs of the DPLC Algorithm.
The goal of DPLC is to broadcast discriminative yet privacy-preserving knowledge through a federated network among clients.
This part investigates its effectiveness in terms of similarities between the noise-free center p and the perturbed one ˆ p by the Gaussian Mechanism with respect to different parameters.
We ﬁrst derive the class centers W from the Caucasian sub-dataset of BUPT-Balancedface by φ 0 .
Taking W as input,
DPLC generates 1000 ˆ p (cid:48) s for each p , yielding the distribution of cosine similarities shown in Fig.
As a common sense, two face features holding cosine similarity over 0 .
7 are recognized as from the same identity with a high probability.
Fig.
4a reveals that ρ > 1 . 2 can always lead to accurate ˆ p with (cid:15)
= 1 and Q = 1 .
If { ρ, Q } = { 1 . 3 , 1 } , we conclude that ˆ p , p are similar with privacy cost (cid:15) over 0 .
3 , as indicated by Fig.
4b.
Although Q = 1 leads to good recognition performance, Fig.
4c studies the effect when more queries are required ( Q = 3 ) at different cost (cid:15) .
At ρ = 1 . 3 , we can generate three clusters with descending sizes (1773, 743, 581) w.r.t the query index.
By choosing a strict privacy cost (cid:15) = 0 .
33 , queries 2 and 3 are too noisy to carry useful information.
Setting (cid:15)
=to a reasonable privacy level, however, all queries will convey accurate descriptions of features.
Ablation Studies on Hyper-parameters.
We conduct several ablation studies on PrivacyFace and present recognition performances on IJB-C in Fig.
Models are trained by ArcFace with default parameters if not speciﬁcally stated.
In Fig.
5a, poor recognition performance and unstable training process can be observed for conventional FL method φ mainly due to the insufﬁcient number of local classes and inconsistent stationary points achieved by different clients.
These drawbacks can be signiﬁcantly relieved by PrivacyFace φ + ˆ p .
Fig.
5b reveals that performances of PrivacyFace improves as privacy cost (cid:15) increases.
The performance is nearly saturated when (cid:15) > 0 . 3 , closing to the method φ + p without privacy protection.
This end-to-end evidence indicates DPLC achieves high privacy-utility trade-offs as analyzed in Fig.
4b.
The effect of cluster margin ρ is explored in
Fig.
5c and the optimal ρ is around 1.3.
A small ρ leads to over-ﬁned clusters with inadequate class centers, adding requirements to increase noise and hurt recognition.
Alternatively, the performance would drop by adopting a large ρ which generates trivial clusters with high occupancy ratio.
[Figure]
Performances on Benchmarks.
Tab.
1 presents veriﬁcation performances on various benchmarks.
By ﬁnetuning on BUPT-Balancedface, φ , φ + ˆ p and φ + p all achieve performance boosts.
Compared to the conventional FL method ( φ ), performances of our PrivacyFace ( φ + ˆ p ) are consistently higher regardless of which training loss used.
Speciﬁcally, improvements on TAR@FAR=1e-4 on IJB-B and IJB-C are 3.54% and 3.56% with CosFace, and 9.63% and 10.26% with ArcFace, which are signiﬁcant and demonstrate the superiority of our method.
We also observe that φ + ˆ p and φ + p achieve very close results in all benchmarks.
That implies the robustness of the proposed method.
Additional experiments can be found in appendix.
We implement FedSGD (Shokri & Shmatikov, 2015) in Sec.
A.2.1 to show the scalability of the PrivacyFace.
Sec.
A.2.2 compares the DPLC with a naive approach while Sec.
A.2.3 discusses the necessity of FL methods in federated setting.
In
Sec.
A.2.4, we analyze experimental attacks to further verify privacy guarantees of our framework
[Table] [Table]
Cost Analysis.
PrivacyFace introduces little computational cost thanks to the efﬁcient DPLC algorithm as well as the consensus-aware loss.
Apart from the backbone (over 200M) to distribute as in the conventional FL, the extra variables to communicate are ˆ p ’s, which only occupy about 16K storage.
Thus, additional communication cost is negligible.
Moreover, the total privacy cost is still of a low level with number M(cid:15)
=
10 ( i.e. , communication rounds times the cost for each round).
C ONCLUSIONS
With the carefully designed DPLC algorithm and a novel consensus-aware recognition loss, we improve federated learning performances on face recognition by communicating auxiliary embedding centers among clients, while achieving rigorous differential privacy.
The framework runs efﬁciently with lightweight communication/computational overheads.
Besides, PrivacyFace can be potentially extended to other metric-learning tasks such as re-identiﬁcation and image retrieval.
In the future, more efforts can be spent on designing a more accurate clustering algorithm in conjunction with the
FL optimization, e.g., adaptive querying on the confusion areas instead of a brute-force sampling.
R EPRODUCIBILITY S TATEMENT
Sec.
3.1 provides key proofs as well as properties of the proposed method.
Additional proofs are described in Sec.
A.1 of the appendix.
The involved training/test datasets and training conﬁgurations are detailed in Sec.
4 when using
FedAvg scheme.
Similar settings are applied for FedSGD in Sec.
A.2.1 in appendix.
Sec.
A.2.4 of the appendix shows visualizations for two potential attacks: K-nearest neighbor attack and inversion attack.
We as well present involved datasets, network structures, training losses as well as training schedules for these attacks.
Those are sufﬁcient for reproducibility.
A A PPENDIX A. M ATHEMATICAL P ROOFS
In this section, we present extra proofs for PrivacyFace.
A.1. P ROOF FOR T HEOREM
Proof.
A spherical cap is a portion of a d-dimensional sphere with radius r cut off by a plane.
As shown in (Li, 2011), the surface area of a spherical cap is
[Equation]
Here h is the height of the cap, and A, A d are the surface area of the spherical cap and the sphere, respectively.
Because the set { f : arccos( f T p ) ≤ ρ, f ∈ S d } is the spherical cap with height h
= r · (1 − cos ρ ) and the radius is 1 in our problem, it can be derived that the occupancy ratio is
[Equation]
A.1. A WEAK VERSION OF T HEOREM
In this part, we present a straightfward but weak version of the Theorem 2:
Theorem 3.
Deﬁne a function p (cid:44) f
( S )
= 1 |S| (cid:80) i ∈S w i .
Then the Gaussian Mechanism ˆ p (cid:44)
M ( S ) =
1 |S| (cid:80)
[Equation]
Proof.
S stores indexes of class centers which have cosine similarities larger than cos ρ with a center (denoted as w o in this proof).
Assuming that S , S (cid:48) are neighbors differed at w, w (cid:48) , then
[Equation]
According to Deﬁnition 3, we conclude that M ( S ) = 1 |S| (cid:80) i ∈S w i + N (0 , σ 2 I d ) is ( (cid:15), δ ) -DP
if σ
[Equation]
This lower bound of σ is 4 |S|·
[Equation]
(cid:113) (1 − cos ρ ) ln( 1 . 25 δ ) , which is larger than our ﬁnal bound because
[Equation]
A tighter bound can reduce the privacy cost with same level of noise added.
For example, the privacy cost in our experiments can be dropped by around 21% if using ρ = 1 . 3 (from (cid:15)
= 1 to (cid:15) = 0 .
A.1. P ROOF FOR T HEOREM
Lemma 1.
For any x , y , z ∈ R d , the following inequality holds:
[Equation]
Proof.
Let x = ( x 1 , x 2 , · · · , x d ) , y =
( y 1 , y 2 , · · · , y d ) ,
z = ( z 1 , z 2 , · · · , z d ) .
Then
[Equation]
i
[Equation]
x
[Equation]
· d (cid:88)
j = z 2 j − ( d (cid:88) i
= x i z i )=
d (cid:88) i =
d (cid:88)
j = x 2 i z 2 j − d (cid:88) i
= x i z i · d (cid:88)
j = x j z j = d (cid:88) i =
d (cid:88)
j = x 2 i z 2 j − d (cid:88) i =
d (cid:88)
j = x i z i x j z j = d (cid:88) i =
d (cid:88)
j = ( x 2 i z 2 j − x i z i x j z j ) = 1d (cid:88)
i =
d (cid:88)
j = ( x 2 i z 2 j + x 2 j z 2 i − 2 x i z i x j z j ) =
[Equation]
d (cid:88) i =
d (cid:88)
j = ( x i z
j − x j z i )
Similarly, it’s easy to prove y T yz T z − y T
zy T z = 12 (cid:80) di =
[Equation]
dj =
[Equation]
y i
z j − y j z i ) 2 .
In addition, we
[Equation]
Then we can deduce that
[Equation]
Here the second step is the Cauchy–Schwarz inequality.
Lemma 2.
Assume that w , w (cid:48) , w o ∈ R d and
[Equation]
π 2 .
Then
[Equation]
holds
[Equation]
Proof.
Let
[Equation]
= cos γ .
We thereby need to prove cos γ ≥ cos( α + β ) = cos α cos β − sin α sin β = cos
[Equation]
cos
[Equation] [Equation]
α )(1 − cos 2 β ) , which is equivalent to
[Equation]
If cos α cos β ≤ cos γ , the inequality equation 7 holds.
If cos α cos β > cos γ , then
[Equation]
⇐⇒
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
The last step is correct according to Lemma 1.
A. E XTRA E XPERIMENTS A.2. P RIVACY F ACE WITH F ED SGD
Algorithm 2 is mainly built on FedAvg (McMahan et al., 2017a) where model parameters are communicated between clients and the server.
In this part, we show that our PrivacyFace is also compatible with another popular FL method called FedSGD (Shokri & Shmatikov, 2015).
FedSGD mainly differs FedAvg in two aspects: (1) During training locally, the model is frozen to ensure gradients across mini-batches are from a same model.
(2) Gradients rather than model parameters are aggregated and distributed among clients and the server.
Apart from changing FedAvg to FedSGD, we let other training setting consistent with those described before and present the results in Tab. 2.
Compared to the conventional FL method φ , performances of our PrivacyFace φ + ˆ p are close to those on RFW and noteworthily higher in IJB-B/IJB-C benchmarks.
Besides, φ + ˆ p and φ + p achieve very close results.
All these phenomenons are in accord with those observed when using FedAvg, which shows the scalability of the proposed PrivacyFace.
However, FedSGD does not achieve that signiﬁcant improvements as FedAvg.
A possible reason is that FedSGD naturally requires more communication rounds than FedAvg to achieve comparable results, which may be caused by low-frequent updates during training.
Experiments in McMahan et al.
(2017a) showed that FedSGD requires about 23 times more communication rounds to achieve similar performances with FedAvg.
With such a fact, the improvements using FedSGD in PrivacyFace is still remarkable.
[Table]
A.2. C OMPARISON WITH A N AIVE DP A PPROACH
A naive alternative to DPLC is to directly make each class center of W to be differential private by adding noise.
However, that can lead to a large privacy cost due to two reasons: 1.
With the
same noise
added
, the
privacy
cost is
propotional
to the
l
-sensitivity based
on deﬁnition 2.
The l 2 -sensitivity of the naive approach is 2 as maxinum distances between two class centers distributed in a unit sphere are 2.
In contrast, the l 2 -sensitivity DPLC is|S| (cid:112) 2 − 2 cos(2 ρ ) .
In our experiment, we let |S| > 512 and ρ = 1 . 4 , which makes the l 2 -sensitivity to be smaller than 0.0024.
The training dataset are of a large number of identities.
According to the composition rule (Deﬁnition 3), the privacy cost grows linearly with respect to the queries ( i.e. , number of identities).
In DPLC, low privacy cost as the clustering mechanism highly reduce the number of queries.
During training, the number of query is 1.
Therefore, the privacy cost in an local client (withclasses) of DPLC is 1 · 0 .
024 7000 · 2 ≈ 1 .
7 e − 7 of that of the naive approach.
We experimentally compare the performances of our method and the naive approach.
For fair comparisons, the privacy cost in each round is ﬁxed to be 1 and all training settings are consistent with those in the main text.
Tab. 3 are the results, where the φ + ˆ W represents the naive approach.
It can be seen that φ + ˆ W can achieve similar results on RFW benchmark as learning local discriminative features with introduced ﬁxed class centers are relatively easy (for example, if introduced class center distributed in the south of the embedding space, placing local features to the north is a solution).
However, as the class centers from other clients are very noisy, the global embedding space will become a mess.
φ + ˆ W even achieves much worse results than the conventional FL methods φ on the challenging benchmarks.
The tar decreases 23.99% on IJB-B and 25.16% on IJB-C when far is 1e-4.
In contrast, PrivacyFace increases the performances by 9.63% and 10.26% respectively, which shows the superiority of our framework.
[Table]
A.2. N ECESSITY OF F EDERATED L EARNING
Besides using federated learning
[Table]
one can also choose to ﬁnetune client-expert models on local datasets.
We also implement this approach by training four independent models on sub-datasets of
BUPT-Balancedface and test their performances on benchmarks.
The training procedure is consistent with what we described before in Sec. 4.
Speciﬁcally
[Table]
we ﬁnetune each model from φ 0 by
ArcFace for 10 epochs with learning rate 0.001, batch size 512 and weight decay 5e-4.
We present our results in Tab. 4, where the “African/Asian/Caucasian/Indian” in the ﬁrst column means the model trained on the corresponding sub-dataset of BUPT-Balancedface.
Models ﬁnetuned on African and Caucasian sub-dataset achieve satisfying results on local benchmarks while those on
Asian and Indian have even worse performances than the initial model φ 0 .
Moreover, all these local models perform poorly on the IJB-B and IJB-C benchmarks, which indicates the lack of generality.
To conclude, local training highly relies on the properties and qualities of local datasets
[Table]
and can not lead to general models.
In contrast, models trained by conventional FL method and PrivacyFace can achieve competitive performances on all benchmarks, and therefore are more robust to various scenarios.
[Table]
A.2. V ISUALIZATION
In this section, we experimentally examine the privacy attacks to demonstrate that class centers lead to privacy leakage while our sanitized clusters are resistent to these attacks.
We mainly consider two types of attacks: K-nearest neighbor attack and reversion attack.
K-Nearest Neighbor Attack We assume that an attacker owns a gallery of faces with enormous identities and attacks the exposed class centers by ﬁnding their neighbors.
To simulate the K-nearest neighbor attack, class centers are extracted on the BUPT-Balancedface dataset and three faces are sampled from each identity on a large dataset called MS1M-V2 (Deng et al., 2019).
Fig.
6 presents the results.
It can be observed that the K-nearest neighbor attack successfully ﬁnd faces of the corresponding identity from the class centers, which enables the attacker to know which identities are contained in the training dataset.
The observation further veriﬁes the fact that sharing class centers leads to privacy leakage in face recognition.
Reversion Attack We design a reconstruction network (modiﬁed from DCGAN (Radford et al., 2015)) as described in Tab. 5 which projects a 512 dimensional embedding to a face image.
CASIAWebFace dataset is used as the training dataset.
Speciﬁcally, we extract features from images by the initial model φ 0 .
With supervision of features and images, we train the network by Adam optimizer with L1 loss, which penalizes the absolute value of pixel differences between reconstructed and real images.
The learning rate is initialized from 0 .
001 and divided by 10 every 10 epochs, and we stop the training at the 50th epoch.
The weight decay is 5 e − 4 and batch size is 512 during our training.
[Table]
We test our model on BUPT-Balancedface dataset and present reconstruction visualizations.
In each image pair of Fig.
7, the left one is the average of faces from one identity while the right one is the reconstructed image from the corresponding class center.
It can be seen that reconstructed images reveal the real individual identity, which demonstrates that w carries human face privacy and therefore should be protected.
Fig.
8 shows the reconstructions from p and ˆ p under various privacy cost (cid:15) .
The left column is the images by p , which are much blurred than those from w and only the group properties can be identiﬁed.
For example, the ﬁrst image seems like an African male while 𝑝 ̂𝑝 𝜖 = 0.
𝜖 = 0.
𝜖 = 0. 𝜖 = 0.
𝜖
= 1.
𝜖
= 1.
𝜖
= 1.
the second last one are very likely to be Caucasian female.
The output ˆ p from our DPLC algorithm can further protect privacy as one can hardly recognize a face from the generated images, especially when (cid:15) is small.
The hardness of a successful identiﬁcation is negative related to (cid:15) according to the visualization.
A.2. C OMPARISONS WITH F ED F ACE
FedFace (Aggarwal et al., 2021) is a recent work applying federated learning to face recognition. In each communication round, the FedFace uploads all local class centers to the server and consequently utilizes the spreadout regularizer to separate class centers from different clients. In this section, we adjust FedFace to ﬁt our federated setting in the main text.
Speciﬁcally, in each communication round, each local client uploads its 7K class centers to the server.
The spreadout regularizer then computes distances of 588 million ( 7 K × 7 K × 3 × 4 ) pairs of class centers and back propagates the model parameters once.
The updated parameters are then distributed to each of the clients.
However, directly computing all distances consumes 11,962M GPU memory, which is out of the capacity of our machine.
Therefore, we sample 1K out of 7K class centers from each client for spreadout regularizer in our machine.
For hyper-parameters of the spreadout regularizer, the authors recommend loss weight λ
= 10 and margin v = 0 .
For fair comparisons, we further runexperiments with λ ∈ { 0 . 5 , 1 , 1 . 5 , 5 , 10 , 15 } and v ∈ { 0 . 3 , 0 . 6 , 0 . 9 } and report the best results of
FedFace.
[Table]
6 summarizes the overall comparisons between FedFace and PrivacyFace.
We observe that
• Performance .
FedFace only boosts the FedAvg by 0 .
82% , 2 . 27% on TAR@FAR=1e-4 for IJB-B and IJB-C.
In contrast, PrivacyFace achieves 9 .
63% , 10 . 26% , respectively.
Besides the sampling strategy, the main reason can be that FedFace only separates classes from different clients for one time (the spreadout regularizer) in a communication round, while the model is locally updated 0.3 million times (number of local images) without considering the conﬂicts among clients.
In contrast, our consensus-aware recogntion loss takes the conﬂicts into account for each local update.
• Privacy cost .
In each round, FedFace uploads 7 , 000 original class centers to the server.
Therefore, the privacy cost is 7000 × ∞ = ∞ .
For PrivacyFace, the number is only 1 from the outputted noisy cluster center in the client.
• Additional communication cost .
For each client in communication round, FedFace additionally uploads 7 , 000 512-d class centers to the server and sends back the updated class centers.
In PrivacyFace, a local client uploads one 512-d cluster centers to the server and downloads 4 cluster centers from the server.
The additional cost from FedFace is 7000+ 1+ = 2 , 800 times of that from PrivacyFace.
• Additional computational cost .
The additional computations in FedFace is from the spreadout regularizer, which calculates 588 million pairs of 512-d features and updates the model parameters on the server side by one-time back propagation.
That requires enormous computation in each round.
In contrast, only extra items in the consensus-aware loss requires additional computation in PrivacyFace, which is of tiny volume.
• Additional usage of GPU memory .
Even though we sample 1 , 000 out of 7 , 000 clients to reduce GPU usage of the spreadout regularizer, FedFace still consumes 1 , 525 M extra GPU memories.
One the other hand, the additional usage of PrivacyFace is negligible, as shown in Tab. 6.
A.2. A BLATION S TUDIES ON I NITIAL M ODELS
In this section, we examine the effects of different initial models on the proposed method.
Besides the φ 0 we used before, we further adopt two intermediate checkpoints at epoch 20, 10 when training on the CASIA-WebFace (Yi et al., 2014) and denote them as φ 1 , φ 2 , respectively.
One additional model φ 3 is trained on MS1M-V2 (Deng et al., 2019) for 10 epochs with learning rate 0.1 and the
ArcFace loss.
Note that we do not follow the conventional training conﬁguration where the learning rate is decreased stepwisely until 0.001.
That’s because such a conﬁguration will lead to a strong model that does not require ﬁnetuning on BUPT-BalancedFace (Wang & Deng, 2020).
With other training settings same as those in the main text, we ﬁnetune the models on BUPTBalancedFace (Wang & Deng, 2020) with the ArcFace loss.
Tab.
7 presents the comparisons of plain
FedAvg and PrivacyFace.
On the IJB-B/IJB-C benchmarks, PrivacyFace can consistently achieve better results than the plain FedAvg.
For TAR@FAR=1e-4 on IJB-C, PrivacyFace can boost the performances by 10 .
% with the four initial models.
It can be observed that improvements are affected by the initial model.
For example, the poorest initial model φ 2 achieves the lowest improvements.
The reason may be that the training loss are overwhelmed by separating local classes as initial features are not discriminative enough.
Consequently, increasing inter-client distances is of the secondary importance to the training in this scenario.
For the best initial model φ 3 , the performances on IJB-B/C decreases when using federated learning.
This is because the ﬁnetuning process also focuses on improving local performances as the global consensus is already reached.
In this senario, FedAvg decreases the TAR@FAR=1e-4 by more than 40% on the benchmarks.
In contrast, PrivacyFace can highly alleviate the performance degradations by taking global consensus into account during training.
Finetuing on BUPT-BalancedFace mainly beneﬁts the performances on the RFW benchmarks as these two datasets are both designed for racial bias.
If the initial model is good enough, the ﬁnetuning procedure will focus more on ﬁtting the racial issue instead of improving the general performances.
This may be the reason why model φ 3 leads to the largest improvements.
[Table]
A.2. E FFECTS OF C LIENT D ISCONNECTIONS
Previously we use a perfect federated setting where all clients are available during training.
In realworld applications, some clients may drop for certain rounds of communication due to network issues.
We further simulate the case by randomly select one of the four clients and exclude it from the training for an ofﬂine probability in each communication round.
For example, if we set the ofﬂine probability to 100%, there will be exactly one client ofﬂine per round.
Note here we only block one client because the total number of clients is small.
We keep other settings unchanged and compare performances of PrivacyFace and plain FedAvg on the IJB-C benchmark with different ofﬂine probability.
We refer the ofﬂine policy as disconnecting patterns during training, and use the same ofﬂine policy for each pair of experiments on PrivacyFace and FedAvg for fair purpose.
To reduce the randomness, we run each experiment with 3 different ofﬂine policies and report the mean
TAR@FAR=1e-4. Fig. 9 presents our results. When the ofﬂine probability is 25% and 50%, PrivacyFace improves the performances of federated learning by about 6%.
The improvement decreases to 2.4% with 75% ofﬂine probability. When each round witnesses one random client ofﬂine, the performances of two methods are below 45% while the improvement brought by our approach is only 0.4%. That reveals the performance boosts of PrivacyFace can be inﬂuenced by the network conditions. [Figure]
In PrivacyFace, we utilize the differential privacy to improve the performance of conventional federated learning in face recognition. Our work basically distills differentially private information from clients to achieve that purpose. We notice that there are plenty of recent works on differential private federated learning (DP-FL) methods (Geyer et al., 2017; McMahan et al., 2017b; Truex et al., 2019) and would like to emphasize the differences:
PrivacyFace and current DP-FL methods aim to solve different privacy leakages.
As different data points lead to different model updates, DP-FL methods essentially add noise to model update to prevent data from being inspected. Recall the model update is essentially the average of gradients, which has a small l 2 -sensitivity (deﬁnition 2) thanks to the tremendous number of data points. Then the protection is relatively easy with the low privacy cost (propotional to the l 2 -sensitivity).
In contrast, PrivacyFace aims to protects class centers in face recognition which are directly linked to individual privacy.
The problem is much harder as these class centers are not naturally averaged like model updates and therefore leads to large privacy cost.
The targets to protect is different.
Current DP-FL methods protect the data points in each local dataset.
In contrast, PrivacyFace protects some of the model parameters ( i
e. , the classiﬁer weights), which is the ﬁrst work in federated learning to our best knowledge.
Current DP-FL methods theoretically decrease performances of federated learning as they use noisy updates.
However, PrivacyFace distill differentially private information from each client.
The extra information can broadcasted to improve the performances of federated learning face recognition.
We can directly combine DP-FL method and PrivacyFace, which lead to a more secure framework protecting both data points as well as class centers together.