S PHERE F ACE 2: B INARY C LASSIFICATION IS A LL Y OU N EED FOR D EEP F ACE R ECOGNITION
Anonymous authors
Paper under double-blind review
A BSTRACT
State-of-the-art deep face recognition methods are mostly trained with a softmaxbased multi-class classiﬁcation framework.
Despite being popular and effective, these methods still have a few shortcomings that limit empirical performance.
In this paper, we start by identifying the discrepancy between training and evaluation in the existing multi-class classiﬁcation framework and then discuss the potential limitations caused by the “competitive” nature of softmax normalization.
Motivated by these limitations, we propose a novel binary classiﬁcation training framework, termed SphereFace2 .
In contrast to existing methods, SphereFace2 circumvents the softmax normalization, as well as the corresponding closed-set assumption.
This effectively bridges the gap between training and evaluation, enabling the representations to be improved individually by each binary classiﬁcation task.
Besides designing a speciﬁc well-performing loss function, we summarize a few general principles for this “one-vs-all” binary classiﬁcation framework so that it can outperform current competitive methods.
We conduct comprehensive experiments on popular benchmarks to demonstrate that SphereFace2 can consistently outperform current state-of-the-art deep face recognition methods.
I NTRODUCTION
Recent years have witnessed the tremendous success of deep face recognition (FR), largely owing to rapid development in training objectives [4, 13, 14, 27, 28, 33–35, 38].
Current deep FR methods are typically based on a multi-class learning objective, e.g. , softmax cross-entropy loss [4, 13, 14, 23, 33– 35].
Despite its empirical effectiveness, there is an obvious discrepancy between such a multi-class classiﬁcation training and open-set pair-wise testing, as shown in Fig.
In contrast to multi-class classiﬁcation, pair-wise veriﬁcation is a binary problem where we need to determine whether a pair of face images belongs to the same person or not.
This signiﬁcant discrepancy may cause the training target to deviate from the underlying FR task and therefore limit the performance.
This problem has also been noticed in [25, 27], but they still try to address it under the multi-class (or triplet) learning framework which still fundamentally differs from pair-wise veriﬁcation in testing.
On the other hand, multi-class classiﬁcation training assumes a closed-set environment where all the training data must belong to one of the known identities, which is fundamentally different from open-set testing.
[Figure]
In order to address these limitations, we propose a novel deep face recognition framework completely based on binary classiﬁcation.
SphereFace [13] is one of the earliest works that explicitly performs multi-class classiﬁcation on the hypersphere ( i.e. , angular space) in deep FR.
In light of this, we name our framework SphereFace2 because it exclusively performs binary classiﬁcations (hence the “2”) on the hypersphere.
Unlike multi-class classiﬁcation training, SphereFace effectively bridges the gap between training and testing, because both training and testing perform pair-wise comparisons.
Moreover, SphereFace2 alleviates the closed-set assumption and views the training as an open-set learning problem.
For example, training samples that do not belong to any class are still useful and can serve as negative samples in SphereFace2, while they cannot be used at all in the multi-class classiﬁcation framework.
Speciﬁcally, given K classes in the training set,
SphereFace2 constructs K binary classiﬁcation objectives where data from the target class are viewed as positive samples and data from the remaining classes are negative samples.
The spirit is similar to “one-vs-all” classiﬁcation [24].
An intuitive illustration comparing typical multi-class training, our proposed SphereFace2, and open-set evaluation is given in Fig.
In multi-class classiﬁcation training, the softmax (cross-entropy) loss introduces competition among different classes, because the softmax normalization requires all the class logits are summed to one. Therefore, increasing the conﬁdence of one class will necessarily decrease the conﬁdence of some other classes, which can easily lead to over-conﬁdent predictions [6].
In contrast to the competitive nature in all the softmax losses, our SphereFace2 encourages each binary classiﬁcation task to individually improve the discriminativeness of the representation without any competition.
[Table]
The signiﬁcance of our method can also be understood from a different perspective.
Depending on how samples interact with each other in training, deep FR methods can be categorized into triplet-based learning and pair-based learning.
Triplet-based learning simultaneously utilizes an anchor, positive samples and negative samples in its objective, while pair-based learning uses either positive pairs or negative pairs in one shot.
Based on whether a proxy is used to represent a set of samples, both triplet-based and pair-based methods can learn with or without proxies.
A categorization of deep FR methods is given in Table 1.
Proxy-free methods usually require expansive pair/triplet mining and most of them [8, 19, 22, 27, 29, 47] use a hybrid loss that includes the standard softmax loss.
Typical examples of triplet-based learning with proxy include the softmax loss and its popular variants [4, 13, 33–35] where a classiﬁer is a class proxy and the learning involves an anchor ( i.e. , the deep feature x ), the positive proxy ( i.e. , the target classiﬁer W y ) and the negative proxies ( i.e. , the other classiﬁers W j , j (cid:54) = y ).
Most popular deep FR methods use proxies by default, since they greatly speed up training and improve data-efﬁciency (especially on large-scale datasets).
Distinct from previous deep FR, we believe SphereFace2 is the ﬁrst work to adopt a pair-wise learning paradigm with proxies.
[Figure]
An outstanding difference between triplet-based and pairbased learning is the usage of a universal threshold.
In
Fig.
2, we show that tripletbased learning compares the similarity scores between different pairs, while pair-based learning compares a similarity score and a universal threshold.
As a pair-based method,
SphereFace2 too optimizes the difference between similarity scores and a universal threshold.
By learning the threshold to distinguish positive and negative pairs during training, it naturally also generalizes to open-set FR.
In our framework, we cast a principled view on designing loss functions by systematically discussing the importance of positive/negative sample balance, easy/hard sample mining and angular margin, and combine these ingredients to propose an effective loss function for SphereFace2.
In fact, most popular variants of the softmax loss [4, 33–35] also perform easy/hard sample mining and incorporate angular margin.
Moreover, we observe that the distribution of similarity scores from positive pairs is quite different from that of negative pairs, in that it exhibits larger variance, which makes the threshold hard to determine.
To address this, we propose a novel similarity adjustment method that can adjust the distribution of pair similarity and effectively improve the generalizability.
The ﬂexibility of binary classiﬁcations in SphereFace2 brings a few additional advantages.
First, in contrast to the classiﬁer layer with the softmax loss which is highly non-trivial to parallelize [1], the classiﬁer layer in SphereFace2 can be naturally parallelized across different GPUs.
While the gradient update in the softmax loss needs to involve all the classiﬁers, the gradient update for classiﬁers in
SphereFace2 is independent and fully decoupled from each other.
The gradient with respect to each classiﬁer only depends on the feature x and the binary classiﬁer itself.
Therefore, SphereFace naturally addresses the problem of the softmax loss in training on a large number of face identities [1].
Second, the pair-wise labels used in SphereFace2 are in fact a weaker form of supervision than the class-wise labels used in all the softmax-based losses.
This property is closely connected to the open-set assumption and also makes SphereFace2 less sensitive to class-wise label noise.
For example, SphereFace2 does not require the class-wise labels of the negative samples in each binary classiﬁcation.
Instead, as long as we know that the identities of negative samples are different from the positive samples, SphereFace2 will still work well.
Our experiments also empirically verify the robustness of SphereFace2 against class-wise label noise.
Our major contributions are:
• SphereFace2 constructs a novel binary classiﬁcation framework for deep FR.
To our knowledge, SphereFace2 is the ﬁrst work in deep FR to adopt a pair-wise learning paradigm with proxies.
We further summarize a series of general principles for designing a good loss function for SphereFace2.
• The decoupling of the binary classiﬁcations leads to a natural parallelization for the classiﬁer layer, making SphereFace2 more scalable than softmax-based losses.
• The pair-wise labels used in SphereFace2 are a weaker supervision than the class-wise labels used in the softmax loss, yielding better robustness to class-wise label noise.
Related Work .
Deep FR methods are either proxy-based or proxy-free, as shown in Table 1.
Proxyfree learning includes contrastive loss [3, 7, 27] and triplet loss [25].
Both losses directly optimize the similarity between samples, so they are highly unstable and data-inefﬁcient.
Proxy-free learning is more widely used in deep metric learning [21, 26, 32, 36, 37, 40].
Proxy-based learning uses a set of proxies to represent different groups of samples and usually works better for large-scale datasets.
Typical examples include the softmax loss and its variants [4, 13, 28, 30, 31, 33–35] where each proxy is used to represent a class.
SphereFace [13] suggests that large-margin classiﬁcation is more aligned with open-set FR, and proposes to learn features with large angular margin.
CosFace [33, 35] and ArcFace [4] introduce alternative ways to learn large-margin features with improved stability.
Although large-margin classiﬁcation brings the training target closer to the task of open-set FR, the discrepancy still exists and it is unclear how much the margin should be to close the gap.
Moreover, a large margin inevitably leads to training instability.
In contrast, SphereFace2 naturally avoids these problems and aligns the training target with open-set FR by adopting pair-wise learning with proxies.
T HE S PHERE F ACE 2 F RAMEWORK 2. O VERVIEW AND P RELIMINARIES
The goal of SphereFace2 is to align the training target with open-set veriﬁcation so that our training is more effective in improving open-set generalization in deep FR.
To this end, SphereFace2 explicitly incorporates pair-wise comparison into training by constructing K binary classiﬁcation tasks ( K is the number of identities in the training set).
The core of SphereFace2 is the binary classiﬁcation reformulation of the training target.
In the i -th binary classiﬁcation, we construct the positive samples with the face images from the i -th class and the negative samples with face images from other classes.
Speciﬁcally, we denote the weights of the i -th binary classiﬁer by W i , the deep feature by x and its corresponding label by y .
A naive loss formulation is
[Equation]
which is a combination of K standard binary logistic regression losses.
Instead, we perform the binary classiﬁcation on the unit hypersphere by normalizing both classiﬁers W i , ∀ i and feature x .
The loss function now becomes
[Equation]
where θ i is the angle between the i -th binary classiﬁer W i and the sample x .
The biases b i , ∀ i are usually removed in common practice [4, 13, 15, 33, 35], since they are learned for a closed set and cannot generalize to unknown classes.
However, we actually ﬁnd them very useful in SphereFace2, as will be discussed later.
For now, we temporarily remove them for notational convenience.
One of the unique advantages of such a parameterization for binary classiﬁers is that it constructs the i -th class positive proxy with W i and the negative proxy with − W i .
Depending on the label, the training will minimize the angle between x and W i or between x and − W i in order to minimize the loss.
This parameterization of positive and negative proxies immediately guarantees minimum hyperspherical energy [12] that has been shown to beneﬁt generalization.
Moreover, our parameterization in
SphereFace2 has the same number of parameters for the classiﬁer layer as the previous multi-class training, and does not introduce extra overhead.
However, naively minimizing the loss in Eq.
(1) will not give satisfactory results.
Therefore, we explore in the next subsection how to ﬁnd a desirable loss function that works well in the SphereFace2 framework.
2. A P RINCIPLED V IEW ON L OSS F UNCTION
We emphasize that the exact form of our loss function is in fact not crucial, and the core of SphereFace is the spirit of binary classiﬁcation ( i.e. , pair-wise learning with proxies).
Following such a spirit, there are likely many alternative losses that work as well as ours.
Besides proposing a speciﬁc loss function, we summarize our reasoning for designing a good loss function via a few general principles.
Positive/negative sample balance .
The very ﬁrst problem we need to address for SphereFace2 is how to balance the positive and negative samples.
In fact, balancing positive/negative samples has also been considered in triplet loss, contrastive loss and softmax-based losses.
Contrastive loss achieves the positive/negative sample balance by selecting the pairs.
Based on the uniﬁed view in [30], triplet-based methods including both triplet loss and softmax-based losses can naturally achieve positive/negative sample balance, since these losses require the presentation of a balanced number of both positive and negative samples.
From Eq.
(1), the gradients from positive samples and negative samples are highly imbalanced because only one out of K terms computes the gradient for positive samples.
A simple yet effective remedy is to introduce a weighting factor λ to balance gradients for positive and negative samples:
[Equation]
where λ ∈ [0 , 1] is a hyperparameter that determines the balance between positive and negative samples.
One of the simplest ways to set λ is based on the fraction of the loss terms, i.e. , λ = K − 1 K when there are K classes in total.
1      s=1s=2s=3s= - -0. r=
r=8 r=2r=3r=1.
cos( [Figure] y )
[Figure]
[Figure]
y
[Figure]
Easy/hard sample mining .
Another crucial criterion for a good loss function is the strategy for mining easy/hard samples, since it is closely related to the convergence speed and quality.
It is commonly perceived that softmax-based losses are free of easy/hard sample mining, unlike triplet and contrastive losses.
However, this is inaccurate.
We construct a simple example to illustrate how the softmax-based loss mines easy/hard samples.
We assume a set of cosine logits from four classes is [cos( θ 1 ) , cos( θ 2 ) , cos( θ 3 ) , cos( θ 4 )] and the target class is y =1 .
Then we also compute the s -normalized softmax loss L n = − log( exp( s · cos( θ y ))
(cid:80) i exp( s · cos( θ i )) ) by ﬁxing cos( θ i )
=0 .
i (cid:54) =
y and varying cos( θ y ) from − 1 to 1 .
From the results shown in Fig. 3(a), we can observe that as the scale s increases, the loss for hard samples will be higher and also more sensitive than the loss for easy samples.
This makes the neural network focus on optimizing the angles of hard samples and therefore can improve convergence and generalization.
The scaling strategy is widely used as a common practice in most softmax-based margin losses [4, 30, 33–35], playing an implicit role of easy/hard sample mining.
For the standard softmax cross-entropy loss, easy/hard sample mining is dynamically realized by the norm of features and classiﬁers.
In our framework, we need a similar strategy to mine easy/hard samples.
Inspired by the rescaled softplus function [5], we use an extra hyperparameter r to adjust the curvature of the loss function based on the objective L b :
[Equation]
where larger r implies stronger focus on hard samples.
We consider an example of one binary classiﬁcation and L e becomes 1 r · log (cid:0) 1+exp( − r · cos( θ y )) (cid:1) when λ =1 .
We then plot how the loss value changes by varying cos( θ y ) from − 1 to 1 under different r in Fig. 3(b).
We observe that when r becomes larger, the loss for easy samples gets closer to zero while the loss for hard samples remains large.
Therefore, r can help to mine and reweight easy/hard samples during training.
[Figure]
Angular margin .
Learning deep features with large angular margin is arguably one of the most effective criteria to achieve good generalizability on open-set face recognition.
SphereFace [13] introduced angular margin to deep face recognition by considering a multiplicative margin.
CosFace [33, 35] and ArcFace [4] further considered an additive angular margin which makes the loss function easier to train.
In light of these works, we introduce a novel two-sided angular margin with two adjustable parameters to the SphereFace2 framework:
[Equation]
) where m p controls the size of the margin for positive samples and m n controls the size of the margin for negative samples.
Larger m p and m n lead to larger additive angular margin, sharing similar spirits to CosineFace [33, 35].
The other forms of angular margin are also applicable in our framework, but exhaustively trying them is simple repetitive work and also beyond the scope of this paper.
However, quite different from the angular margin in the softmax-based losses, the angular margin in
Eq.
3 has a universal conﬁdence threshold 0 ( i.e. , cos( θ i )=0 ).
Both angular margins m p and m n are introduced with respect to this decision boundary cos( θ i )=0 , see Fig. 4(b).
This property has both advantages and disadvantages.
One of the unique advantages is that our angular margin for each class is added based on a universally consistent conﬁdence threshold and does not depend on the other classiﬁers, while the angular margin in softmax-based losses will be largely affected by the neighbor classiﬁers.
However, it is extremely challenging to achieve the universal threshold 0 , which results in training difﬁculty and instability.
To improve training stability, the bias term that has long been forgotten in softmax-based losses comes to the rescue.
We combine the biases back:
[Equation]
where b i denotes the bias term for the binary classiﬁer of the i -th class.
Since the class-speciﬁc bias is not useful in the open-set testing, we will simply use the same bias b for all the classes.
The bias b now becomes the universal conﬁdence threshold for all the binary classiﬁcations and the baseline decision boundary becomes r · cos( θ y )
+
b =0 instead of r · cos
( θ y )=0 , making the training more stable and ﬂexible.
The ﬁnal decision boundary is r · (cos( θ y ) − m p )+ b =0 for the positive samples and r · (cos( θ y )+ m n )+ b =0 for the negative samples.
More importantly, the threshold b is still universal and consistent for each class.
An illustration of Eq.
4 is given in Fig.
Alternatively, we can interpret Eq.
4 as a learnable angular margin where the conﬁdence threshold is still 0 .
Then we can view m p − br as the positive margin and m n + br as the negative margin.
Because b is learnable, we only need to focus on m p + m n which yields only one effective hyperparameter.
For convenience, we simply let m p = m n = m and only need to tune m in practice.
[Figure]
Similarity adjustment .
In Fig. 5(a), we observe a large inconsistency between positive and negative pairs in the distribution of cosine similarity.
The similarity distribution of negative pairs has smaller variance and is more concentrated, while the positive pairs exhibit much larger variation in similarity score.
The distributional discrepancy between positive and negative pairs leads to a large overlap of similarity scores between them, making it difﬁcult to give a clear threshold to separate the positive and negative pairs.
This is harmful to generalization.
Fig.
5(a) also empirically shows the similarity scores mostly lie in the range of [ − 0 . 2 , 1] .
These observations motivate us to (1) reduce the overlap of the similarity scores between positive and negative pairs, and (2) increase the empirical dynamic range of the similarity score such that the pair similarity can be distributed in a larger space.
t=1t=2t=3t=
Cosine Similarity cos(θ)
A d j u s t e d
S i m il a r it
y g
( c o s ( θ ))
To this end, we propose a novel similarity adjustment method.
Our basic idea is to construct a monotonic decreasing function g z ) where z ∈ [ − 1 , 1] and g ( z ) ∈ [ − 1 , 1] and then use g (cos( θ )) instead of the original cos( θ ) to adjust the mapping from angle to similarity score during training.
Considering that the originally learned cos( θ ) mostly lies in the range of [ − 0 . 2 , 1] , we require g ( z ) to map [ − 0 .
2 , 1] to a larger range ( e.g. , [ − 0 .
9 , 1] ), so that if cos( θ ) is learned similarly as before and still gives the empirical dynamic range of [ − 0 . 2 , 1] , we can end up with g (cos( θ )) whose empirical dynamic range becomes [ − 0 .
Speciﬁcally, g ( z ) is parameterized as g ( z )=2 (cid:0) z +12 (cid:1) t −where we typically use z =cos( θ ) ∈ [ − 1 , 1] .
In practice, we simply replace the original cosine similarity with g (cos( θ )) during training.
t controls the strength of similarity adjustment.
When t =1 , g (cos( θ )) reduces to the standard cosine similarity.
In Fig.
5, we show that the similarity distribution can be modiﬁed by increasing the parameter t .
As t increases, the overlap between the positive and negative pairs is reduced and their similarity distributions also become more separable.
Moreover, the empirical dynamic range of the similarity score is also approximately increased from [ − 0 .
2 , 1] to [ − 0 .
The empirical results validate the effectiveness of the proposed similarity adjustment.
Final loss function .
After combining all the principles and simplifying hyperparameters, we have
[Equation]
where g ( · ) has a hyperparameter t .
In total, there are four hyperparameters λ , r , m and t .
Each has a unique geometric interpretation, making them easy to tune.
Following our design principles, there could be many potential loss functions that share similar properties to ours.
Our framework opens up new possibilities to advance deep face recognition.
2. G EOMETRIC I NTERPRETATION
This subsection provides a comprehensive discussion and visualization to justify our designs and explain the geometric meaning of each hyperparameter.
By design, r is the radius of the hypersphere where all the learned features live and is also the magnitude of the features.
The bias b for the i -th class moves the baseline decision boundary along the direction of its classiﬁer W i .
The parameter m controls the size of the induced angular margin.
We set the output feature dimension as 2 and plot the 2D features trained by SphereFace2 with different margin m in Fig.
The visualization empirically veriﬁes the following arguments.
[Figure]
) (2) (3) (1) r(cos(θ 1 )-m)+b=0 (2) r cos(θ 1 )+b= (3) r(cos(θ 1 )+m)+b=0 .
r cos(θ 1 )+b=0 .
The bias b moves the decision boundary .
From Fig.
7, we can observe that the bias b can be effectively learned to move the decision boundary along the classiﬁer direction and lead to a new universal conﬁdence − b for all the classes.
The bias b makes the training easier and more stable while still preserving the unique property that all classes share the same conﬁdence for classiﬁcation.
Compared to other deep FR methods, the universal conﬁdence in SphereFace2 can help to learn a consistent positive/negative pair separation and explicitly encourage a unique and consistent veriﬁcation threshold during training.
m , r control the angular margin .
Fig.
7 visualizes the baseline decision boundary (denoted as (2) in Fig. 7(b)) and the decision boundary for the positive/negative samples (denoted as (1)/(3) in
Fig. 7(b)).
The distance between the positive and negative decision boundary is 2 rm , producing an effective angular margin.
The results show that the empirical margin matches our expected size well.
Larger m leads to larger angular margin .
From Fig.
7, we can empirically compare the deep features learned with m =0 and m =0 . 2 and conﬁrm that larger m indeed incorporates larger angular margin between different classes.
[Figure]
We also visualize 3D deep features and the decision planes for one class with r =30 and m =0 .
2 in Fig.
We can observe that samples from each class are well separated with large angular margins, which is consistent with the 2D case.
The empirical angular margin also perfectly matches the induced one ( i.e. , the distance between the positive and negative plane).
The results further verify our empirical conclusions drawn from the 2D visualization.
2. E FFICIENT M ODEL P ARALLELIZATION ON GPU S
As it becomes increasingly important for deep FR methods to train on largescale datasets with million-level identities, a bottleneck is the storage of the classiﬁer layer since its space complexity grows linearly with the number of identities.
A common solution is to distribute the storage of the classiﬁers W i , ∀ i and their gradient computations to multiple
GPUs.
Then each GPU only needs to compute the logits for a subset of the classes.
However, the normalization in softmax-based losses inevitably requires some data communication overhead across different GPUs, resulting in less efﬁcient parallelization.
In contrast, the gradient computations in SphereFace2 are class-independent and can be performed locally within one GPU.
Thus no communication cost is needed.
The decoupling among different classiﬁers makes SphereFace suitable for multi-GPU model parallelization.
Speciﬁcally, the softmax normalization in the softmax loss involves the computation of all the classiﬁers, so computing the gradient w.r.t.
any classiﬁer will still require the weights of all the other classiﬁers, which introduces communication overhead across GPUs.
In contrast, the loss for
SphereFace2 (Eq.
(5)) can be rewritten as L = (cid:80)
Ki =
1 f i ( W i , x ) where f i ( · , · ) is some differentiable function.
To compute ∂L∂ W i , we only need to compute ∂f i ( W i , x ) W i which does not involve any other classiﬁers.
Therefore, this gradient computation can be performed locally and does not require any communication overhead.
Appendix E compares the gradient of SphereFace2 and the softmax loss.
E XPERIMENTS AND R ESULTS
Preprocessing .
Each face image is cropped based on the 5 face landmarks detected by MTCNN [43] using a similarity transformation.
The cropped image is of size 112 × 112 .
Each RGB pixel ( [0 , 255] ) is normalized to [ − 1 , 1] .
We put the details of training and validation datasets in Appendix A.
CNNs .
We adopt the same CNNs from [13, 35] for fair comparison.
We use 20-layer CNNs in ablations and 64-layer CNNs for the comparison to existing state-of-the-art methods.
Training and Testing .
We use SGD with momentum 0 .
9 by default.
We adopt VGGFace2 [2] as the same training set for all the methods.
VGGFace2 contains 3.1M images from 8.6K identities, as shown in Table 9.
The training faces are horizontally ﬂipped for data augmentation.
We strictly follow the speciﬁc protocol provided in each dataset for evaluations.
Given a face image, we extract a 512D embedding.
The ﬁnal score is computed by the cosine distance of two embeddings.
The nearest neighbor classiﬁer and thresholding are used for face identiﬁcation and veriﬁcation, respectively.
3. A BLATION S TUDY AND E XPLORATORY E XPERIMENTS
We perform an ablation study on four validation sets: LFW, AgeDB-30, CA-LFW, and CP-LFW.
The statistics of these datasets are summarized in Table 9.
Following the provided evaluation protocols, we report 1:1 veriﬁcation accuracy of 6,000 pairs (3,000 positive and 3,000 negative pairs) for each dataset.
In addition, we combine these datasets and compute the overall veriﬁcation accuracy, which serves as a more accurate metric to evaluate the models.
[Table]
Design Principles .
We start with ablations on the designing principles of the loss function.
As shown in Table 2, it is quite effective to improve the performance following the principles to design a loss function.
Speciﬁcally, the naive binary classiﬁcation (Eq. 1) fails to converge, since the learning objective and the gradients are both dominated by the negative pairs.
To address this, we set λ to K − 1 K and report the results in the ﬁrst row of Table 2.
As can be seen, while the model is trainable and starts to converge, the results show signiﬁcant room for improvement.
After some tuning of the hyperparameter λ ,
SphereFace2 can achieve 78.03% accuracy on the combined dataset.
Then we gradually incorporate the hard sample mining and angular margin into SphereFace2, improving the veriﬁcation accuracy from 78.03% to 89.65%, and then to 93.87% With similarity adjustment, SphereFace2 yields 94.28% accuracy on the combined dataset.
Table 2 clearly shows the effectiveness of each ingredient
[Table] [Table]
Hyperparameters.
There are four hyperparameters ( λ , r , m , and t ) in SphereFace2.
Since r and m have been extensively explored in [4, 33–35], we follow previous practice to ﬁx r and m to 30 and 0.4 respectively.
Our experiments mainly focus on analyzing the effect of λ and t .
From Table
[Table]
the performance of SphereFace2 remains stable for λ from 0.5 to 0.9, where a good balance for the positive and negative pairs is achieved.
Then we evaluate different t for λ =
7 , 0 . 8] and report the results in Table 4.
As can be seen from the results, adjusting the similarity scores further boosts model accuracy.
The effect of different t is also illustrated in Fig.
More detailed ablation studies are included in Appendix C and C.
[Table]
State-of-art loss functions .
We compare SphereFace2 with current state-ofart loss functions in Table 5.
We note that these current best-performing methods [4, 13, 30, 35] are based on multiclass classiﬁcation and belong to the triplet learning paradigm.
In contrast, our SphereFace2 is based on binary classiﬁcation and adopts the pair-wise learning paradigm.
We re-implement
SphereFace [13] with feature normalization for fair comparison.
We observe that the veriﬁcation accuracies on LFW are saturated around 99.5%.
On both AgeDB-30 and CA-LFW datasets, SphereFace2 achieves the best accuracy, outperforming the second best results by a signiﬁcant margin.
The results on the combined datasets also validate the effectiveness of SphereFace2.
3. E VALUATIONS ON L ARGE - SCALE B ENCHMARKS
We use three challenging face recognition benchmarks, IJB-B, IJB-C
[Table]
and MegaFace to evaluate
SphereFace2 (with λ =0 .
7 , r =40 , m =0 .
4 , t =3 ). We use 64-layer CNNs for all the methods here.
[Table]
IJB datasets .
IJB-B [39] has 21.8K still images (including 11.8K faces and 10k non-faces) and 55K frames from 7K videos.
The total number of identities is 1,845.
We follow the standard 1: veriﬁcation and 1:N identiﬁcation protocols for experiments.
The protocol deﬁnes 12,115 templates, where each template consists of multiple images and/or frames.
Matching is performed based on the deﬁned templates.
Speciﬁcally, 10,270 genuine matches and 8M impostor matches are constructed in 1:1 veriﬁcation protocol, and 10,270 probes and 1,875 galleries are constructed in 1:N identiﬁcation protocol.
IJB-C is an extension of IJB-B, comprising 3,531 identities with 31.3K still images and 117.5K frames from 11.8K videos.
The evaluation protocols of IJB-C are similar to
IJB-B.
The details of the protocols are summarized in Appendix.
We report the true accept rates (TAR) at different false accept rates (FAR) for veriﬁcation, and true positive identiﬁcation rates (TPIR) at different false positive identiﬁcation rates (FPIR) for identiﬁcation, as shown in Table 6.
[Table]
We make several observations based on the evaluation results of IJB-B (Table 6).
First, SphereFace2 produces signiﬁcant improvements over other state-of-the-art methods, especially at low FARs and
FPIRs.
Speciﬁcally, SphereFace2 outperforms CosFace by 5.37% at FAR=1eand 2.70% at FAR=1e-4 in 1:1 veriﬁcation, 3.09% at FPIR=1e-2 and 3.02% at
FPIR=1e-1 in 1:N identiﬁcation.
These signiﬁcant performance gains suggest that the pair-wise learning paradigm is very useful in improving the robustness of a face recognition system.
Similar observations can also be found in the results of IJB-C (Table 7) and the ROC curves results (Fig. 9).
Second, the performance is getting saturated for veriﬁcation rate at FAR=1e-3.
Compared to other methods, SphereFace2 can still improve the results by 0.67% - 1.02% on IJB-B and 0.43% - 1.04% on IJB-C. Third, the rank-1 identiﬁcation performance of SphereFace2 is slightly better than CosFace, ArcFace, Circle Loss, and comparable to SphereFace.
Overall, SphereFace performs signiﬁcantly better than current best-performing methods on these two challenging datasets.
[Table]
MegaFace .
We further evaluate SphereFace2 on the MegaFace dataset.
This is a challenging testing benchmark to evaluate the performance of face recognition methods at the million scale of distractors.
It contains a gallery set with more thanmillion images from 690K different identities, and a probe set with 3,530 images from 530 identities.
MegaFace provides two testing protocols for identiﬁcation and veriﬁcation.
We evaluate on both and report the results in Table 8.
The gains are consistent with the IJB datasets.
Under the same training setup, SphereFace2 outperforms current state-of-the-art methods by a large margin.
3. N OISY L ABEL L EARNING
[Table]
[Figure]
Since the pair-wise labels used in
SphereFace2 provide weaker supervision than the class-wise labels, we perform experiments to evaluate the robustness of SphereFace2 in label-noisy training.
We randomly alter 20%, 40%, 60% and 80% of the labels for each class.
The four noisy datasets are used to train CosFace, ArcFace [Figure] and SphereFace2 separately.
We evaluate the trained models on the combined validation set.
From Fig 10 (left), SphereFace2 shows stronger robustness to noisy labels than CosFace and ArcFace, as its performance degrades signiﬁcantly more slowly as the ratio of noisy labels increases from 0 to 0.8.
M ODEL P ARALLELIZATION
[Figure]
d
[Figure]
s p e r [Figure] We follow [4] to parallelize the loss computations for CosFace, ArcFace, and SphereFace2.
These methods are trained with 1 million identities.
Fig.(right) shows how the number of processed images per second changes with different numbers of GPUs.
Note that we do not include the feature extraction here.
CosFace and ArcFace have negligible difference on running time.
When a single GPU is used ( i.e. , no model parallelization), CosFace and ArcFace are slightly faster than SphereFace2.
As the number of GPUs increases, the acceleration of SphereFace2 is more signiﬁcant, due to less communication cost over GPUs.
The near linear acceleration for SphereFace2 is owing to its proxy-based pair-wise formulation.
R EPRODUCIBILITY S TATEMENT
Experiments.
To ensure the reproducibility of the empirical results, we include all the necessary details (including all hyperparameters) in the experiments.
On acceptance, we will publicly release our codebase to ensure reproducibility of our experiments
Appendix A S TATISTICS FOR THE U SED D ATASETS
[Table]
B S TATISTICS OF IJB T EST P ROTOCOLS
[Table]
C M ORE H YPERPARAMETER E XPERIMENTS
We additionally provide the results under different hyperparameters.
We follow the same settings as in the main paper by training a 20-layer CNN [13] on VGGFace2.
First, we vary the hyperparameter r from 20 to 50 and the results in Table 11 show that both r = 30 and r = 40 work reasonably well.
Second, we vary the hyperparameter m from 0.2 to 0.5 and evaluate how the size of margin affects the performance.
The results in Table 11 show that m = 0 .
3 generally achieves the best performance.
Last, we evaluate how the hyperparameter t in similarity adjustment may affect the performance.
Table 11 shows that similarity adjustment is generally helpful for performance ( i.e. , t = 2 , 3 , 4 ,works better than t = 1 ) and t = 3 achieves the best combined performance. [Table]
D S IMILARITY A DJUSTMENT FOR OTHER METHODS
To empirically show the comparison between SphereFace2 and other methods with SA, we present the experiments on IJBB and IJBC datasets.
As shown in Table 12, similarity adjustment works well for SphereFace2, and applying it to multi-class classiﬁcation losses is not as useful as in SphereFace
[Table]
E G RADIENT C OMPARISON
We compare the gradient between the multi-class softmax-based loss and SphereFace2 in Table 13.
We observe that the gradient updates in SphereFace2 can be performed with only the corresponding classiﬁer weights ( i
[Figure]
e. , no summation over classiﬁer weights of different classes).
Therefore, the classiﬁer layer in the SphereFace2 framework can be back-propagated with the local GPU and involve no communication overhead.
[Figure]