S AMPLE AND C OMPUTATION R EDISTRIBUTION FOR E FFICIENT F ACE D ETECTION
A BSTRACT
Although tremendous strides have been made in uncontrolled face detection, accurate face detection with a low computation cost remains an open challenge.
In this paper, we point out that computation distribution and scale augmentation are the keys to detecting small faces from low-resolution images.
Motivated by these observations, we introduce two simple but effective methods: (1) Computation Redistribution (CR), which reallocates the computation between the backbone, neck and head of the model; and (2) Sample Redistribution (SR), which augments training samples for the most needed stages.
The proposed Sample and Computation Redistribution for Face Detection ( SCRFD ) is implemented by a random search in a meticulously designed search space.
Extensive experiments conducted on WIDER FACE demonstrate the state-of-the-art accuracy-efﬁciency trade-off for the proposed SCRFD family across a wide range of compute regimes.
In particular, SCRFD-34GF outperforms the best competitor, TinaFace, by 4 .
78% (AP at hard set) while being more than 3 × faster on GPUs with VGA-resolution images.
Code is available at: https://github.com/deepinsight/insightface/ tree/master/detection/scrfd .
I NTRODUCTION
Face detection is a long-standing problem in computer vision with many applications, such as face alignment (Bulat & Tzimiropoulos, 2017; Deng et al., 2019b), face reconstruction (Feng et al., 2018; Gecer et al., 2021), face attribute analysis (Zhang et al., 2018; Pan et al., 2018), and face recognition (Schroff et al., 2015; Deng et al., 2019a; 2020a).
Following the pioneering work of (Viola & Jones, 2004), numerous face detection algorithms have been designed.
Among them, the single-shot anchor-based approaches (Najibi et al., 2017; Zhang et al., 2017b; Tang et al., 2018; Li et al., 2019; Ming et al., 2019; Deng et al., 2020b; Liu et al., 2020; Zhu et al., 2020) have recently demonstrated very promising performance.
In particular, on the most challenging face detection dataset, WIDER FACE (Yang et al., 2016), the average precision (AP) on its hard validation set has been boosted to 93 .
4% by TinaFace (Zhu et al., 2020).
Even though TinaFace (Zhu et al., 2020) achieves impressive results on unconstrained face detection, it employs large-scale ( e.g . 1 , 650 pixels) testing, which consumes huge amounts of computational resources.
In addition, TinaFace design is based on a generic object detector ( i.e . RetinaNet (Lin et al., 2017b)), directly taking the classiﬁcation network as the backbone, tiling dense anchors on the multi-scale feature maps ( i.e . P2 to P7 of neck), and adopting heavy head designs.
Without considering the prior of faces, the network design of TinaFace is thus redundant and sub-optimal.
One approach of optimizing such networks’ performance is computation redistribution.
Since directly taking the backbone of the classiﬁcation network for object detection is sub-optimal, the recent CR-NAS (Liang et al., 2020) reallocates the computation across different resolutions to obtain a more balanced Effective Receptive Field (ERF), leading to higher detection performance.
In
BFbox (Liu & Tang, 2020), a face-appropriate search space is designed, based on the observation of scale distribution gap between general object detection and face detection.
In ASFD (Zhang et al.,
[Figure]
The proposed SCRFD outperforms a range of state-of-the-art open-sourced methods by using much fewer ﬂops
2020a)
, a differential architecture search is employed to discover optimized feature enhance modules for efﬁcient multi-scale feature fusion and context enhancement.
Even though (Liu & Tang, 2020; Zhang et al., 2020a) have realized the limitation of directly applying general backbone, neck and head settings to face detection, CR-NAS (Liang et al., 2020) only focuses the optimization on backbone, BFbox (Liu & Tang, 2020) neglects the optimization of head, and ASFD (Zhang et al., 2020a) only explores the best design for neck.
Another optimization approach, is the sample redistribution across different scales.
Due to the extremely large scale variance of faces in real-world scenarios, different scale augmentation strategies are employed to introduce scale adaptation into the face detector.
The most widely used scale augmentation approaches include random square crop (Zhang et al., 2017b;
Deng et al., 2020b; Zhu et al., 2020) and data anchor sampling (Tang et al., 2018).
Nevertheless, the scale augmentation parameters in these methods are manually designed for all different network structures.
Therefore, traditional multi-scale training in face detection is also tedious and sub-optimal.
Since VGA resolution ( 640 × 480 ) is widely used for efﬁcient face detection on numerous mobile phones and digital cameras, we focus on efﬁcient face detection from low-resolution images in this paper.
In Fig 1(a), we give the cumulative face scale distribution on the WIDER FACE validation dataset.
Under the VGA resolution, most of the faces ( 78 . 93% ) in WIDER FACE are smaller than 32 × 32 pixels.
Under this speciﬁc scale distribution, both network structure and scale augmentation need to be optimized.
In this work, we present a meticulously designed methodology of search space optimization, that addresses both the redistribution between the backbone, neck and head, and the sample redistribution between the most needed scales.
As the structure of a face detector determines the distribution of computation and is the key in determining its accuracy and efﬁciency, we ﬁrst discover principles of computation distribution under different ﬂop regimes.
Inspired by (Radosavovic et al., 2020), we control the degrees of freedom and reduce the search space.
More speciﬁcally, we randomly sample model architectures with different conﬁgurations on backbone (stem and four stages), neck and head.
Based on the statistics of these models, we compute the empirical bootstrap (Efron &
Tibshirani, 1994) and estimate the likely range in which the best models fall.
To further decrease the complexity of the search space, we divide the computation ratio estimation for backbone and the whole detector into two steps.
To handle extreme scale variations in face detection, we also design a search-able zoom-in and zoom-out space, speciﬁed by discrete scales and binary probabilities.
In experiments, the proposed computation redistribution and sample redistribution yield signiﬁcant and consistent improvement on various compute regimes, even surpassing a range of state-of-the-art face detectors by using much fewer ﬂops as shown in Fig. 1(b).
To sum up, this paper makes following contributions:
• We have proposed a simpliﬁed search space, as well as a two-step search strategy for computation redistribution across different components (backbone, neck and head) of a face detector.
The proposed computation redistribution method can easily boost detection performance through random search.
• We have designed a search-able zoom-in and zoom-out space for face-speciﬁc scale augmentation, which automatically redistributes more training samples for shallow stages, enhancing the detection performance on small faces.
• Extensive experiments conducted on WIDER FACE demonstrate the signiﬁcantly improved accuracy and efﬁciency trade-off of the proposed SCRFD across a wide range of compute regimes.
R ELATED W ORK
Face Detection.
To deal with extreme variations ( e.g . scale, pose, illumination and occlusion) in face detection (Yang et al., 2016), most of the recent single-shot face detectors focus on improving the anchor sampling/matching or feature enhancement.
SSH (Najibi et al., 2017) builds detection modules on different feature maps with a rich receptive ﬁeld.
S 3 FD (Zhang et al., 2017b) introduces an anchor compensation strategy by offsetting anchors for outer faces.
PyramidBox (Tang et al., 2018) formulates a data-anchor-sampling strategy to increase the proportion of small faces in the training data.
DSFD (Li et al., 2019) introduces small faces supervision signals on the backbone, which implicitly boosts the performance of pyramid features.
Group sampling (Ming et al., 2019) emphasizes the importance of the ratio for matched and unmatched anchors.
RetinaFace
(Deng et al., 2020b) employs deform-able context modules and additional landmark annotations to improve the performance of face detection.
HAMBox (Liu et al., 2020) ﬁnds that many unmatched anchors in the training phase also have strong localization ability and proposes an online high-quality anchor mining strategy to assign high-quality anchors for outer faces.
BFbox (Liu & Tang, 2020) employs a single-path one-shot search method (Guo et al., 2019) to jointly optimize the backbone and neck for face detector.
ASFD (Zhang et al., 2020a) explores a differential architecture search to discover optimized feature enhance modules for efﬁcient multi-scale feature fusion and context enhancement.
All these methods are either designed by expert experience or partially optimized on backbone, neck and head.
By contrast, we search for computation redistribution across different components (backbone, neck and head) of a face detector across a wide range of compute regimes.
Neural Architecture Search.
Given a ﬁxed search space of possible networks, Neural Architecture
Search (NAS) automatically ﬁnds a good model within the search space.
DetNAS (Chen et al., 2019b) adopts the evolution algorithm for the backbone search to boost object detection on COCO (Lin et al., 2014).
By contrast, CR-NAS (Liang et al., 2020) reallocates the computation across different stages within the backbone to improve object detection.
NAS-FPN (Ghiasi et al., 2019) uses reinforcement learning to search the proper FPN for general object detection.
As there is an obvious distribution gap between COCO (Lin et al., 2014) and WIDER FACE (Yang et al., 2016), the experience in the above methods is not directly applicable for face detection but gives us an inspiration that the backbone, neck and head can be optimized to enhance the performance of face detection.
Inspired by RegNet (Radosavovic et al., 2020), we optimize the computation distribution on backbone, neck and head based on the statistics from a group of random sampled models.
We successfully reduce the search space and ﬁnd the stable computation distribution under a particular complex regime, which signiﬁcantly improves the model’s performance.
M ETHODOLOGY
To efﬁciently and accurately detect small faces from low-resolution images ( e.g . VGA 640 × 480 ), we propose two methodologies that, when combined, outperform the state-of-the-art.
In Sec. 3.1, we explore the computation redistribution across different stages of backbone, as well as different components ( i.e . backbone, neck and head) of the whole detector, given a pre-deﬁned computation budget.
Then, in Sec. 3.2, we investigate the redistribution of positive training samples across different scales of feature maps by searching optimized scale augmentations.
C OMPUTATION R EDISTRIBUTION
As illustrated in Fig.
2, we apply our search method on a network consisting of (1) RetinaNet (Lin et al., 2017a), with ResNet (He et al., 2016) as the backbone, (2) Path Aggregation Feature Pyramid
Network (PAFPN) (Liu et al., 2018) as the neck, and (3) stacked 3 × 3 convolutional layers for the head.
Despite the generally simple structure, the total number of possible network conﬁgurations of the search space becomes unwieldy.
Therefore, we attempt to simplify the tremendous search space and at a low-dimensional design space consisting of simple and effective networks
3.1. S EARCH S PACE D ESIGN
Inspired by RegNet (Radosavovic et al., 2020), we explore the structures of face detectors, assuming ﬁxed standard network blocks ( i.e ., basic residual or bottleneck blocks with a ﬁxed bottleneck ratio of 4 ).
In our case, the structure of a face detector includes:
• the backbone stem , three 3 × 3 convolutional layers with w 1 output channels ( He et al. , 2019a
• the backbone body , four stages ( i e . C2, C3, C4 and C5) operating at progressively reduced resolution, with each stage consisting of a sequence of identical blocks.
For each stage i , the degrees of freedom include the number of blocks
d i ( i.e . network depth) and the block width w i ( i e . number of channels).
• the neck , a multi-scale feature aggregation module by a top-down path and a bottom-up path with n channels (Liu et al., 2018).
• the head , with h i channels of m blocks to predict face scores and regress face boxes.
The search space can be initially designed as follows.
As the channel number of the stem is equal to the block width of the ﬁrst residual block in C2, the degree of freedom of the stem w 1 can be merged into w 2 .
In addition, we employ a shared head design for three-scale of feature maps and ﬁx the channel number for all 3 × 3 convolutional layers within the heads.
Therefore, we reduce the degrees of freedom to three within our neck and head design: (1) output channel number n for neck, (2) output channel number h for head, and (3) the number of 3 × 3 convolutional layers m .
We perform uniform sampling of n ≤ 256 , h ≤ 256 , and m ≤ 6 (both n and h are divisible by 8).
The backbone search space has 8 degrees of freedom as there are 4 stages and each stage i hasparameters: the number of blocks
d i and block width w i .
Following RegNet (Radosavovic et al., 2020), we perform uniform sampling of d i ≤ 24 and w i ≤ 512 ( w i is divisible by 8).
As state-ofthe-art backbones have increasing widths (Radosavovic et al., 2020), we also constrain the search space, according to the principle of w
i +1 ≥ w i .
3.1. E STIMATION M ETRIC
Based on above simpliﬁcations, our search space becomes more compact and efﬁcient.
We repeat the random sampling in our search space until we obtain 320 models in our target complexity regime, and train each model on the WIDER FACE (Yang et al., 2016) training set for 80 epochs.
Then, we test the Average Precision (AP) of each model on the validation set.
Based on these 320 pairs of model statistics ( x i , AP i ) , where x i is the computation ratio of a particular component and AP i the corresponding performance, we can compute the empirical bootstrap (Efron & Tibshirani, 1994) to estimate the likely range in which the best models fall.
More speciﬁcally, we repeatedly sample with replacement 25% of the pairs for 10 3 times and select the pair with maximum AP in each sampling.
Afterwards, we compute the 95% conﬁdence interval for the maximum value and the median gives the most likely best computation ratio.
[Figure] [Figure]
The computation distribution of CRFD-2.5GF within backbone follows Fig.
In (d), the yellow rectangles in C2 to C5 represents the basic residual block.
The width of rectangles corresponds to the computation cost.
After computation redistribution, more computations are allocated to shallow stages ( i.e . C2 and C3).
3.1. T WO - STEP S EARCH
To further decrease the complexity of search space, we divide our network structure search into the following two steps: (1) CRFD 1 : search the computational distribution for the backbone only, while ﬁxing the settings of the neck and head to the default conﬁguration, and (2) CRFD 2 : search the computational distribution over the whole face detector ( i.e . backbone, neck and head), with the computational distribution within the backbone, following the optimized CRFD 1 .
By optimizing in both manners, we achieve the ﬁnal optimized network design for the computation-constrained face detection.
In the example below, we constrain CRFD to 2.5 GFlops ( CRFD-2.5GF ), in order to illustrate our two-step searching strategy.
Computation redistribution on backbone.
For CRFD 1 -2.5GF , we ﬁx the output channel of the neck at 32 and use two stacked 3 × 3 convolutions with 96 output channels.
As the neck and head conﬁgurations do not change in the whole search process of CRFD 1 , we can easily ﬁnd the best computation distribution of the backbone.
As described in Fig.
3, we show the distribution of 320 model APs (on the WIDER FACE hard validation set) versus the computation ratio over each component ( i.e . stem, C2, C3, C4 and C5) of backbone.
After applying an empirical bootstrap (Efron & Tibshirani, 1994), a clear trend emerges, showing that the backbone computation is reallocated to the shallow stages ( i.e . C2 and C3).
Computation redistribution on backbone, neck and head.
In this step, we only keep the randomly generated network conﬁgurations whose backbone settings follow the computation distribution from CRFD 1 as shown in Fig.
In this case, there are another three degrees of freedom ( i.e . output channel number n for neck, output channel number h for head, and the number m of 3 ×convolutional layers in head).
We repeat the random sampling in our search space, until we obtain 320 qualifying models in our target complexity regime ( i.e . 2.5 GFlops).
As evident in Fig.
4, most
[Table]
) CRFD-10GF
1 2 3 4 5 6 7 8 9 10 11 12 13 Block Index
W i d t h stem=24 d i =[3,5,3,2] w i =[24,48,48,80] neck=24 h i =[64,64] (h) CRFD-2.5GF 1 2 3 4 5 6
8 9 10 11 Block Index
W i d t h stem=48 d i =[3,2,1,5] w i =[48,160,216,312]
neck=24 h
i =[96,96] (i) CRFD-1.0GF 1 2 3 4 5 6 7 8 9 10 11 12 13 Block Index
W i d t h stem=16 d i =[2,3,2,6] w i =[
neck=16 h
i =[64,64] (j) CRFD-0.5GF of the computation is allocated in the backbone, with the head following and the neck having the lowest computation ratio.
Fig. 4(d) also depicts the comparison between the hand-crafted model architecture and the computation redistributed network
S AMPLE R EDISTRIBUTION
As face detection features large scale variations (from several pixels to thousand pixels), there exist two widely used scale augmentation strategies, random square crop (Zhang et al., 2017b; Deng et al., 2020b; Zhu et al., 2020) and data anchor sampling (Tang et al., 2018).
In the random square crop strategy, square patches are cropped from the original image with a random size between [0 . 3 , 1] of the short edge and then resized into 640 × 640 to generate larger training faces.
By contrast, data anchor sampling strategy aims to generate more small scale faces by down-sampling the original image, bringing a large amount of padded area.
Even though both random square crop and data anchor sampling can achieve promising results on the WIDER FACE dataset, the scale augmentation parameters are manually designed for all different network structures.
Therefore, the training sample distribution on the feature pyramids can be sub-optimal for a particular network structure.
To handle extreme scale variations in face detection, we also design a search-able zoom-in and zoom-out space, speciﬁed by the scale s i and probability p i .
The scale s i represents the zooming ratio, sampled from a discrete set S
= { s min , s min + 0 . 1 , · · · , s max − 0 .
1 , s max } .
For a particular training image in each iteration, square patches are cropped from the original images with a zooming ratio s i of the short edge of the original images.
If the square patch is larger than the original image, average RGB values will ﬁll the missing pixels.
To shrink the scale search space, we employ a binary probability set p i ∈ { 0 , 1 } .
Under this setting, the probability-based scale search is simpliﬁed into a discrete scale sampling from a ﬁxed set.
As the interval of the discrete scale set is only 0 .
1 , adjacent scales will have the probability of 1 . 0 to approximate a higher probability around a particular scale.
In this paper, we employ random search under the estimation metric of AP on WIDER FACE to construct the best scale augmentation set.
More speciﬁcally, we set s min = 0 .
1 and s max = 3 . 0 .
Then, we randomly select 8 to 20 discrete scale values to construct each scale augmentation set and train CRFD models under 320 different scale augmentation sets.
Finally, the scale augmentation set with the highest detection performance is selected for optimized scale augmentation.
E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Training.
For the scale augmentation, square patches are cropped from the original images with a random size from a pre-deﬁned scale set, and then these patches are resized to 640 × 640 for training.
Besides scale augmentation, the training data are also augmented by color distortion and random horizontal ﬂipping, with a probability of 0 . 5 .
For the anchor setting, we tile anchors of { 16 , 32 } , { 64 , 128 } , and { 256 , 512 } on the feature maps of stride 8 , 16 , and 32 , respectively.
The anchor ratio is set as 1 . 0 .
In this paper, we employ Adaptive Training Sample Selection (ATSS) [Table] (Zhang et al., 2020b) for positive anchor matching.
In the detection head, weight sharing and Group
Normalization (Wu & He, 2018) are used.
The losses of classiﬁcation and regression branches are
Generalized Focal Loss (GFL) (Li et al., 2020) and DIoU loss (Zheng et al., 2020), respectively.
Our experiments are implemented in PyTorch, based on the open-source MMDetection (Chen et al., 2019a).
We adopt the SGD optimizer (momentum 0.9, weight decay 5e-4) with a batch size of 8 ×and train on eight Tesla V100.
The learning rate is linearly warmed up to 0 .
015 within the ﬁrstepochs.
During network search, the learning rate is multiplied by 0 .
1 at the 55 -th, and 68 -th epochs.
The learning process terminates on the 80 -th epoch.
For training of both baselines and searched conﬁgurations, the learning rate decays by a factor of 10 at the 440 -th and 544 -th epochs, and the learning process terminates at the 640 -th epoch.
All the models are trained from scratch without any pre-training.
Testing.
For fair comparisons with other methods, we employ three testing strategies, including single-scale VGA resolution ( 640 × 480 ), single-scale original resolution, and multi-scale testing.
The results of DSFD (Li et al., 2019), RetinaFace (Deng et al., 2020b), TinaFace (Zhu et al., 2020),
Faceboxes (Zhang et al., 2017a), libfacedetection (Feng et al., 2021) and LFFD (He et al., 2019b) are reported by testing the released models, while the HAMBox (Liu et al., 2020) and BFBox (Liu & Tang, 2020) models are shared from the author.
4. A BLATION S TUDY
In Tab. 1, we present the performance of models on the WIDER FACE dataset by gradually including the proposed computation and sample redistribution methods.
Our manually-designed baseline model, ResNet-2.5GF, gets APs of 91 .
49% , and 67 .
32% under three validation scenarios.
Computation redistribution.
After separately employing the proposed computation redistribution on the backbone and the whole detector, the AP on the hard set improves to 69 .
78% and 70 .
This indicates that (1) the network structure directly inherited from the classiﬁcation task is suboptimal for the face detection task, and (2) joint computation reallocation on the backbone, neck and head outperforms computation optimization applied only on the backbone.
Furthermore, the proposed two-step computation redistribution strategy achieves AP of 71 .
37% , surpassing one-step computation reallocation on the whole detector by 0 .
As we shrink the whole search space by the proposed two-step strategy and our random model sampling number is ﬁxed at 320 , the twostep method is possible to ﬁnd better network conﬁgurations from the large search space.
In Tab.
1, we also compare our method with the single path one-shot NAS method (BFBox (Liu & Tang, 2020)) and the evolutionary search method (Appendix A.2), under the constraint of 2.5 GFlops.
BFBox aims to design a face-appropriate search space by combing some excellent block designs, such as bottleneck block, densenet block and shufﬂenet block.
However, such a combination generates a complex and redundant search space, which inevitably involves a vast body of low-performance candidate architectures.
The evolutionary approach iteratively adopts mutations and crossover to gradually generate better architecture candidates from the randomly initialized search space, which also contains a large number of under-performing architectures.
By contrast, CRFD-2.5GF utilizes an empirical bootstrap to estimate the optimized computation distribution of the best-performing architecture candidates, which directly eliminates the low-quality architectures from the initialized search
[Table]
space. Therefore
[Table]
CRFD-2.5GF can obviously outperform the BFBox and evolutionary method by 1 .
96% and 1 .
75% on the hard track
[Table]
Sample redistribution.
For scale augmentation, we ﬁrst manually extend the default scale set { 0 . 3 , 0 . 45 , 0 .
by adding larger scales { 1 . 2 , 1 . 4 , 1 . 6 , 1 . 8 , 2 . 0 } .
By adding this hand-crafted sample redistribution, the hard set APs signiﬁcantly increase by 7 . 15% for the baseline and 6 . 5% for the proposed CRFD , indicating the beneﬁt from allocating more training samples on the feature map of stride 8 .
By employing the optimized scale augmentation from searching, the hard set AP further increases by 0 .
48% for the proposed CRFD .
For SCRFD-2.5GF , the best scale augmentation set searched is { 0 . 5 , 0 . 7 , 0 . 8 , 1 . 0 , 1 . 1 , 1 . 2 , 1 . 4 , 1 . 5 , 1 . 8 , 2 . 0 , 2 . 3 , 2 . 6 } .
As we can see from these discrete scales, faces around the original scale are preferred for training, along with an appropriate probability and ratio of zooming-out.
4. C OMPUTATION R EDISTRIBUTION ACROSS D IFFERENT C OMPUTE R EGIMES
Besides the complexity constraint of 2.5 GFlops, we also utilize the same two-step computation redistribution method to explore the network structure optimization for higher compute regimes ( e.g . 10 GFlops and 34 GFlops) and lower compute regimes ( e.g . 0.5 GFlops and 1.0 GFlops).
In
Fig.
5, we show the computation redistribution and the optimized network structures under different computation constraints.
Our ﬁnal architectures have almost the same ﬂops as the baseline networks.
From these redistribution results, we can draw the following conclusions: (1) more computation is allocated in the backbone and the computation on the neck and head is compressed; (2) more capacity is reallocated in shallow stages due to the speciﬁc scale distribution on WIDER FACE; (3) for the high compute regime ( e.g . 34 GFlops), the explored structure utilizes the bottleneck residual block and we observe signiﬁcant depth scaling, instead of width scaling in shallow stages.
Scaling the width is subject to over-ﬁtting due to the larger increase in parameters (Bello et al., 2021).
By contrast, scaling the depth, especially in the earlier layers, introduces fewer parameters compared to scaling the width; (4) for the mobile regime (0.5 GFlops), allocating the limited capacity in the deep stage ( e.g . C5) for the discriminative features captured in the deep stage, can beneﬁt the shallow small face detection by the top-down neck pathway.
[Table]
4. A CCURACY AND E FFICIENCY C OMPARISONS ON WIDER FACE
As shown in Tab. 2 and Tab. 3, we compared the proposed SCRFD with other state-of-the-art face detection algorithms ( e.g . DSFD (Li et al., 2019), RetinaFace
(Deng et al., 2020b), BFBox (Liu &
Tang, 2020), HAMBox (Liu et al., 2020) and TinaFace (Zhu et al., 2020)) as well as light-weight face methods (
e.g . Faceboxes (Zhang et al., 2017a), libfacedetection (Feng et al., 2021) and
LFFD
(He et al., 2019b)).
Overall, all of the proposed SCRFD models provide considerable improvements compared to the hand-crafted baseline models ( e.g . ResNet-2.5GF and MobileNet-0.5GF), by optimizing the network structure as well as the scale augmentation, across a wide range of compute regimes.
When we ﬁx the testing scale at 640 as in Tab. 2, the proposed SCRFD-34GF outperforms all these state-of-the-art methods on the three subsets, especially for the hard track, which contains a large number of tiny faces.
More speciﬁcally, SCRFD-34GF surpasses TinaFace by 4 .
78% while being more than 3 × faster on GPUs.
In addition, the computation cost of SCRFD-34GF is only around 20% of TinaFace.
As SCRFD-34GF scales the depth in the earlier layers, it also introduces fewer parameters, resulting in a much smaller model size ( 9 . 80 M ).
Compared to the hand-crafted baseline (ResNet-34GF), the proposed computation redistribution and sample redistribution improve the
AP by 1 . 27% and 0 .
92% , indicating the superiority of SCRFD over manual designs.
Compared to the single path one-shot NAS method, SCRFD-34GF outperforms BFBox by 15 .
81% , while using a more compact model size.
As the search space of BFBox is complex, there exists a large number of low-performance architectures.
In addition, BFBox only searches the backbone and neck without considering the optimization on the head.
For multi-scale testing, SCRFD-34GF slightly outperforms TinaFace but consumes much less computation.
For the low-compute regimes in Tab. 3,
SCRFD-0.5GF signiﬁcantly outperforms RetinaFace-MobileNet0.25 by 21 .
19% on the hard AP, while consuming only 63 .
34% computation and 45 .
57% inference time under the VGA resolution.
When the evaluation is conducted on the original image, SCRFD-0.5GF surpasses LFFD by 3 . 6% on the hard AP, while consuming only 5 .
5% ﬂops.
C ONCLUSIONS
In this work, we present a sample and computation redistribution paradigm for efﬁcient face detection.
Our results show signiﬁcantly improved accuracy and efﬁciency trade-off by the proposed
SCRFD across a wide range of compute regimes, when compared to the current state-of-the-art.
Acknowledgements.
We would like to thank Hui Ni from Tencent for preparing the mobile demo of SCRFD
Stefanos Zafeiriou acknowledges support from the EPSRC Fellowship DEFORM (EP/S010203/1), FACER2VM (EP/N007743/1) and a Google Faculty Fellowship.
A A PPENDIX A. T INA F ACE R EVISITED
Based on RetinaNet (Lin et al., 2017a), TinaFace (Zhu et al., 2020) employs ResNet-50 (He et al., 2016) as backbone, and Feature Pyramid Network (FPN) (Lin et al., 2017a) as neck to construct the feature extractor.
For the head design, TinaFace ﬁrst uses a feature enhancement module on each feature pyramid to learn surrounding context through different receptive ﬁelds in the inception block (Szegedy et al., 2015).
Then, four consecutive 3 × 3 convolutional layers are appended on each feature pyramid.
Focal loss (Lin et al., 2017b) is used for the classiﬁcation branch, DIoU loss (Zheng et al., 2020) for the box regression branch and cross-entropy loss for the IoU prediction branch.
To detect tiny faces, TinaFace tiles anchors of three different scales, over each level of the FPN ( i.e . { 2 4 / 3 , 2 5 / 3 , 2 6 / 3 } × { 4 , 8 , 16 , 32 , 64 , 128 } , from level P 2 to P 7 ).
The aspect ratio is set as 1 . 3 .
During training, square patches are cropped from the original image and resized to 640 × 640 , using a scaling factor randomly sampled from [0 . 3 , 0 . 45 , 0 . 6 , 0 . 8 , 1 . 0] , multiplied by the length of the original image’s short edge.
During testing, TinaFace employs single scale testing, when the short and long edges of the image do not surpass [1100 , 1650] .
Otherwise, it employs with short edge scaling at [500 , 800 , 1100 , 1400 , 1700] , shift with directions [(0 , 0) , (0 , 1) , (1 , 0) , (1 , 1)] and horizontal ﬂip.
As shown in Fig. 6(a) and Tab. 4, we compare the performance of TinaFace under different testing scales.
For the multi-scale testing, TinaFace achieves an impressive AP of 93 .
4% , which is the current best performance on the WIDER FACE leader-board.
For large single-scale testing ( 1650 ), the AP slightly drops at 93 .
but the computation signiﬁcantly decreases to 1021 .
82 GFlops.
On the original scale ( 1024 ), the performance of TinaFace is still very high, obtaining an AP of 91 .
4% with 508 .
47 GFlops.
Moreover, when the testing scale decreases to VGA level ( 640 ), the AP signiﬁcantly reduces to 81 .
4% , with the computation further decreasing at 172 .
95 GFlops.
In Fig. 6(b), we illustrate the computation distribution of TinaFace on the backbone, neck and head components with a testing scale of 640 .
From the view of different scales of the feature pyramid, the majority of the computational costs (about 68% ) are from stride 4 , as the resolution of feature map is quite large ( 120 × 160 ).
From the view of the different components of the face detector, most of the computational costs (about 79% ) are from the head, since the backbone structure is directly borrowed from the ImageNet classiﬁcation task (Deng et al., 2009), without any modiﬁcation.
Even though TinaFace achieves state-of-the-art performance on tiny face detection, the heavy computational cost renders it unsuitable for real-time applications.
A. D ETAILS OF E VOLUTIONARY B ASELINE
To compare the proposed SCRFD with the other network search methods in Tab. 1, we design the evolutionary baseline (Real et al., 2019) as follows:
A population of networks P are randomly initialized.
We set | P | = 50 . 2.
Each network architecture from P is trained on the WIDER FACE training data and then the APs on the WIDER FACE validation dataset are tested.
Architectures with top performance are selected from P as parents P .
To generate child networks C , we employ the mutation and crossover policies.
Here, we set |P| = 10 and | C | = 50 . 4.
Each network architecture from C is trained on the WIDER FACE training data and then the APs on the WIDER FACE validation dataset are calculated.
The worst 50 individuals from the populations of P ∪ C are dropped and then we get the new evolutionary population P .
We repeat steps 3, 4 and 5 for 20 times, resulting in 1000 network architectures as well as their validation APs.
The architecture with the highest AP is selected as the ﬁnal result.
[Figure]
A. A LGORITHM OF C OMPUTATION R EDISTRIBUTION
In Algorithm 1, we show the details of the proposed two-step computation redistribution method.
Algorithm 1: Search algorithm for
Input: Constraint of computation cost Y (in GFlops); Number of random network architectures N ; Dataset for training D train and validation D val ; Evaluation metric AP .
Output: Best architecture A ∗
Initialize the architecture set A = Ø while length( A ) <
N do net =
RandomSampling ( { d i , w i } ) ; /* d i and w i denote block number and channel number, i = 2 , 3 , 4 , 5 .
*/ if net.
Flops ≤ 1 .
02 ∗ Y
And net.
Flops ≥ 0 .
98 ∗ Y then A .Append(net) end end
ParallelTrain ( A , D train )
CR 1 =
Bootstrap ( A , APs ) | APs =
Evaluate ( A , D val )
Initialize the architecture set A = Ø while length( A ) <
N do net =
RandomSampling ( { CR 1 , n, m, h } ) ; /* n, m, and h denote channel in neck, block and channel in head.
*/ if net.
Flops ≤ 1 .
02 ∗ Y
And net.
Flops ≥ 0 .
98 ∗ Y then A .Append(net) end end
ParallelTrain ( A , D train )
CR 2 =
Bootstrap ( A , APs ) | APs =
Evaluate ( A , D val )
Output: Best architecture A ∗ = choose top 1( CR 2 , A ) .
A. D ETAILED N ETWORK C ONFIGURATIONS
In Tab. 5, we give the detailed network conﬁgurations for baselines and the proposed CRFD across different compute regimes.
A. S TATISTICS AFTER S AMPLE R EDISTRIBUTION
As illustrated in Fig. 7(a), there are more faces below the scale of 32 after the proposed automatic scale augmentation strategy is used.
Moreover, even though there will be more extremely tiny faces ( e.g . < 4 × 4 ) under the proposed scale augmentation, these ground-truth faces will be neglected during training due to unsuccessful anchor matching.
As shown in Fig. 7(b), positive anchors within one epoch signiﬁcantly increase at the scale of 16 and 32 .
With more training samples redistributed to the small scale, the branch to detect tiny faces can be trained more adequately.
[Table] [Table]
[Figure] [Figure]
[Table] [Table]
[Figure] [Figure] [Figure]
T r u e p o s i t i v e
[Figure]
t
[Figure] [Figure]
WIDER FACE The WIDER FACE dataset (Yang et al., 2016) consists of 32 , 203 images and 393 , 703 face bounding boxes with a high degree of variability in scale, pose, expression, occlusion and illumination.
The WIDER FACE dataset is split into training ( 40% ), validation ( 10% ) and testing ( 50% ) subsets by randomly sampling from 61 scene categories.
Based on the detection rate of EdgeBox (Zitnick & Doll ´ ar, 2014), three levels of difﬁculty ( i.e . Easy, Medium and Hard) are deﬁned by incrementally incorporating hard samples.
AFW The AFW dataset (Zhu & Ramanan, 2012) contains 205 high-resolution images withfaces (Mathias et al., 2014) collected from Flickr.
Images in this dataset contain cluttered backgrounds with large variations in viewpoint.
PASCAL The PASCAL face dataset (Mathias et al., 2014) is collected from the PASCALperson layout subset, includes 1 , 335 labeled faces in 851 images with large facial appearance and pose variations ( e.g . large in-plane rotation).
FDDB The FDDB dataset (Jain & Learned-Miller, 2010) is a collection of labeled faces from Faces in the Wild dataset.
It contains a total of 5 , 171 face annotations on 2 , 845 images.
The dataset incorporates a range of challenges, including difﬁcult pose angles, out-of-focus faces and low-resolution.
A. C ROSS D ATASET E VALUATION AND V ISUALIZATION
Besides the evaluation on the WIDER FACE (Yang et al., 2016) data set, we also conduct cross dataset evaluation and test the proposed SCRFD models on AFW (Zhu & Ramanan, 2012), PASCAL (Mathias et al., 2014) and FDDB (Jain & Learned-Miller, 2010), under the VGA resolution.
As shown in Fig 8, SCRFD-34GF achieves 99 .
945% AP on AFW, 99 .
597% AP on PASCAL, and 99 .
25% on FDDB, surpassing BFBox (Liu & Tang, 2020) and
HAMBox (Liu et al., 2020).
Even though the face scale distributions on these three datasets are different from WIDER FACE, the proposed SCRFD-34GF still obtains state-of-the-art performance across different datasets, showing impressive robustness of the proposed computation and sample redistribution approaches.
In addition, SCRFD-2.5GF also obtains impressive performance on different datasets with much lower computation cost ( 99 . 821% AP on AFW, 98 .
911% AP on PASCAL, and 99 .
02% AP on FDDB).
Fig.
9 shows qualitative results generated by SCRFD-2.5GF .
As can be seen, our face detector works very well in both indoor and outdoor crowded scenes under different conditions ( e.g . appearance variations from pose, occlusion and illumination).
The impressive performance across a wide range of scales indicate that SCRFD-2.5GF has a very high recall and can detect faces accurately even without large scale testing.
[Figure]