Cluster and Aggregate: Face Recognition with Large Probe Set
Anonymous Author(s)
AffiliationAddress email
Abstract
Feature fusion plays a crucial role in unconstrained face recognition where inputs(probes) comprise of a set of N low quality images whose individual qualitiesvary.
Advances in attention and recurrent modules have led to feature fusion thatcan model the relationship among the images in the input set.
However, attentionmechanisms cannot scale to large N due to their quadratic complexity and recurrentmodules suffer from input order sensitivity.
We propose a two-stage feature fusionparadigm, Cluster and Aggregate , that can both scale to large N and maintainthe ability to perform sequential inference with order invariance.
Specifically,
Cluster stage is a linear assignment of N inputs to M global cluster centers, and
Aggregation stage is a fusion over M clustered features.
The clustered features playan integral role when the inputs are sequential as they can serve as a summarizationof past features.
By leveraging the order-invariance of incremental averagingoperation, we design an update rule that achieves batch-order invariance, whichguarantees that the contributions of early image in the sequence do not diminish astime steps increase.
Experiments on IJB-B and IJB-S benchmark datasets show thesuperiority of the proposed two-stage paradigm in unconstrained face recognition.
Introduction
Face Recognition (FR) matches a set of input query imagery, known as probe , to enrolled identitydatabase, known as gallery . Verification is to confirm the claimed probe’s identity and identificationis to identify the unknown probe’s identity by searching a known database [41].
In either case, aprobe can go beyond an image and include a set of images, videos or their combinations [4]. Thus
FR involves fusing features of multiple images or videos to create a discriminative feature for a probe. Due to the interest in unconstrained surveillance scenarios e.g. IJB-S [21] the role of fusion isbecoming more important.
Unconstrained FR is often based on probes from low quality images andvideos.
It is challenging due to two issues: 1) individual video frames can be of poor quality, causingerroneous FR model prediction and 2) the number of images in a probe can be very large, e.g. , aprobe video in IJB-S may have 500 , 000 frames.
Feature fusion across all frames in the probe isespecially crucial if frame-based predictions are unreliable.
While prior works [23,36] address thefirst issue of prediction in low quality images, the large size of probe set was not addressed.
Fig.
1a) illustrates the problem caused by the absence of proper feature fusion.
The contribution of goodquality image can be made insignificant in the presence of many other poor quality images in the set.
This paper aims to learn a fusion function that maps an unordered set of N probe features { f i } Nof the same person to a single fused output f . Note that f i =
E ( x i ) is the feature extracted fromthe i -th sample in the set, using a fixed feature extractor E .
The task of fusing multiple featuresinvolves 1) estimating the quality of individual features and 2) modeling the intra-set relationship ofthe features.
Prior feature fusion works utilize either simple average pooling [10,39], reinforcementlearning [29], recurrent models [15] or self-attention [14,28,31,49,53].
IJB-A (9)
IJB-B (21)
IJB-C (23)
IJB-S (7114)
Naïve PFE CFAN RSA CAFace
Contribution of badquality images
Contribution of goodquality images
Unconstrained Setting (low quality + multiple frames) …
E
E E
E E
E E
E …
Intra-Set Relation Large N Probe
Order Invariance
Single Image
Intra-Set Attention
Recurrent Module
Proposed c) Benefit of feature fusion with large N >>
[Figure]
[Figure]
of prior works
Proposed ( ⋅ ) : avg probe size % inc. over Naïve
Typically to compute the intra-set relationship among inputs of an arbitrary size N , one would adoptset-to-set functions such as Multihead Self Attention (MSA) [28,47,49], enabling inputs to propagateinformation among themselves.
The downside of this approach is its computational cost of O ( N 2 )which becomes infeasible when N exceeds a few thousand.
Also, when the inputs are sequential as ina live video feed, it is nontrivial to model the intra-set relationship except to compute attention overall past frames at each time step.
Recurrent methods [15,18] are useful in the sequential inferencebut their drawback is set order inconsistency, i.e. , as the number of sequential steps T increases, thecontribution of early frames in a set decreases.
Fig.
1 b) contrasts various fusion methods.
A feature fusion framework that can consider both 1) intra-set relationship for a large N and 2)efficient sequential inference is necessary in the real-world unconstrained FR scenarios.
Fig.
1 c)shows the average probe sizes of four datasets.
IJB-S [21]’s probe size is too large for intra-setattention such as RSA [28] to perform inference with all frames concurrently.
We present a feature fusion framework, Cluster and Aggregate (CAFace), that achieves two abovementioned criteria.
It consists of two modules: Cluster Network (CN) and Aggregation Network (AGN).
CN makes soft assignments of N features into M fixed number of clusters, i.e. , { f i } N → { f ′ j } Mwhere M << N .
While N varies from one set to other, M is fixed.
AGN combines M clusteredfeatures into a single feature f , i.e. , { f ′ j } M → f .
Conceptually, M intermediate cluster featuresserve as a summarization of N inputs and AGN models the intra-set relationship among { f ′ j } M .
The proposed framework depends on learning global cluster assignments { f i } N → { f ′ j } M that areconsistent across different probes.
Thus, we propose learning shared cluster centers that are inputindependent .
These centers govern the clustering assignments.
But, it is not obvious which clusteringcriterion is the best for feature fusion.
Thus, we design CN to discover learned clusters with anend-to-end differentiable framework that allows AGN to back-propagate the gradients to CN. Thecluster assignments are learned to maximize the FR performance.
We also design an input pipeline, Style Input Maker (SIM) that can helps CN perform class (identity) agnostic clustering efficiently.
The purpose of introducing an intermediate stage { f ′ j } M is to facilitate the sequential inference.
Thekey design of CN is to formulate { f ′ j } M as a linear combination of { f i } N .
This guarantees that evenwhen the input sequence of set length N is divided into T smaller batches of set length N ′ , { f ′ j } Mcan be sequentially updated with batch-order invariance.
This is due to our update rule, inspired bythe order invariance property of the averaging operation, as in Eq.
When the inputs are sequential,we feed only the new features to CN and update the cached { f ′ j } M .
It achieves the similar effect ashaving used all previous features simultaneously.
Fig.
2 shows the contrast with previous approaches.
For readability, we will interchange the set notation { f i } N with the matrix notation F ∈ R N × C .
In summary, the contributions of this paper include:
• A novel feature fusion framework for both large N feature fusion and efficient sequential inference.
To our knowledge, this is the first approach to utilize linearly combinedintermediate clusters to achieve batch-order invariance with intra-set relationship modeling.
Weighted average
Global Centers 𝑪 ∈ ℝ !
×
# 𝑭 𝒕𝟏 ∈ ℝ & ! ×’ 𝑭 𝒕𝟐 ∈ ℝ & !
×’ 𝑭 𝒕𝟏) ∈ ℝ ! ×’ 𝑭 𝒕𝟐) ∈ ℝ !
×’
UpdateRule 𝑭 𝒕𝟏𝟐) ∈ ℝ !
×’ 𝒇 ∈ ℝ *×’ 𝑭 𝒕𝟑 ∈ ℝ & !
×’ UpdateRule 𝑭 𝒕𝟏𝟐𝟑) ∈ ℝ !
×’ 𝑭 𝒕𝟑) ∈ ℝ ! ×’ 𝒇 ∈ ℝ *×’ 𝒇 ∈ ℝ *×’ 𝑭 ∈ ℝ &×’
[Figure]
AggregationNetwork AggregationNetwork
[Figure]
• An task-driven clustering mechanism that can discover latent clustering centers that maximize the task performance.
In our case, the task is FR.
We achieve the task-driven clustering
with an assignment algorithm using the global query and decoupled key and value structure.
• We show the superiority of CAFace in unconstrained face recognition on multiple datasets.
Related Work
Feature Fusion (Unordered Set).
The simplest way of feature fusion is to average over a set offeatures { f i } N [10,39].
In this case, the features with larger norms play a bigger role, and it generallyworks since easy samples tend to show larger norms [36,40].
To learn the weights, CFAN and QANutilize the self-attention mechanism, a learned weighted averaging mechanism [14,31].
The drawbackof these approaches is the lack of an intra-set relationship during the weight calculation process.
Previous works that adopt the intra-set attention mechanism are Non-local Neural network and
RSA [28,49].
These works use intermediate feature maps U i of size R C M × H × W during aggregationbecause feature maps provide rich and complementary information that can be refined by taking thespatial relationship into account.
However, the drawback is in the heavy computation in the attentioncalculation.
For a set of N features maps, an attention module involves making ( N × H × W ) 2 sizedaffinity map.
Our Cluster Network utilizes a compact style vector from SIM and makes N 2 sizedaffinity map which greatly increases the computation efficiency in attention computation.
DAC [29] and MARN [15] propose RL-based and RNN-based quality estimators, respectively.
Yet,they fail to be agnostic to input stream order and are unsuitable for modeling long-range dependencies.
Our method can split the N inputs into T smaller batches and still achieve batch-order invariance.
Video Recognition (Ordered Set).
The feature fusion for recognition has a resemblance to video- based recognition [30], but set inputs cannot always expect the temporal dependencies to be available.
Therefore, most video-based approaches for tasks such as action recognition or quality enhancement [2,3,24,33,38,56] focus on exploiting the relationship between nearby frames, whereas feature fusion approaches do not define them.
In video-based FR, the general trend is to focus more on assessing the quality of individual frames as opposed to exploring the relationship among nearby frames.
Some examples of video-based FR utilize n-order statistics [34], affine hulls [8,19,54], SPD matrices [20] and manifolds [17,48].
Recently, probabilistic representation such as PFE [43] gained popularity[1,9,42,43] since the variance in distribution serves as a quality estimation for individual frames.
Attention Mechanism.
Multihead Self Attention (MSA) [47] is a widely adopted set-to-set function that models intra-set relationships via an affinity map.
It is also a key component in transformer architectures which outperform CNNs in various vision tasks [7,11,13,27,32,46,55].
The underlyingmechanism of MSA which uses the affinity of query and key to update the value is versatile in itsapplication beyond recognition and has led to its usage in memory retrieval and grouping [5,44,52].
The unique property of the proposed Cluster Network is in the linear combination of value assignmentwhich enables batch-order invariance using an incremental average update rule.
Unlike MSA whichrequires concurrent inputs during inference for intra-set relationship, ours can split the inference andestablish a connection across batches without decreasing the contribution of early inputs. 𝑺 ! = 𝑨𝑺 [Figure] [Figure]
[Figure]
[Figure]
[Figure]
×
[Figure] 𝑁 ! images [Figure]
identity
Cluster Network (CN) 𝑸𝒖𝒆𝒓𝒚:𝑪 𝑲𝒆𝒚:𝑺 𝑽𝒂𝒍𝒖𝒆: (𝑺,𝑭)
Global Centers
Transformer+SoftMax 𝑪 ∈ ℝ &×$
Assignment map 𝑨
Aggregation Network (AGN) 𝑷 ∈ ℝ &×% ℝ &×$ 𝑭 !
=
𝑨𝑭
M×𝑁′ ℝ &×% MLPMIxer weight weightedaverage 𝒇: [Figure] (CAFace)
[Figure]
Proposed Approach
The Cluster and Aggregate paradigm seeks to divide the large N inference into partitioned inferenceswhile still obtaining the result as seeing all inputs at the same time. This can be achieved if 1) eachpartitioned inference can update the intermediate representation with necessary information and 2)the order of inference does not affect the final outcome, so the information in early batches is notforgotten. In essence, the intermediate representation serves as a communication channel acrossbatches.
We achieve this by designing a Cluster Network (CN) and Aggregation Network (AGN).
Fig.
2 c) shows the proposed paradigm. In this section, we will elaborate on how we obtain the globalassignment that is consistent across batches and how the update rule can be batch-order invariant.
Weformally layout a few assumptions for the Cluster and Aggregate paradigm in the face recognition(FR) task as shown in Fig. Let { x i } N be a set of N facial images from the same person.
The task is to produce a single featurevector f from { x i } N that is discriminative for the recognition task. We assume that a single imagebased pretrained face recognition model E : x i → f i is available following the settings of previousworks [14,28,43].
For readability, we will interchange the set notation { f i } N with the matrix notation F ∈ R N × C where { f i } N is the input is a set of length N and F simplifies equations.
For clarity, wedenote N to be the probe size (the number of images in a set) and N ′ to be the partitioned set sizewhen N is large.
During training, we fix the number of images for fusion as N ′ .
Note that the shapeof inputs during training would have one more dimension, training batch size B , i.e. F ∈ R B × N ′ × C . Training batch size refers to the number of persons sampled in a mini-batch, different from thenumber of images per person, N ′ . We drop the training batch size dimension in equations for brevity.
3.1 Architecture
Cluster Network (CN).
Cluster Network is responsible for mapping inputs
F ∈ R N ′ × C of variablesize N ′ to F ′ ∈ R M × C of a fixed size, M .
A natural choice for the architecture would be Transformer [13,47] as it is a set to set function.
However, there are two problems with it.
It cannothandle large inputs due to the quadratic complexity of MSA. 2) When the inputs are partitioned andinferred sequentially, the intra-set information across the batch is lost, as MSA computes the affinitywithin the given inputs. CN solves this problem by modifying Transformer with 1) shared queriesand 2) linear value mapping. These changes result in a clustering mechanism.
We first consider the following generic attention equation [47] with query Q , key K and value V .
[Equation]
where W q , W k , W v are learnable weights and d is the channel dimension of K .
The row-wise
Softmax ensures that the output is the weighted average of all projected values W v V for each query
[Figure]
𝑶𝒖𝒕 index.
We modify this to
[Equation]
First, unlike K and V which are inputs, the query is now a shared learnable parameter C initializedat the beginning of training.
Secondly, removing W v and the column-wise Softmax ensure that foreach query index of C , the output is the (soft) selection of values V .
These two modifications resultin a learned soft assignment mechanism where C serves as the global shared center.
We name theassignment map as A .
The difference between A from row and column SoftMax is shown in Fig. 4a) and b).
We then divide A by the weight of samples assigned to each center (row-sum of A ) as in
Cluster
[Equation]
( K , V
[Equation] = [Equation] [Equation]
Note that Cluster C ( K , V ) is linear in V , while the prediction of A is nonlinear. To further addthe nonlinearity of A to the Cluster Network, we first embed the keys K with shallow Transformerbefore clustering.
The combined result CN ( · ) is the learned soft assignment of values according tothe affinity between keys and global queries. More details on the architecture is provided in Supp. Style Input Maker (SIM).
So far, we have discussed the generic Cluster Network algorithm. Forface recognition, we still need to decide keys K and values V for feature fusion. It is clear that Vshould be F , the facial identity features, as it is what we are interested in merging. It is possible to use
F for K as well, but K should ideally contain useful information for fusion and be compatible withqueries which are the global center C .
However, f i is optimized to be invariant to any characteristicsother than the identity.
Thus it lacks input image style which encompasses various image traits suchas brightness, contrast, quality, pose, or a domain differences from the training data. In light of the success of using first and second-order feature statistics as an image style [22,26,37],we propose SIM for extracting style information using the intermediate representations of featureextractors. The benefit of modeling keys K in clustering with style over using identity is shown in Sec. 4.2.
We also ablate the benefit of further including feature norm || f i || in K as it is sometimesused to approximate the confidence of the prediction [23,36].
Let U i ∈
R C M × H × W be the intermediate feature.
We capture image style by a style vector γ {+i ∈ R 
+}
[Equation]
where Γ = [ µ sty , σ sty ] ,
[Equation]
A learnable matrix W s ∈ R C M × 2 controls the importance of µ sty and σ sty via element-wise multiplication ⊙ .
Simply put, SIM is a shallow network on spatial mean and standard deviation of U i .
Onecan take U i from more than one intermediate locations and in such a case, we concatenate them.
To verify whether the feature norm would further benefit the fusion process, we embed the featurenorm
|| f
i || 2 to a 64 -dim vector, following the convention of Sinusoidal conversion [47], which isanalogous to the position embeddings in ViT [13].
The norm embedding n i is a 64 -dim vector andthe details can be found in Supp.
B. Finally, the output SIM is the concatenation, s i =
[ γ i , n i ] wheres i ∈ R d and d = 64 + 64 = 128 .
For readability, we denote the set { s i } N ′ ∈ R N ′ × 128 as S .
In summary, we decouple style S and identity F and use S as keys to map
[Equation]
which are the intermediates that will be used for subsequent fusion in AGN or stored for sequentialinference.
We also map S to S ′ using the same assignment.
Fig.
3 shows the overall diagram.
Aggregation Network (AGN).
The Aggregation Network is responsible for fusing a fixed numberof M inputs, F ′ and S ′ into a single fused output f with intra-set relationship.
We adopt MLPMixer [45] as it can efficiently propagate information for the fixed-size input.
For interpretability, wepredict weights P ∈ R M × C that combines F ′ to f ∈ R C . Specifically, f = AGN ( S ′ , F ′ ) is
[Equation]
where [ S , C ] denotes the concatenation along the channel dimension.
The magnitude of P is aninterpretable quantity showing the importance of each cluster during fusion.
The final output f is aweighted average of F ′ whose weight is P .
The details of MLP-Mixer [45] can be found in Supp.
Sequential Inference.
A key characteristic of CAFace is its ability to divide the inputs into T -stepsequential inference of smaller set length N ′ when N is large, and still achieve similar results as theconcurrent inference. It is possible as the intermediates F ′ and S ′ are linear combinations of F , Srespectively, although estimating the combination weights A is non-linear. This allows us to formulatethe update rule as the incremental weighted average whose innate property is order-invariant.
Consider partitioned inputs F 1 , ..., F T , with corresponding predicted weights A 1 , ..., A T .
Since bydefinition (Eq. 5), F ′ t = A t F t / (cid:80) N ′ j A t, ( i,j ) , we can write the cumulative intermediate, (cid:99) F T ′ as
[Equation]
This formulation requires storing all inputs of timestep 1 , ..., T .
We can easily convert this to
[Equation]
, where
[Equation]
which requires storing only the cumulative row-summed assignment map a T − 1 and a cumulativeintermediate (cid:98) F ′ T − 1 of the previous time-step.
The same logic applies to S ′ as well.
Note that thisoperation, by design, is invariant to inference order (batch-order) as the final result will always bethe total weighted average. However, we do not obtain element-wise permutation invarinace asthe prediction of A t will change with different inputs.
We test the susceptibility to element-wisepermutation in Sec.
4.3 and it has minimal impact on the overall performance.
3.2 Loss Function
Template Loss.
The objective is to make the fused output f be close to the ground truth class centerf GT .
The task can be viewed as correctly inferring the true class center in the presence of low qualityfeatures { f i } N .
Here f GT is dependent on the pretrained feature extractor E and can be either takenfrom the last FC layer of E or computed using per-subject average of the embeddings f i in thetraining data. Our loss function can be viewed as the cosine distance version of the Center loss [50].
In training, we randomly sample B number of subjects and N ′ images per subject. Let the superscriptin F ( b ) denote the b -th subject. The loss to increase the cosine similarity is,
[Equation]
Set Permutation Consistency Loss.
The Cluster And Aggregate paradigm achieves batch-wise orderinvariance by formulation, but it does not achieve element-wise permutation invariance, as noted in
Sec. 3.1.
Therefore, we explore the element-wise permutation’s added benefit using an additional lossfunction.
Thus, our final loss is L = λ
c L t + λ p L p , and the set permutation consistency loss L p is
[Equation]
It lets the splitted inference outcome similar to the concurrent inference.
Sec. 4.2 shows the benefit of L p but it is small, meaning the batch-order invariance from model design is already powerful.
[Figure]
[Figure]
Experiments
4.1 Datasets and Implementation Details
We use WebFace4M [57] as our training dataset.
It is a large-scale dataset with 4 .
2 M facial imagesfrom 205 , 990 identities.
The single image based pretrained face recognition model E has beentrained with the whole training dataset.
To train the aggregation module, we use its randomlysampled subset, consisting of 813 , 482 images from 10 , 000 identities.
We do not use VGG-2 [6] or MS1MV2 [12,16] as they were withheld by their distributors due to privacy and other issues.
For the pretrained face recognition model E , we use the IResNet-101, trained with ArcFace loss [12].
Since the performance of the aggregation depends on the quality of E , we set E to be the same forall experiments.
E produces an embedding vector f i ∈ R 512 for each image.
To offer variations inthe training data features, we randomly augment the dataset with cropping, blurring, and photometricaugmentations.
The training hyper-parameters such as optimizers are detailed in Supp.
A.
We test on IJB-B [51], IJB-C [35] and IJB-S [21] datasets.
IJB-B is a widely used FR test setcontaining both high-quality images and low-quality videos of celebrities (see Fig.
5 for examples).
IJB-C is an updated version of IJB-B with more complex motions in the video.
IJB-S is a surveillancevideo dataset, benchmarking extremely low-quality image/video face recognition.
The probe andgallery set size can exceed 500 , 000 .
Within the set, there are many low-quality images, making IJB-Svery challenging and suitable for measuring the feature fusion framework (see Fig.
6 for examples).
For IJB-S, we use protocols, Surv.-to-Single, Surv.-to-Booking and Surv.-to-Surv.
The first/secondword in the protocol refers to the probe/gallery image source.
Surv. is the surveillance video, ‘Single’is the frontal high-quality enrollment image and ‘Booking’ refers to the 7 high-quality enrollmentimages.
For the ablation study, Sec. 4.2, we report the average of all 9 metrics listed in Tab. 3.
4.2 Ablation and Analysis
Effect of Different Style Input.
In Tab. 1, we ablate the efficacy of various components in SIM whichprepares the input for CN.
The table shows that using f i for clustering is harmful to performance and [Table] increases the no. of parameters for CN.
It also shows that using the additional norm embedding n ialong with s i produces the best results for IJB-B and IJB-S datasets.
However, the margin is small,so simply using s i would suffice for the setting that requires faster computation.
Effect of L p .
As noted in Sec.
3.2, we further propose to constrain the set permutation consistencywith the additional loss L p .
The ablation between λ p = 0 and λ p = 1 is shown in Tab. 1.
Effect of Number of Clusters.
In Tab. 1, the effect of the number of clusters M is shown.
The IJB-Sperformance peaks when M = 4 .
When M = 2 , the summary representations F ′ and S ′ have onlytwo assignment options where one cluster takes the poor quality images with low weights.
The behavior could be interpreted as performing an outlier detection, which is powerful enough to give highperformance.
When M > 2 , F ′ and S ′ have the capacity to store richer history of previous frameswhich would be beneficial in sequential inference.
IJB-S has large N probes which require dividing theinference into batches.
Higher IJB-S performance when M = 4 indicates that the freedom to assignsamples to different clusters is important in the sequential setting. A similar phenomenon is observed96.
N ' = 4 N ' = N ' = N ' =
N ' =
N ' = N ' =
N
M= M=
M=
[Figure]
in Fig.
The performance gap widens for M = 2 as we reduce the batch size to make more sequential steps in inference.
Weight Visualizations.
An example of clustering assignmentswhen M = 4 could be viewed in Figs.
5 and 6.
Fig.
5 showshow samples are soft-assigned to different clusters along withthe weight estimation.
The cluster weight is calculated byaveraging along 512 dimensions of P ∈ R M × 512 .
Note thateach column sums to 1 but F ′ and S ′ are calculated by averaging each row.
Thus, the relative contribution of samples ineach row is important.
Fig.
6 shows the actual contribution ofeach sample during fusion.
The contribution can be calculatedby multiplying the magnitudes of A and P .
Note that in thepresence of many poor quality images, selecting a few goodones is very important and the sample weight of our methodcan effectively select a subset of samples during fusion.
4.3 Comparison with SoTA methods
To compare with prior feature aggregation methods, we use the same feature extractor E as in Sec.
4.1,for a fair comparison.
Average is the conventional embedding f i averaging scheme that is adoptedin the absence of a learned aggregation model.
It is equivalent to the stand-alone ArcFace modelperformance.
The rest of the methods learn an additional network for fusing the set of features.
In Tab. 2, we show the performance of various feature fusion methods on IJB-B. CAFace achieves alarge performance gain in all TAR@FAR metrics.
CFAN [14] and PFE [43] do not use any intra-setrelationship, as they learn to predict the confidence of a single image.
RSA [28] calculates intra-setattention of feature maps, which is computationally costly and incapable of sequential inference.
CAFace obtains the best results with the least number of parameters.
The results for IJB-C show asimilar trend and are included in Supp, along with comparison of required computation resources.
In Tab. 3, we compare feature aggregation models in the IJB-S dataset that has large N low qualityimages/videos in probes.
RSA [28] cannot load all images in the probes concurrently for large N .
Asan alternative, we divide the probes into a manageable size of N ′ = 256 and average the results. Since
RSA does not have a sequential update mechanism, dividing large N probes reduces the performance, [Table] [Table] which shows why the sequential capacity is important.
CAFace also divides the probes image intobatch size of N ′ = 256 images yet achieves a large margin improvement in IJB-S.
It shows that ourtwo-stage mechanism is very effective in the large N setting.
Particularly, the performance gain inthe hardest protocol, Surveillance to Surveillance, is the largest.
We also randomly shuffle imageswithin the probes 5 times and measure the mean and std.
of the performance in the last row.
Theresult shows that our model is robust to input ordering.
We also include an experiment on a highquality image dataset, IJB-A [25], in Supp, and note that the performance gain with feature fusion isnegligible.
As noted in Fig.
1 c), the improvement over the baseline (averaging) goes up with theincreased number of images in the probe, which highlights the importance of large N scalability.
4.4 Computation Efficiency
Since CAFace is build on top of a single image feature extractor E , we show the relative FPS of
CAFace with respect to the FPS of E in Tab. 2.
We compute FPS with the batchsize of 256 . It showsthat the single image based quality estimation methods, PFE and CFAN are the fastest. And RSAwith intra-set attention is the slowest. CAFace can achieve a relatively good speed and obtain the bestperformance.
More analysis on the speed and resource consumption can be found in the Supp.
D.  Conclusions We address the two problems arising from the feature fusion of large N inputs, a common scenario inunconstrained FR.
With large N features, modeling intra-set relationships with attention mechanismsare prohibitive due to computational constraints while sequential inference suffers from the reducedcontribution of early frames.
In this work, we explore the possibility of dividing N inputs into Tsmaller unordered batches whose inference result is the same as concurrent N inference.
To thisend, we introduce a two-stage cluster and aggregate paradigm.
The clustering stage, inspired byorder-invariance of incremental mean operation, is designed to linearly combine N inputs to M globalcluster centers, whose assignment is invariant to the batch-order.
The aggregation stage efficientlyproduces a fused output from M clustered features, while utilizing the intra-set relationship.
We showour proposed CAFace outperforms baselines on unconstrained face datasets such as IJB-B and IJB-S.
Limitations.
Cluster and Aggregate is a feature fusion framework that learns the weights of individualinputs, given a fixed feature extractor E .
Weight estimation, in other words, is an interpolation amongthe given set of features which is a double-edged sword, as it gives the interpretability, but is notcapable of extrapolation .
Therefore, when the given feature extractor E is sub-optimal, it could befavorable to relax the constraint and let the model extrapolate for better performance.
Potential Negative Societal Impacts.
We believe that the machine learning community as a wholeshould strive to minimize the negative societal impacts.
Large scale face recognition training datasetsinevitably comprise web-crawled images which are without formal consent, or IRB review.
Werefrained from using any dataset withdrawn by its creators such as VGG-2 [6] or MS1MV2 [16] toavoid any known copyright issues.
We hope that the FR community can collectively move towardcollecting datasets with informed consent, fostering R&D without societal concern.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec. 5(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The proof for thebatch order invariance property is in Sec. 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See Tab.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] We discuss the computation
efficency and resources needed for running our model in Supp.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] When we used the
datasets, we cited the corresponding papers.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We do not provide any new dataset.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We discuss the ethical issues related to the training data collected
from the web in Sec. 5.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] We discuss the copy-right issues in Sec. 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We do not use crowdsourcing in our paper.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No] We dicsuss the difficulty of obtaining
consents for large scale face datasets in Sec. 5.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] We do not collect any dataset.