Cluster and Aggregate: Face Recognition with Large Probe Set
Abstract
Feature fusion plays a crucial role in unconstrained face recognition where inputs (probes) comprise of a set of N low quality images whose individual qualities vary.
Advances in attention and recurrent modules have led to feature fusion that can model the relationship among the images in the input set.
However, attention mechanisms cannot scale to large N due to their quadratic complexity and recurrent modules suffer from input order sensitivity.
We propose a two-stage feature fusion paradigm, Cluster and Aggregate , that can both scale to large N and maintain the ability to perform sequential inference with order invariance.
Specifically,
Cluster stage is a linear assignment of N inputs to M global cluster centers, and
Aggregation stage is a fusion over M clustered features.
The clustered features play an integral role when the inputs are sequential as they can serve as a summarization of past features.
By leveraging the order-invariance of incremental averaging operation, we design an update rule that achieves batch-order invariance, which guarantees that the contributions of early image in the sequence do not diminish as time steps increase.
Experiments on IJB-B and IJB-S benchmark datasets show the superiority of the proposed two-stage paradigm in unconstrained face recognition.
Code and pretrained models are available in Link.
Introduction
Face Recognition (FR) matches a set of input query imagery, known as probe , to enrolled identity database, known as gallery .
Verification is to confirm the claimed probe’s identity and identification is to identify the unknown probe’s identity by searching a known database [41].
In either case, a probe can go beyond an image and include a set of images, videos or their combinations [4].
Thus
FR involves fusing features of multiple images or videos to create a discriminative feature for a probe.
Due to the interest in unconstrained surveillance scenarios, e.g. IJB-S [21], the role of fusion is becoming more important.
Unconstrained FR is often based on probes from low quality images and videos.
It is challenging due to two issues: 1) individual video frames can be of poor quality, causing erroneous FR model prediction and 2) the number of images in a probe can be very large, e.g. , a probe video in IJB-S may have 500 , 000 frames.
Feature fusion across all frames in the probe is especially crucial if frame-based predictions are unreliable.
While prior works [23,36] address the
IJB-A (9)
IJB-B (21)
IJB-C (23)
IJB-S (7114)
Naïve PFE CFAN RSA CAFace
Contribution of badquality images
Contribution of goodquality images
Unconstrained Setting (low quality + multiple frames) …
E
E E
E E
E E
E …
Intra-Set Relation Large N Probe
Order Invariance
Single Image
Intra-Set Attention
Recurrent Module
Proposed c) Benefit of feature fusion with large N >>
[Figure]
Proposed ( ⋅ ) : avg probe size % inc. over Naïve first issue of prediction in low quality images, the large size of probe set was not addressed. Fig.a) illustrates the problem caused by the absence of proper feature fusion. The contribution of good quality image can be made insignificant in the presence of many other poor quality images in the set.
This paper aims to learn a fusion function that maps an unordered set of N probe features { f i } N of the same person to a single fused output f . Note that f i = E ( x i ) is the feature extracted from the i -th sample in the set, using a fixed feature extractor E .
The task of fusing multiple features involves 1) estimating the quality of individual features and 2) modeling the intra-set relationship of the features. Prior feature fusion works utilize either simple average pooling [10,39], reinforcement learning [29], recurrent models [15] or self-attention [14,28,31,49,53]. Typically, to compute the intra-set relationship among inputs of an arbitrary size N , one would adopt set-to-set functions such as Multihead Self Attention (MSA) [28,47,49], enabling inputs to propagate information among themselves.
The downside of this approach is its computational cost of O ( N 2 ) which becomes infeasible when N exceeds a few thousand.
Also, when the inputs are sequential as in a live video feed, it is nontrivial to model the intra-set relationship except to compute attention over all past frames at each time step.
Recurrent methods [15,18] are useful in the sequential inference but their drawback is set order inconsistency, i.e. , as the number of sequential steps T increases, the contribution of early frames in a set decreases.
Fig.
1 b) contrasts various fusion methods.
A feature fusion framework that can consider both 1) intra-set relationship for a large N and 2) efficient sequential inference is necessary in the real-world unconstrained FR scenarios.
Fig.
1 c) shows the average probe sizes of four datasets.
IJB-S [21]’s probe size is too large for intra-set attention such as RSA [28] to perform inference with all frames concurrently.
We present a feature fusion framework, Cluster and Aggregate (CAFace), that achieves two abovementioned criteria.
It consists of two modules: Cluster Network (CN) and Aggregation Network (AGN).
CN makes soft assignments of N features into M fixed number of clusters, i.e. , { f i } N → { f ′ j } M where M << N .
While N varies from one set to other, M is fixed.
AGN combines M clustered features into a single feature f , i.e. , { f ′ j } M → f .
Conceptually, M intermediate cluster features serve as a summarization of N inputs and AGN models the intra-set relationship among { f ′ j } M .
The proposed framework depends on learning global cluster assignments { f i } N → { f ′ j } M that are consistent across different probes.
Thus, we propose learning shared cluster centers that are input independent .
These centers govern the clustering assignments.
But, it is not obvious which clustering criterion is the best for feature fusion.
Thus, we design CN to discover learned clusters with an end-to-end differentiable framework that allows AGN to back-propagate the gradients to CN.
The cluster assignments are learned to maximize the FR performance.
We also design an input pipeline,
Style Input Maker (SIM) that can helps CN perform class (identity) agnostic clustering efficiently.
The purpose of introducing an intermediate stage { f ′ j } M is to facilitate the sequential inference.
The key design of CN is to formulate { f ′ j } M as a linear combination of { f i } N .
This guarantees that even when the input sequence of set length N is divided into T smaller batches of set length N ′ , { f ′ j } M can be sequentially updated with batch-order invariance.
This is due to our update rule, inspired by
Weighted average
Global Centers 𝑪 ∈ ℝ !
×
# 𝑭 𝒕𝟏 ∈ ℝ & !
×’ 𝑭 𝒕𝟐 ∈ ℝ & !
×’ 𝑭 𝒕𝟏) ∈ ℝ !
×’ 𝑭 𝒕𝟐) ∈ ℝ !
×’
UpdateRule 𝑭 𝒕𝟏𝟐) ∈ ℝ !
×’ 𝒇 ∈ ℝ *×’ 𝑭 𝒕𝟑 ∈ ℝ & !
×’
UpdateRule 𝑭 𝒕𝟏𝟐𝟑) ∈ ℝ !
×’ 𝑭 𝒕𝟑) ∈ ℝ !
×’
𝒇 ∈ ℝ *×’ 𝒇 ∈ ℝ *×’ 𝑭 ∈ ℝ &×’ 𝑭 ∈ ℝ &×’ a) Individual Paradigm
[Figure]
AggregationNetwork AggregationNetwork
[Figure]
the order invariance property of the averaging operation, as in Eq. 8.
When the inputs are sequential, we feed only the new features to CN and update the cached { f ′ j } M .
It achieves the similar effect as having used all previous features simultaneously.
Fig.
2 shows the contrast with previous approaches.
For readability, we will interchange the set notation { f i } N with the matrix notation F ∈ R N × C .
In summary, the contributions of this paper include:
• A novel feature fusion framework for both large N feature fusion and efficient sequential inference.
To our knowledge, this is the first approach to utilize linearly combined intermediate clusters to achieve batch-order invariance with intra-set relationship modeling.
• An task-driven clustering mechanism that can discover latent clustering centers that maximize the task performance.
In our case, the task is FR.
We achieve the task-driven clustering with an assignment algorithm using the global query and decoupled key and value structure.
• We show the superiority of CAFace in unconstrained face recognition on multiple datasets.
Related Work
Feature Fusion (Unordered Set).
The simplest way of feature fusion is to average over a set of features { f i } N [10,39].
In this case, the features with larger norms play a bigger role, and it generally works since easy samples tend to show larger norms [36,40].
To learn the weights, CFAN and QAN utilize the self-attention mechanism, a learned weighted averaging mechanism [14,31].
The drawback of these approaches is the lack of an intra-set relationship during the weight calculation process.
Previous works that adopt the intra-set attention mechanism are Non-local Neural network and
RSA [28,49].
These works use intermediate feature maps U i of size
R C M × H × W during aggregation because feature maps provide rich and complementary information that can be refined by taking the spatial relationship into account.
However, the drawback is in the heavy computation in the attention calculation.
For a set of N features maps, an attention module involves making ( N × H × W ) 2 sized affinity map.
Our Cluster Network utilizes a compact style vector from SIM and makes N 2 sized affinity map which greatly increases the computation efficiency in attention computation.
DAC [29] and MARN [15] propose RL-based and RNN-based quality estimators, respectively.
Yet, they fail to be agnostic to input stream order and are unsuitable for modeling long-range dependencies.
Our method can split the N inputs into T smaller batches and still achieve batch-order invariance.
Video Recognition (Ordered Set).
The feature fusion for recognition has a resemblance to videobased recognition [30], but set inputs cannot always expect the temporal dependencies to be available.
Therefore, most video-based approaches for tasks such as action recognition or quality enhancement [2,3,24,33,38,56] focus on exploiting the relationship between nearby frames, whereas feature fusion approaches do not define them.
In video-based FR, the general trend is to focus more on assessing the quality of individual frames as opposed to exploring the relationship among nearby frames.
Some examples of video-based FR utilize n-order statistics [34], affine hulls [8,19,54], SPD matrices [20] and manifolds [17,48].
Recently, probabilistic representation such as PFE [43] gained popularity [1,9,42,43] since the variance in distribution serves as a quality estimation for individual frames.
𝑺 ! = 𝑨𝑺
StyleInputMaker (SIM) …
Identity 𝑭 … Style 𝑺 … ℝ " !
×
[Figure]
𝑁 !
images
[Figure]
identity
Cluster Network (CN) 𝑸𝒖𝒆𝒓𝒚:𝑪 𝑲𝒆𝒚:𝑺 𝑽𝒂𝒍𝒖𝒆: (𝑺,𝑭)
Global Centers
Transformer+SoftMax 𝑪 ∈ ℝ &×$
Assignment map 𝑨
Aggregation Network (AGN) 𝑷 ∈ ℝ &×% ℝ &×$ 𝑭 !
=
𝑨𝑭
M×𝑁′ ℝ &×%
MLPMIxer weight weightedaverage 𝒇:
[Figure]
(CAFace)
[Figure]
[Figure]
Attention Mechanism.
Multihead Self Attention (MSA) [47] is a widely adopted set-to-set function that models intra-set relationships via an affinity map.
It is also a key component in transformer architectures which outperform CNNs in various vision tasks [7,11,13,27,32,46,55].
The underlying mechanism of MSA which uses the affinity of query and key to update the value is versatile in its application beyond recognition and has led to its usage in memory retrieval and grouping [5,44,52].
The unique property of the proposed Cluster Network is in the linear combination of value assignment which enables batch-order invariance using an incremental average update rule.
Unlike MSA which requires concurrent inputs during inference for intra-set relationship, ours can split the inference and establish a connection across batches without decreasing the contribution of early inputs.
Proposed Approach
The Cluster and Aggregate paradigm seeks to divide the large N inference into partitioned inferences while still obtaining the result as seeing all inputs at the same time.
This can be achieved if 1) each partitioned inference can update the intermediate representation with necessary information and 2) the order of inference does not affect the final outcome, so the information in early batches is not forgotten.
In essence, the intermediate representation serves as a communication channel across batches.
We achieve this by designing a Cluster Network (CN) and Aggregation Network (AGN).
Fig.
2 c) shows the proposed paradigm.
In this section, we will elaborate on how we obtain the global assignment that is consistent across batches and how the update rule can be batch-order invariant.
We formally layout a few assumptions for the Cluster and Aggregate paradigm in the face recognition (FR) task as shown in Fig.
Let { x i } N be a set of N facial images from the same person.
The task is to produce a single feature vector f from { x i } N that is discriminative for the recognition task.
We assume that a single image based pretrained face recognition model E : x i → f i is available following the settings of previous works [14,28,43].
For readability, we will interchange the set notation { f i } N with the matrix notation
F ∈ R N × C where { f i } N is the input is a set of length N and F simplifies equations.
For clarity, we denote N to be the probe size (the number of images in a set) and N ′ to be the partitioned set size when N is large.
During training, we fix the number of images for fusion as N ′ .
Note that the shape of inputs during training would have one more dimension, training batch size B , i.e. F ∈ R B × N ′ × C .
Training batch size refers to the number of persons sampled in a mini-batch, different from the number of images per person, N ′ .
We drop the training batch size dimension in equations for brevity.
3.1 Architecture
Cluster Network (CN).
Cluster Network is responsible for mapping inputs
F ∈ R N ′ × C of variable size N ′ to F ′ ∈ R M × C of a fixed size, M .
A natural choice for the architecture would be Transformer [13,47] as it is a set to set function.
However, there are two problems with it.
1) It cannot handle large inputs due to the quadratic complexity of MSA.
2) When the inputs are partitioned and [Figure] 𝑶𝒖𝒕 inferred sequentially, the intra-set information across the batch is lost, as MSA computes the affinity within the given inputs.
CN solves this problem by modifying Transformer with 1) shared queries and 2) linear value mapping.
These changes result in a clustering mechanism.
We first consider the following generic attention equation [47] with query Q , key K and value V .
[Equation]
where W q , W k , W v are learnable weights and d is the channel dimension of K .
The row-wise
Softmax ensures that the output is the weighted average of all projected values W v V for each query index.
We modify this to
[Equation]
First, unlike K and V which are inputs, the query is now a shared learnable parameter C initialized at the beginning of training.
Secondly, removing W v and the column-wise Softmax ensure that for each query index of C , the output is the (soft) selection of values V .
These two modifications result in a learned soft assignment mechanism where C serves as the global shared center.
We name the assignment map as A .
The difference between A from row and column SoftMax is shown in Fig.a) and b).
We then divide A by the weight of samples assigned to each center (row-sum of A ) as in
Cluster C ( K , V ) =
A (cid:80)
[Equation]
Note that Cluster C ( K , V ) is linear in V , while the prediction of A is nonlinear.
To further add the nonlinearity of A to the Cluster Network, we first embed the keys K with shallow Transformer before clustering.
The combined result CN ( · ) is the learned soft assignment of values according to the affinity between keys and global queries.
More details on the architecture is provided in Supp.
Style Input Maker (SIM).
So far, we have discussed the generic Cluster Network algorithm.
For face recognition, we still need to decide keys K and values V for feature fusion.
It is clear that V should be F , the facial identity features, as it is what we are interested in merging.
It is possible to use
F for K as well, but K should ideally contain useful information for fusion and be compatible with queries which are the global center C .
However, f i is optimized to be invariant to any characteristics other than the identity.
Thus it lacks input image style which encompasses various image traits such as brightness, contrast, quality, pose, or a domain differences from the training data.
In light of the success of using first and second-order feature statistics as an image style [22,26,37], we propose SIM for extracting style information using the intermediate representations of feature extractors.
The benefit of modeling keys K in clustering with style over using identity is shown in
Sec. 4.2.
We also ablate the benefit of further including feature norm
||
f
i || in K as it is sometimes used to approximate the confidence of the prediction [23,36].
Let U i ∈
R C M × H × W be the intermediate feature.
We capture image style by a style vector γ i ∈ R[-
-]
[Equation]
where Γ =
[ µ sty , σ sty ] ,
[Equation]
A learnable matrix W s ∈ R C M × 2 controls the importance of µ sty and σ sty via element-wise multiplication ⊙ .
Simply put, SIM is a shallow network on spatial mean and standard deviation of U i .
One can take U i from more than one intermediate locations and in such a case, we concatenate them.
To verify whether the feature norm would further benefit the fusion process, we embed the feature norm
|| f
i || 2 to a 64 -dim vector, following the convention of Sinusoidal conversion [47], which is analogous to the position embeddings in ViT [13].
The norm embedding n i is a 64 -dim vector and the details can be found in Supp.
B. Finally, the output SIM is the concatenation, s i =
[ γ i , n i ] where s i ∈ R d and d = 64 + 64 = 128 .
For readability, we denote the set { s i } N ′ ∈ R N ′ × 128 as S .
In summary, we decouple style S and identity F and use S as keys to map
[Equation]
which are the intermediates that will be used for subsequent fusion in AGN or stored for sequential inference.
We also map S to S ′ using the same assignment.
Fig.
3 shows the overall diagram.
Aggregation Network (AGN).
The Aggregation Network is responsible for fusing a fixed number of M inputs, F ′ and S ′ into a single fused output f with intra-set relationship.
We adopt MLPMixer [45] as it can efficiently propagate information for the fixed-size input.
For interpretability, we predict weights P ∈ R M × C that combines F ′ to f ∈ R C .
Specifically, f = AGN ( S ′ , F ′ ) is
[Equation]
where [ S , C ] denotes the concatenation along the channel dimension.
The magnitude of P is an interpretable quantity showing the importance of each cluster during fusion.
The final output f is a weighted average of F ′ whose weight is P .
The details of MLP-Mixer [45] can be found in Supp.
Sequential Inference.
A key characteristic of CAFace is its ability to divide the inputs into T -step sequential inference of smaller set length N ′ when N is large, and still achieve similar results as the concurrent inference.
It is possible as the intermediates F ′ and S ′ are linear combinations of F , S respectively, although estimating the combination weights A is non-linear.
This allows us to formulate the update rule as the incremental weighted average whose innate property is order-invariant.
Consider partitioned inputs F 1 , ..., F T , with corresponding predicted weights A 1 , ..., A T .
Since by definition (Eq. 5), F ′ t =
A t F t / (cid:80) N ′ j
A t, ( i,j ) , we can write the cumulative intermediate, (cid:99) F T ′ as
[Equation]
This formulation requires storing all inputs of timestep 1 , ..., T .
We can easily convert this to
[Equation]
, where
[Equation]
which requires storing only the cumulative row-summed assignment map a T − 1 and a cumulative intermediate (cid:98) F ′ T − 1 of the previous time-step.
The same logic applies to S ′ as well.
Note that this operation, by design, is invariant to inference order (batch-order) as the final result will always be the total weighted average.
However, we do not obtain element-wise permutation invarinace as the prediction of A t will change with different inputs.
We test the susceptibility to element-wise permutation in Sec.
4.3 and it has minimal impact on the overall performance.
3.2 Loss Function
Template Loss.
The objective is to make the fused output f be close to the ground truth class center f GT .
The task can be viewed as correctly inferring the true class center in the presence of low quality features { f i } N .
Here f GT is dependent on the pretrained feature extractor E and can be either taken from the last FC layer of E or computed using per-subject average of the embeddings f i in the training data.
Our loss function can be viewed as the cosine distance version of the Center loss [50].
In training, we randomly sample B number of subjects and N ′ images per subject.
Let the superscript in F ( b ) denote the b -th subject.
The loss to increase the cosine similarity is,
[Equation]
[Equation]
[Figure]
Set Permutation Consistency Loss.
The Cluster And Aggregate paradigm achieves batch-wise order invariance by formulation, but it does not achieve element-wise permutation invariance, as noted in Sec. 3.1.
Therefore, we explore the element-wise permutation’s added benefit using an additional loss function.
Thus, the set permutation consistency loss L p is
[Equation]
It lets the splitted inference outcome similar to the concurrent inference.
Sec.
4.2 shows the benefit of
L p
but it is small, meaning the batch-order invariance from model design is already powerful.
The final loss is
[Equation]
(11) where λ p is the scaling terms for L p respectively.
Experiments 4.1 Datasets and Implementation Details
We use WebFace4M [57] as our training dataset.
It is a large-scale dataset with 4 .
M facial images from 205 , 990 identities.
The single image based pretrained face recognition model E has been trained with the whole training dataset.
To train the aggregation module, we use its randomly sampled subset, consisting of 813 , 482 images from 10 , 000 identities.
We do not use VGG-2 [6] or
MS1MV2 [12,16] as they were withheld by their distributors due to privacy and other issues.
For the pretrained face recognition model E , we use the IResNet-101, trained with ArcFace loss [12].
Since the performance of the aggregation depends on the quality of E , we set E to be the same for all experiments.
E produces an embedding vector f i ∈ R 512 for each image.
To offer variations in the training data features, we randomly augment the dataset with cropping, blurring, and photometric augmentations.
The training hyper-parameters such as optimizers are detailed in Supp.
A.
We test on IJB-B [51], IJB-C [35] and IJB-S [21] datasets.
IJB-B is a widely used FR test set containing both high-quality images and low-quality videos of celebrities (see Fig.
5 for examples).
IJB-C is an updated version of IJB-B with more complex motions in the video.
IJB-S is a surveillance video dataset, benchmarking extremely low-quality image/video face recognition.
The probe and gallery set size can exceed 500 , 000 .
Within the set, there are many low-quality images, making IJB-S very challenging and suitable for measuring the feature fusion framework (see Fig.
6 for examples).
For IJB-S, we use protocols, Surv.-to-Single, Surv.-to-Booking and Surv.-to-Surv.
The first/second word in the protocol refers to the probe/gallery image source.
Surv. is the surveillance video, ‘Single’ is the frontal high-quality enrollment image and ‘Booking’ refers to the 7 high-quality enrollment images.
For the ablation study, Sec. 4.2, we report the average of all 9 metrics listed in Tab. 4.
4.2 Ablation and Analysis
Effect of Different Style Input.
In Tab. 1, we ablate the efficacy of various components in SIM which prepares the input for CN.
The table shows that using f i for clustering is harmful to performance and increases the no. of parameters for CN.
It also shows that using the additional norm embedding n i along with s i produces the best results for IJB-B and IJB-S datasets.
However, the margin is small, so simply using s i would suffice for the setting that requires faster computation.
Effect of L p .
As noted in Sec.
3.2, we further propose to constrain the set permutation consistency with the additional loss L p .
The ablation between λ p = 0 and λ p = 1 is shown in Tab. 1. [Figure] [Table]
Effect of Number of Clusters.
In Tab. 1, the effect of the number of clusters M is shown.
The IJB-S performance peaks when M = 4 .
When M = 2 , the summary representations F ′ and S ′ have only two assignment options where one cluster takes the poor quality images with low weights.
The behavior could be interpreted as performing an outlier detection, which is powerful enough to give high performance.
When M > 2 , F ′ and S ′ have the capacity to store richer history of previous frames which would be beneficial in sequential inference.
IJB-S has large N probes which require dividing the inference into batches.
Higher IJB-S performance when M = 4 indicates that the freedom to assign samples to different clusters is important in the sequential setting.
A similar phenomenon is observed 96.
N ' = 4 N ' =
N ' =
N ' = N ' =
N ' = N ' =
N
M= M=
M=
[Figure]
in Fig.
The performance gap widens for M = 2 as we reduce the batch size to make more sequential steps in inference.
Weight Visualizations.
An example of clustering assignments when M = 4 could be viewed in Figs.
5 and 6.
Fig.
5 shows how samples are soft-assigned to different clusters along with the weight estimation.
The cluster weight is calculated by averaging along 512 dimensions of P ∈ R M × 512 .
Note that each column sums to 1 but F ′ and S ′ are calculated by averaging each row.
Thus, the relative contribution of samples in each row is important.
Fig.
6 shows the actual contribution of each sample during fusion.
The contribution can be calculated by multiplying the magnitudes of A and P .
Note that in the presence of many poor quality images, selecting a few good ones is very important and the sample weight of our method can effectively select a subset of samples during fusion.
4.3 Comparison with SoTA methods
To compare with prior feature aggregation methods, we use the same feature extractor E as in Sec.
4.1, for a fair comparison.
Average is the conventional embedding f i averaging scheme that is adopted [Table] [Table] [Table]
in the absence of a learned aggregation model.
It is equivalent to the stand-alone ArcFace model performance.
The rest of the methods learn an additional network for fusing the set of features.
In Tab. [Table] we show the performance of various feature fusion methods on IJB-B. CAFace achieves a large performance gain in all TAR@FAR metrics.
CFAN [14] and PFE [43] do not use any intra-set relationship, as they learn to predict the confidence of a single image.
RSA [28] calculates intra-set attention of feature maps, which is computationally costly and incapable of sequential inference.
CAFace obtains the best results with the least number of parameters.
In Tab. 3, the performance in
IJB-C dataset is also shown with the similar observation as in IJB-B. We also include an additional backbone, AdaFace [23] to highlight how CAFace can work across different backbones. More performance comparisons can be found in Supp.
C. In Tab. 4, we compare feature aggregation models in the IJB-S dataset that has large N low quality images/videos in probes.
RSA [28] cannot load all images in the probes concurrently for large N .
As an alternative, we divide the probes into a manageable size of N ′ = 256 and average the results.
Since
RSA does not have a sequential update mechanism, dividing large N probes reduces the performance, which shows why the sequential capacity is important.
CAFace also divides the probes image into batch size of N ′ = 256 images yet achieves a large margin improvement in IJB-S.
It shows that our two-stage mechanism is very effective in the large N setting.
Particularly, the performance gain in the hardest protocol, Surveillance to Surveillance, is the largest.
We also randomly shuffle images within the probes 5 times and measure the mean and std.
of the performance in the last row.
The result shows that our model is robust to input ordering.
We also include an experiment on a high quality image dataset, IJB-A [25], in Supp. C, and note that the performance gain with feature fusion is negligible.
As noted in Fig.
1 c), the improvement over the baseline (averaging) goes up with the increased number of images in the probe, which highlights the importance of large N scalability.
4.4 Resource and Computation Efficiency
Since CAFace is build on top of a single image feature extractor E , we show the relative FPS of
CAFace with respect to the FPS of E in Tab. 2.
The relative FPS reported in Tab. 2 is computed with the input sequence length N = 256 .
It shows that the single image based quality estimation methods, PFE and CFAN are the fastest.
And RSA with intra-set attention is the slowest.
CAFace can achieve a relatively good speed and obtain the best performance.
Another aspect of computation requirement is GPU memory usage. In the second column of Tab. 5, we show the maximum sequence length N that each method can take simultaneously to perform the feature fusion. It shows that RSA with the inner-set attention cannot handle a sequence length N larger than 384 .
This is a drawback that prevents the method from fusing large N features.
On the other hand, CAFace can take a large N sequence upto 12 , 000 simultaneously.
Note that for the sequence length larger than this can still be handled because CAFace has a sequential inference scheme as described in Sec. 3.1.
In other words, we can divide the input into smaller set size N ′
[Table]
and the intermediate representation is updated to account for all elements in a set.
Tab. 5 also shows the relative FPS of the fusion model compared to the backbone FPS under different sequence lengths N .
The details on the experiment setting can be found in the Supp.
D.
[Table]
Conclusions
We address the two problems arising from the feature fusion of large N inputs, a common scenario in unconstrained FR.
With large N features, modeling intra-set relationships with attention mechanisms are prohibitive due to computational constraints while sequential inference suffers from the reduced contribution of early frames.
In this work, we explore the possibility of dividing N inputs into T smaller unordered batches whose inference result is the same as concurrent N inference.
To this end, we introduce a two-stage cluster and aggregate paradigm.
The clustering stage, inspired by order-invariance of incremental mean operation, is designed to linearly combine N inputs to M global cluster centers, whose assignment is invariant to the batch-order.
The aggregation stage efficiently produces a fused output from M clustered features, while utilizing the intra-set relationship.
We show our proposed CAFace outperforms baselines on unconstrained face datasets such as IJB-B and IJB-S.
Limitations.
Cluster and Aggregate is a feature fusion framework that learns the weights of individual inputs, given a fixed feature extractor E .
Weight estimation, in other words, is an interpolation among the given set of features which is a double-edged sword, as it gives the interpretability, but is not capable of extrapolation .
Therefore, when the given feature extractor E is sub-optimal, it could be favorable to relax the constraint and let the model extrapolate for better performance.
Potential Negative Societal Impacts.
We believe that the machine learning community as a whole should strive to minimize the negative societal impacts.
Large scale face recognition training datasets inevitably comprise web-crawled images which are without formal consent, or IRB review.
We refrained from using any dataset withdrawn by its creators such as VGG-2 [6] or MS1MV2 [16] to avoid any known copyright issues.
We hope that the FR community can collectively move toward collecting datasets with informed consent, fostering R&D without societal concern.
Acknowledgments..
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2022-21102100004.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of
ODNI, IARPA, or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.