Physically-Based Face Rendering for NIR-VIS Face Recognition
Anonymous Author(s)
AffiliationAddress
Abstract
Near infrared (NIR) to Visible (VIS) face matching is challenging due to thesignificant domain gaps as well as a lack of sufficient data for cross-modality modeltraining.
To overcome this problem, we propose a novel method for paired NIR-VISfacial images generation.
Specifically, we reconstruct 3D face shape and reflectancefrom a large 2D facial dataset and introduce a novel method of transforming the VISreflectance to NIR reflectance.
We then use a physically-based renderer to generatea vast, high-resolution and photorealistic dataset consisting of various poses andidentities in the NIR and VIS spectra.
Moreover, to facilitate the identity featurelearning, we propose an IDentity-based Maximum Mean Discrepancy (ID-MMD)loss, which not only reduces the modality gap between NIR and VIS images at thedomain level but encourages the network to focus on the identity features insteadof facial details, such as poses and accessories.
Extensive experiments conductedon four challenging NIR-VIS face recognition benchmarks demonstrate that theproposed method can achieve comparable performance with the state-of-the-art(SOTA) methods without requiring any existing NIR-VIS face recognition datasets.
With slightly fine-tuning on the target NIR-VIS face recognition datasets, ourmethod can significantly surpass the SOTA performance.
Introduction
To overcome the problem that the conventional VISible (VIS) images face recognition methodsgenerally fail to achieve a satisfactory performance under poor illumination, face recognition across
Near InfraRed (NIR) images and VIS images has recently gained increasing attention in the computervision community [18, 19, 11, 12].
However, due to the lack of sufficient NIR-VIS data, the trainingof the NIR-VIS face recognition network is prone to be over-fitting [19].
To alleviate overfitting previous works attempt to generate large-scale NIR-VIS face images bytransferring the VIS images to NIR ones [27, 35, 47, 45].
However, the image-to-image translationbased methods are limited by the number of data in the source domain and the diversity of generatedimages [11].
Recently, unconditional generative models are employed to synthesize heterogeneousface image pairs from noise [11, 12] and achieve state-of-the-art performance by adopting the variousintra-changes, such as poses and illumination, of the target NIR-VIS datasets during the generation.
Although the intra-class diversity is considered, only one NIR-VIS pair is generated for each identityin [12], which limits the potential of synthesized face images in the NIR-VIS face recognition task.
However, when generating multiple NIR-VIS image pairs for a given identity by using [12], weobserve that the identity consistency cannot be well preserved, as shown in Fig. 1(a).
Additionally, theappearance variations of generated images are based on the target NIR-VIS face recognition datasets,
[Figure]
which means, different facial images are synthesized to fit different target datasets.
Such datasetspecific face generation degrades the generalizability of the NIR-VIS face recognition networks.
Totackle the above problems, we propose a novel physically-based facial image generation method,where high-quality NIR-VIS facial image pairs are generated based on acquired renderable 3D facialassets.
Rendering photorealistic 3D facial datasets enables us to acquire paired labelled trainingdata with, with controllable identity, pose, expression and illumination.
In contrast with generativemethods, the rendered identity does not change at all when varying other parameters, which greatlyaids training.
However, acquiring human rendering assets requires tremendous manually work, eitherby scanning systems [32, 16] or artists [40].
The available datasets are either small in size [2] or donot contain relightable reflectance [44, 3], such as the diffuse albedo, specular albedo and specularnormals.
Recent works [25, 26, 31, 9] have introduced methods that produce high-quality renderableassets from arbitrary facial images.
Moreover, Wood et al .
[40] showed that high-quality syntheticfacial data, can be successfully used for computer vision tasks, including landmark localization andfacial parsing.
However, to our knowledge, there exists no dataset or method capable of producingrenderable 3D faces, in both VIS and NIR domains.
Utilizing a state-of-the-art facial reflectance acquisition method [26], we generate numerous suchfacial assets and transform them from VIS to NIR, and then render both under the same conditions,in order to generate high-quality training data.
Since our novel transformation method is appliedper-pixel on high-resolution reflectance maps, the identity of the subject is perfectly preserved in both
NIR and VIS.
Faces generated by the proposed methods are illustrated in Fig. 1(b).
As can be seen,our NIR-VIS face generation outperforms [12] (Fig. 1(a)) in preserving identity consistency as wellas retaining the diversity of facial appearances.
The generated high-quality NIR-VIS facial image dataset is then used, along with a VIS facerecognition dataset, to train the NIR-VIS face recognition network.
To facilitate the identity featurelearning as well as reduce the modality discrepancy, an IDentity-based Maximum Mean Discrepancy(ID-MMD) loss is proposed, which pulls the feature centroids of the same identity in the NIR domainand the VIS domain closer.
Assisted by the ID-MMD loss, the gap between NIR images and VISones is bridged at the domain level, and meanwhile, the network is encouraged to focus on identityfeatures rather than facial details of instances, such as poses and accessories.
Overall, our maincontributions can be summarized as:
• A method capable of generating vast amounts of paired NIR and VIS facial images, of various identities, poses and illumination via 3D facial reconstruction and a novel VIS-to-NIR transformation for facial reflectance, is proposed.
• To bridge the gap between the NIR images and VIS images, we propose an IDentity-based Maximum Mean Discrepancy (ID-MMD) loss, which not only reduces the modality discrepancy
at the domain level but encourages the network to attend to identity features instead of facialdetails.
• Extensive experiments on four NIR-VIS face recognition benchmarks demonstrate that thepropose method achieves comparable performance with the state-of-the-art methods, withoutrequiring any existing NIR-VIS face recognition dataset.
With slightly fine-tuning the modelson the target NIR-VIS face recognition datasets, our method significantly surpasses the SOTAperformance.
Background and Related Work
NIR-VIS Face Recognition To facilitate the NIR-VIS face recognition, earlier works focuses onlearning modality-invariant features [18, 19, 21, 20].
[18] extracts features for NIR and VIS imagesvia a shared feature extractor pretrained on large-scale VIS face images, where the modality-invariantidentity information is then filtered out by an orthogonal constraint.
Similarly, DFAL [21] and OMDRA [20] attempt to purify the identity information via decoupling identity-related representationsfrom modality-invariant ones.
To further reduce the gap between two modalities, [19] minimizes the Wasserstein distance of NIR and VIS feature distributions while SMCL [39] designs a center-basedloss to regulate the relationship between the syncretic modality and the NIR(VIS) one.
However, dueto the limited amount of NIR-VIS data, the aforementioned methods are generally in the mire of theover-fitting problem.
To address the problem, generative models are involved to facilitate the NIR-VIS face recognitiontask by transferring VIS face images to NIR ones via the image-to-image translation [27, 35, 47]or synthesizing heterogeneous face image pairs [12].
[27] produces VIS faces from NIR imagesvia learning the patch-to-patch mapping between NIR-VIS image pairs.
[35] transfers NIR faceimages to the corresponding VIS ones via a two-path Generative Adversarial Networks (GAN)-basedframework.
[47] proposes a multi-stream feature-level fusion technique based on GAN to synthesizevisible images from polarimetric thermal images.
However, the improvement brought by the “one-toone" face synthesis strategy is limited by the number of images within the NIR-VIS face recognitiondatasets, as well as the lack of the attribute diversity of generated images.
Henceforth, instead ofadopting the conditional image generation, [12] employs the Variational AutoEncoders (VAE) tosynthesize a paired NIR-VIS face images for a given new identity.
The identity representations areobtained from an identity sampler, which is pre-trained on a large-scale face recognition dataset.
Inspired by [12], generating a large number of paired NIR-VIS face images is beneficial to the
NIR-VIS face recognition.
However, we notice that, when generating multiple NIR-VIS face imagepairs from a given identity representation via [12], the identity consistency is not well preserved.
Suchdisadvantage hinders the potential of generated face images to boost the recognition performancesince they cannot provide an explicit guidance for identity features learning.
Therefore, we propose anovel NIR-VIS face image generation method, where a physically-based renderer is used to generatea vast photorealistic dataset.
NIR-VIS Rendering Synthetic datasets have long been used in face analysis problems with moderatesuccess [48, 13, 34]. DA-GAN [48] pioneered realistic profile face generation for pose-invariant facerecognition.
A recent work by Wood et al .
[40], achieves state-of-the-art performance in variousfacial analysis tasks, by using hand-crafted high-quality photorealistic facial avatars.
3D facialappearance methods [25, 26, 31, 9] leverage deep generative models and differentiable rendering toreconstruct facial assets that can be photorealistically rendered.
Moreover, another work [4] projectsincomplete renderings to the latent space of a deep generative model and reconstructs correspondingphotorealistic facial images.
Although potent in the VIS spectrum, such methods are not directlyusable in NIR.
The literature around NIR rendering from VIS assets remains limited.
Wu et al .
[43] introduced awavelength-dependent Bidirectional Reflectance Distribution Function (BRDF) for Mid-Wavelength
Infrared (MWIR) landscape scene simulation.
Moreover, Aguerre et al .
[1] simulate urban thermalimaging.
While closest to our work, none of the above can directly render photorealistic faces in
NIR-VIS.
In that manner, we employ the recent works in photorealistic synthetic face and combine itwith our novel VIS-to-NIR transformation to unlock the potential for NIR-VIS facial matching.
Proposed Method
In this work, in order to overcome the over-fitting problem caused by the available small-scale NIR-VIS face recognition datasets, we generate a vast, high-resolution and photorealistic datasetconsisting of large-scale identities with various poses and illumination in the NIR and VIS spectra(Sec. 3.1).
Then, the generated NIR-VIS facial image pairs, along with a large-scale VIS facerecognition dataset, are used to train the NIR-VIS face recognition network, without the aid of anyexisting NIR-VIS datasets (Sec. 3.2).
[Figure]
VIS Rendering
NIR Rendering
3.1 NIR-VIS Facial Rendering
We introduce a method able to generate vast amounts of photorealistic pairs of NIR-VIS face images,to facilitate the heterogeneous face recognition task.
Instead of using 2D generative models, wereconstruct a set of 3D facial assets capable of photorealistic rendering in the VIS domain.
Meanwhile,we transform these into the NIR domain with a novel VIS-NIR transformation method and renderpaired images, as described in the following sections.
In this manner, we can create paired labelled NIR-VIS facial images, of virtually unlimited identities, in arbitrary poses and illumination conditions.
VIS Reflectance Data Acquisition Recent works [25, 26, 31, 9] have achieved high-quality renderready VIS face reconstructions, from arbitrary faces.
In that manner, we acquire CelebA [30], adataset of over 200k facial images I i , and fit a textured 3D Morphable Model (3DMM) a GANFITbased [14] fitting F , to acquire a reconstructed texture and mesh S i .
We employ an image-to-imagetranslation network A , based on AvatarMe++ [26], that disentangles facial reflectance maps fromthe reconstructed 3DMM texture maps.
In that manner, for each image I i , we acquire the shape
S
i ,diffuse albedo
A VISD i , specular albedo
A VISS i and specular normals N VISS i , in the VIS domain:
[Equation]
VIS to NIR Reflectance Transformation Human skin is a dielectric material and exhibits bothdiffuse and specular reflectance.
In the VIS spectrum, the diffuse albedo A VISD describes the amountof light emitted per RGB channel when a medium is lit by uniform white illumination.
On the otherhand, the specular albedo A VISS describes the intensity of the incoming illumination that is reflected,based on the direction of the specular normals N VISS .
To transform these spatially-varying reflectancevalues to the NIR spectrum, we define the following empirical model, based on the assumptionthat the physical properties of the skin can be linearly described by the wavelength of the incidentillumination, in the VIS ( 380 − 700 nm ) and a NIR illumination ( 850 nm ).
It is well-established that for specular normals measured under white illumination, shorter wavelengthof incident illumination exhibit sharper surface details, due to the subsurface scattering of light inskin [32, 24].
For a green wavelength w G and red-wavelength w R we calculate the width σ of agaussian kernel G required to minimize the error between the green channel N GS and red normals
[Figure]
N RS (Eq. 2).
Then, for a NIR wavelength w NIR , we linearly scale σ , based on the distance betweenthe red and NIR wavelength and apply a gaussian kernel on VIS specular normals (green-channel)
N VISS to acquire the NIR specular normals
N NIRS :
[Equation]
The NIR sensor is monochrome and its response is more similar to that of the VIS red channel.
Given that a facial VIS diffuse albedo
A VISD is measured under uniform white light, its red channel
A RD measures the skin’s response to the red-wavelengths w R .
Assuming again a linear relationshipbetween wavelength and spectral response between red and infrared, we define the NIR diffuse albedo
A NIRD , as the smoothed red channel albedo A RD .
In contrast to Eq. 2, here we use a Bilateral Filter[36], in order to preserve the edges of the facial details.
Finally, we retain the VIS specular albedo as A VISS =
A NIRS , assuming it is wavelength independent in the visible spectrum [32, 24].
However, we decrease the overall specular roughness,proportionally to the distance of the target NIR wavelength from the mean of the visible spectrum,following [43].
Following the above, we define the complete NIR-VIS transformation function as
A NIRD , A NIRS , N NIRS =
T ( A VISD , A VISS , N VISS , w NIR ) .
Paired NIR-VIS
Rendering The importance of explicitly extracting diffuse and specular reflectancemaps, lies in the fact that we can employ photorealistic rendering algorithms, such as GGX [37]in our case.
We collect a set of n environment maps E 1 , . . .
, E n ∈ E , that define omnidirectionalincoming illumination of various realistic scenes [5].
We define a VIS physically-based renderer R V IS ( S , R VIS , M , E ) −→ I ∈ R
h × w × 3 , where h, w is the size of the rendered image, S is thefacial shape, R VIS = (cid:2)
A VISD , A VISS , N VISS (cid:3) is the reflectance, M is a rotation matrix for the shapeand E an environment map.
For NIR rendering, we define a similar but monochrome renderer R NIR ( S , R NIR , M , E ) −→ I ∈ R h × w × 1 .
Finally, as NIR sensors typically rely on a flood illuminator placed adjacent to the lens, we create an environment map E f with only a flood illuminator placedat the camera direction.
Then, for a map E i , we define the equivalent NIR map as E NIRi =
E i + E f
|| E
i + E f || .
We then use the generated VIS and NIR facial assets and the VIS and NIR renderers, to generatesets of paired NIR-VIS facial images, under arbitrary illumination and pose.
For each subject iand each pose-illumination set j , we use a random subject with shape S i and reflectance R VISiand R NIRi , and sample a) a random environment map E j ∈ E , which is rotated along the Yaxis by a random angle θ j ∈ [0 , 360] , b) a random rotation matrix M j and render the pairedimages I V IS i,j = R V IS i ( S i , R VISi , M j , E j ) and I NIRi,j = R
NIRi ( S i , R NIRi , M j , E NIRj ) .
We use theenvironment map to generated arbitrary background that match in VIS and NIR.
3.2 NIR-VIS Face Recognition
As a departure from most NIR-VIS face recognition works, the generated heterogeneous facialdataset is employed, along with a large-scale VIS face recognition dataset, to train a NIR-VIS facerecognition network explicitly.
Specifically, we present the large-scale VIS face recognition datasetcontaining C identities as X = { x i } Ni =1 , and the corresponding identity label as Y =
{ y i } Ci =1 .
Similarly, the synthesized NIR-VIS dataset with C s identities is denoted as X s = { x i } N s i =1 withlabel Y s =
{ y i } C s i =1 .
N and N s denote the number of images in the VIS dataset and the synthesized
NIR-VIS dataset, respectively.
Note that, since the synthesized identities are from CelebA [30], thereare no overlapped identities between the VIS face recognition dataset and the synthesized one.
Giventhe final training set built by X and X s , following [12], a widely-used face recognition network [41]is trained under the supervision of the identity loss and the proposed IDentity-based Maximum Mean
Discrepancy (ID-MMD) loss.
The NIR-VIS face recognition task is conducted with the identityfeatures derived from the network.
Identity Loss To improve the discrimination power of the face recognition network, we employ themargin-based softmax loss [38] L id during training, which is denoted as follows,
[Equation]
where cosθ j,i =
W jT f i , f i is the normalized feature of i -th image, and W j is the normalized weightvector of the j -th class.
θ j,i refers to the angle between W j and f i .
m is the cosine margin and sis the feature scale.
N r
and N s are the training sample numbers of real VIS faces and synthesized
NIR-VIS faces.
ID-MMD Loss To overcome the main challenge of the NIR-VIS face recognition task, i.e., thecross-modality discrepancy, the Maximum Mean Discrepancy (MMD) loss [15] designed for thetransfer learning task is adopted.
Formally, given a mini-batch containing M NIR images and N VISimages, the MMD loss L mmd is formulated as follows, [Equation] j = ϕ
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
H , (4) where ϕ ( · ) represents the kernel function that maps the original data to a Reproducing Kernel Hilbert
Space (RKHS) H .
Although the MMD loss reduces the domain discrepancy by aligning NIR-VISfeature distributions, rigidly adopting such domain-level constraint to the NIR-VIS face recognitionnetwork training is sub-optimal, since it considers each modality as a whole and ignores the identityfeature distribution within the modality, as shown in Fig 3.
To solve the problem, an explicit solution is reducing the distance between each NIR-VIS imagepair of the same identity in the latent space, i.e., minimizing the Pairwise Mean Square Error(PMSE)
L pmse between the NIR-VIS features.
Concretely, random sampling P identities from theheterogeneous dataset, and for each identity, sampling K NIR images and K VIS images, to formthe mini-batch with 2 × PK images.
The L pmse is denoted as follows,
[Equation]
) where f nir/visp,k denotes the normalized feature of k -th NIR/VIS image of p -th identity.
Although theidentity distribution is considered, such pairwise loss reduces the modality discrepancy at the instancelevel, where the network is highly likely to focus on facial details, such as poses and accessories,rather than identity features.
Taking the girl wearing glasses in Fig.
3 as an example, the PMSE lossmay reduce the feature distance between the NIR-VIS image pair by encouraging the network toattend to the frontal pose or the glasses, instead of the identity features.
To address the problems of the domain-based (MMD loss) and the instance-based (PMSE) modalitydiscrepancy reduction loss, we propose an ID-based MMD loss L idmmd , which bridges the modalitygap by reducing the distance between the NIR-VIS feature centroids of the same identity in the
RKHS.
Formally, for the given mini-batch, the proposed L idmmd is denoted as follows,
[Equation]
The proposed identity-based modality discrepancy reduction loss not only reduces the modality gapbetween NIR and VIS images, but also encourages the features of the same identity within eachmodality to be compactly distributed, i.e., reducing the intra-modality discrepancy.
Overall, theobjective of the NIR-VIS face recognition network is denoted as L =
L id + λ ∗ L idmmd , where λ isthe balancing parameter set as 100 during training. [Figure]
Experiments
4.1 Databases and Protocols
Four NIR-VIS face recognition datasets are used to evaluate the proposed method.
Specifically,the CASIA NIR-VIS 2.0 [28] (725 identities) and the LAMP-HQ [45] (573 identities) are the mostchallenging NIR-VIS face recognition datasets due to the huge facial appearance diversity in poses,illumination, and ages.
Following [28, 45], the ten-fold experiments are conducted on both datasets.
For each fold, approximately 50 % identities are randomly selected as the training set and the restare adopted as the testing set.
Note that, there is no overlap between the training and testing sets.
Following [12, 45], the verification rate (VR)@false accept rate (FAR)=0.01 % , VR@FAR=0.1 % , andthe Rank-1 accuracy, are employed for evaluation.
In terms of two low-shot NIR-VIS face recognitiondatasets: the Oulu-CASIA NIR-VIS [23] and the BUAA-VisNir [22], the identities within the datasetsare split as 20/20 and 50/100, respectively, for the setting of the training/testing set.
Considering thesmall data scale, the VR@FAR=0.1 % and the Rank-1 accuracy are adopted as the evaluation metrics.
4.2 Experimental Details
NIR-VIS Face Generation.
To acquire the 3D facial assets, we use AvatarMe++[26], trained with
RealFaceDB [25] at textures of 1024 × 1024 pixels, on a GANFIT [14] base.
The dataset used is CelebA [30], however other 2D facial datasets can be used to extend generalization.
For the rendering,we employ highly parameterizable commercial rendering software Marmoset Toolbag [33].
For eachidentity from CelebA, we synthesize 20 VIS and NIR facial image pairs, under different poses andillumination.
NIR-VIS Face Recognition Following [12], we utilize LightCNN-29 [41] as the NIR-VIS facerecognition backbone.
To make a fair comparison with [12], which adopts about 5 million imagesfrom the MS-Celeb-1M [17] dataset for pre-training, we use a subset from the large-scale VISdataset [49], i.e., WebFace4M [49], for training. WebFace4M contains 4 million images of 200krandomly chosen identities from WebFace260M [49]. Additionally, instead of using 128 × 128 facialimages as inputs, all the face images are aligned and cropped to 112 × 112 [6] in the paper.
During training, we first train the network with identity loss L id for 20 epochs on the WebFace4Mand the synthesized dataset.
Then, we fine-tune the network on the synthesized images with bothidentity loss L id and ID-MMD loss L idmmd for 5 epochs.
The batch size is set as 512 .
Duringfine-tuning, 32 identities are randomly selected to form a mini-batch, and for each identity, 8 VIS and8 NIR images are randomly selected.
Stochastic gradient descent (SGD) is adopted as the optimizer,where the momentum is set to 0.9 and the weight decay is set to 1 e - 4 .
The learning rate is set to 1 e - 2initially and decays by a factor of 0.5 per 10 epochs.
4.3 Ablation Study
NIR Reflectance Generation.
We show the importance of each component in our proposed NIRreflectance generation method, by reconstructing subjects from LAMP-HQ [45] and showing theeffectiveness of our algorithm against alternative approaches.
In Fig.
4 and Tab. 1, we compare ourmethod with monochrome rendering of VIS assets or remove some of our transformations.
Not onlyour method is capable of producing photorealistic NIR renderings, but can do is in various poseswhile preserving the subject identity, since it is based on rendering 3D assets.
[Table]
reconstructions with our method and the ablation alternatives shown in Fig.
[Table]
Identity Consistency and Diversity.
Following DVG-Face [12], we analyze the identity consistency,the identity diversity and the distribution consistency of the generated NIR-VIS images via evaluationmetrics - Mean Similarity (MS), Mean Instance Similarity (MIS), and Frechet Inception Distance(FID), respectively.
To make a fair comparison, we randomly select 1000 identities from NIR-VISimages generated by DVG-Face and ours. For each identity, 16 NIR images and 16 VIS images arerandomly selected for evaluation.
The results are reported in Table 2.
Note that, apart from measuring MS between each NIR-VIS image pair as DVG-Face, we also measure the feature similarity acrossmultiple images generated for a given identity, which are indicated by 1 v 1 and 1 v N in Table 2,respectively. The higher intra-class (identity) similarity (MS) proves the superiority of our generationin preserving identity consistency. Additionally, our method achieves a lower inter-class similarity(MIS), which ensures identity diversity.
Meanwhile, a lower FID evaluated by LightCNN [41, 12]facilitates the adaptation to real-world NIR-VIS face recognition datasets.
Effectiveness of Generated Data
To prove the proposed NIR-VIS face generation method cansignificantly facilitate the NIR-VIS face recognition, we compare the performances of models trainedwith the proposed ID-MMD loss under different percentages { 0 % , 10 % , 50 % , 100 % } of generateddata.
In Table 3, the test results on LAMP-HQ show that the generated images continuously contributeto the performance improvement and the best result is achieved when all generated data are involved.
Comparisons of Domain Adaption Losses.
To compare our ID-MMD loss with other modalitydiscrepancy reduction losses, the ablation studies are conducted on the LAMP-HQ dataset [45].
Specifically, PMSE loss, MMD loss, and ID-MMD loss, are employed to supervise the learning ofthe NIR-VIS face recognition network.
The corresponding recognition performances are reported in
Table 4.
It can be observed that, the network achieves the best performance when optimizing by theproposed ID-MMD loss, surpassing the PMSE loss by 1.56 % and the MMD loss by 1.04 % when VR@FAR=0.01 % . [Table] [Table]
4.4 Comparisons with State-of-the-art Methods
We extensively compare our method with the state-of-the-art (SOTA) methods on four NIR-VISface recognition benchmarks.
The performances are reported in Table 5.
Following [12], we set the NIR-VIS face recognition baselines {+LightCNN-
+}
Unlike [12], our baseline models, 1) aretrained on the WebFace4M [49], 2) take 112 × 112 face images as input, 3) are trained under themargin-based softmax loss [38].
As can be seen in Table 5, our baseline models achieve a comparableperformance with [41] at a lower computational cost.
We show that, with the aid of the synthesized NIR-VIS face images and the modality discrepancyreduction loss, our method greatly improves the baseline performances on four benchmark datasets. To intuitively illustrate the effectiveness of proposed method, we visualize the feature similaritydistribution of positive pairs (belonging to the same identity) and negative pairs (belonging todifferent identities) of LAMP-HQ in Fig. Benefiting from the generated data and ID-MMDloss, the similarities between positive pairs increase while that between negative pairs decrease.
[Table]
{+0 ± 1 .98 . 0 ± 0 . 9 ± 0 .94 .100 .99 .100 . 
+}
*LC-29( L id ) 98 . 8 ± 0 .99 . 7 ± 0 .99 . 8 ± 0 .94 .
5 ± 0 .98 . 7 ± 0 .
{+4 99 . 0 ± 
 *LC-29( L id + L idmmd ) 99 . 9 ± 0 . 1 100 . 0 ± 0 . 0 99 . 9 ± 0 .+}
6 ± 0 . 4 ± 0 . 1 ± 0 .99 .100 .99 .100 . 1.0 Similarity
[Figure]
e
[Figure]
u e n c y LC-29 ( id ) neg
[Figure]
[Figure] [Figure]
[Figure] [Figure]
As can be seen, without requiring any existing NIR-VIS face recognition datasets, our method has achieved comparable performanceswith SOTA methods on the CASIA NIR-VIS 2.0 and the OuluCASIA NIR-VIS datasets, and has surpassed the SOTA performance by a large margin on the other two.
Especially, on thechallenging LAMP-HQ dataset, our method outperforms the SOTAone [45] by 19.8 % in terms of VR@FAR=0.1 % .
The performances can be further boosted after slightly fine-tuningthe models on the target datasets.
Specifically, as reported in thelast two rows of Table 5, by adopting the identity loss ( L id ) duringfine-tuning, VR@FAR=0.01 % increases from 92.0 % to 94.5 % on LAMP-HQ.
After imposing the proposed ID-MMD loss ( L idmmd )during the fine-tuning on target NIR-VIS face recognition datasets,the performances are further boosted.
Concretely, an improvementof 6.6 % on the VR@FAR=0.01 % can be observed on LAMPHQ.
Additionally, on the two low-shot NIR-VIS face recognitiondatasets that fewer identities are contained, i.e., the Oulu-CASIA
NIR-VIS and BUAA-VisNir datasets
, we surpass the second bestmethod [12] by 1.8 % and 0.8 % in terms of VR@FAR=0.1 % , respectively.
In summary, after fine-tuning on the target NIR-VISface recognition datasets, our method outperforms all other competitors on four benchmarks.
Conclusion
To address the problem of insufficient NIR-VIS data for the cross-modality face recognition networktraining, this paper proposes a novel NIR-VIS face generation method, which enables the generationof vast amounts of photorealistic paired NIR-VIS facial images with various poses and illuminationwhile preserving the identity consistency.
Such merit enables the use of the generated dataset, alongwith a large-scale VIS face recognition dataset, to train the NIR-VIS face recognition network,which can achieve comparable performance with the state-of-the-arts without requiring any existing
NIR-VIS face recognition datasets.
Additionally, to bridge the domain gap between NIR imagesand VIS images during training, an IDentity-based Maximum Mean Discrepancy (ID-MMD) loss isproposed, which reduces the modality discrepancy at the domain level and encourages the network tofocus on identity features rather than facial details.
Qualitative and quantitative experiment results onfour NIR-VIS face recognition benchmarks show the superiority of the proposed method.
Finally,understanding the social impacts of facial matching, our method is modeled for and limited to mobilephone NIR sensors, and thus its use is aimed at more user-friendly experience for such devices.
Checklist
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code and
model will be submitted as supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See Table 5.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec.4.
(b) Did you mention the license of the assets?
[Yes] See Sec.4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable [Table] information or offensive content?
[No] [Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]