Dense Interspecies Face Embedding
Abstract
Dense Interspecies Face Embedding (DIFE) is a new direction for understanding faces of various animals by extracting common features among animal faces including human face.
There are three main obstacles for interspecies face understanding: (1) lack of animal data compared to human, (2) ambiguous connection between faces of various animals, and (3) extreme shape and style variance.
To cope with the lack of data, we utilize multi-teacher knowledge distillation of CSE and StyleGAN2 requiring no additional data or label.
Then we synthesize pseudo pair images through the latent space exploration of StyleGAN2 to ﬁnd implicit associations between different animal faces.
Finally, we introduce the semantic matching loss to overcome the problem of extreme shape differences between species.
To quantitatively evaluate our method over possible previous methodologies like unsupervised keypoint detection, we perform interspecies facial keypoint transfer on MAFL and AP-10K. Furthermore, the results of other applications like interspecies face image manipulation and dense keypoint transfer are provided. The code is available at https://github.com/kingsj0405/dife.
Introduction
Face understanding is a longstanding research topic in computer vision, and there have been extensive studies and applications such as face recognition, facial expression estimation, face photo manipulation, and virtual humans [6, 52, 4, 57, 3].
The vast majority of successful works on faces have focused on human faces due to its practical implication for a variety of applications and the accessibility to a large amount of annotated data.
There have been few attempts for understanding faces of other animal species to facilitate more applications such as ecological research, pet identiﬁcation and farm automation [15, 38, 13].
However, the task of understanding animal faces is more challenging than that for the human face due to the lack of sufﬁcient data with annotations.
In the literature on full-body understanding, joint learning of interspecies data has been studied as a remedy by transferring the knowledge of well-annotated domain such as human to other species [37, 30, 31].
However, the existing approaches are focused on full-body data, therefore their performance on facial data is limited.
In this work, we are interested in the problem of learning the representation of interspecies facial semantics, which is a common representation shared across interspecies facial data.
Learning such representation allows us to extend face understanding beyond the human face, as it enables discovering facial semantics of other species without enough annotations through transferring knowledge from well-annotated human data.
As many animal species share similar face geometry, the representation can also beneﬁt from learning multiple species jointly.
Since sharing photos of animals through social networks has become more popular, learning the interspecies face representation can be used for [Figure] various applications.
For instance, one can search animal photos using a photo of human face as a query to ﬁnd animal faces with similar pose.
An animal face animation can be synthesized by using that of a human as a template, which can be useful for media production.
To achieve our goal, we propose to learn Dense Interspecies Face Embedding (DIFE) as shown in
Fig. 1(a).
Our approach is inspired by CSE [31], which was proposed to learn continuous surface embedding (CSE) of full-body shared across interspecies data.
While the interspecies embedding in [31] is promising, it is difﬁcult to use it for faces as the embedding in the facial region is too coarse to discriminate facial landmarks accurately.
To overcome this problem, we propose a multi-teacher knowledge distillation method for learning DIFE.
Our key idea is to enforce a student network to learn a ﬁne-grained interspecies facial embedding from two teacher networks: CSE and StyleGAN2 [22].
Since the CSE has a useful representation of interspecies data, we use it as a teacher to have the student network learn a common embedding space shared in multiple species.
For a ﬁne-grained face embedding, we simultaneously exploit StyleGAN2 as the secondary teacher.
StyleGAN2 is originally trained to generate high-quality face images [22].
Recent studies have demonstrated that the features of StyleGAN2 have abundant information on facial attributes such as postures and expressions [14, 40, 41].
Thus, the goal of our knowledge distillation is to transfer such information to the student network for learning ﬁne-grained facial semantics for multiple species.
To further improve the performance of DIFE, we propose a data augmentation method to synthesize pseudo-paired images, and the semantic matching loss to enhance the semantic correspondence between images.
For the data augmentation, we synthesize images, where the face poses of the synthesized images are similar to that of input images using StyleGAN2.
This can be achieved in our framework as DIFE is tied to the latent features of multiple StyleGAN2 networks in different domains.
Thus, DIFE of the input image can be used as a face geometry condition for the latent space exploration of StyleGAN2 to generate images for both intra- and inter-domain pairs.
In addition, inspired by [44], a semantic matching loss is proposed to use the pseudo-paired data to enforce the
DIFE to learn pixel-level ﬁne-grained matching.
We demonstrate the effectiveness of our method on the cross-domain keypoint transfer task using interspecies data such as people, dogs, cats, and wild animals in MAFL [56] and AP-10K [50].
We evaluate our method on both sparse and dense keypoint transfers.
In addition, we apply our method for an image editing task of overlaying a virtual object on various animal faces.
Our contribution can be summarized as follows:
• We present a cross-domain face understanding study that considers not only human faces but also the faces of animals.
Our approach makes it possible to avoid tedious and expensive animal data collection through domain adaptation, setting the stage for a variety of crossdomain applications.
• We present a novel multi-teacher knowledge distillation paradigm that extracts and combines the information from models with different architectures and data domains.
Our framework is designed to learn continuous face embedding across interspecies data from CSE and ﬁne-grained face embedding from StyleGAN2.
Combining the knowledge from the two, our method learns dense interspecies face embedding (DIFE).
• We further introduce a method for synthesizing paired data for learning the semantic matching using the latent space exploration of StyleGAN2.
With the novel approach, we can reﬁne the representation of our face embedding for semantic correspondence between various instances and species.
Related Works 2.1 Domain-speciﬁc Face Understanding
We ﬁrst review works that aimed to identify the visual correspondence between various instances and scenes within one data category, human or animal.
For faces, detecting facial landmarks such as the eyes and the nose has been studied extensively to ﬁnd correspondences between subjects.
Initial works [43, 28, 51, 54, 48, 5] ﬁnd facial landmarks using facial landmark annotations in a fully supervised way.
Regarding animal faces, Yang et al.
[49] detected sheep facial landmarks based onannotated keypoints.
Although most works show impressive results under extreme conditions, the lack of annotated animal data limits the application compared to human faces.
Recently, there have been active researches [45, 53, 16, 44, 18, 7, 20] on ﬁnding landmarks from a collection of images in an unsupervised way, which is applicable to various domains, including human body, face, etc.
This stream of works relies on the fact that landmarks are spatially equivariant to the transformation.
Some works learn landmarks by comparing warped images with the original ones [45, 53], and others cast the problem into a conditional image generation using video frames depicting the same instances in different poses [16, 18].
The most related work to our method is
DVE [44] in that they consider not only equivariance between the same instances but also intracategory variations.
All of aforementioned methods can detect landmarks within the same object categories, but the landmarks are not convertible between different object categories (e.g. humans and cats).
2.2 Interspecies Understanding
There are studies that aim to discover the relationship between animal species, whose structures are related but geometrically diverse.
To the best of our knowledge, [36] is the only work that conducted semantic matching between animal faces.
Animal facial landmark detector is trained by transferring the knowledge from the pre-trained human landmark detector.
The network is ﬁnetuned using horses and sheep data with 5 keypoints annotations.
In comparison, we propose an unsupervised approach for interspecies facial semantic matching with the higher granularity of the keypoints.
Closely relevant to our work is the stream of works in [37, 30, 31], as they ﬁnd the relationship of body poses between animals.
Sanakoyeu et al.
[37] transfers the knowledge of the human body to the chimpanzee using DensePose [11].
The CSE [30] utilizes the manually annotated 3D meshes of each category and continuous surface embedding to relate several animal categories ﬂexibly.
In the following work [31], they utilize the cycle-consistency between meshes and images to further the performance.
Although they show impressive results on the complete body, they have difﬁculty in achieving a ﬁne-level sharpness on faces, making it impossible to discern different parts of faces.
While our work also takes the advantage of CSE [31] to detect interspecies correspondences, we improve the accuracy of facial correspondence with the help of the latent features of StyleGAN2 [22] without additional annotations.
2.3 StyleGAN2 Utilization
Our work is also related to StyleGAN utilization works in that we use features of pretrained StyleGAN2 [22] to obtain ﬁne-grained facial features.
StyleGANs [21, 22] show highly photo-realistic results in the unconditional image generation problem with the power of well disentangled latent spaces.
There have been abundant works to exploit the pre-trained StyleGAN for various tasks, including image manipulation [46, 14, 41, 39], 3D reconstruction [34], image segmentation [25, 2], and semantic matching [35].
Because StyleGAN2 has plenty of face geometry many works try to extract the information with knowledge distillation [2, 33, 46].
Labels4Free [2] and Segmentation in style [33] utilizes the feature of StyleGAN2 to train segmentation model in unsupervised manner. In StyleGAN2 distillation [46],
[Figure] StyleGAN manipulation model is trained with synthesized data.
can tell have used
StyleGAN models learned in different data domains at the between separate StyleGAN
Method
Let x d ∈ R
H
× W × 3 denote an image of a face in a domain d ∈ { 0 , 1 , ..., D − 1 } , where H and W are the height and the width of the image and D denote the number of domains.
The goal of our task is to obtain a dense interspecies face embedding E ∈ R H (cid:48)
× W (cid:48) × n of the image x d through an encoder Φ , which is described as E =
Φ( x d ) .
Here, n is the embedding dimension for dense interspecies face embedding.
To train the encoder Φ in an unsupervised manner, we propose two training approaches: the multi -teacher knowledge distillation, and the semantic matching through pseudo-paired data synthesis.
Since all images and labels are synthesized by the teacher models whose training data are not suitable for our task, the training can be considered as an unsupervised learning.
In the following, we describe our dense interspecies face embedding (DIFE) and its training in detail.
3.1 Multi-Teacher Knowledge Distillation
Fig. 2(a) illustrates the overview of the multi-teacher knowledge distillation on multiple domains consisting of humans, dogs, and cats.
Desirable properties of the interspecies face embedding are two folds.
First, the representation is shared by various face images of different domains.
Faces in multiple domains should have similar embeddings if their geometry is similar.
Second, the embedding contains ﬁne-grained details of facial semantics, so that different parts of a face are distinguishable.
To learn such properties, we ﬁrst train DIFE by distilling knowledge from CSE [31].
As the representation of CSE has high-quality co-embeddings of diverse animal species, we exploit it to learn a common space for cross-domain faces.
Speciﬁcally, we optimize a knowledge distillation loss that minimizes the distance between the DIFE and the CSE embedding of the same input.
Formally, the loss is described as
[Equation]
where C d is the CSE embedding of x d .
Note that x d is an image synthesized by StyleGAN2 described as dotted arrow in Fig. 2(a).
Although the above loss enforces our embedding to learn a shared space for multiple domains of faces, the representation of CSE is coarse in the region of the face since it is originally learned to distinguish different parts of a full body.
To further learn ﬁne-grained details of faces, we add another knowledge distillation loss using the features of StyleGAN2.
Speciﬁcally, we exploit the semantic information of faces extracted from a pre-trained StyleGAN2 to enhance the representation of DIFE.
Using synthetic images generated by the StyleGAN2 as input, we minimize the distance between the embedding of the input and its corresponding StyleGAN2 feature map, which is formulated as
[Equation]
where F d is the StyleGAN2 feature map, and D d is the domain-speciﬁc embedding of x d for the domain d .
Since the dimension of DIFE is different from that of StyleGAN2 and each feature of
StyleGAN2s is in the different domain, we convert DIFE to a domain-speciﬁc embedding using a domain converter Ψ d , as follows: D d = Ψ d ( E d ) .
When learning from multiple domains, we optimize L K 1 = (cid:80) Dd L dK 1 and L K [-
-]2 = (cid:80) Dd L dK 2 for all of the domains.
3.2 Semantic Matching
Pseudo-paired data synthesis.
Fig.
2(b) shows how to synthesize pseudo-paired data x d (cid:48) , which has the identical face geometry with the original image x d .
The paired domain d (cid:48) can be different from the original domain d .
From a given human face image x d generated by StyleGAN2, we ﬁrst extract DIFE using the encoder Φ .
Then, with the domain converter Ψ d (cid:48) , we transform the interspecies embedding into a domain-speciﬁc embedding D d (cid:48) .
In Fig.
2(b)
, we convert DIFE into the dog-speciﬁc embedding.
Note that the domain-speciﬁc embedding lies in the similar vector space with StyleGAN2, since the domain converter is trained by the StyleGAN2 knowledge distillation.
Finally, we synthesize a pseudo-paired image x d (cid:48) by feeding the domain-speciﬁc embedding D d (cid:48)
to
StyleGAN2.
Rather than directly feeding the embedding, we introduce a latent space exploration method, which generates more realistic images by adjusting the latent space of the domain-speciﬁc embeddings into the natural image manifold.
In the latent space exploration, given a domain-speciﬁc embedding D d (cid:48) , the goal is to ﬁnd a latent code w d (cid:48) ∈ W d (cid:48) , which satisﬁes the following two conditions.
Here, W d (cid:48) is a latent space of
StyleGAN2 trained on the data domain d (cid:48) .
First, the pseudo-paired data x d (cid:48) generated from the found latent code w d (cid:48) should have the same face geometry with the original image x d .
This is achieved by ﬁnding w d (cid:48) with small difference between the StyleGAN2 feature F d (cid:48) ( w d (cid:48) ) that is generated from w d (cid:48) , and the domain-speciﬁc embedding D d (cid:48) .
Second, the pseudo-paired data needs to be realistic and natural.
To this end, following [1], we take the mean of latent codes ¯ w , and enforce our latent code to be similar to ¯ w .
¯ w is calculated by sampling 4096 latent codes.
With the regularization, the generated image looks natural by locating the latent code in the realistic image manifold.
Finally, the latent code w d (cid:48) is found by optimizing the following equation:
[Equation]
Here, the feature map F d (cid:48) ( w ) is extracted from StyleGAN2 by feeding the latent code w to optimize.
After the optimization, the pseudo-paired data x d (cid:48) is synthesized by StyleGAN2 with w d (cid:48) .
Semantic matching loss.
The purpose of the semantic matching loss is to boost the matching performance of DIFE at pixel level with the synthesized pseudo-paired data.
The pixel location u ∈ Ω on 2D cartesian coordinate system Ω = { 0 , 1 , ..., H (cid:48) − 1 } × { 0 , 1 , ..., W (cid:48) − 1 } is used to explain the loss.
Theoretically, the pixel embedding E du at location u in DIFE of the original image has the same semantic meaning with the corresponding pixel embedding E d (cid:48) u of the pseudo-paired image.
However, the shape difference between different domains makes it infeasible to exactly match the pixel locations.
For example, the eyes of the original image x d and the eyes of the pseudo-paired data x d (cid:48) in Fig. 2(b) do not perfectly match.
To alleviate this issue, we introduce a soft distance measure similar to [44].
Intuitively, we compare the embedding from a pixel on the original image to the weighted embedding of a small region on the pseudo-paired image, as described as the contour in
Fig. 2(b).
We ﬁnd the region to compare based on the similarity σ ( E d , E d (cid:48) , u i , u j ) between the pixel embeddings as:
[Equation]
where u i and u j are the pixel coordinates of E d and E d (cid:48) , respectively.
Finally, the semantic matching loss L d,d (cid:48)
M is deﬁned as follows:
[Equation]
When learning from multiple domains, we optimize intra semantic matching loss L M 1 = (cid:80) Dd L d,dM and inter semantic matching loss L M {+
+} = (cid:80) Dd (cid:80) Dd (cid:48)
(cid:54) = d L d,d (cid:48) M for all permutations of the domains.
3.3 Training Objective
We can formulate our ﬁnal training objective with hyper-parameters as:
[Equation]
The matching losses L M 1 and L M 2 cannot be applied in the early stage of training, since the domainspeciﬁc embedding is different from the features of StyleGAN2s.
Therefore, we ﬁrst impose L K 1 and
L K 2 on the encoder and domain converter, and after initialization, we start to generate pseudo-paired data and utilize semantic matching losses L M 1 and L M 2 .
Experiments 4.1 Interspecies Keypoint Transfer
Given an image in the source domain with keypoints, the goal of interspecies keypoint transfer is to ﬁnd the corresponding locations of the keypoints on another image in the target domain.
We mainly evaluate our method on this task to demonstrate that our embedding is not only a high quality representation of cross-domain faces, but also beneﬁcial for knowledge transfer to unlabeled domains.
The task is basically achieved by ﬁnding the pixel-level correspondence between the embeddings of the source and the target images.
The performance is quantitatively evaluated by calculating the error between the positions of the transferred keypoints and their corresponding ground truth.
We further describe mathematical formulations of the evaluation in detail in the supplementary material.
Datasets.
At the training time, DIFE is trained with synthetic images and their target labels generated by pre-trained CSE and StyleGAN2 networks.
We use four datasets for evaluation: MAFL [56], AP-10K [50], WFLW [47] and AnimalWeb [23].
The MAFL is a dataset of human faces that is composed of 20k images.
The test set of MAFL excluding the shared images with the training set of CelebA is used (400 images).
The AP-10K dataset is composed of 10k images of various animal species which is split into three disjoint subsets, i.e., train, validation, and test sets, with the ratio of 7:1:2 per animal species.
As the images contain the full-body of animals, we crop the region of their faces by computing the bounding box using facial landmarks in the dataset.
For the evaluation, we make pairs of two different species by randomly selecting images from each domain.
As a result, we collect 100 pairs of humans and dogs, 89 pairs of human and cats, 47 pairs of humans and wild animals, and 89 pairs of dogs and cats. The same preprocessing is applied to WFLW and
AnimalWeb to evaluate our method on more dense keypoint annotations including the mouth.
[Table]
[Figure]
Implementation details.
We pre-train CSE with DensePose-COCO [26] and DensePose-LVIS [12], which are the datasets for full-body keypoints of human and animal respectively.
StyleGAN2 is pre-trained with FFHQ [22] for human and AFHQ [9] for animals.
DIFE encoder uses the hourglass network [32] for a fair comparison with DVE [44], and the domain converter consists of three consecutive 7 × 7 conv layers.
We use Adam optimizer [24] with the learning rate of 10 − 3 , batch size of 12, and max training step of 1M with early stopping.
The lambda values for each loss functions are ﬁxed to λ 1 = 10 − 2 , λ 2 = 10 0 , λ 3 = 10 − 2 , and λ 4 = 10 − 2 in all experiments.
All experiments are carried out on one NVIDIA Titan XP.
Baselines.
We compare our method with three baselines: CSE [31], DVE [44] and CATs [8].
Even though CSE is originally learned for full-body embeddings of interspecies data, we compare to the method to demonstrate the effectiveness of our method on face images.
DVE is an unsupervised learning approach for learning pixel-level facial embeddings, which is designed for human faces in the original work.
For the comparison, we train the method with interspecies data.
Both the training set of AP-10K and synthesized images of StyleGAN2 were used, and the latter is selected for better performance.
We implemented the baselines using the original source code available on their websites with the default hyperparameters.
CATs is the state-of-the-art for ﬁnding visual semantic correspondence.
We use the pre-trained model from the original paper trained on SPair-71k [29] that contains annotations including the landmarks of dogs, cats, and other animals.
Quantitative results.
Table 1 presents the quantitative results of CSE, DVE, and DIFE on humans, dogs, cats, and wild animals showing that our method achieves better performance in all data domains. [Table]
As can be seen in the table, transferring between human and other animals is more challenging due to the larger shape differences between faces of human and animals.
Compared to other baselines,
DIFE makes a signiﬁcant improvement with the help of the semantic loss.
The errors in all methods are lower for Dog+Cat, with DIFE performing better than other baselines.
We have also tested a challenging case of the keypoint transfer between three domains by training the models at once with human, dogs, and cats, in which DIFE also outperforms other baselines.
For the Human+Wild case, we utilized the model that was trained with human+cat, because it was difﬁcult to train with human and wild animals due to the extreme shape and style variance.
Since the wild animal domain includes animals that resemble cats, such as tigers, lions, and cheetahs, using our model trained with human and cats performed reasonably, showing that our method is applicable for out-of-domain cases. Our method also outperforms the CATs which indicates the applicability of our method to discovering landmarks in unlabeled animal domains.
Table 2 shows the quantitative result of the interspecies keypoint transfer on WFLW [47] and
AnimalWeb [23] with 9 landmarks including the corners of the eye and mouth.
Our method shows the best performance compared to previous methods on every domain pair.
Qualitative results.
Fig.
3 shows some qualitative results of the keypoint transfer.
As expected,
CSE has difﬁculty in ﬁnely distinguishing different parts of faces as it has limited face prior for animals due to the sparse annotation of DensePose-LVIS.
While DVE tends to recognize the eyes and the noses to some extent, it does not properly overcome the shape difference, showing some mismatches.
DIFE, with its ability to understand cross-domain association and face embedding, can ﬁgure out exact visual correspondence on interspecies faces.
More visualizations of the interspecies keypoint transfer including the results on WFLW and AnimalWeb can be found in the appendix.
Fig.
4 visualizes the embeddings of different methods.
CSE shows similar values across domains, but less distinction between different parts of a face.
On the other hand, StyleGAN2 features exhibit distinctiveness between different parts of a face, but corresponding parts between different domains show a large difference in values, making it difﬁcult to align between different domains.
DIFE makes the best of both worlds, showing distinctive values within a face, while having similar values between corresponding face parts between different domains. [Figure] [Figure] [Table]
[Table]
The effect of loss functions.
Table 3 shows an ablation study to verify the role of each loss.
Pixel errors are reduced by adding each loss, and any absence of a loss leads to the performance drop.
As we cannot synthesize pseudo-paired data without the knowledge distillation of StyleGAN2, the result for the absence of L K 2 is not provided.
The absence of L K 1 leads to a large performance drop of
DIFE
[Table]
and we ﬁnd that the role of knowledge distillation of CSE is essential in the early stages of learning.
The semantic matching loss also helps to increase the performance by learning more ﬁne visual correspondence across the data domains.
Which layer in StyleGAN2 is good for the knowledge distillation?
Table 4 shows the result of selecting a different layer of StyleGAN2 for acquiring the features.
Fig.
5 shows that the features from the higher layer is more ﬁne-grained but domain-speciﬁc, which makes the domain converter to become confused.
Also, because the dimension of the domain-speciﬁc embedding follows the dimension of StyleGAN2 features n d , selecting the ninth layer limits the representation power of the domain embedding.
On the other hand, the feature from the lower layer is easy for the knowledge distillation with the abundant representation power with large n d , but the information about the face is limited because of its coarseness.
With this ablation study, we selected the seventh layer of
StyleGAN2 to learn face prior of each data domain.
4.3 Applications
Image manipulation.
Fig.
6 presents image manipulation examples based on DIFE.
Putting objects like glasses and masks requires the positions of face parts, and the result shows that DIFE ﬁnds exact locations to put the objects onto the images, even on animal faces. Interspecies face parsing Fig.
7 shows the results of interspecies face parsing based on DIFE. Following the segmentation in style [33], we use k-means clustering by DIFE for unsupervised face parsing. The eye, nose, mouth, and hairy parts are discovered with a simple method which means DIFE has proper semantic information for the interspecies face.
Dense keypoint transfer.
Although DIFE extracts embeddings from all pixels, we cannot accurately evaluate all visual correspondences due to the lack of annotations.
However, we can ﬁgure out more dense visual correspondence qualitatively by transferring generated grid keypoints .
In Fig. 8,
[Figure]
the result of transferring dense keypoints from a human source to another human, dog, cat, and wild animal targets can be found.
Pseudo-paired data synthesis.
Fig.
9 shows examples of pseudo-paired data with various data domains and postures.
Given original images, we perform pseudo-paired data synthesis on the intra-domain and the inter-domain.
We can see that the generated paired images are in similar poses with the original ones, while having minor texture variations.
Conclusion
In this paper, we introduced a study for extracting common features from faces of several species, including humans, as a dense embedding.
In this process, we presented a mechanism for extracting and learning desired information from separate teacher models, even though the teacher models have different architectures and data domains.
Furthermore, we proposed a new data augmentation technique with a latent space exploration of StyleGAN2 and the semantic matching loss to overcome the extreme shape difference between species.
Various experiments and applications were offered for the evaluation, and our approach has been shown to be superior to other baselines.
For the future research directions, we need to deal with multiple animals more than three domains efﬁciently, which is not focused on this paper due to the absence of pre-trained StyleGAN2.
A research on the generalization method for the out-of-domain data is needed to address the situation where data cannot be obtained or the animal species is unknown.
Our method is dependent on the performance of pre-trained CSE and StyleGAN2.
Lastly, there are typical face understanding failure cases like occlusion or dark illumination.
As the ﬁrst work to tackle the difﬁcult problem of interspecies face embedding, more follow-up studies are necessary to overcome current limitations.
Nevertheless, we believe that this work has set the stage for a new research topic of cross-domain study for interspecies face understanding.
Broader Impact.
This study aims to understand the interspecies face, which can be potentially extended to various applications.
For example, our method can assist to ﬁnd lost pets, check the health status of farm animals through analyzing the facial expressions.
Identifying wild animals for ecological experiments is another expected application of our method.
Moreover, the proposed method can potentially be extended to the automatic motion capture, that can be used in the ﬁlm production or content creation for AR/VR.
We note that the studies using DIFE could potentially violate the personal privacy or animal rights.
The studies extending DIFE should carefully consider the above potential misuses, and bring positive impacts to the society.
Acknowledgments
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00124, Development of Artiﬁcial Intelligence Technology for Self-Improving Competency-Aware Learning Capabilities), and No. 2020-0-01361, Artiﬁcial Intelligence Graduate School Program (Yonsei University). A Metric for Keypoint Transfer In the keypoint transfer experiment, we calculate the pixel error between the transferred keypoint ˆ u t and the ground truth target keypoint u t to evaluate facial embedding quantitatively.
First, source embedding E s and target embedding E t are extracted from the source image and target image.
Then, the ground truth source keypoint u s is transferred to the target via cosine distance matching of pixel embedding formulated as
[Equation]
with 2D cartesian coordinate system Ω = { 0 , 1 , ..., H (cid:48) − 1 } × { 0 , 1 , ..., W (cid:48) − 1 } . After the keypoint transfer, pixel error e p is calculated by the euclidean distance between the transferred keypoint and the label of target formulated as following:
[Equation]
With lower pixel error, we can assure the learned embedding works well as visual descriptors as matching reliably different face geometry of various species [44]. B The reproducibility of our work
For reproducibility of our work, we provide the code blocks in the zipped supplementary ﬁle. The codes related to dataset, model, loss, training pipeline and experiment are enclosed. To make sure the training stability of our pipeline, we execute multiple training and interspecies keypoint transfer experiment with respect to the random seed.
All values for Section 4 are computed in three randomly seeded training and the statistics for some experiments are shown in Table 5.
[Table]
[Figure]
[Table]
We carry out the keypoint regression experiment setup following DVE [44] and On Equivariant
And Invariant [7] to evaluate the robustness and the accuracy of DIFE on intra-species landmark detection. In the experiment, we train a single FC layer with a frozen pre-trained embedder to predict keypoint. The NME values of previous methods are also brought from On Equivariant And Invariant. Even though our embedder is trained on synthesized datasets, not the target dataset, DIFE shows compatible performance with the early study results of each category meaning DIFE is the apposite baseline for cross-domain face understanding.
The qualitative results are also provided in Fig. D Detailed Explanation for Method Algorithm 1 shows our detailed training pipeline as pseudo-code. From line 6 to line 18, the pseudocode for multi-teacher knowledge distillation is provided.
At the start of training, latent code z is sampled with normal distribution. The synthesized image x d 1 , x d 2 and the feature of StyleGAN2 F d 1 , F d 2 is generated with the StyleGAN2 G d 1 , G d 2 . Here d 1 and d 2 means the the domain of instance.
The continuous surface embedding C d 1 , C d 2 is extracted with the CSE E . Therfore, DIFE E d 1 , E dand domain-speciﬁc embedding D d 1 (cid:41) d 1 , D d 2 (cid:41) d 2 are extracted with the encoder Φ and the domain converter Ψ d 1 , Ψ d 2 . We utilize superscript d (cid:41) d (cid:48) to express the instance of domain d (cid:48) is originated from the image of d which is useful to describe pseudo-paired data synthesis later. Finally, CSE knowledge distillation loss L K 1 and StyleGAN2 knowledge distillation loss L K 2 are calculated with
1: (0) Prepare DIFE training 2: Set the learning rate η 3: Load pre-trained weights For parameters of CSE E , StyleGAN2 G d 1 , G d4: Initialize network parameters θ Φ , θ Ψ d 1 , θ Ψ d5: for the number of iterations do 6: (1-1) Synthesize data and labels with teacher models 7: Sample latent code z ∼ N (0 , 1) 8: x d 1 , F d 1 ← G d 1 ( z ) 9: x d 2 , F d 2 ← G d 2 ( z ) 10: C d 1 , C d 2 ← E ( x d 1 , x d 2 ) 11: (1-2) Extract embedding 12: E d 1 , E d 2 ← Φ( x d 1 , x d 2 ) 13:
D d 1 (cid:41) d 1 ← Ψ d 1 ( E d 1 ) 14: D d 2 (cid:41) d 2 ← Ψ d 2 ( E d 2 ) 15: (1-3) Calculate losses For multi-teacher knowledge distillation 16: L K 1 =
L K ( E d 1 , C d 1 ) + L K ( E d 2 , C d 2 ) 17: L d 1 K 2 = L K ( D d 1 (cid:41) d 1 , F d 1 ) 18:
L d 2 K 2 = L K ( D d 2 (cid:41) d 2 , F d 2 ) 19: if not initialization period then 20: (2-1) Intra pseudo-paired data synthesis 21: x d 1 (cid:41) d 1 ← G d 1 ( D d 1 (cid:41) d 1 ) with latent space exploration 22: x d 2 (cid:41) d 2 ← G d 2 ( D d 2 (cid:41) d 2 ) with latent space exploration 23: (2-2) Inter pseudo-paired data synthesis 24: D d 2 (cid:41) d 1 ← Ψ d 1 ( E d 2 ) 25: D d 1 (cid:41) d 2 ← Ψ d 2 ( E d 1 ) 26: x d 2 (cid:41) d 1 ← G d 1 ( D d 2 (cid:41) d 1 ) with latent space exploration 27: x d 1 (cid:41) d 2 ← G d 2 ( D d 1 (cid:41) d 2 ) with latent space exploration 28: (2-3) Extract embedding 29:
[Equation] 1 (cid:41) [Equation] [Equation]
(cid:41) [Equation] [Equation] 1 (cid:41)
[Equation] [Equation] (cid:41) [Equation]
[Equation] d 2 (cid:41) d [Equation] Calculate losses For semantic matching 31:
[Equation] [Equation]
33: else 34:
[Equation] [Equation]
36: end if 37: (3) Update network parameters 38: [Equation] [Equation] [Equation]
41: end for knowledge distillation loss L K formulated as following:
[Equation]
The following codes from line 20 to line 32 presents the procedure of the pseudo-paired data synthesis and the semantic matching.
The if statement on line 18 means pseudo-paired data is not synthesized in initialization period because the data synthesis requires proper face geometry of the domain-speciﬁc embedding.
After the initialization period, the intra pseudo-paired data x d 1 (cid:41) d 1 , x d 2 (cid:41) d 2 and inter pseudo-paired data x d 1 (cid:41) d 2 , x d 2 (cid:41) d 1 are generated with latent space exploration described at Section 3.2.
With encoder Φ , each DIFE for pseudo-paired data is extracted. At last semantic matching loss
L M are utilized to get intra semantic matching loss L M 1 and inter semantic matching loss L M 2 .
OriginalImage [Figure]
Here the equation for semantic matching loss with cosine similarity σ deﬁned at Appendix B: [Equation] At the end of iteration, the parameters of the encoder θ Φ is updated with all losses and the parameters of the domain converter θ Ψ d 1 , θ Ψ d 2 are updated with StyleGAN2 knowledge distillation loss.
We provide more examples of pseudo-paired data on various combinations of original and pair domains in Fig.
Each three group of rows shows examples of pseudo-paired data for human+dog, human+cat, and dog+cat. With various pseudo-paired data, we observe the generated image has the same face geometry with the original image.
In inter pseudo-paired data, there is a little difference like the location of the ear or the shape of the eye, but the posture of the face is always aligned. By matching them, ambiguous connections between the interspecies faces can be learned in unsupervised manner.
[Figure]
The detailed licenses of utilized codes and datasets are as followings:
- Detectron2, DensePose-CSE [31]: The license of code is Aphache 2.0 License which is an open source license. - StyleGAN, FFHQ [21]: The individual images were published in Flickr by their respective authors under either Creative Commons BY 2.0, Creative Commons BY-NC 2.0, Public Domain Mark 1.0, Public Domain CC0 1.0, or U.S. Government Works license. All of these licenses allow free use, redistribution, and adaptation for non-commercial purposes.
- AFHQ [9]: The dataset is available under Creative Commons BY-NC 4.0 license by NAVER Corporation.
There is no problem to use, copy, tranform and build upon the material for non-commercial purposes as long as giving appropriate credit by citing our paper, and indicating if changes were made.
- AP-10K [50]: The dataset is available under Creative Commons BY-NC 4.0 license which is same with AFHQ. - CelebA [27], MAFL [56]: The name of license is unknown but datasets are available for non-commercial research purposes.