U NSUPERVISED D OMAIN A DAPTATION FOR D ISTANCE M ETRIC L EARNING
A BSTRACT
Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain.
However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space.
This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated.
Inspired by a theoretical analysis, we ﬁrst reformulate the disjoint classiﬁcation task, where the source and target domains correspond to non-overlapping class labels , to a veriﬁcation one.
To handle both within and cross domain veriﬁcations, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space.
Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of
FTNs on the target domain.
In experiments, we ﬁrst illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.
I NTRODUCTION
Despite strong performances on facial analysis using deep neural networks (Taigman et al., 2014;
Sun et al., 2014; Schroff et al., 2015; Parkhi et al., 2015) learning a model that generalizes across variations in attributes like ethnicity, gender or age remains a challenge.
For example, it is reported by Buolamwini & Gebru (2018) that commercial engines tend to make mistakes at detecting gender for images of darker-skinned females.
Such biases have enormous social consequences, such as conscious or unconscious discrimination in law enforcement, surveillance or security (WIRED, 2018a;b; NYTimes, 2018; GIZMODO, 2018).
A typical solution is to collect and annotate more data along the underrepresented dimension, but such efforts are laborious and time consuming.
This paper proposes a novel deep unsupervised domain adaptation approach to overcome such biases in face veriﬁcation and identiﬁcation.
Deep domain adaptation (Long et al., 2013; 2015; 2016; Tzeng et al., 2015; Ganin et al., 2016; Sohn et al., 2017; Haeusser et al., 2017; Luo et al., 2017) allows porting a deep neural network to a target domain without extensive labeling efforts.
Currently, there are two predominant approaches to deep domain adaptation.
The ﬁrst approach, domain divergence reduction learning, is motivated by the works of (Ben-David et al., 2007; 2010).
It aims to reduce the source-target domain divergence using domain adversarial training (Ganin et al., 2016; Sohn et al., 2017; Tran et al., 2018) or maximum mean discrepancy minimization (Tzeng et al., 2015; Long et al., 2015; 2016), while leveraging supervised loss from labeled source examples to maintain feature space discriminative power.
Since the theoretical basis of this approach (Ben-David et al., 2007) assumes a common task between domains, it is usually applied to a classiﬁcation problem where the source and target domains share the same label space and task deﬁnition.
The second approach considers domain adaptation as a semi-supervised learning problem and applies techniques such as entropy minimization (Grandvalet & Bengio, 2005) or self-ensembling (Laine & Aila, 2017; Tarvainen & Valpola, 2017; French et al., 2018) on target examples to encourage decisive and consistent predictions.
However, neither of those are applicable if the label spaces of source and target domains do not align.
As a motivating example, consider a cross-ethnicity generalization of face recognition problem, where the source ethnicity (e.g., Caucasian) contains labeled examples and the target ethnicity (e.g.,
African-American) contains only unlabeled examples.
When it is cast as a classiﬁcation problem, the tasks of the two domains are different due to disjoint label spaces.
Moreover, examples from different ethnicity domains almost certainly belong to different identity classes.
To satisfy such additional label constraints, representations of examples from different domains should ideally be distant from each other in the embedding space, which conﬂicts with the requirements of domain divergence reduction learning as well as entropy minimization on target examples with source domain class labels.
In this work, we aim at learning a shared representation space between a source and target domain with disjoint label spaces that not only remains discriminative over both domains but also keep representations of examples from different domains well-separated, when provided with additional label constraints.
Firstly, to overcome the limitation of domain adversarial neural network (DANN) (Ganin et al., 2016), we propose to convert disjoint classiﬁcation tasks (i.e., the source and target domains correspond to non-overlapping class labels) into a uniﬁed binary veriﬁcation task.
We term adaptation across such source and target domains as cross-domain distance metric adaptation (CD2MA) .
We demonstrate a generalization of the theory of domain adaptation (Ben-David et al., 2007) to our setup, which bounds the empirical risk for within-domain veriﬁcation of two examples drawn from the unlabeled target domain.
While the theory does not guarantee veriﬁcation between examples from different domains, we propose approaches that also address such cross-domain veriﬁcation tasks.
To this end, we introduce a Feature Transfer Network (FTN) that separates the target features from the source features while simultaneously aligning them with an auxiliary domain of transformed source features.
Speciﬁcally, we learn a shared feature extractor that maps examples from different domains to representations far apart.
Simultaneously, we learn a feature transfer module that transforms the source representation space to another space used to align with the target representation space through a domain adversarial loss.
By forging this alignment, the discriminative power from the augmented source representation space would ideally be transferred to the target representation space.
The veriﬁcation setup also allows us to introduce a novel entropy minimization loss in the form of N -pair metric loss (Sohn, 2016), termed multi-class entropy minimization (MCEM), to further leverage unlabeled target examples whose label structure is not known.
MCEM samples pairs of examples from a discovered label structure within the target domain using an ofﬂine hierarchical clustering algorithm such as HDBSCAN (Campello et al., 2013), computes the N -pair metric loss among these examples (Sohn, 2016), and backpropagates the resulting error derivatives.
In experiments, we ﬁrst perform on a controlled setting by adapting between disjoint sets of digit classes.
Speciﬁcally, we adapt from 0–4 of MNIST-M (Ganin et al., 2016) dataset to 5–9 of MNIST dataset and demonstrate the effectiveness of FTN in learning to align and separate domains.
Then, we assess the impact of our proposed unsupervised CD2MA method on a challenging cross-ethnicity face recognition task, whose source domain contains face images of Caucasian identities and the target domain of non-Caucasian identities, such as African-American or East-Asian.
This is an important problem since existing face recognition datasets show signiﬁcant label biases towards Caucasian ethnicity, leading to sub-optimal recognition performance for other ethnicities.
The proposed method demonstrates signiﬁcant improvement in face veriﬁcation and identiﬁcation compared to a source-only baseline model and a standard DANN.
Our proposed method also closely matches the performance upper bounds obtained by training with fully labeled source and target domains.
R ELATED W ORK
Research efforts in deep domain adaptation have explored a proper metric to measure the variational distance between two domains and subsequently regularize neural networks to minimize this distance.
For example, maximum mean discrepancy (Long et al., 2013; 2016; Tzeng et al., 2014; Fernando et al., 2015; Tzeng et al., 2015; Sun & Saenko, 2016) estimates the domain difference based on kernels.
As another example, domain adversarial neural networks (Ganin et al., 2016; Bousmalis et al., 2016; 2017; Sohn et al., 2017; Luo et al., 2017; Tran et al., 2018), measuring the distance using a trainable and ﬂexible discriminator often parameterized by an MLP, have been successfully adopted for several computer vision applications, such as semantic segmentation (Hoffman et al., 2016;
Tsai et al., 2018; Zhang et al., 2018) and object detection (Chen et al., 2018).
Most of those works assume a common classiﬁcation task between two domains, whereas we tackle a cross-domain distance metric adaptation problem where label spaces of source and target domains are different.
Moreover, our problem setting, an adaptation from labeled source to unlabeled target with disjoint label spaces, contains ﬂavors from both domain adaptation (DA) and transfer learning (TL), following the nomenclature of (Pan et al., 2010).
The difference in input distribution between source and target domains and the lack of labels in the target domain are similar to that of DA or transductive TL (Pan et al., 2010), while the difference in label distribution and task deﬁnitions between two domains is akin to inductive TL (Pan et al., 2010; Daumé III, 2007).
In our work, we formalize this problem in domain adaptation framework using veriﬁcation as a common task.
This is a key contribution that allows theoretical analysis on the generalization bound as presented in Section 3 and Appendix A, while allowing novel applications like cross-ethnicity face recognition.
In terms of task objective, (Hu et al., 2015; Ganin et al., 2016; Sohn et al., 2017) also deal with domain adaptation in distance metric learning, but neither learns a representation space capable of separating the source and target domains.
Resembling CD2MA, Luo et al.
(2017) considers domain adaptation with disjoint label spaces, but the problem is still cast as classiﬁcation with an assumption that the target label space is known and a few labeled target examples are provided for training.
In terms of network design, residual transfer network (Long et al., 2016), which learns two classiﬁers differ by a residual function for the source and the target domain, is closely related.
However, it only tackles the scenario where source and target domains share a common label space for classiﬁcation.
R EVISITING THE T HEORY OF D OMAIN A DAPTATION FOR V ERIFICATION
Under the domain adaptation assumption, Ben-David et al.
(2007) show that the empirical risk on the target domain X T is bounded by the empirical risk on the source domain X S and the variational distance between the two domains, provided that the source and the target domains share the classiﬁers.
Therefore, this bound is not applicable to our CD2MA setup where the label spaces of two domains are often different.
To generalize those theoretical results to our setting, we reformulate the veriﬁcation task as a binary classiﬁcation task shared across two domains.
This new binary classiﬁcation task takes a pair of images as an input and predicts the label of 1 if the pair of images shares the same identity and 0 otherwise.
Furthermore, if we now deﬁne the new source domain to be pairs of source images and the new target domain to be pairs of target images, then Theorem 1 and 2 from (Ben-David et al., 2007) can be directly carried over to bound the new target domain binary classiﬁcation error in the same manner.
That is, the empirical with-in target domain veriﬁcation loss is bounded by with-in source domain veriﬁcation loss and the variational distance between X S × X S and X T × X T .
1 Note that inputs to the binary classiﬁer are pairs of images from the same domain.
Thus, this setup only addresses adaptation of within-domain veriﬁcation to unlabeled target domains.
There are two implications from the theoretical insights on domain adaptation using veriﬁcation as a shared classiﬁcation task.
Firstly, domain adversarial training, reducing the discrepancy between the source and the target product spaces, coupled with supervised source domain binary classiﬁcation loss (i.e., veriﬁcation loss using source domain labels) can yield target representations with high discriminative power when performing within-domain veriﬁcation.
Note that in practice we approximately reduce the product space discrepancy by generic adversarial learning as done in (Ganin et al., 2016; Sohn et al., 2017).
Secondly, there is no guarantee that the aligned source and target feature spaces possess any discriminative power for cross-domain veriﬁcation task.
Thus, additional actions in the form of a feature transfer module and domain separation objective are required to address this issue.
These two consequences together motivate the design of our proposed framework, which is introduced in the next section.
F EATURE T RANSFER N ET : L EARNING TO A LIGN AND S EPARATE D OMAINS
In this section, we ﬁrst deﬁne the CD2MA problem setup and motivate our proposed feature transfer network (FTN).
Then we elaborate on the training objectives that help our model achieve its desired properties.
Lastly, we provide practical considerations to implement our proposed algorithm.
4. P ROBLEM S TATEMENT AND A LGORITHM O VERVIEW
Recall the description of CD2MA, given source and target domain data distributions X S and X T , our goal is to verify whether two random samples x, x (cid:48) drawn from either of the two distributions (and we do not know which distribution x or x (cid:48) come from a priori) belong to the same class.
There are 3 scenarios of constructing a pair: x, x (cid:48)
∈ X S , x, x (cid:48) ∈ X T , or x ∈ X S , x (cid:48) ∈ X T .
We refer the task of the ﬁrst two cases as within-domain veriﬁcation and the last as cross-domain veriﬁcation.
[Figure] If x, x (cid:48) ∈ X S (or X T ), we need a source (or target) domain classiﬁer 2 .
For the source domain, we are provided with adequate labeled training examples to learn a competent classiﬁer.
For the target domain, we are only given unlabeled examples.
However, with our extension of Theorem 1 andfrom (Ben-David et al., 2007), discriminative power of the classiﬁer can be transferred to the target domain by adapting the representation spaces of X T × X T and X S × X S , that is, we can utilize the same competent classiﬁer from the source domain to verify target domain pairs if two domains are well-aligned.
For the third scenario where x ∈ X S but x (cid:48) ∈ X T , we assume that the two examples cannot be of the same class, which is true for problems such as cross-ethnicity face veriﬁcation.
Our proposed framework, Feature Transfer Network (FTN), is designed to solve all these veriﬁcation scenarios in an uniﬁed framework.
FTN is composed of multiple modules as illustrated in Figure 1.
First, a feature generation module f : X → Z denoted as “Gen” in Figure 1 ideally maps X S and X T to distinguishable representation spaces, that is, f ( X S ) and f ( X T ) are far apart.
To achieve this, we introduce a domain separation objective.
3 Next, the feature transfer module g : Z → Z denoted as “Tx” in Figure 1 transforms f ( X S ) to g ( f ( X S ))
for it to be aligned with f ( X T ) .
To achieve this, we introduce a domain adversarial objective .
Finally, we apply veriﬁcation losses on f ( X S ) and g ( f ( X S )) using classiﬁers h f , h g :
Z × Z → { 0 , 1 } .
During testing, we compare the metric distance between f ( x ) and f ( x (cid:48) ) .
Overall, we achieve the following desired capabilities:
• If x, x (cid:48) are from different domains, f ( x ) and f ( x (cid:48) ) will be far away due to the functionality of the feature generation module.
• If x, x (cid:48) ∈ X S , then f ( x ) and f ( x (cid:48) ) will be close if they belong to the same class and far away otherwise, due to the discriminative power acquired from optimizing h f .
• If x, x (cid:48) ∈ X T , then f ( x ) and f ( x (cid:48) ) will be close if they belong to the same class and far otherwise, due to the discriminative power acquired by optimizing h g with domain adversarial training.
4. T RAINING O BJECTIVES
We ﬁrst deﬁne individual learning objectives of the proposed Feature Transfer Network and then present overall training objectives of FTN.
For ease of exposition, all objectives are to be maximized.
Veriﬁcation Objective.
For a pair of source examples, we evaluate the veriﬁcation losses at two representations spaces f ( X S ) and g ( f ( X S )) using classiﬁers h f
and h g as follows:
[Equation]
[Equation]
where g i = g ( f ( x i )) , f i = f
( x i ) and y 12 = 1 if x 1 and x 2 are from the same class and 0 otherwise.
While classiﬁers h f , h g can be parameterized by neural networks, we aim to learn a generator f and g whose embeddings can be directly used as a distance metric.
Therefore, we use non-parameteric classiﬁers h f
= σ ( f (cid:62) 1 f 2 ) , h g = σ ( g (cid:62) 1 g 2 ) where σ ( a ) = 1 1+exp( − a ) .
Domain Adversarial Objective.
Let D 1 : Z → (0 , 1) be a domain discriminator.
As mentioned earlier, D 1 is trained to discriminate distributions f ( X T ) and g ( f ( X S )) and then produces gradient for them to be indistinguishable.
The learning objectives are written as follows:
[Equation]
Note that when feature transform module is an identity mapping, i.e., g ( f ( x )) =
f ( x ) , Equation (3) deﬁnes the training objective of standard DANN.
Domain Separation Objective.
The goal of this objective is to distinguish between source and target at representation spaces of generation module.
To this end, we formulate the objective using another domain discriminator D 2 : Z → (0 , 1) :
[Equation]
Note that, in L sep , the source space f ( X S ) is not only pushed apart from the target space f
( X T ) but also from the augmented source space g ( f ( X S )) to ensure that g learns meaningful transformation of source domain representation beyond identity transformation.
Training FTN.
Now we are ready to present the overall training objectives L f and L g :
[Equation]
(5) with λ 1 for domain adversarial objective and λ 2 for domain separation objective.
We use L D 1 in
Equation (3) for D 1 and L D 2 =
L sep for D 2 .
We alternate updating between D 1 and ( f, g, D 2 ) .
4. P RACTICAL C ONSIDERATIONS
Preventing Mode Collapse via Feature Reconstruction Loss.
The mode collapsing phenomenon with generative adversarial networks (GANs) (Goodfellow et al., 2014) has received much attention (Salimans et al., 2016).
In the context of domain adaptation, we also ﬁnd it critical to treat the domain adversarial objective with care to avoid similar optimization instability.
In this work, we prevent the mode collapse issue for domain adversarial learning with an additional regularization method similar to (Sohn et al., 2017).
Assuming the representation of the source domain is already close to optimal, we regularize the features of source examples to be similar to those from the reference network f ref : X → Z , which is pretrained on labeled source data and ﬁxed during the training of f .
Furthermore, we add a similar but less emphasized ( λ 4 < λ 3 ) regularization to target examples, simultaneously avoiding collapsing and allowing more room for target features to diverge from the original representations.
Finally, the feature reconstruction loss is written as follows:
[Equation]
We empirically ﬁnd that without the feature reconstruction loss, the training would become unstable, reach an early local optimum and lead to suboptimal performance (see Section 6 and Appendix C).
Thus, we always include the feature reconstruction loss to train DANN or FTN models unless stated otherwise.
Replacing Veriﬁcation Loss with N -pair Loss.
Our theoretical analysis in Section 3 (and Appendix A) suggests to use a veriﬁcation loss that compares similarity between a pair of images.
In practice, however, the pairwise veriﬁcation loss is too weak to learn a good deep distance metric.
Following (Sohn, 2016), we propose to replace the veriﬁcation loss with an N -pair loss, deﬁned as follows:
[Equation]
where x n and x + n are from the same class and x n and x + k , n (cid:54)
=
k , are from different classes.
Replacing
L vrf into L N [Equation] the training objective of FTN with N -pair loss is written as follows:
[Equation]
[Figure]
E NTROPY M INIMIZATION VIA H IERARCHICAL C LUSTERING
Entropy minimization (Grandvalet & Bengio, 2005) is a popular training objective in unsupervised domain adaptation: unlabeled data is trained to minimize entropy of a class prediction distribution so as to form features that convey conﬁdent decision rules.
However, it is less straightforward how to apply entropy minimization when label spaces for source and target are disjoint.
Motivated from
Section 3, we extend entropy minimization for distance metric adaptation using veriﬁcation as a common task for both domains:
[Equation]
(9) where p ij (cid:44) p ij ( f ) =
σ ( f ( x i ) (cid:62) f
( x j )) .
This formulation encourages a more conﬁdent prediction for verifying two unlabeled images, whether or not coming from the same class.
However, recall that for the source domain, we use N -pair loss instead of pair-wise veriﬁcation loss for better representation learning.
Therefore, we would like to similarly incorporate the concept of Npair loss on the target domain by forging a multi-class entropy minimization (MCEM) objective.
This demands N pair examples to be sampled from the target domain.
As the target domain is unlabeled, we ought to ﬁrst discover a plausible label structure, which is done off-line via HDBSCAN (Campello et al., 2013; McInnes et al., 2017), a fast and scalable density-based hierarchical clustering algorithm.
The returned clusters provide pseudo-labels to individual examples of the target domain, allowing us to sample N pair examples to evaluate the following MCEM objective:
[Equation]
where x n and x + n are from the same cluster and x n and x + k , n (cid:54)
=
k are from different clusters.
The objective can be combined with L f in Equation (8) to optimize f .
E XPERIMENTS
In this section, we ﬁrst experiment on digit datasets as a proof of concept and compare
our proposed
FTN to DANN.
Then, we tackle the problem of cross-ethnicity generalization in the context of face recognition to demonstrate the effectiveness of FTN.
In all experiments, we use N -pair loss as deﬁned in Equation (8) to update f and g for better convergence and improved performance.
We also use the same learning objectives for DANN while ﬁxing g to the identity mapping and λ 2 = 0 .
6. P ROOF OF C ONCEPT : MNIST-M (0–4) TO MNIST (5–9)
To provide insights on the functionality of FTN, we conduct an experiment adapting the digits 0– from MNIST-M
(Ganin et al., 2016) to 5–9 from MNIST.
In other words, the two domains in our setting not only differ in foreground and background patterns but also contain non-overlapping digit classes, contrasting the usual adaptation setup with a shared label space.
Our goal is to learn a feature space that separates the digit classes not only within each domain, but also across the two.
We construct a feature generator f composed of a CNN encoder followed by two fully-connected (FC) layers and a feature transfer module g composed of MLP with residual connections.
Outputs of f and g are then fed to discriminators D 1 and D 2 parameterized by MLPs to induce domain adversarial and domain separation losses respectively.
We provide more architecture details in Appendix B.1.
We visualize t-SNE plots of generator features in Figure 2.
Without an adaptation (Figure 2(a)), features of digits from the target domain are heavily mixed with those from the source domain as [Table] [Table] [Table]
well as one another.
The model reaches 1 .
3% veriﬁcation error in the source domain but as high as 27 .
3% in the target domain.
Though DANN in Figure 2(b) shows better separation with a reduced target veriﬁcation error of 2 . 2% , there still exists signiﬁcant overlap between digit classes across two domains, such as 3/5, 4/9, 0/6 and 2/8.
As a result, a domain classiﬁer trained to distinguish source and target on top of generator features can only attain 11 .
5% classiﬁcation error.
In contrast, the proposed FTN in Figure 2(c) shows 10 clean clusters without any visual overlap among 10 digits classes from either source or target domain, implying that it not only separates digits within the target domain ( 2 . 1% veriﬁcation error), but also differentiates them across domains ( 0 . 3% domain classiﬁcation error).
6. C ROSS E THNICITY F ACE V ERIFICATION AND R ECOGNITION
The performances of face recognition engines have signiﬁcantly improved thanks to recent advances in deep learning for image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015;
Szegedy et al., 2015;
He et al., 2016) and publicly available large-scale face recognition datasets (Yi et al., 2014; Guo et al., 2016).
However, most public datasets are collected from the web by querying celebrities, with signiﬁcant label bias towards Caucasian ethnicity.
For example, more than 85% of identities are Caucasian for CASIA Web face dataset (Yi et al., 2014).
Similarly, 82% are
Caucasian (CAU) for MS-Celeb-1M (MS-1M) dataset (Guo et al., 2016), while there are only 9 . 7%
African-American (AA), 6 . 4% East-Asian (EA) and less than 2% Latino and South-Asian combined.
Such imbalance across ethnicity in labeled training data can result in signiﬁcant drop in identiﬁcation performance on data-scarce minorities: the second row of Table 1 shows a model trained on Caucasian dominated dataset performs poorly on the other ethnicities.
As expected, if the training data is composed of only Caucasian identities as source domain, the performance over the target domains consisting of the other ethnicities further deteriorates (see row 1 of Table 1).
Provided the available labeled source domain contains only Caucasian identities, we subsequently demonstrate that our method can effectively leverage unlabeled data from the non-Caucasian target ethnicity to substantially improve their face veriﬁcation performances.
Experimental Setup.
We perform an adaptation from CAU to a mixture of AA and EA.
Our experiments use the MS-1M dataset.
We ﬁrst remove identities that both appear in the training and testing sets.
The resulting training set consists of 4 .
M images from 60 K CAU identities, 398 K images from 7 K AA identities, and 308 K images from 4 .
6 K EA identities.
For domain adaptation experiments, we use labeled CAU images and unlabeled AA, EA images for training.
For supervised experiments to obtain performance lower and upper bound, we use labeled CAU images to train Sup C and labeled CAU, AA, EA images to train Sup C,A,E .
We adopt a 38-layer ResNet (He et al., 2016) for the feature generation module.
Feature transfer module and discriminators are parameterized with MLPs similarly to Section 6.1.
We use 4096 -pair loss for training, including for the supervised CNNs.
It is worth mentioning that our network architecture and training scheme result in strongly competitive face recognition performance, comparing to other state-of-the-art methods such as FaceNet (Schroff et al., 2015) on YouTube Faces (Wolf et al., 2011) ( 97 .
32% (ours) vs 95 .
and Neural Aggregation Network (Yang et al., 2017) on IJB-A (see row 2 of Table 3).
The complete network architecture and training details are provided in Appendix B.2.
Evaluation.
We report the performance of the baseline and our proposed models on two standard face recognition benchmarks LFW (Huang et al., 2007) and IJB-A (Klare et al., 2015).
Note that these datasets also exhibit signiﬁcant ethnicity bias.
To highlight the effectiveness of the proposed adaptation approach, we construct individual test set for CAU, AA, EA, each of which contains 10 face images from 200 identities.
We refer to our testing set as the Cross-Ethnicity Faces (CEF) dataset.
We apply two evaluation metrics on CEF dataset, veriﬁcation accuracy and identiﬁcation accuracy.
For veriﬁcation, following the standard protocol (Huang et al., 2007), we construct 10 splits, each containing 900 positive and 900 negative pairs, and compute the accuracy on each split using the threshold found from the other 9 splits.
For identiﬁcation, a pair composed of the reference and the query images from the same identity is considered correct if there is no image from different identity that has higher similarity to the reference image than the query image.
We evaluate identiﬁcation accuracy per ethnicity ( 200 -way) as well as across all ethnicities ( 600 -way).
Results.
The results on CEF are summarized in Table 1.
Cross domain identiﬁcation accuracy is reported in Table 2, where we use AA and EA as negative classes when evaluating accuracy on CAU and vice versa, as a measure to indicate domain discrepancy.
Among adaptation models, DANN without feature reconstruction loss (DANN \L recon ) shows unstable training and easily degenerate, which leads to only marginal improvement upon Sup C .
Similar trend is observed while training FTN.
Therefore, to ensure training stability, we impose L recon as a regularization term for all adaptation models.
More analysis on the effectiveness of L recon is provided in Appendix C.
When testing on AA and EA with model trained on only the labeled source CAU domain (Sup C ), we observe signiﬁcant performance drops in Table 1.
Meanwhile, in Table 2, cross domain identiﬁcation accuracy is much higher than within domain identiﬁcation accuracy, i.e., 96.14% of AA vs. CAU is much higher than 71.92% of AA identiﬁcation in Table 1, indicating 1) signiﬁcant discrepancy between the feature spaces of the source and target domains and 2) lack of discriminative power for within domain veriﬁcation task on target ethnicity.
Comparing to Sup C , both DANN and FTN show moderate improvement when testing on AA and EA from CEF (Table 1), demonstrating the effectiveness of domain adversarial learning in transferring within domain veriﬁcation capability from labeled source domain to unlabeled target domain.
Despite the improvement, DANN suffers a notable drawback from adversarial objective which attempts to align identities from different domains, resulting a poor cross domain identiﬁcation accuracy as shown in Table 2.
In contrast, the proposed FTN achieves much higher cross domain identiﬁcation accuracy, demonstrating both within and cross domain discriminative power.
Additionally, in combination with the multi-class entropy minimization (FTN+MCEM), we further boost the veriﬁcation and identiﬁcation accuracy over FTN on AA and EA as well as approach the accuracy of Sup C,A,E , the performance upper bound.
This indicates that the HDBSCAN-based hierarchical clustering provides high quality pseudo-class labels for MCEM to be effective.
Indeed, the clustering algorithm achieves F-score as high as 96 . 31% and 96 .
34% on AA and EA.
We provide more in-depth analysis on the clustering strategy in Appendix D.
Finally, Table 3 reports the performance of face recognition models on standard veriﬁcation and recognition benchmarks.
We observe similar improvements with our proposed distance metric adaptation when only using labeled CAU, i.e., source domain, as training data.
Once the task becomes more challenging thus demands more discriminative power, the advantage of our method becomes more evident, such as in the case of open-set recognition and veriﬁcation at low FAR.
C ONCLUSION
We address the challenge of unsupervised domain adaptation when the source and the target domains have disjoint label spaces by formulating the classiﬁcation problem into a veriﬁcation task. We
[Table] propose a Feature Transfer Network, allowing simultaneous optimization of domain adversarial loss and domain separation loss, as well as a variant of N -pair metric loss for entropy minimization on the target domain where the ground-truth label structure is unknown, to further improve the adaptation quality.
Our proposed framework excels at both within-domain and cross-domain veriﬁcation tasks.
As an application, we demonstrate cross-ethnicity face veriﬁcation that overcomes label biases in training data, achieving high accuracy even for unlabeled ethnicity domains, which we believe is a result with vital social signiﬁcance.
Appendix A D ERIVATION FOR G ENERALIZATION B OUND OF T ARGET D OMAIN V ERIFICATION L OSS
Let ( X , F ) and ( Z , G ) be measurable input and feature spaces respectively and a feature extractor
R : X → Z be a measurable function.
Let µ be a probability measure on X corresponding to the data distribution.
Let ( X 1 , F , µ 1 ) =
( X 2 , F , µ 2 ) =
( X , F , µ ) and µ 12 = µ 1 × µ 2 on X 1 ×X 2 be the unique product measure (Forrest).
Similarly, we construct Z 1 ×Z 2 where ( Z 1 , G ) =
( Z 2 , G ) =
( Z , G ) .
Since
R is measurable, R 2 :
X 1 ×X 2 → Z 1 ×Z 2 where R 2 ( x 1 , x 2 )
=
( R ( x 1 ) , R ( x 2 )) is also measurable (see Lemma 1 for the proof).
Then we can obtain an induced probability measure for Z 1 ×Z 2 from
R 2 , denoted as ˜ µ 12 = µ 12 ◦
( R 2 ) − 1 (Proposition 1.34 from (Lalley, 2017)).
Let Y : X 1 ×X 2 → { 0 , 1 } , where 1 represents the pair from the same identity and 0 otherwise, 6 be the stochastic target function for ground truth labeling, φ ( x 1 , x 2 ) =
E [ Y ( x 1 , x 2 )] be the expectation of the label at ( x 1 , x 2 ) , and ˜ φ ( z 1 , z 2 ) =
E [ φ ( x 1 , x 2 ) | R ( x 1 )=
z 1 , R ( x 2 )=
z 2 ] be the conditional expectation of φ given the value of R 2 ( x 1 , x 2 ) =
( z 1 , z 2 ) .
Now, consider two domains, namely the source domain with probability measure µ S over X and induced probability measure ˜ µ S over
Z 1 ×Z 2 , as well as target domain counterparts µ T and ˜ µ T .
Provided with a deterministic hypothesis class H ⊆ { g : Z 1 ×Z 2 → { 0 , 1 }} of VC-dimension d , suppose there exists a function h ∈ H that can predict both source and target domains reasonably well.
Then, we can quantify ˜ φ to be λ -close to H :
[Equation]
λ [Equation] where
[Equation]
We are ready to deﬁne the variational distance between the two domains with respect to H :
[Equation]
So far, we have successfully prepared the components in our veriﬁcation setup to meet the assumptions and the format required by Theorem 1 from (Ben-David et al., 2007).
We may now directly apply the theorem:
Theorem 1.
Randomly sample a labeled set of size m by applying R 2 to samples from X 1 ×X 2 with labels deﬁned according to Y , with probability at least 1 − δ , ∀ h ∈ H
[Equation]
Furthermore, d H (˜ µ S , ˜ µ T ) can be empirically approximated by ﬁnite samples from both domain (Kifer et al., 2004), using the binary classiﬁer from H that can best distinguishes pairs of samples between two domains.
Following Theorem 2 from (Ben-David et al., 2007), let ˜ U S and ˜ U T consist of n random pairs of samples from source and target each, with probability at least 1 − δ , we have :
[Equation]
n , where d H ( ˜ U S , ˜ U T ) = 2 (cid:16) 1 − 2 min h ∈H 12 n (cid:80) [Equation] ni
[Equation]
|
[Equation]
[Equation]
,i
[Equation]
z
[Equation]
For completeness of our analysis, we formalize and prove in Lemma 1 that R 2 is measurable.
Lemma 1.
Let ( X , F , µ ) and ( Z , G , ˜ µ ) be measurable spaces and let ( X × X , σ ( F × F ) , µ × µ ) , ( Z × Z , σ ( G × G ) , ˜ µ × ˜ µ ) be their product spaces with the product measures.
Let R : X → Z be a measurable function, then R 2 : X × X
→
Z × Z where R 2 ( x 1 , x 2 )
=
( R ( x 1 ) , R ( x 2 )) is also measurable.
Proof.
As the σ -algebra of Z × Z is generated by G × G , we only need to show that the pre-image of any generator is measurable.
Let G 1 × G 2 ∈ G × G , then it is easy to see that ( R 2 ) − 1 ( G 1 × G 2 ) =
R − 1 ( G 1 ) × R − 1 ( G 2 ) .
Since R is a measurable function, hence R − 1 ( G 1 ) and
R − 1 ( G 2 ) are measurable and so is R − 1 ( G 1 )
× R − 1 ( G 2 ) measurable.
[Figure]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
B N ETWORK A RCHITECTURE AND T RAINING D ETAILS B. T OY E XPERIMENTS : MNIST-M ( 0 − 4 ) TO MNIST ( 5 − 9 )
Following (Haeusser et al., 2017), we preprocess the data by subtracting a channel-wise pixel mean and dividing by channel-wise standard deviation of pixel values.
For MNIST examples, we also apply color-intensity inversion.
All images are resized into 32 × 32 with 3 channels.
Our feature generator module is composed of 6 convolution layers and 3 max-pooling layers followed by 2 fully-connected layers.
We use ReLU (Nair & Hinton, 2010) after convolution layers.
The output dimension of the feature generator module is 128 and is normalized to have L2-norm of 2 .
The full description of the generator module is in Table S1.
The feature transfer module maps 128 dimensional vector into the same dimensional vector using two fully-connected layers ( 128 − 256 − 256 − 128 ) and residual connection as in Figure 1(a).
Discriminator architectures are similar to that in Figure 1(b) but with fully-connected layers whose output dimensions are 128 instead of 256 .
We use Adam stochastic optimizer with learning rate of 0 .
0003 , λ 1 = 0 .
3 and λ 2 = 0 .
03 to train FTN.
B
C ROSS E THNICITY F ACE V ERIFICATION AND R ECOGNITION
Our experimental protocols, such as data preprocessing and network architecture, closely follow those of (Sohn et al., 2017).
We preprocess face images by detecting (Yang et al., 2016), aligning (Yu et al., 2016), and cropping to provide face images of size 110 × 110 .
The data is prepared for network training by random cropping into 100 × 100 with horizontal ﬂip with a 50% chance and converting into gray-scale.
Our feature generation module contains 38 layers of convolution with several residual blocks and max pooling layers.
We use ReLU (Nair & Hinton, 2010) for most of the layers in combination with maxout nonlinearities (Goodfellow et al., 2013).
We add 7 × 7 average pooling layer on top of the last convolution layer.
The output of the feature generation module is 320 dimensional vector and is normalized to have L2-norm of size 12 .
The full description of the model is in Table S2.
The feature transfer module maps 320 dimensional output vector from feature generation module into the same dimensional vector using two fully-connected layers and residual connection.
The architecture of feature transfer module is described in Figure 1(a).
Discriminators have similar network architecture besides different numbers of neurons and omitted residual connection.
All models, including supervised CNNs (Sup C , Sup C,A,E ), are trained with 4096 -pair loss.
For Sup C and Sup C,A,E , we use Adam stochastic optimizer (Kingma & Ba, 2015) with the learning rate of 0 .
0003 for the ﬁrst 12 K updates and 0 . 0001 and 0 .
00003 for the next two subsequent 3 K updates.
Our feature generation module is initialized with the Sup C model, which is also used as a reference network for feature reconstruction loss as described in Section 4.3.
Other modules of our model, such as feature generation module and discriminators, are initialized randomly.
All modules are then updated with the learning rate of 0 . 00003 .
Hyperparameters of different models are summarized in
Table S
[Table] [Table]
C I MPACT OF F EATURE R ECONSTRUCTION L OSS ON D OMAIN A DVERSARIAL T RAINING
We demonstrate the effectiveness of feature reconstruction loss in stabilizing the domain adversarial training in DANN framework.
We train four different DANN models with different conﬁgurations of λ 3 and λ 4 .
We visualize in Figure S2 the performance curves of identiﬁcation accuracy evaluated on the AA, EA [Table] and CAU ethnicities of CEF dataset.
Note that we stop training early on when the performance start to degrade signiﬁcantly.
Therefore, x -axis, the number of training epoch, of different curves are different.
y -axis represents the identiﬁcation accuracy.
As we see in Figure S [Table] the performance of all models on the target ethnicities start to improve in the beginning of training from those of the pretrained reference network.
Soon after, however, the accuracy starts to drop when values of either λ 3 or λ 4 are set to 0 .
Note that even in that situation the performance on the CAU set still remains high, which implies the failure of discriminative information transfer.
On the other hand, our proposed feature reconstruction loss with non-zero values of λ 3 and λ 4 (Figure 2(d)) shows much more stable performance curve.
Nonetheless, values of λ 3 and λ 4 should be carefully selected since the feature generation module of DANNs or FTNs will remain almost the same to the reference network when they are set too strong and the effectiveness of the domain adversarial loss will be reduced.
In our experiment, we use λ 3 = 0 .
1 and λ 4 = 0 .
01 for
DANN, λ 3 = 0 .
03 and λ 4 = 0 .
01 for FTN.
For FTN with entropy minimization we further reduce λ 4 = 0 .
003 to give more ﬂexibility in updating model parameters based on entropy loss.
[Figure]
D P ERFORMANCE OF U NSUPERVISED H IERARCHICAL C LUSTERING
In this section, we provide analysis on the performance of our clustering strategy by measuring the clustering accuracy.
Speciﬁcally, we measure the veriﬁcation precision and recall as follows:
[Equation]
where y i is the ground-truth class label of an example x i , and ˆ y i is an index of an assigned cluster.
Precision computes the proportion of positive pairs among pairs assigned to the same cluster, i.e., purity of returned clusters, and recall computes the proportion of positive pairs assigned to the same cluster.
Ideally, we expect high precision and high recall, i.e., high F-score, to ensure examples with the same class labels are assigned to the same cluster.
Note that we only use clusters of size 5 or larger as new target classes and discard examples assigned to a cluster whose size is less than 5 .
Here, in addition to our proposed clustering strategy, we also evaluate the clustering performance that clusters target examples by ﬁnding a nearest classes or examples from the source domain, which are shown to be effective for zero-shot learning (Vinyals et al., 2016) or semi-supervised domain adaptation with disjoint source and target classes (Luo et al., 2017).
In this case, we call two examples from the target domain are assigned to the same cluster if the nearest source examples are the same.
We also measure the clustering performance by matching the nearest source classes.
The summary result is provided in Table S4.
Firstly, we observe extremely low precision when using source domain examples or clusters as a proxy to relate target examples.
We believe that this idea of “clustering by ﬁnding the nearest source classes” works under a cross-category similarity assumption between disjoint classes of source and target domains.
In other words, it assumes that there exists a certain source class closer to examples from certain target class, so that those examples from the same target class can be clustered around that source class, even though those matching source and target classes are indeed different (e.g., 3/5, 2/8, 4/9, and 0/6 in Section 6.1).
Unfortunately, such an assumption does not hold for our problem, maybe due to the huge number of identity classes ( 60 K ) in the source domain.
On the other hand, using hierarchical clustering on target features achieves signiﬁcantly higher precision and recall.
Especially, when using embedding vectors of Sup C , we achieve 100% precision, which means that all clusters are pure even though some ground-truth classes might be separated into multiple clusters.
We observe slightly lower precision using FTN features but much higher recall, achieving higher F-score overall.
Further, the number of examples returned with FTN feature ( 253 K and 195 K for AA and EA, respectively) is higher than with Sup C feature ( 217 K and 165 K ).
Repeating the process using feature of FTN+MCEM model improves the F-score while returning more target examples that are with cluster assignment ( 276 K and 214 K ).
This not only shows the
[Table]
improved discriminative quality of features by FTNs, but also suggests a potential tool for automatic labeling of unlabeled data by iterative training of FTN model and hierarchical clustering.
E V ISUALIZATION OF E THNICITY A NNOTATED I MAGE S AMPLES
We visualize few images from each ethnicity subset in Figure S3 for annotation quality assurance.
[Figure]