D ISJOINT M APPING N ETWORK FOR C ROSS - MODAL M ATCHING OF V OICES AND F ACES
A BSTRACT
We propose a novel framework, called Disjoint Mapping Network (DIMNet), for cross-modal biometric matching, in particular of voices and faces.
Different from the existing methods, DIMNet does not explicitly learn the joint relationship between the modalities.
Instead, DIMNet learns a shared representation for different modalities by mapping them individually to their common covariates.
These shared representations can then be used to ﬁnd the correspondences between the modalities.
We show empirically that DIMNet is able to achieve better performance than the current state-of-the-art methods, with the additional beneﬁts of being conceptually simpler and less data-intensive.
The code is made available at https://github.com/ydwen/DIMNet .
I NTRODUCTION
A person’s face is predictive of their voice.
Biologically, the genetic, physical and environmental inﬂuences that affect the face also affect the voice.
Humans have been shown to be able to associate voices of unknown individuals to pictures of their faces (Kamachi et al., 2003).
Humans also show improved ability to memorize and recall voices when previously exposed to pictures of the speaker’s face, but not imposter faces (McAllister et al., 1993; Schweinberger et al., 2007; 2011).
Cognitively, studies indicate that neuro-cognitive pathways for voices and faces share common structure (Ellis, 1989), possibly following parallel pathways within a common recognition framework (Belin et al., 2004; 2011).
The above studies lend credence to the hypothesis that it may be possible to ﬁnd associations between voices and faces algorithmically as well.
With this in perspective, this paper focuses on the task of devising computational mechanisms for cross-modal matching of voice recordings and images of the speakers’ faces.
The speciﬁc problem we look at is the one wherein we have an existing database of samples of people’s voices and images of their faces, and we aim to automatically and accurately determine which voices match to which faces.
This problem has seen signiﬁcant research interest, in particular since the recent introduction of the VoxCeleb corpus (Nagrani et al., 2017), which comprises collections of video and audio recordings of a large number of celebrities.
The existing approaches (Nagrani et al., 2018b;a; Kim et al., 2018) have generally attempted to directly relate subjects’ voice recordings and their face images, in order to ﬁnd the correspondences between the two.
Nagrani et al.
(2018b) formulates the mapping as a binary selection task: given a voice recording, one must successfully select the speaker’s face from a pair of face images (or the reverse – given a face image, one must correctly select the subject’s voice from a pair of voice recordings).
They model the mapping as a neural network that is trained through joint presentation of voices and faces to determine if they belong to the same person.
In Kim et al.
; Nagrani et al.
(2018a), the authors attempt to learn common embeddings ( i.e. , vector representations) for voices and faces that can be compared to one another to identify associations.
The networks that compute the embeddings are also trained through joint presentation of voices and faces, to maximize the similarity of embeddings derived from them if they belong to the same speaker.
In all cases, the voice and face are implicitly assumed to directly inform about one another.
In reality, though, it is unclear how much these models capture the direct inﬂuence of the voice and face on one another, and how much is explained through implicit capture of higher-level variables such as gender, age, ethnicity etc., which individually predict the two.
These higher-level variables,
C o n t r a s t i v e
L o ss
T r i p l e t L o ss
M u l t
i -t a s k
C l a ss i f
i c a t i o n
Multiple Covariates Supervision
[Figure]
Identity
Nationality
[Figure]
P a i r C o n s t r u c
[Figure]
i
[Figure]
T r i p l e
[Figure]
t
[Figure]
B i n a r y
C l a ss i f
i c a t i o n
Concat
[Figure]
not?
P a i r C o n s t r u c t i o n
[Figure]
[Figure] [Figure]
Dependence between different modalities and
[Figure]
covariates which we will refer to as covariates 1 can, in fact, explain much of our ability to match voices to faces (and vice versa) under the previously mentioned “select-from-a-pair” test (where a voice must be used to distinguish the speaker’s face from a randomly-chosen imposter).
For instance, simply matching the gender of the voice and the face can result in an apparent accuracy of match of up to 75% in a gender-balanced testing setting.
Even in a seemingly less constrained “veriﬁcation” test, where one must only verify if a given voice matches a given face, matching them based on gender alone can result in an equal error rate of 33% (Appendix B).
Even matching the voice and the face by age ( e.g. matching older-looking faces to older-sounding voices) could result in match accuracy that’s signiﬁcantly better than random.
Previous studies (Nagrani et al., 2018b; Kim et al., 2018) attempt to disambiguate the effect of multiple covariates through stratiﬁed tests that separate the data by covariate value.
The results show that at least some of the learned associations are explained by the covariate, indicating that their learning approaches do utilize the covariate information, albeit only implicitly.
In this paper, we propose a novel framework to learn mappings between voices and faces that do not consider any direct dependence between the two, but instead explicitly exploit their individual dependence on the covariates.
We deﬁne covariate as the identity-sensitive factors that can simultaneously affect voice and face, e.g. nationality, gender, identity (ID), etc.
We do not require the value these factors take to be the same between the training and test set, since what we are learning is the nature of the covariation with the variable in general, not merely the covariation with the speciﬁc values the variable takes in the training set.
In contrast to existing methods where supervision is provided through the correspondence of voices and faces, our learning framework, Disjoint Mapping
Network (DIMNet), obtains supervision from common covariates, applied separately to voices and faces, to learn common embeddings for the two.
The comparison between the existing approaches and DIMNets are illustrated in Fig.
DIMNet comprises individual feature learning modules which learn identically-dimensioned features for data from each modality, and a uniﬁed input-modality-agnostic classiﬁer that attempts to predict covariates from the learned feature.
Data from each modality are presented separately during learning; however the uniﬁed classiﬁer forces the feature representations learned from the individual modalities to be comparable.
Once trained, the classiﬁer can be removed and the learned feature representations are used to compare data across modalities.
The proposed approach greatly simpliﬁes the learning process and, by considering the modalities individually rather than as coupled pairs, makes much more effective use of the data.
Moreover, if multiple covariates are known, they can be simultaneously used for the training through multi-task learning in our framework (see Fig. 2).
Compared to current methods (Nagrani et al., 2018b;a; Kim et al., 2018), DIMNets achieve consistently better performance, indicating that direct supervision through covariates is more effective in
[Figure]
these settings.
We ﬁnd that of all the covariates, ID provides the strongest supervision.
The results obtained from supervision through other covariates also match what may be expected.
Our contributions are summarized as follows:
• We propose DIMNets, a framework that formulates the problem of cross-modal matching of voices and faces as learning common embeddings for the two through individual supervision from one or more covariates, in contrast to current approaches that attempt to map voices to faces directly.
An overview of our framework is given in Fig.
• In this framework, we can make full use of multiple kinds of label information (provided by covariates) with a multi-task objective function.
• We achieve the state-of-the-art results on multiple tasks.
We are also able to isolate and analyze the effect of the individual covariate on the performance.
Moreover, we note that the proposed framework is applicable in any setting where matching of different types of data which have common covariates is required.
T HE P ROPOSED F RAMEWORK
Our goal is to learn common vector representations for both voices and faces, that permit them to be compared to one another.
In the following sections we ﬁrst describe how we learn them from their relationship to common covariates .
Subsequently, we describe how we will use them for comparison of voices to faces.
2. L EVERAGING C OVARIATES TO L EARN E MBEDDINGS
The relationship between voices and faces is largely predicted by covariates – factors that individually relate to both the voice and the face.
To cite a trivial example, a person’s gender relates their voice to their face: male subjects will have male voices and faces, while female subjects will have female voices and faces.
More generally, many covariates may be found that relate to both voice and face (Lippert et al., 2017).
Our model attempts to ﬁnd common representations for both face images and voice recordings by leveraging their relationship to these covariates (rather than to each other).
We will do so by attempting to predict covariates from voice and face data in a common embedding space, such that the derived embeddings from the two types of data can be compared to one another.
Let V represent a set of voice recordings, and F represent a set of face images.
Let C be the set of covariates we consider.
For the purpose of this paper, we assume that all covariates are discrete valued (although this is not necessary).
Every voice recording in V and every face in F can be related to each of the covariates in C .
For every covariate C ∈ C we represent the value of that covariate for any voice recording v as C ( v ) , and similarly the value of the covariate for any face f as C ( f ) .
For example, C could be ID, gender, or nationality.
When C is ID, C ( v ) and C ( f ) are the
ID of voice v and face f , respectively.
Let F v ( v ; θ v ) : v (cid:55)→ R d be a voice embedding function with parameters θ v that maps any voice recording v into a d -dimensional vector.
Similarly, let F f ( f ; θ f ) be a face embedding function that maps any face f into a d -dimensional vector.
We aim to learn θ v and θ f such that the embeddings of the voice and face for any person are comparable.
For each covariate C ∈ C we deﬁne a classiﬁer H C ( x ; φ C ) with parameter φ C , which assigns any input x ∈ R d to one of the values taken by C .
The classiﬁer H C ( · ) is agnostic to which modality its input x was derived from; thus, given an input voice v , it operates on features F v ( v ; θ v ) derived from the voice, whereas given a face f , it operates on F f ( f ; θ f ) .
For each v (or f ) and each covariate C , we deﬁne a loss L ( H C ( F v ( v ; θ v ); φ C ) , C ( v )) between the covariate predicted by H C ( . ) and the true value of the covariate for v , C ( v ) .
We can now deﬁne a total loss L over the set of all voices V and the set of all faces F , over all covariates as
[Equation]
In order to learn the parameters of the embedding functions, θ f
and θ v , we perform the following optimization.
[Equation]
2. D ISJOINT M APPING N ETWORKS
In DIMNet, we instantiate F v ( v ; θ v ) , F f ( f ; θ f ) and H C ( x ; φ C ) as neural networks.
Fig.
2 shows the network architecture we use to train our embeddings.
It comprises three components.
The ﬁrst, labelled Voice Network in the ﬁgure, represents F v ( v ; θ v ) and is a neural network that extracts ddimensional embeddings of the voice recordings.
The second, labelled Face Network in the ﬁgure, represents F f ( f ; θ f ) and is a network that extracts d -dimensional embeddings of face recordings.
The third component, labelled Classiﬁcation Networks in the ﬁgure, is a bank of one or more classiﬁcation networks, one per covariate considered.
Each of the classiﬁcation networks operates on the d -dimensional features output by the embedding networks to classify one covariate, e.g. gender.
The training data comprise voice recordings and face images.
Voice recordings are sent to the voiceembedding network, while face images are sent to the face-embedding network.
This switching operation is illustrated by the switch at the input in Fig.2.
In either case, the output of the embedding network is sent to the covariate classiﬁers.
As can be seen, at any time the system either operates on a voice, or on a face, i.e. the operations on voices and faces are disjoint.
During the learning phase too, the updates of the two networks are disjoint – loss gradients computed when the input is voice only update the voice network, while loss gradients derived from face inputs update the face network, while both contribute to updates of the classiﬁcation networks.
In our implementation, speciﬁcally, F v ( · ) is a convolutional neural network that operates on MelSpectrographic representations of the speech signal.
The output of the ﬁnal layer is pooled over time to obtain a ﬁnal d -dimensional representation.
F f ( · ) is also a convolutional network with a pooled output at the ﬁnal layer that produces a d -dimensional representation of input images.
The classiﬁers
H C ( · ) are all simple multi-class logistic-regression classiﬁers comprising a single softmax layer.
Finally, in keeping with the standard paradigms for training neural network systems, we use the cross-entropy loss to optimize the networks.
Also, instead of the optimization in Eq. 2, the actual optimization performed is the one below.
The difference is inconsequential.
[Equation]
2. T RAINING THE DIMN ET
All parameters of the network are trained through backpropagation, using stochastic gradient descent.
During training, we construct the minibatches with a mixture of speech segments and face images, as the network learns more robust cross-modal features with mixed inputs.
Taking voice as an example, we compute the voice embeddings using F v ( v ; θ v ) , and obtain the losses using classiﬁers H C ( · ) for all the covariates.
We back-propagate the loss gradient to update the voice network as well as the covariate classiﬁers.
The same procedure is also applied to face data: the backpropagated loss gradients are used to update the face network and the covariate classiﬁers.
Thus, the embedding functions are learned using the data from their modalities individually, while the classiﬁers are learned using data from all modalities.
2. U SING THE E MBEDDINGS
Once trained, the embedding networks F v ( v ; θ v ) and F f ( f ; θ f ) can be used to extract embeddings from any voice recording or face image.
Given a voice recording v and a face image f , we can now compute a similarity between the two through the cosine similarity S ( v, f ) =
[Table]
We can employ this similarity to evaluate the match of any face image to any voice recording.
This enables us, for instance, to attempt to rank a collection of faces f 1 , · · · , f K in order of estimated match to a given voice recording v , according to S ( v, f i ) , or conversely, to rank a collection of voices v 1 , · · · , v K according to their match to a face f , on order of decreasing S ( v i , f ) .
E XPERIMENTS
We ran experiments on matching voices to faces, to evaluate the embeddings derived by DIMNets.
The details of the experiments are given below and Appendix A.
Datasets.
Our experiments were conducted on the Voxceleb (Nagrani et al., 2017) and VGGFace (Parkhi et al., 2015) datasets, which are speciﬁed in appendix A.1.
We use the intersection of the two datasets, i.e. subjects who ﬁgure in both corpora, for our ﬁnal corpus, which thus includes 1,225 IDs with 667 males and 558 females from 36 nationalities.
The data are split into train/validation/test sets, following the settings in Nagrani et al.
(2018b).
Details can be found in Appendix A.1.
We use
ID, gender and nationality as our covariates, all of which are provided by the datasets.
Separated data preprocessing pipelines are employed to audio segments and face images (see Appendix A.2).
Training.
The detailed network conﬁgurations are elaborated in appendix A.3.
Note that the classiﬁcation networks are single-layer softmax units with as many outputs as the number of unique values the class can take (2 for gender, 32 for nationalities, and 924 for IDs in our case).
The networks are trained to minimize the cross entropy loss, following the typical settings of stochastic gradient descent (SGD) in appendix A.
Testing.
We use the following protocols for evaluation:
• 1:2 Matching.
Here, we are given a probe input from one modality (voice or face), and a gallery of two inputs from the other modality (face or voice), including one that belongs to the same subject as the probe, and another of an “imposter” that does not match the probe.
The task is to identify which entry in the gallery matches the probe.
We report performance in terms of matching accuracy – namely what fraction of the time we correctly identify the right instance in the gallery.
To minimize the inﬂuence of random selection, we construct as many testing instances as possible through exhaustive enumeration all positive matched pairs (of voice and face).
To each pair, we include a randomly drawn imposter in the gallery.
We thus have a total of 4,678,897 trials in the validation set, and 6,780,750 trials in the
test
set.
• 1: N Matching.
This is the same as the 1:2 matching, except that the gallery now includes N −imposters.
Thus, we must now identify which of the N entries in the gallery matches the probe.
Here too results are reported in terms of matching accuracy.
We use the same validation and test sets as the 1:2 case, by augmenting each trial with N − 2 additional imposters.
So the number of trials in validation and test sets is the same as earlier.
• Veriﬁcation.
We are given two inputs, one a face, and another a voice.
The task is to determine
if they are matched, i.e. both belong to the same subject.
In this problem setting the similarity between the two is compared to a threshold to decide a match.
The threshold can be adjusted to trade off false rejections ( F R ), i.e. wrongly rejecting true matches, with false alarms ( F A ), i.e. wrongly accepting mismatches.
We report results in terms of equal error rate , i.e. when F R =
F A .
We construct our validation and test sets from those used for the 1:2 matching tests, by separating each trial into two, one comprising a matched pair, and the other a mismatched pair.
Thus, our validation and test sets are exactly twice as large as those for the 1:2 test.
• Retrieval.
The gallery comprises a large number of instances, one or more of which might match the probe.
The task is to order the gallery such that the entries in the gallery that match the probe lie at the top of the ordering.
Here, we report performance in terms of Mean Average Precision (MAP) (Manning et al., 2008).
Here we use the entire collection of 58,420 test faces as the gallery for each of our 21,799 test voices, when retrieving faces from voices.
For the reverse (retrieving voices from faces), the numbers are reversed.
Each result is obtained by averaging the performances of 5 models, which are individually trained
Covariates in Training and
Testing
We use the three covariates provided in the dataset, namely identity (I), gender (G), and nationality (N) for our experiments.
The treatment of covariates differs for training and test.
• Training.
For training, supervision may be provided by any set of (one two or three) covariates.
We consider all combinations of covariates, I, G, N, (I,G), (I,N), (G,N) and (I,G,N).
Increasing the number of covariates effectively increases the supervision provided to training.
All chosen covariates were assigned a weight of 1.0. • Testing.
As explained in Appendix B, simply recognizing a covariate such as gender can result in seemingly signiﬁcant matching performance.
For instance, just recognizing the subjects’ gender from their voice and images can result in a 33% EER for veriﬁcation, and 25% error in matching for the 1 : 2 tests.
In order to isolate the effect of covariates on performance hence we also stratify our test data by them.
Thus we construct 4 testing groups based on the covariates, including the unstratiﬁed (U) group, stratiﬁed by gender (G), stratiﬁed by nationality (N), and stratiﬁed by gender and nationality (G, N).
In each group the test set itself is separated into multiple strata, such that for all instances within any stratum the covariate
values are the same
. 3. C ROSS - MODAL M ATCHING
[Table]
In this section we report results on the 1:2 and 1: N matching tests.
In order to ensure that the embedding networks do indeed leverage on accurate modelling of covariates, we ﬁrst evaluate the classiﬁcation accuracy of the classiﬁcation networks for the covariates themselves.
Tableshows the results.
The rows of the table show the covariates used to supervise the learning.
Thus, for instance, the row labelled “DIMNet-I” shows results obtained when the networks have been trained using ID alone as covariate, the row labelled “DIMNet-G” shows results when supervision is provided by gender, “DIMNetIG” has been trained using ID and gender, etc.
The columns of the table show the speciﬁc covariate being evaluated.
Since the identities of subjects in the training and test set do not overlap, we are unable to evaluate the accuracy of ID classiﬁcation.
Note that we can only test the accuracy of the classiﬁcation network for a covariate if it has been used in the training.
Thus, classiﬁcation accuracy for gender can be evaluated for DIMNet-G, DIMNetGN and DIMNet-IGN, while that for nationality can be evaluated for DIMNet-N, DIMNet-GN and
DIMNet-IGN.
The results in Table 1 show that gender is learned very well, and in all cases gender recognition accuracy is quite high.
Nationality, on the other hand, is not a well-learned classiﬁer, presumably because the distribution of nationalities in the data set is highly skewed (Nagrani et al., 2018b), with nearly 65% of all subjects belonging to the USA.
It is to be expected therefore that nationality as a covariate will not provide sufﬁcient supervision to learn good embeddings.
1 : 2 matching.
Table 2 shows the results for the 1:2 matching tests.
In the table, the row labelled “SVHF-Net” gives results obtained with the model of Nagrani et al.
(2018b).
The columns are segregated into two groups, one labelled “voice → face” and the other labelled “face → voice”.
In the former, the probe is a voice recording, while the gallery comprises faces.
In the later the modalities are reversed.
Within each group the columns represent the stratiﬁcation of the test set.
“U” represents test sets that are not stratiﬁed, and include the various covariates in the same proportion that they occur in the overall test set.
The columns labelled “G” and “N” have been stratiﬁed by gender and nationality, respectively, while the column “G, N” represents data that
[Table]
have been stratiﬁed by both gender and nationality.
In the stratiﬁed tests, we have ensured that all data within a test instance have the same value for the chosen covariate.
Thus, for instance, in a test instance for voice → face in the “G” column, the voice and both faces belong to the same gender.
This does not reduce the overall number of test instances, since it only requires ensuring that the gender of the imposter matches that of the probe instance.
We make several observations.
First, DIMNet-I performs better than SVHF-Net, improving the accuracies by 2.45%-4.02% for the U group, and 7.01%-8.38% for the G group.
It shows that mapping voices and faces to their common covariates is an effective strategy to learn representations for cross-modal matching.
Second, DIMNet-I produces signiﬁcantly better embeddings that DIMNet-G and DIMNet-N, highlighting the rather unsurprising fact that ID provides the more useful information than the other two covariates.
In particular, DIMNet-G respectively achieves 72.90% and 72.47% for voice to face and face to voice matching using only gender as a covariate.
This veriﬁes our hypothesis that we can achieve almost 75% matching accuracy by only using the gender.
These numbers also agree with the performance expected from the numbers in Table 1 and the analysis in Appendix B.
As expected, nationality as a covariate does not provide as good supervision as gender.
DIMNet-IG is marginally better than DIMNet-I, indicating that gender supervision provides additional support over ID alone.
Third, we note that while DIMNet-I is able to achieve good performance on the dataset stratiﬁed by gender, DIMNet-G only achieves random performance.
The performance achieved by DIMNet-G on the U dataset is hence completely explained by gender matching.
Once again, the numbers match our expectations (Appendix B).
[Figure]
1 : N matching.
We also experiment for N > 2 .
Unlike
SVHF-Net (Nagrani et al., 2018b) that needs to train different models for different N in this setting, we use the same model for different N .
The results in Fig.
3 shows accuracy as a function of N for various models.
All the results in Fig.
3 are consistent with Table 2.
As expected, the performance of all methods degrades with increasing N .
In general, DIMNets that use ID as supervision outperform SVHF-Net by a considerable margin, showing that DIMNets are able to make best use of the
ID information.
We obtain the best results when both ID and gender are used as supervision covariates.
However, The results obtained using only gender information as covariate is much worse, which is also consistent with our analysis in Appendix B.
3. C ROSS - MODAL V ERIFICATION
[Table]
For veriﬁcation, we need to determine whether an audio segment and a face image are from the same
ID or not.
We report the equal error rate (EER) for veriﬁcation in Table 3.
In general, DIMNets that use ID as a covariate achieve an EER of about 25%, which is considerably lower than the 33% expected if the veriﬁcation were based on gender matching alone.
The results in Table 3 show that using both gender and ID information as covariates can further improve the performance over using ID alone, well validating the superiority of our multi-task learning framework.
Using proper combination of covariates is crucial to the performance.
ID is arguably
[Figure]
the most effective covariate supervision.
More interestingly, nationality is seen to be an ineffective covariate, while gender alone as a covariate produces results that well matches our expectation
[Figure]
3. C ROSS - MODAL R ETRIEVAL
[Figure]
[Table]
We also perform retrieval experiments using voice or face as query.
Table 4 lists the mean average precision (mAP) of the retrieval for various models.
The columns in the table represent the covariate being retrieved.
Thus, for example, in the “ID” column, the objective is to retrieve gallery items with the same ID as the query, whereas in the “gender” column the objective is to retrieve the same gender.
We note that ID-based DIMNets produce the best features for retrieval, with the best performance obtained with DIMNet-IG.
Also, as may be expected, the covariates used in training result in the best retrieval of that covariate.
Thus, DIMNet-G achieves an mAP of nearly 98% on gender, though on retrieval of ID it is very poor.
As in other experiments, nationality remains a poor covariate in general.
Compared to gender (2 classes) and nationality (unbalanced 28 classes), retrieving ID is a challenging problem given the large amount of identities (182 classes).
The signiﬁcant and consistent improvements over chance-level results show that the DIMNet models do learn some useful associations between voices and faces.
3. C OMPARISONS TO THE CURRENT STATE - OF - THE - ART
[Table]
We compare DIMNet with the state of the art (Nagrani et al., 2018a).
The results are reported in
Table 5.
Note that it is fair comparison because the DIMNet models in this section are trained with and evaluated on the same released datasets in Nagrani et al.
(2018a).
Detailed statistics and splits of the dataset can be found in Appendix A.1.
There are two evaluation protocols, including SeenHeard and Unseen-Unheard scenarios.
The identities of the training and testing set have overlaps in
Seen-Heard scenario (closed-set), while they are fully disjoint in Unseen-Unheard scenario (openset).
For each scenario, there are 5 testing groups based on the covariates, including the unstratiﬁed group (U), group, stratiﬁed by gender (G), stratiﬁed by nationality (N), stratiﬁed by age (A), and stratiﬁed by (G, N, A).
We compute the area under the curve (AUC) for different testing groups.
It is clear that DIMNets produce better embeddings than Nagrani et al.
(2018a)
for pair-wise veriﬁcation on both seen-heard and unseen-unheard scenarios.
Speciﬁcally, DIMNets achieve 8%-15% absolute and 3%-10% absolute improvements on seen-heard and unseen-unheard test set, respectively.
Compared to DIMNet-IG, DIMNet-I performs better on the seen-heard test set while DIMNet-IG is better on the unseen-unheard test set.
It implies that introducing useful covariates improves the generalization capability of DIMNet.
D ISCUSSIONS AND C ONCLUDING R EMARKS
We have proposed that it is possible to learn common embeddings for multi-modal inputs, particularly voices and faces, by mapping them individually to common covariates.
In particular, the proposed DIMNet architecture is able to extract embeddings for both modalities that achieves consistently better performance than the methods that directly map faces to voices.
The approach also provides us the ability to tease out the inﬂuence of each of the covariates of voice and face data, in determining their relation.
The results show that the strongest covariate, not unexpectedly, is ID.
The results also indicate that prior results by other researchers who have attempted to directly match voices to faces may perhaps not be learning any direct relation between the two, but implicitly learning about the common covariates, such as ID, gender, etc.
Our experiments also show that although we have achieved possibly the best reported performance on this task, thus far, the performance is not anywhere close to prime-time.
In the 1 : N matching task, performance degrades rapidly with increasing N , indicating a rather poor degree of true match.
To better understand the problem, we have visualized the learned embeddings from DIMNet-I in
Fig.
4 to provide more insights.
The visualization method we used is multi-dimensional scaling (MDS) (Wickelmaier, 2003), rather than the currently more popular t-SNE (van der Maaten & Hinton, 2008).
This is because MDS tends to preserve distances and global structure, while t-SNE attempts to retain statistical properties and highlights clusters, but does not preserve distances.
From Fig.
4, we immediately notice that the voice and face data for a subject are only weakly proximate.
While voice and face embeddings for a speaker are generally relatively close to each other, they are often closer to other subjects.
Interestingly, the genders separate (even though gender has not been used as a covariate for this particular network), showing that at least some of the natural structure of the data is learned.
Fig.
4 shows embeddings obtained from both training and test data.
We can observe similar behaviors in both, showing that the the general characteristics observed are not just the outcome of overﬁtting to training data.
The visualization in Fig.
4 also shows that there is still signiﬁcant room for improvement.
For example, it may be possible to force compactness of the distributions of voice and face embeddings through modiﬁed loss functions such as the center loss (Wen et al., 2016) or angular softmax loss (Liu et al., 2016; 2017a;b) or through an appropriately designed loss function that is speciﬁc to this task.
A PPENDIX A E XPERIMENTAL DETAILS A. DATASET
The Voxceleb dataset consists of 153,516 audio segments from 1,251 speakers.
Each audio segment is taken from an online video clip with an average duration of 8.2 seconds.
For the face dataset, we used a manually ﬁltered version of VGGFace .
After face detection, there remain 759,643 images from 2,554 subjects.
The data are split into train/validation/test sets, following the settings in
Nagrani et al.
(2018b).
Details are shown in Table
[Table]
The visual data used in Section 3.4 is densely extracted from the video in VoxCeleb dataset at 25/ fps.
It contains 100,000 segmented speaking face-tracks obtained by SyncNet (Chung & Zisserman, 2016), leading to 1,218,575 frames (images).
For fair comparison, we follow the train/val/test split strategy from Nagrani et al.
(2018a) in our experiments.
The evaluations are performed based on the provided lists (Nagrani et al., 2018a), which specify the testing pairs of voices and faces.
A. PREPROCESSING
We employ separated data preprocessing pipelines for audio segments and face images.
For audio segments, we use an energy-based voice activity detector (Povey et al., 2011) to isolate speechbearing regions of the recordings.
Subsequently, 64-dimensional log mel-spectrograms are generated, using an analysis window of 25ms, with hop of 10ms between frames.
We perform mean and variance normalization of each mel-frequency bin.
For training, we randomly crop out regions of varying lengths of 300 to 800 frames (so the size of the input spectrogram ranges from 300 × 64 to 800 × 64 for each mini-batch, around 3 to 8 seconds).
For the face data, facial landmarks in all images are detected using MTCNN (Zhang et al., 2016).
The cropped RGB face images of size 128 × 128 × 3 are obtained by similarity transformation.
Each pixel in the RGB images is normalized by subtracting 127 .
5 and then dividing by 127 .
We perform data augmentation by horizontally ﬂipping the images with 50% probability in minibatches (effectively doubling the number of face images).
A. TRAINING
The details of network architectures are shown in Table 7.
For the voice network, we use 1D convolutional layers, where the convolution is performed along the axis that corresponds to time.
The face network employs 2D convolutional layers.
For both, the convolutional layers are followed by batch normalization (BN) (Ioffe & Szegedy, 2015) and rectiﬁed linear unit activations (ReLU) (Krizhevsky et al., 2012).
The ﬁnal face embedding is obtained by averaging the feature maps from the ﬁnal layer, i.e. through average pooling .
The ﬁnal voice embedding is obtained by averaging the feature maps at the ﬁnal convolutional layer along the time axis alone.
We follow the typical settings of SGD for optimization.
Minibatch size is 256.
The momentum and weight decay values are 0.9 and 0.001 respectively.
To learn the networks from scratch, the learning rate is initialized at 0.1 and divided by 10 after 16K iterations and again after 24K iterations.
The training is completed at 28K iterations.
[Table]
A. E XPERIMENTS ON THE EMBEDDING DIMENSION
To investigate the affect of embedding dimension to the performance, we train DIMNet-I models with various embedding dimensions of 32, 64, 128, 256, and 512.
Table 8 shows the results on 1: matching experiment (voice → face).
[Table]
It could be observed that the performance of cross-modal matching is very stable within a wide range of embedding dimension, showing that the accuracy is not sensitive to the embedding dimension.
A PPENDIX B E XPECTED PERFORMANCE BASED ON GENDER MATCHING
In this appendix we discuss the performance to be expected in the matching and veriﬁcation tests, when the matching is done based purely on gender.
We assume below that in any distribution of human-subject data, the division of subjects between male and female genders to be half and half.
It is to be noted that gender is merely an illustrative example here; the analysis can be extended to other covariates.
For a more detailed analysis of covariates with more values and unbalanced distributions, please refer to Wen et al.
B. A CCURACY OF 1:2 MATCHING BASED ON GENDER
We show that the equal-error-rate for 1:2 matching can be as high as 25%, through gender matching alone.
The problem is as follows: a probe input (voice or face), and a gallery consisting of two inputs (face or voice), one of which is from the same subject as the probe.
We must identify which of the two is the true match.
B.1. P ERFECT GENDER IDENTIFICATION
Consider the situation where we are able to identify the gender of the subject of the data (face or voice) perfectly.
There are two possibilities: (a) both probe instances are the same gender, and (b) they are different genders.
Each of the two possibilities occurs with a probability of 0.
We employ the following simple strategy: If the two gallery instances are different genders, then we select the instance whose gender matches the probe.
In this case, clearly, the probability of error is 0.
If the two instances are the same gender, we select one of them randomly with a probability of 0.5.
The probability of error here is 0.5.
Thus, the overall probability of error is
Prob ( error ) = 0 .
5 × 0 + 0 .
5 × 0 .
5 = 0 .
B.1.
I MPERFECT GENDER IDENTIFICATION
Now let us consider the situation where gender identiﬁcation itself is imperfect, and we have error rates e f and e v in identifying the gender of faces and voices, respectively.
Assume the error rates are known.
We will assume below that gallery entries are faces, and probe entries are voices.
(The equations are trivially ﬂipped to handle the converse case).
Since we are aware that we sometimes make mistakes in identifying gender, we modify our strategy as follows: when the two gallery items are found to have different genders, we select the entry with the same gender as the probe P of the time (so that if the gender classiﬁcation was correct, we would have a match error rate of (1- P )).
When both gallery items are found to be the same gender, we choose randomly.
The actual error can now be computed as follows.
The gallery items are both of the same gender in 0.5 of the trials, and of mismatched gender in the remaining 0.5 of the trials.
When both gallery items have the same gender, regardless of the strategy chosen, the probability of error is 0.5 (by symmetry).
When both gallery items are of mismatched gender, we have 8 combinations of correctness of gender-classiﬁcation.
Table 9 lists all eight, along with the probability of matching error (in the ﬁnal column).
Taking type 1 as an example, we have probability (1 − e v )(1 − e f ) 2 that the gender
[Table]
of both probe and galleries
are correctly classiﬁed.
In this case, our strategy gives us an error of (1 − P ) .
For type 2, the gender of probe and one of the gallery items is correctly classiﬁed, while the other gallery item is misclassiﬁed, we have an error of 0.5.
If we go through all the cases, the total error Prob ( error ) can be computed as
Prob
[Equation]
error
[Equation]
=
[Equation] [Equation]
[Equation] [Equation]
Our objective is to minimize Prob ( error ) , so we must choose P to minimize the above term.
I.e. we must solve
[Equation]
Its easy to see that the solution for P is 1.0 if its multiplicative factor is negative in the above equation, and
otherwise, i.e.
P =
[Equation] [Equation]
else
The corresponding match error rates are Prob ( error )
= 0 .
25 + 0 .
5( e f
+ e v − 2 e v e f ) and 0 .
+ e f e f − 0 .
5( e v + e f ) respectively.
Although complicated looking, the solution is, in fact, quite intuitive.
When gender classiﬁcation is either better than random for both modalities ( i.e. e f , e v > 0 . 5 ) or worse than random for both ( e f , e v < 0 . 5 ), the best strategy is to select the gallery item that matches the gender of the probe.
If either of these is random ( i.e. either e f or e v is 0.5) , the choice of P does not matter, and the error is 0.5.
If one of the two is correct more than half the time, and the other is wrong more than half the time ( e.g. e f < 0 .
5 , e v > 0 .
), the optimal choice is to select the gallery item that is classiﬁed as mismatched in gender with the probe.
B. A CCURACY OF 1:N MATCHING BASED ON GENDER
We now consider the best achievable performance on 1: N matching, when the only information known is the gender of the voices and faces.
B.2. P ERFECT GENDER IDENTIFICATION
Consider the situation where the gender of the faces and voices in each test trial is perfectly known.
We employ the following strategy: we randomly select one of the gallery instances that have the same gender as the probe instance.
If there are K imposter gallery instances of the same gender as the probe instance, the expected accuracy is 1 K +1 .
The probability of randomly having K of N −imposters of the same gender as the probe is given by
[Equation]
The overall accuracy is given by:
[Equation]
giving us the error
[Equation]
N
[Equation]
B.2. I MPERFECT GENDER IDENTIFICATION
Consider now that the gender recognition is erroneous for voices with probability e v and for faces with probability e f .
Note that regardless of the error in gender recognition, the probability of any noisy gallery entry having any gender remains 0 .
To account for the possible error in gender classiﬁcation, we consider the following stochastic policy: with probability P
we select one of the gallery entries with the same gender assigned to probe (by the gender classiﬁer), and with probability 1 − P we choose one of the entries with the opposite gender assigned to the probe.
Let α represent the probability that the genders assigned to probe and the corresponding gallery entry by their respective classiﬁers are identical.
[Equation]
The equation above considers both possibilities: that both the probe and its matching gallery entry are correctly classiﬁed, and that both of them are misclassiﬁed.
It follows that the probability and its matching gallery entries are assigned different genders is 1 − α .
Given that we have selected the correct gender for retrieval from the the gallery ( i.e. that the gender we have selected is the same as that assigned to the gallery entry matching the probe by the face classiﬁer), using the same analysis as in Section B.2.1, we obtain the following probability of being correct:
[Equation]
The probability of selecting the correct gender is given by
[Equation]
Since the probability of being correct when we choose the wrong gender is 0, the overall probability of being correct is
[Equation]
Maximizing the probability requires us to solve
[Equation]
which gives us the optimal P as
[Equation]
and the optimal error as
[Equation]
B. EER OF VERIFICATION BASED ON GENDER
Here we show that the equal-error-rate for veriﬁcation (determining if the the subjects in two recordings are the same) can be as high as 33%, through gender matching alone.
The problem is as follows: we are given a pair of inputs, one (features extracted from) a face, and the other a voice.
We must determine whether they are both from the same speaker.
The test set include some number of “positives”, where both do belong to the same subject, and some “negatives”, where both do not.
If a positive is falsely detected as a negative, we have an instance of false rejection.
If a negative is wrongly detected as a positive, we have an instance of false acceptance.
Let F R represent the ‘false rejection rate”, i.e. the fraction of all positives that are wrongly rejected.
Let F A represent the “false acceptance rate”, i.e. the fraction of negatives that are wrongly accepted.
Any classiﬁer can generally be optimized to trade off F R against F A .
The “Equal Error Rate” (EER) is achieved when F R = F A .
Among the “positive” test pairs, both voice and face in each pair have the same gender.
We assume the “negative” test instances are drawn randomly, i.e. , 0.5 of all negative pairs have the same gender, while the remaining 0.5 do not.
B.3. P ERFECT GENDER IDENTIFICATION
Consider the situation where we know the subject’s gender for both the voices and faces (or, alternately, are able to identify the gender from the voice or face perfectly).
We employ the following strategy: if the gender of the voice and face are different, we declare it as a negative 100% of the time.
If the two are from the same gender, we randomly call it a positive P of the time, where 0 ≤ P ≤ 1 . 0 .
Using this strategy, the false acceptance rate is:
[Equation]
Here we’re considering that using our strategy we never make a mistake on the 50% of negative pairs that have mismatched genders, but are wrong P of the time on the negative pairs with matched genders.
Among the positives, where all pairs are gender matched, our strategy of accepting only a fraction
P of them as positives will give us a false rejection rate F R = 1 − P .
The equal error rate is achieved when F R = F A , i.e.
[Equation]
P =
[Equation]
P, giving us P = 23 , i.e. the best EER is achieved when we accept gender-matched pairs two-thirds of the time.
The EER itself is 0 .
5 P = 13 .
Thus, merely by being able to identify the gender of the subject accurately, we are able to veriﬁcation
EER of 0.33.
B.3. I MPERFECT GENDER IDENTIFICATION
Now let us consider the situation where gender identiﬁcation itself is imperfect, and we have error rates e f and e v in identifying the gender of the face and the voice, respectively.
Assume these error rates are known.
To account for this, we modify our strategy: when we ﬁnd the genders of the voice and face to match, we accept the pair as positive P of the time, but when they are mismatched we still accept them as positive Q of the time.
Let α represent the probability that we will correctly call the polarity of the gender match between the voice and the face.
I.e. α is the probability that if the two have the same gender, we will correctly state that they have the same gender, or if they are of opposite gender, we will correctly state they are of opposite gender.
[Equation]
This combines two terms: that we call the genders of both the voice and face correctly, and that we call them both wrongly (which also results in ﬁnding the right polarity of the relationship).
Its easy to see that 0 ≤ α ≤ 1 , and to verify that when gender identiﬁcation is perfect, α = 1 . 0 .
The probability of calling the polarity of the gender relationship wrongly is 1 − α .
Among the positive test pairs, all pairs are gender matched.
We will correctly call α of these as gender matched.
Using our strategy, our error on these instances is (1 − P ) .
We will incorrectly call
1 − α of these as gender mismatched, and the error on these instances is (1 − Q ) .
So the overall false rejection rate is given by
[Equation]
Among the negative pairs, half are gender matched, and half are gender mismatched.
Using the same logic as above, the error on the gender-matched negative pairs is αP + (1 − α ) Q .
Among the gender mismatched pairs the error is αQ + (1 − α ) P .
The overall false acceptance rate is given by
[Equation]
[Equation]
Equating F A and F R as the condition for EER, we obtain
[Equation]
Since at EER, the EER equals F A , and we would like to minimize it, we obtain the following solution to determine the optimal P and Q :
[Equation]
For α > 0 .
5 it is easy to see that the solution to this is obtained at
[Equation]
For α < 0 .
5 the optimal solution is at
[Equation]
That is, when the probe and gallery classiﬁers are likely to make the same error more than half the time, the optimal solution is to always reject pairs detected as having mismatched genders, and to accept matched-gender pairs 2 1+2 α of the time.
The optimal EER is 1
+2 α .
When they are more likely to make different errors, the optimal solution is to always reject pairs detected as having matched genders, and to accept mismatched-gender pairs
[Equation]
α of the time.
The optimal EER now is
[Equation]
Note that if an operating point other than EER were chosen to quantify performance ( e.g. F A = βF R for β (cid:54) = 1 , or for some ﬁxed F A or F R ), the above analysis can be modiﬁed to accommodate it, provided a feasible solution exists.