Original: F ROM I NFERENCE TO G ENERATION : E ND - TO - END F ULLY S ELF - SUPERVISED G ENERATION OF H UMAN F ACE FROM S PEECH
Revised: F ROM I NFERENCE TO G ENERATION : E ND - TO - END F ULLY S ELF - SUPERVISED G ENERATION OF H UMAN F ACE FROM S PEECH

Original: ABSTRACT
Revised: A

Revised: BSTRACT

Original: This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations.
Revised: This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations.

Original: To this end, we propose a multi-modal learning framework that links the inference stage and generation stage.
Revised: To this end, we propose a multi-modal learning framework that links the inference stage and generation stage.

Original: First, the inference networks are trained to match the speaker identity between the two different modalities.
Revised: First, the inference networks are trained to match the speaker identity between the two different modalities.

Original: Then the trained inference networks cooperate with the generation network by giving conditional information about the voice.
Revised: Then the trained inference networks cooperate with the generation network by giving conditional information about the voice.

Original: The proposed method exploits the recent development of GANs techniques and generates the human face directly from the speech waveform making our system fully end-to-end.
Revised: The proposed method exploits the recent development of GANs techniques and generates the human face directly from the speech waveform making our system fully end-to-end.

Original: We analyze the extent to which the network can naturally disentangle two latent factors that contribute to the generation of a face imageone that comes directly from a speech signal and the other that is not related to itand explore whether the network can learn to generate natural human face image distribution by modeling these factors.
Revised: We analyze the extent to which the network can naturally disentangle two latent factors that contribute to the generation of a face imageone that comes directly from a speech signal and the other that is not related to itand explore whether the network can learn to generate natural human face image distribution by modeling these factors.

Original: Experimental results show that the proposed network can not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech.
Revised: Experimental results show that the proposed network can not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech.

Original: Finally, the correlation between the generated face and the corresponding speech is quantitatively measured to analyze the relationship between the two modalities.
Revised: Finally, the correlation between the generated face and the corresponding speech is quantitatively measured to analyze the relationship between the two modalities.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Utilizing audio-visual cues together to recognize a person’s identity has been studied in various ﬁelds from neuroscience (Hasan et al., 2016; Tsantani et al., 2019) to practical machine learning applications (Nagrani et al., 2018b;a; Wen et al., 2019a; Shon et al., 2019).
Revised: Utilizing audio-visual cues together to recognize a person’s identity has been studied in various ﬁelds from neuroscience (Hasan et al., 2016; Tsantani et al., 2019) to practical machine learning applications (Nagrani et al., 2018b;a; Wen et al., 2019a; Shon et al., 2019).

Original: For example, some neurological studies have found that in some cortical areas, humans recognize familiar individuals by combining signals from several modalities, such as faces and voices (Hasan et al., 2016).
Revised: For example, some neurological studies have found that in some cortical areas, humans recognize familiar individuals by combining signals from several modalities, such as faces and voices (Hasan et al., 2016).

Original: In conjunction with the neurological studies, it is also a well known fact that a human speech production system is directly related to the shape of the vocal tract (Mermelstein, 1967; Teager & Teager, 1990).
Revised: In conjunction with the neurological studies, it is also a well known fact that a human speech production system is directly related to the shape of the vocal tract (Mermelstein, 1967; Teager & Teager, 1990).

Original: Inspired by the aforementioned scientiﬁc evidence, we would like to ask three related questions from the perspective of machine learning: 1) Is it possible to match the identity of faces and voices?
Revised: Inspired by the aforementioned scientiﬁc evidence, we would like to ask three related questions from the perspective of machine learning: 1) Is it possible to match the identity of faces and voices?

Original: (inference) 2)
Revised: (inference) 2)

Original: If so, is it possible to generate a face image from a speech signal?
Revised: If so, is it possible to generate a face image from a speech signal?

Original: (generation) 3)
Revised: (generation) 3)

Original: Can we ﬁnd the relationship between the two modalities only using cross-modal self-supervision with the data “in-the-wild”?
Revised: Can we ﬁnd the relationship between the two modalities only using cross-modal self-supervision with the data “in-the-wild”?

Original: To answer these questions, we design a two-step approach where the inference and generation stages are trained sequentially.
Revised: To answer these questions, we design a two-step approach where the inference and generation stages are trained sequentially.

Original: First, the two inference networks for each modality (speech encoder and face encoder) are trained to extract the useful features and to compute the cross-modal identity matching probability.
Revised: First, the two inference networks for each modality (speech encoder and face encoder) are trained to extract the useful features and to compute the cross-modal identity matching probability.

Original: Then the trained inference networks are transferred to the generation stage to pass the information about the speech, which helps the generation network to output the face image from the conditioned speech.
Revised: Then the trained inference networks are transferred to the generation stage to pass the information about the speech, which helps the generation network to output the face image from the conditioned speech.

Original: We believe, however, that it is impossible to perfectly reconstruct all the attributes in the image of a person’s face through the characteristics of the voice alone.
Revised: We believe, however, that it is impossible to perfectly reconstruct all the attributes in the image of a person’s face through the characteristics of the voice alone.

Original: This is due to factors that are clearly unrelated to one’s voice, such as lighting, glasses, and orientation, that also exist in the natural face image.
Revised: This is due to factors that are clearly unrelated to one’s voice, such as lighting, glasses, and orientation, that also exist in the natural face image.

Original: To reﬂect the diverse characteristics presented in the face images “in-the-wild”, we therefore model the generation process by incorporating two latent factors into the neural network.
Revised: To reﬂect the diverse characteristics presented in the face images “in-the-wild”, we therefore model the generation process by incorporating two latent factors into the neural network.

Original: More speciﬁcally, we adopted conditional generative adversarial networks (cGANs) (Mirza & Osindero, 2014; Miyato & Koyama, 2018) so that the generator network can produce a face image that is dependent not only on the paired speech condition, but also on the stochastic variable.
Revised: More speciﬁcally, we adopted conditional generative adversarial networks (cGANs) (Mirza & Osindero, 2014; Miyato & Koyama, 2018) so that the generator network can produce a face image that is dependent not only on the paired speech condition, but also on the stochastic variable.

Original: This allows the latent factors that contribute to the overall facial attributes to be disentangled into two factors: one that is relevant to the voice and the other that is irrelevant.
Revised: This allows the latent factors that contribute to the overall facial attributes to be disentangled into two factors: one that is relevant to the voice and the other that is irrelevant.

Original: Adopting cGANs negligently still leaves a few problems.
Revised: Adopting cGANs negligently still leaves a few problems.

Original: For example, the condition in a cGANs framework is typically provided as embedded conditional vectors through the embedding look-up table for one-hot encoded labels (Brock et al., 2019; Miyato & Koyama, 2018).
Revised: For example, the condition in a cGANs framework is typically provided as embedded conditional vectors through the embedding look-up table for one-hot encoded labels (Brock et al., 2019; Miyato & Koyama, 2018).

Original: The raw signals such as speech, however, cannot be taken directly from the embedding look-up table, so an encoder module is required.
Revised: The raw signals such as speech, however, cannot be taken directly from the embedding look-up table, so an encoder module is required.

Original: Therefore, the trained speech encoder from the inference step is reused to output a pseudo conditional label that is used to extract meaningful information relevant to the corresponding face.
Revised: Therefore, the trained speech encoder from the inference step is reused to output a pseudo conditional label that is used to extract meaningful information relevant to the corresponding face.

Original: Then the generator and the discriminator are trained in an adversarial way by utilizing the pseudo-embedded conditional vectors obtained from the trained speech encoder in the ﬁrst step.
Revised: Then the generator and the discriminator are trained in an adversarial way by utilizing the pseudo-embedded conditional vectors obtained from the trained speech encoder in the ﬁrst step.

Original: Another problem with applying the conventional cGANs for generating faces from voice arises from the fact that the distinction between different speakers can be quite subtle, which calls for a need for a more effective conditioning method.
Revised: Another problem with applying the conventional cGANs for generating faces from voice arises from the fact that the distinction between different speakers can be quite subtle, which calls for a need for a more effective conditioning method.

Original: To mitigate this problem, we propose a new loss function, relativistic identity cGANs
Revised: To mitigate this problem, we propose a new loss function, relativistic identity cGANs

Original: (relidGANs) loss, with modiﬁcation of the relativistic GANs (JolicoeurMartineau, 2019), allowing us to generate the face with a more distinct identity.
Revised: (relidGANs) loss, with modiﬁcation of the relativistic GANs (JolicoeurMartineau, 2019), allowing us to generate the face with a more distinct identity.

Original: Each step will be described in greater detail in Section 3.
Revised: Each step will be described in greater detail in Section 3.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We propose simple but effective end-to-end inference networks trained on audio-visual data without any labels in a self-supervised manner that perform a cross-modal identity matching task.
Revised: We propose simple but effective end-to-end inference networks trained on audio-visual data without any labels in a self-supervised manner that perform a cross-modal identity matching task.

Original: A cGANs-based generation framework is proposed to generate the face from speech, to be seamlessly integrated with the trained networks from inference stage.
Revised: A cGANs-based generation framework is proposed to generate the face from speech, to be seamlessly integrated with the trained networks from inference stage.

Original: A new loss function, so called a relidGANs loss, is designed to preserve a more consistent identity between the voices and the generated images.
Revised: A new loss function, so called a relidGANs loss, is designed to preserve a more consistent identity between the voices and the generated images.

Original: An extensive analysis is conducted on both inference and generation tasks to validate our proposed approaches.
Revised: An extensive analysis is conducted on both inference and generation tasks to validate our proposed approaches.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: There has been an increasing interest in the self-supervised learning framework within the machine learning research community.
Revised: There has been an increasing interest in the self-supervised learning framework within the machine learning research community.

Original: While the focus of the many studies has been concentrated on applications of matching the audio-visual correspondence such as the presence or absence of objects in a video or a temporal alignment between two modalities (Chung et al., 2017; Afouras et al., 2018b;a;
Revised: While the focus of the many studies has been concentrated on applications of matching the audio-visual correspondence such as the presence or absence of objects in a video or a temporal alignment between two modalities (Chung et al., 2017; Afouras et al., 2018b;a;

Original: Ephrat et al., 2018), growing attention has come to matching the speaker identity between the human face and voice.
Revised: Ephrat et al., 2018), growing attention has come to matching the speaker identity between the human face and voice.

Original: Inference The cross-modal identity matching task between face and voice has been recently studied in machine learning community.
Revised: Inference The cross-modal identity matching task between face and voice has been recently studied in machine learning community.

Original: Nagrani et al.
Revised: Nagrani et al.

Original: (2018b) proposed a convolutional-neural-network (CNN) based biometric matching network by concatenating the two embedding vectors from each modality and classifying them with an additional classiﬁer network.
Revised: (2018b) proposed a convolutional-neural-network (CNN) based biometric matching network by concatenating the two embedding vectors from each modality and classifying them with an additional classiﬁer network.

Original: Though showing promising results, this style of training is limited as the model is not ﬂexible in that the number of concatenated vectors used in the training cannot be changed in the test phase.
Revised: Though showing promising results, this style of training is limited as the model is not ﬂexible in that the number of concatenated vectors used in the training cannot be changed in the test phase.

Original: Next, Nagrani et al.
Revised: Next, Nagrani et al.

Original: (2018a) modeled the concept of personal identity nodes (Bruce & Young, 1986) by mapping each modality into the shared embedding space and using triplet loss to train the encoders to allow more ﬂexible inference.
Revised: (2018a) modeled the concept of personal identity nodes (Bruce & Young, 1986) by mapping each modality into the shared embedding space and using triplet loss to train the encoders to allow more ﬂexible inference.

Original: Lastly, Wen et al. (2019a) proposed DIMNet where two independent encoders are trained to map the speech and face into the shared embedding space, classifying each of them independently utilizing the supervised learning signals from labels such as identity, gender and nationality.
Revised: Lastly, Wen et al. (2019a) proposed DIMNet where two independent encoders are trained to map the speech and face into the shared embedding space, classifying each of them independently utilizing the supervised learning signals from labels such as identity, gender and nationality.

Original: Generation There have been a few concurrent works that tackled the similar problem of generating a face image from speech, which we think are worth noting here.
Revised: Generation There have been a few concurrent works that tackled the similar problem of generating a face image from speech, which we think are worth noting here.

Original: Duarte et al.
Revised: Duarte et al.

Original: (2019) proposed a
Revised: (2019) proposed a

Original: GANs-based framework to generate a face image from the speech.
Revised: GANs-based framework to generate a face image from the speech.

Original: But the intention of their work was not to generate a face from unseen speaker identity but more of seeking the possibility of crossmodal generation between the speech and face itself.
Revised: But the intention of their work was not to generate a face from unseen speaker identity but more of seeking the possibility of crossmodal generation between the speech and face itself.

Original: Oh et al.
Revised: Oh et al.

Original: (2019) proposed a similar work called speech2face.
Revised: (2019) proposed a similar work called speech2face.

Original: The pre-trained face recognition network and the neural parametric decoder were used to produce a normalized face (Parkhi et al., 2015; Cole et al., 2017).
Revised: The pre-trained face recognition network and the neural parametric decoder were used to produce a normalized face (Parkhi et al., 2015; Cole et al., 2017).

Original: After that, the speech encoder was trained to estimate the input parameter of the face decoder directly.
Revised: After that, the speech encoder was trained to estimate the input parameter of the face decoder directly.

Original: Because the training process still requires the pre-trained modules trained with supervision, we do not consider this a fully-selfsupervised approach.
Revised: Because the training process still requires the pre-trained modules trained with supervision, we do not consider this a fully-selfsupervised approach.

Original: Lastly, the most similar work to ours was recently proposed by Wen et al.
Revised: Lastly, the most similar work to ours was recently proposed by Wen et al.

Original: (2019b) where they utilized a pre-trained speech identiﬁcation network as a speech encoder, and used a GANs-based approach to produce the face conditioned on the speech embedding.
Revised: (2019b) where they utilized a pre-trained speech identiﬁcation network as a speech encoder, and used a GANs-based approach to produce the face conditioned on the speech embedding.

Original: Our proposed method differs from the abovementioned approaches in the following ways.
Revised: Our proposed method differs from the abovementioned approaches in the following ways.

Original: First, none of the previous approaches has tried to model the stochasticity in the generation process, but we address this problem by incorporating the stochasticity in the latent space so that the different face images can be sampled even when the speech condition is ﬁxed.
Revised: First, none of the previous approaches has tried to model the stochasticity in the generation process, but we address this problem by incorporating the stochasticity in the latent space so that the different face images can be sampled even when the speech condition is ﬁxed.

Original: Second, modeling the image is more challenging in our work as we aim to train our network to produce larger image size ( × 128) compared to other GANs-based works.
Revised: Second, modeling the image is more challenging in our work as we aim to train our network to produce larger image size ( × 128) compared to other GANs-based works.

Original: Also, we trained the model on the AVSpeech dataset (Ephrat et al., 2018) which includes extremely diverse dynamics in the images.
Revised: Also, we trained the model on the AVSpeech dataset (Ephrat et al., 2018) which includes extremely diverse dynamics in the images.

Original: Finally, the important scope of this work is to seek the possibility of training the whole inference and generation stages only using the self-supervised learning method, which is the ﬁrst attempt to our knowledge.
Revised: Finally, the important scope of this work is to seek the possibility of training the whole inference and generation stages only using the self-supervised learning method, which is the ﬁrst attempt to our knowledge.

Original: The whole pipeline of the proposed approach is illustrated in Fig.
Revised: The whole pipeline of the proposed approach is illustrated in Fig.

Original: [Figure]
Revised: [Figure]

Original:  S ELF - SUPERVISED I NFERENCE AND G ENERATION 3. C ROSS - MODAL I DENTITY M ATCHING
Revised:  S ELF - SUPERVISED I NFERENCE AND G ENERATION 3. C ROSS - MODAL I DENTITY M ATCHING

Original: In order to successfully identify the speaker identity from the two audio-visual modalities, we trained two encoders for each modality.
Revised: In order to successfully identify the speaker identity from the two audio-visual modalities, we trained two encoders for each modality.

Original: First, a speech encoder is trained to extract the information related to a person’s identity from a segment of speech.
Revised: First, a speech encoder is trained to extract the information related to a person’s identity from a segment of speech.

Original: To this end, we use raw-waveform based speech encoder that was shown to extract useful features in speaker veriﬁcation task, outperforming conventional feature such as mel-frequnecy-cepstal-coefﬁcient (MFCC) even when trained using selfsupervised learning in speech modality (Pascual et al., 2019) 1 .
Revised: To this end, we use raw-waveform based speech encoder that was shown to extract useful features in speaker veriﬁcation task, outperforming conventional feature such as mel-frequnecy-cepstal-coefﬁcient (MFCC) even when trained using selfsupervised learning in speech modality (Pascual et al., 2019) 1 .

Original: The ﬁrst layer of the speech encoder is SincNet where the parameterized sinc functions act as trainable band-pass ﬁlters (Ravanelli &
Revised: The ﬁrst layer of the speech encoder is SincNet where the parameterized sinc functions act as trainable band-pass ﬁlters (Ravanelli &

Original: Bengio, 2018).
Revised: Bengio, 2018).

Original: The rest of the layers are composed of 7 stacks of 1d-CNN, batch normalization (BN), and multi-parametric rectiﬁed linear unit (PReLU) activation.
Revised: The rest of the layers are composed of 7 stacks of 1d-CNN, batch normalization (BN), and multi-parametric rectiﬁed linear unit (PReLU) activation.

Original: Next, we used a 2d-CNN based face encoder with residual connections in each layer.
Revised: Next, we used a 2d-CNN based face encoder with residual connections in each layer.

Original: Note that the network structure is similar to the discriminator network of (Miyato & Koyama, 2018).
Revised: Note that the network structure is similar to the discriminator network of (Miyato & Koyama, 2018).

Original: The details of the networks are shown in
Revised: The details of the networks are shown in

Original: Appendix C.
Revised: Appendix C.

Original: Based on the speech encoder S ( · ) and the face encoder F ( · ) , they are trained to correctly identify whether the given face and speech is paired or not.
Revised: Based on the speech encoder S ( · ) and the face encoder F ( · ) , they are trained to correctly identify whether the given face and speech is paired or not.

Original: Speciﬁcally, as a cross-modal identity matching task, we consider two settings as in (Nagrani et al., 2018b), 1.
Revised: Speciﬁcally, as a cross-modal identity matching task, we consider two settings as in (Nagrani et al., 2018b), 1.

Original: V-F: given a segment of speech select one face out of K different faces, 2.
Revised: V-F: given a segment of speech select one face out of K different faces, 2.

Original: F-V: given an image of face select one speech segment out of
Revised: F-V: given an image of face select one speech segment out of

Original: K different speech segments.
Revised: K different speech segments.

Original: The probability that the j -th face f j is matched to the given speech s or vice versa is computed by the inner product of embedding vectors from each module followed by the softmax function as follows:
Revised: The probability that the j -th face f j is matched to the given speech s or vice versa is computed by the inner product of embedding vectors from each module followed by the softmax function as follows:

Original: V - F : p ( y = j |{
Revised: V - F : p ( y = j |{

Original: S
Revised: S

Original: ( s ) ,
Revised: ( s ) ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: F - V : p ( y = j |{S ( s j ) , F ( f )
Revised: F - V : p ( y = j |{S ( s j ) , F ( f )

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: = K
Revised: = K

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: where s denotes a speech segment, f denotes a face image, se denotes a speech embedding vector and fe denotes a face embedding vector.
Revised: where s denotes a speech segment, f denotes a face image, se denotes a speech embedding vector and fe denotes a face embedding vector.

Original: Note that using inner product as a means of computing similarity allows more ﬂexible inference.
Revised: Note that using inner product as a means of computing similarity allows more ﬂexible inference.

Original: For example, the proposed method enables both F-V and V-F settings, no matter with which method the model is trained.
Revised: For example, the proposed method enables both F-V and V-F settings, no matter with which method the model is trained.

Original: Also, it allows setting a different number of K in the test phase than the training phase.
Revised: Also, it allows setting a different number of K in the test phase than the training phase.

Original: Finally, the two encoders are trained solely using the self-supervised learning signal from crossentropy error.
Revised: Finally, the two encoders are trained solely using the self-supervised learning signal from crossentropy error.

Original: 3. G ENERATING F ACE FROM V OICE
Revised: 3. G ENERATING F ACE FROM V OICE

Original: Although we aim to generate the human face from the speech, it is only natural to think that not all attributes of the face image are correlated to the speech.
Revised: Although we aim to generate the human face from the speech, it is only natural to think that not all attributes of the face image are correlated to the speech.

Original: Hence, we assume the latent space of the face to be broken down into two parts, the deterministic variable from the speech encoder c and a random variable z sampled from Gaussian distribution.
Revised: Hence, we assume the latent space of the face to be broken down into two parts, the deterministic variable from the speech encoder c and a random variable z sampled from Gaussian distribution.

Original: Such latent space is modeled using cGANs, a generative model that allows the sampling of face images conditioned on speech.
Revised: Such latent space is modeled using cGANs, a generative model that allows the sampling of face images conditioned on speech.

Original: More speciﬁcally, randomly sampled Gaussian noise z ∈ Rand speech condition c ∈ R 128 are concatenated and used as input for the generator function f fake
Revised: More speciﬁcally, randomly sampled Gaussian noise z ∈ Rand speech condition c ∈ R 128 are concatenated and used as input for the generator function f fake

Original: = G ( z , c ) to sample the face image conditioned on the speech.
Revised: = G ( z , c ) to sample the face image conditioned on the speech.

Original: In addition, adaptive instance normalization (AdaIN) technique is applied as a more direct conditioning method for each layer of the generator network (Huang & Belongie, 2017; Karras et al., 2019).
Revised: In addition, adaptive instance normalization (AdaIN) technique is applied as a more direct conditioning method for each layer of the generator network (Huang & Belongie, 2017; Karras et al., 2019).

Original: The details of the network are described in Appendix C.
Revised: The details of the network are described in Appendix C.

Original: In order to generate the face image that are not only close enough to the real face image distribution, but matches with the given condition, the condition information must be properly fed into the discriminator.
Revised: In order to generate the face image that are not only close enough to the real face image distribution, but matches with the given condition, the condition information must be properly fed into the discriminator.

Original: Many studies have suggested such conditioning approaches for the discriminator (Reed et al., 2016; Odena et al., 2017; Miyato & Koyama, 2018); among them, we adopted a recent conditioning method, projection discriminator (Miyato & Koyama, 2018), which not only suggests a more principled way of conditioning embeddings into the discriminator, but has also been widely used in many successful GANs related works (Brock et al., 2019; Zhang et al., 2018;
Revised: Many studies have suggested such conditioning approaches for the discriminator (Reed et al., 2016; Odena et al., 2017; Miyato & Koyama, 2018); among them, we adopted a recent conditioning method, projection discriminator (Miyato & Koyama, 2018), which not only suggests a more principled way of conditioning embeddings into the discriminator, but has also been widely used in many successful GANs related works (Brock et al., 2019; Zhang et al., 2018;

Original: Miyato et al., 2018).
Revised: Miyato et al., 2018).

Original: The study showed that, writing the discriminator function as D ( f , c ) :
Revised: The study showed that, writing the discriminator function as D ( f , c ) :

Original: = A ( g ( f , c )) , the condition information can be effectively provided to the discriminator using the inner-product of two vectors, c and φ ( f ) , as follows:
Revised: = A ( g ( f , c )) , the condition information can be effectively provided to the discriminator using the inner-product of two vectors, c and φ ( f ) , as follows:

Original: [Equation]
Revised: [Equation]

Original: where A ( · ) denotes an activation function (sigmoid in our case), c denotes condition embedding, φ ( f ) denotes output from the inner layer of discriminator and ψ ( · ) denotes a function that maps input vector to a scalar value (fully-connected layer in our case).
Revised: where A ( · ) denotes an activation function (sigmoid in our case), c denotes condition embedding, φ ( f ) denotes output from the inner layer of discriminator and ψ ( · ) denotes a function that maps input vector to a scalar value (fully-connected layer in our case).

Original: Here we focused on the fact that the conditioning signals can be used as an inner-product of two vectors and replaced it with the same inner-product operation used to compute the identity matching probability in the inference framework.
Revised: Here we focused on the fact that the conditioning signals can be used as an inner-product of two vectors and replaced it with the same inner-product operation used to compute the identity matching probability in the inference framework.

Original: Accordingly, we can rewrite the Eq. 2 by providing the condition with the trained speech encoder c = S ( s ) and substituting φ ( · ) with a trained face encoder F ( · ) from the subsection 3.1 as follows:
Revised: Accordingly, we can rewrite the Eq. 2 by providing the condition with the trained speech encoder c = S ( s ) and substituting φ ( · ) with a trained face encoder F ( · ) from the subsection 3.1 as follows:

Original: [Equation]
Revised: [Equation]

Original: Next, we adopted relativistic GANs (relGANs) loss which was reported to give stable image generation performance.
Revised: Next, we adopted relativistic GANs (relGANs) loss which was reported to give stable image generation performance.

Original: See Jolicoeur-Martineau (2019) for more details.
Revised: See Jolicoeur-Martineau (2019) for more details.

Original: Combining the condition term c T φ ( f ) and relGANs loss, g ( f , c ) can be modiﬁed to g rel ( f real , f fake , c ) as follows:
Revised: Combining the condition term c T φ ( f ) and relGANs loss, g ( f , c ) can be modiﬁed to g rel ( f real , f fake , c ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Eq. 4 is formulated to produce a face from the paired speech, but it can cause catastrophic forgetting on the trained φ ( · ) because the discriminator is no longer trained to penalize the mismatched face and voice.
Revised: Eq. 4 is formulated to produce a face from the paired speech, but it can cause catastrophic forgetting on the trained φ ( · ) because the discriminator is no longer trained to penalize the mismatched face and voice.

Original: Thus, we again modify Eq.
Revised: Thus, we again modify Eq.

Original: 4 so that the discriminator relativistically penalizes the mismatched face and voice more than a positively paired face and voice as follows:
Revised: 4 so that the discriminator relativistically penalizes the mismatched face and voice more than a positively paired face and voice as follows:

Original: [Equation]
Revised: [Equation]

Original: where f real and c + denotes the paired face and speech condition from data distribution, f fake denotes the generated face sample conditioned on c + , and c − denotes the speech condition with mismatched identity to f real using negative sampling.
Revised: where f real and c + denotes the paired face and speech condition from data distribution, f fake denotes the generated face sample conditioned on c + , and c − denotes the speech condition with mismatched identity to f real using negative sampling.

Original: Finally, utilizing the non-saturating loss (Goodfellow et al., 2014; Jolicoeur-Martineau, 2019), the proposed objective function (relidGANs loss) for discriminator L D and generator L G becomes as follows:
Revised: Finally, utilizing the non-saturating loss (Goodfellow et al., 2014; Jolicoeur-Martineau, 2019), the proposed objective function (relidGANs loss) for discriminator L D and generator L G becomes as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: Dataset and Sampling Two datasets were used throughout the experiments.
Revised: Dataset and Sampling Two datasets were used throughout the experiments.

Original: The ﬁrst dataset is the
Revised: The ﬁrst dataset is the

Original: AVSpeech dataset (Ephrat et al., 2018).
Revised: AVSpeech dataset (Ephrat et al., 2018).

Original: It consists of 2.8m of YouTube video clips of people actively speaking.
Revised: It consists of 2.8m of YouTube video clips of people actively speaking.

Original: Among them, we downloaded about 800k video clips from a train set and 140k clips from a test set.
Revised: Among them, we downloaded about 800k video clips from a train set and 140k clips from a test set.

Original: Out of 800k training samples, we used 50k of them as a validation set.
Revised: Out of 800k training samples, we used 50k of them as a validation set.

Original: The face images included in the dataset are extremely diverse since the face images were extracted from the videos “in the wild” (e.g., closing eyes, moving lips, diversity of video quality, and diverse facial expressions) making it challenging to train a generation model.
Revised: The face images included in the dataset are extremely diverse since the face images were extracted from the videos “in the wild” (e.g., closing eyes, moving lips, diversity of video quality, and diverse facial expressions) making it challenging to train a generation model.

Original: In addition, since no speaker identity information is provided, training the model with this dataset is considered fully self-supervised .
Revised: In addition, since no speaker identity information is provided, training the model with this dataset is considered fully self-supervised .

Original: Because of the absence of the speaker identity information, we assumed that each audio-visual clip represents an individual identity.
Revised: Because of the absence of the speaker identity information, we assumed that each audio-visual clip represents an individual identity.

Original: Therefore, a positive audio-visual pair was sampled only within a single clip while the negative samples were randomly selected from any clips excluding the ones sampled by the positive pair.
Revised: Therefore, a positive audio-visual pair was sampled only within a single clip while the negative samples were randomly selected from any clips excluding the ones sampled by the positive pair.

Original: Note that each speech and face image included in the positive pair was sampled from different time frames within the clip.
Revised: Note that each speech and face image included in the positive pair was sampled from different time frames within the clip.

Original: This was to ensure that the encoded embeddings of each modality do not contain linguistic related features (e.g., lip movement or phoneme related features).
Revised: This was to ensure that the encoded embeddings of each modality do not contain linguistic related features (e.g., lip movement or phoneme related features).

Original: The second dataset is the intersection of VoxCeleb (Nagrani & Zisserman, 2017) and VGGFace (Parkhi et al., 2015).
Revised: The second dataset is the intersection of VoxCeleb (Nagrani & Zisserman, 2017) and VGGFace (Parkhi et al., 2015).

Original: VoxCeleb is also a large-scale audio-visual dataset collected on YouTube videos.
Revised: VoxCeleb is also a large-scale audio-visual dataset collected on YouTube videos.

Original: VGGFace is a collection of the images of public ﬁgures gathered from the web, which is less diverse and relatively easier to model compared to the images in AVSpeech dataset.
Revised: VGGFace is a collection of the images of public ﬁgures gathered from the web, which is less diverse and relatively easier to model compared to the images in AVSpeech dataset.

Original: Since both
Revised: Since both

Original: VGGFace and VoxCeleb provide speaker identity information, we used the speakers’ included in the both datasets resulting in 1,251 speakers.
Revised: VGGFace and VoxCeleb provide speaker identity information, we used the speakers’ included in the both datasets resulting in 1,251 speakers.

Original: We used face images from both VoxCeleb and VGGFace, and used speech audio from Voxceleb.
Revised: We used face images from both VoxCeleb and VGGFace, and used speech audio from Voxceleb.

Original: Note that, this dataset provides multiple images and audio for a single speaker; therefore, training a model with this dataset cannot be called a self-supervised training method in a strict sense.
Revised: Note that, this dataset provides multiple images and audio for a single speaker; therefore, training a model with this dataset cannot be called a self-supervised training method in a strict sense.

Original: Implementation In every experiment, we used 6 seconds of audio.
Revised: Implementation In every experiment, we used 6 seconds of audio.

Original: The speech samples that were shorter or longer than 6 seconds were duplicated or randomly truncated so that they became 6 seconds.
Revised: The speech samples that were shorter or longer than 6 seconds were duplicated or randomly truncated so that they became 6 seconds.

Original: At the inference stage, we trained the networks with stochastic gradient descent optimizer (SGD).
Revised: At the inference stage, we trained the networks with stochastic gradient descent optimizer (SGD).

Original: At the generation stage, we trained the networks with Adam optimizer (Kingma & Ba, 2015).
Revised: At the generation stage, we trained the networks with Adam optimizer (Kingma & Ba, 2015).

Original: The discriminator network was regularized using R 1 regularizer (Mescheder et al., 2018).
Revised: The discriminator network was regularized using R 1 regularizer (Mescheder et al., 2018).

Original: More details are described in Appendix B.
Revised: More details are described in Appendix B.

Original: 4. C ROSS - MODAL I NFERENCE E VALUATION R ESULTS
Revised: 4. C ROSS - MODAL I NFERENCE E VALUATION R ESULTS

Original: Fully self-supervised inference accuracy results :
Revised: Fully self-supervised inference accuracy results :

Original: Here we show the evaluation results of the inference networks trained on the AVSpeech dataset.
Revised: Here we show the evaluation results of the inference networks trained on the AVSpeech dataset.

Original: Again, the model was trained with no additional information about speaker identity.
Revised: Again, the model was trained with no additional information about speaker identity.

Original: The evaluations were conducted 5 times by selecting different negative samples each time and reported using the average value of them.
Revised: The evaluations were conducted 5 times by selecting different negative samples each time and reported using the average value of them.

Original: [Table]
Revised: [Table]

Original: We trained our model in two settings (1.
Revised: We trained our model in two settings (1.

Original: V-F and 2.
Revised: V-F and 2.

Original: F-V) and both were trained in 10-way setting (1 positive pair andnegative pairs).
Revised: F-V) and both were trained in 10-way setting (1 positive pair andnegative pairs).

Original: Then the two trained models were tested on both V-F and F-V settings, each of which were tested on 2way and 10-way settings.
Revised: Then the two trained models were tested on both V-F and F-V settings, each of which were tested on 2way and 10-way settings.

Original: The top1-accuracy (%) results of cross-modal identity matching task is shown in Table 1.
Revised: The top1-accuracy (%) results of cross-modal identity matching task is shown in Table 1.

Original: The results show that our network can perform the task of crossmodal identity matching with a reasonable accuracy despite being trained in a fully self-supervised manner.
Revised: The results show that our network can perform the task of crossmodal identity matching with a reasonable accuracy despite being trained in a fully self-supervised manner.

Original: In addition, our model can perform F-V inference with a reasonable accuracy even when it is trained in V-F setting, and vice versa.
Revised: In addition, our model can perform F-V inference with a reasonable accuracy even when it is trained in V-F setting, and vice versa.

Original: Comparison results : We compared our model with two models - SVHFNet (Nagrani et al., 2018b) and DIMNet (Wen et al., 2019a).
Revised: Comparison results : We compared our model with two models - SVHFNet (Nagrani et al., 2018b) and DIMNet (Wen et al., 2019a).

Original: For fair comparisons, we trained our model with the intersection of VoxCeleb and VGGFace datasets.
Revised: For fair comparisons, we trained our model with the intersection of VoxCeleb and VGGFace datasets.

Original: Our model is trained in two train settings (1.
Revised: Our model is trained in two train settings (1.

Original: V-F and 2.
Revised: V-F and 2.

Original: F-V) with 10-way conﬁguration.
Revised: F-V) with 10-way conﬁguration.

Original: [Table]
Revised: [Table]

Original: The comparison results are shown in Table 2.
Revised: The comparison results are shown in Table 2.

Original: The results show that our model performs worse when tested in 2-way setting compared to other models.
Revised: The results show that our model performs worse when tested in 2-way setting compared to other models.

Original: Note that, SVHFNet used a pretrained speaker identiﬁcation network and face recognition network trained in a supervised manner with identity information and DIMNet trained the network in a supervised manner using the labels such as identity, gender, and nationality, and therefore is not trainable without such labels, whereas our model was trained from the scratch and without any such labels.
Revised: Note that, SVHFNet used a pretrained speaker identiﬁcation network and face recognition network trained in a supervised manner with identity information and DIMNet trained the network in a supervised manner using the labels such as identity, gender, and nationality, and therefore is not trainable without such labels, whereas our model was trained from the scratch and without any such labels.

Original: Under a more challenging situation where K increases from two to ten, however, our model shows signiﬁcantly bet- ter performance, which may indicate the proposed method is capable of extracting more informative and reliable features for the cross-modal identity matching task.
Revised: Under a more challenging situation where K increases from two to ten, however, our model shows signiﬁcantly bet- ter performance, which may indicate the proposed method is capable of extracting more informative and reliable features for the cross-modal identity matching task.

Original: 4. C ROSS - MODAL G ENERATION E VALUATION R ESULTS
Revised: 4. C ROSS - MODAL G ENERATION E VALUATION R ESULTS

Original: We conducted three qualitative analyses (QLA’s) and three quantitative analyses (QTA’s) to thoroughly examine the relationship between condition and the generated face samples.
Revised: We conducted three qualitative analyses (QLA’s) and three quantitative analyses (QTA’s) to thoroughly examine the relationship between condition and the generated face samples.

Original: QLA 1.
Revised: QLA 1.

Original: Random samples from ( z , c ) plane : The generated face images from diversely sampled z and ﬁxed speech condition c is shown in Fig.
Revised: Random samples from ( z , c ) plane : The generated face images from diversely sampled z and ﬁxed speech condition c is shown in Fig.

Original: We can observe that each variable shows different characteristics.
Revised: We can observe that each variable shows different characteristics.

Original: For example, it is observable that z controls the orientation of head, background color, haircolor, hairstyle, glasses, etc.
Revised: For example, it is observable that z controls the orientation of head, background color, haircolor, hairstyle, glasses, etc.

Original: Alternatively, we observed that c controls gender, age, ethnicity and the details of the face such as the shape of the face and the shape of the nose.
Revised: Alternatively, we observed that c controls gender, age, ethnicity and the details of the face such as the shape of the face and the shape of the nose.

Original: QLA 2.
Revised: QLA 2.

Original: Generated samples from interpolated speech condition : Next, we conducted an experiment to observe how speech conditions affect facial attributes by exploring the latent variable c with z ﬁxed.
Revised: Generated samples from interpolated speech condition : Next, we conducted an experiment to observe how speech conditions affect facial attributes by exploring the latent variable c with z ﬁxed.

Original: We sampled the speech segments from the AVSpeech test set and linearly interpolated two speech conditions.
Revised: We sampled the speech segments from the AVSpeech test set and linearly interpolated two speech conditions.

Original: After that we generated the face images based on the interpolated condition vectors.
Revised: After that we generated the face images based on the interpolated condition vectors.

Original: The generated face images are shown in Fig.
Revised: The generated face images are shown in Fig.

Original: We can observe the smooth transition of speech condition c in the latent space and can conﬁrm that it has a control over the parts that are directly correlated to the identity of a person such as gender, age, and ethnicity.
Revised: We can observe the smooth transition of speech condition c in the latent space and can conﬁrm that it has a control over the parts that are directly correlated to the identity of a person such as gender, age, and ethnicity.

Original: QLA 3.
Revised: QLA 3.

Original: Generated samples from interpolated stochastic variable : Lastly, we conducted an experiment that shows the role of stochastic variable in the generation process by interpolating two randomly sampled z vectors with c ﬁxed.
Revised: Generated samples from interpolated stochastic variable : Lastly, we conducted an experiment that shows the role of stochastic variable in the generation process by interpolating two randomly sampled z vectors with c ﬁxed.

Original: The speech segments from the AVSpeech test set were used to make a ﬁxed c for each row.
Revised: The speech segments from the AVSpeech test set were used to make a ﬁxed c for each row.

Original: The generated face images are shown in Fig.
Revised: The generated face images are shown in Fig.

Original: We can observe that the latent vector z has a control over the parts that are not directly correlated to the identity of a person such as head orientation, hair color, attire, and glasses.
Revised: We can observe that the latent vector z has a control over the parts that are not directly correlated to the identity of a person such as head orientation, hair color, attire, and glasses.

Original: QTA 1.
Revised: QTA 1.

Original: Correlation between the generated samples and speech conditions : We performed a quantitative analysis to investigate the relationship between the speech condition and the face image it generated.
Revised: Correlation between the generated samples and speech conditions : We performed a quantitative analysis to investigate the relationship between the speech condition and the face image it generated.

Original: To this end, we ﬁrst sampled two different random variables z 1 , z 2 and two different speech condition vectors c 1 , c 2 of different speakers.
Revised: To this end, we ﬁrst sampled two different random variables z 1 , z 2 and two different speech condition vectors c 1 , c 2 of different speakers.

Original: Next, we generated two face images f fake=
Revised: Next, we generated two face images f fake=

Original: G ( z 1 , c 1 ) , f fake= G ( z 2 , c 2 ) using the generator network.
Revised: G ( z 1 , c 1 ) , f fake= G ( z 2 , c 2 ) using the generator network.

Original: Then the two generated samples were encoded using the trained inference network F ( · ) to extract the face embeddings fe 1 = F ( f fake) , fe 2 = F ( f fake) .
Revised: Then the two generated samples were encoded using the trained inference network F ( · ) to extract the face embeddings fe 1 = F ( f fake) , fe 2 = F ( f fake) .

Original: We then calculated the cosine distance between the speech condition vectors ( CD ( c 1 , c 2 ) ), and the cosine distance between the two face embeddings ( CD ( fe 1 , fe 2 ) ).
Revised: We then calculated the cosine distance between the speech condition vectors ( CD ( c 1 , c 2 ) ), and the cosine distance between the two face embeddings ( CD ( fe 1 , fe 2 ) ).

Original: Finally, we computed the Pearson correlation between CD ( c 1 , c 2 ) and CD ( fe 1 , fe 2 ) .
Revised: Finally, we computed the Pearson correlation between CD ( c 1 , c 2 ) and CD ( fe 1 , fe 2 ) .

Original: This is to see if there exists any positive correlation between the embedding spaces of the two different modalities; that is, to see if closer speech embeddings help generate face images whose embeddings are also closer even when the random variable z is perturbed.
Revised: This is to see if there exists any positive correlation between the embedding spaces of the two different modalities; that is, to see if closer speech embeddings help generate face images whose embeddings are also closer even when the random variable z is perturbed.

Original: Fig.
Revised: Fig.

Original: 5 (a) shows the results using the test set of the AVSpeech.
Revised: 5 (a) shows the results using the test set of the AVSpeech.

Original: We can see a positive correlation between the CD ( c 1 , c 2 ) and CD ( fe 1 , fe 2 ) meaning that the generated face images are not randomly sampled.
Revised: We can see a positive correlation between the CD ( c 1 , c 2 ) and CD ( fe 1 , fe 2 ) meaning that the generated face images are not randomly sampled.

Original: Furthermore, we examined whether the CD between two speech condition vectors gets closer when controlling the gender of the speaker, and also the face images generated from the two speech condition vectors.
Revised: Furthermore, we examined whether the CD between two speech condition vectors gets closer when controlling the gender of the speaker, and also the face images generated from the two speech condition vectors.

Original: We tested this on VoxCeleb dataset as it provides gender labels for each speaker identity.
Revised: We tested this on VoxCeleb dataset as it provides gender labels for each speaker identity.

Original: We compared the mean values of CD ( c 1 , c 2 ) and CD ( fe 1 , fe 2 ) in two cases; one setting the gender of the two speakers different and the other one setting the gender the same.
Revised: We compared the mean values of CD ( c 1 , c 2 ) and CD ( fe 1 , fe 2 ) in two cases; one setting the gender of the two speakers different and the other one setting the gender the same.

Original: Fig.
Revised: Fig.

Original: 5 (b) shows a scatter plot of the CD when the two sampled speakers have different genders, and Fig.
Revised: 5 (b) shows a scatter plot of the CD when the two sampled speakers have different genders, and Fig.

Original: 5 (c) shows a case where the genders are set the same.
Revised: 5 (c) shows a case where the genders are set the same.

Original: We found out that the mean value of CD ( c 1 , c 2 ) gets smaller ( 0 . 

Revised: We found out that the mean value of CD ( c 1 , c 2 ) gets smaller ( 0 .

Original: → 0 . 28 ) when we set the gender of two speakers the same and accordingly the
Revised: → 0 . 28 ) when we set the gender of two speakers the same and accordingly the

Original: CD ( fe 1 , fe 2 ) gets also smaller ( 0 . 

Revised: CD ( fe 1 , fe 2 ) gets also smaller ( 0 .

Original: → 0 . 15 ).
Revised: → 0 . 15 ).

Original: [Figure]
Revised: [Figure]

Original: Correlation
Revised: Correlation

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Correlation: 0.
Revised: Correlation: 0.

Original: (c) Voxceleb (same gender) 0.
Revised: (c) Voxceleb (same gender) 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: QTA 2.
Revised: QTA 2.

Original: Testing the generated samples with the inference networks : Here we conducted two inference experiments in V-F setting.
Revised: Testing the generated samples with the inference networks : Here we conducted two inference experiments in V-F setting.

Original: In the ﬁrst experiment, given a positive pair from the AVSpeech test set ( f , s ), two face images ( f and G ( z , S ( s )) ) are passed to the trained inference networks.
Revised: In the ﬁrst experiment, given a positive pair from the AVSpeech test set ( f , s ), two face images ( f and G ( z , S ( s )) ) are passed to the trained inference networks.

Original: The inference networks are then used to determine which of the two images yields a greater cross-modal matching probability with a given speech segment s as follows:
Revised: The inference networks are then used to determine which of the two images yields a greater cross-modal matching probability with a given speech segment s as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 [ · ] denotes an identity function, n denotes a index of test sample, f n, 1 denotes the generated sample G ( z , S ( s n )) ), f n, 2 denotes the ground truth image paired with the speech s n , and N denotes the total number of test samples.
Revised: where 1 [ · ] denotes an identity function, n denotes a index of test sample, f n, 1 denotes the generated sample G ( z , S ( s n )) ), f n, 2 denotes the ground truth image paired with the speech s n , and N denotes the total number of test samples.

Original: Note that z was randomly sampled from Gaussian distribution for each n .
Revised: Note that z was randomly sampled from Gaussian distribution for each n .

Original: Surprisingly, we found out that the inference networks tend to choose the generated samples more than the ground truth face images with a chance of 76.65%, meaning that the generator is able to generate a plausible image given the speech condition c .
Revised: Surprisingly, we found out that the inference networks tend to choose the generated samples more than the ground truth face images with a chance of 76.65%, meaning that the generator is able to generate a plausible image given the speech condition c .

Original: On the other hand, we found out that the chances signiﬁcantly decreases to 47.2% when we do not use the relidGANs loss (using Eq. 4), showing that the proposed loss function helps generate images reﬂecting the identity of a speaker.
Revised: On the other hand, we found out that the chances signiﬁcantly decreases to 47.2% when we do not use the relidGANs loss (using Eq. 4), showing that the proposed loss function helps generate images reﬂecting the identity of a speaker.

Original: Note that, however, this result does not necessarily say that the generator is capable of generating a more plausible image conditioned on one’s voice than the paired face image as this experiment is bounded to the performance of the inference network.
Revised: Note that, however, this result does not necessarily say that the generator is capable of generating a more plausible image conditioned on one’s voice than the paired face image as this experiment is bounded to the performance of the inference network.

Original: In addition, we believe the distribution of real face image is much more diverse than the generated samples, causing the inference network tend towards the generated samples more.
Revised: In addition, we believe the distribution of real face image is much more diverse than the generated samples, causing the inference network tend towards the generated samples more.

Original: Our second experiment is similar to the ﬁrst experiment, but this time we compared the generated images from two different generators; one trained without mismatched identity loss (Eq. 4) and the other with the proposed relidGANs loss.
Revised: Our second experiment is similar to the ﬁrst experiment, but this time we compared the generated images from two different generators; one trained without mismatched identity loss (Eq. 4) and the other with the proposed relidGANs loss.

Original: We found out that the inference networks selected the generated sample from the generator trained with the proposed relidGANs loss with a chance of 79.68%, again showing that the proposed loss function helps to generate samples that reﬂect the identity information encoded in the speech condition.
Revised: We found out that the inference networks selected the generated sample from the generator trained with the proposed relidGANs loss with a chance of 79.68%, again showing that the proposed loss function helps to generate samples that reﬂect the identity information encoded in the speech condition.

Original: Next, we conducted one additional experiment in F-V setting.
Revised: Next, we conducted one additional experiment in F-V setting.

Original: Given a generated image G ( z , S ( s 1 )) from a speech segment s 1 , we measured the accuracy of inference network selecting s 1 out of two audio segments s 1 and s 2 as follows:
Revised: Given a generated image G ( z , S ( s 1 )) from a speech segment s 1 , we measured the accuracy of inference network selecting s 1 out of two audio segments s 1 and s 2 as follows:

Original: N (cid:88) n =  
Revised: N (cid:88) n =

Original: [Equation]
Revised: [Equation]

Original: where f n denotes a generated face image from a speech segment s n, 1 , and s n, 2 denotes a negatively selected speech segment.
Revised: where f n denotes a generated face image from a speech segment s n, 1 , and s n, 2 denotes a negatively selected speech segment.

Original: We found out that the inference networks select s 1 with a chance of 95.14%.
Revised: We found out that the inference networks select s 1 with a chance of 95.14%.

Original: Note that, the accuracy of inference networks in 2-way F-V setting is 88.47%, which means the generator can faithfully generate a face image according to the given speech segment s 1 .
Revised: Note that, the accuracy of inference networks in 2-way F-V setting is 88.47%, which means the generator can faithfully generate a face image according to the given speech segment s 1 .

Original: QTA 3.
Revised: QTA 3.

Original: Face image retrieval : Lastly, we conducted a face retrieval experiment in which the goal was to accurately retrieve a real face image for the speaker using the generated image from their speech segment as a query.
Revised: Face image retrieval : Lastly, we conducted a face retrieval experiment in which the goal was to accurately retrieve a real face image for the speaker using the generated image from their speech segment as a query.

Original: To compose a retrieval dataset, we randomly sampled 100 speakers from the test set of the AVSpeech.
Revised: To compose a retrieval dataset, we randomly sampled 100 speakers from the test set of the AVSpeech.

Original: For each speaker, we extractedface images from a video clip resulting in 5,000 images for the retrieval experiment in total.
Revised: For each speaker, we extractedface images from a video clip resulting in 5,000 images for the retrieval experiment in total.

Original: [Table]
Revised: [Table]

Original: Note that this process of composing the retrieval dataset is same as that of Speech2Face
Revised: Note that this process of composing the retrieval dataset is same as that of Speech2Face

Original: Oh et al.
Revised: Oh et al.

Original: To retrieve the closest face image out of the 5,000 samples, the trained face encoder was used to measure the feature distance between the generated image and each of the 5,000 images.
Revised: To retrieve the closest face image out of the 5,000 samples, the trained face encoder was used to measure the feature distance between the generated image and each of the 5,000 images.

Original: We computed three metrics as a feature distance ( L 1 , L 2 , and cosine distance ( CD )) and the results are reported in Table 3.
Revised: We computed three metrics as a feature distance ( L 1 , L 2 , and cosine distance ( CD )) and the results are reported in Table 3.

Original: Table 3 shows that our model can achieve higher performance than that of Speech2Face on all distance metrics.
Revised: Table 3 shows that our model can achieve higher performance than that of Speech2Face on all distance metrics.

Original: We also measured the performance of the generator trained without mismatched identity loss (mil) (Eq. 4).
Revised: We also measured the performance of the generator trained without mismatched identity loss (mil) (Eq. 4).

Original: The results show that the proposed relidGANs loss function is crucial to generate the face images that reﬂect the identity of the speakers.
Revised: The results show that the proposed relidGANs loss function is crucial to generate the face images that reﬂect the identity of the speakers.

Original: Examples of the top-5 retrieval results are shown in Fig.
Revised: Examples of the top-5 retrieval results are shown in Fig.

Original: Real
Revised: Real

Original: Generated
Revised: Generated

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this work, we proposed a cross-modal inference and generation framework that can be trained in a fully self-supervised way.
Revised: In this work, we proposed a cross-modal inference and generation framework that can be trained in a fully self-supervised way.

Original: We trained cGANs by transferring the trained networks from the inference stage so that the speech could be successfully encoded as a pseudo conditional embedding.
Revised: We trained cGANs by transferring the trained networks from the inference stage so that the speech could be successfully encoded as a pseudo conditional embedding.

Original: We also proposed relidGANs loss to train the discriminator to penalize negatively paired face and speech so that the generator could produce face images with more distinguished identity between different speakers.
Revised: We also proposed relidGANs loss to train the discriminator to penalize negatively paired face and speech so that the generator could produce face images with more distinguished identity between different speakers.

Original: As a future work, we would like to address a data bias problem (e.g., ethnicity, gender, age, etc.)
Revised: As a future work, we would like to address a data bias problem (e.g., ethnicity, gender, age, etc.)

Original: that exists in many datasets.
Revised: that exists in many datasets.

Original: This is a signiﬁcant problem as many publicly available datasets have biased demographic statistics, consequently affecting the results of many algorithms (Buolamwini & Gebru, 2018).
Revised: This is a signiﬁcant problem as many publicly available datasets have biased demographic statistics, consequently affecting the results of many algorithms (Buolamwini & Gebru, 2018).

Original: We believe that this can be solved with the use of a better data sampling strategy in an unsupervised manner such as (Amini et al., 2019).
Revised: We believe that this can be solved with the use of a better data sampling strategy in an unsupervised manner such as (Amini et al., 2019).

Original: In addition, we would like to expand the proposed methods to various multi-modal datasets by generalizing the proposed concept to other modalities.
Revised: In addition, we would like to expand the proposed methods to various multi-modal datasets by generalizing the proposed concept to other modalities.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was supported partly by Next-Generation Information Computing Development Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (MSIT; Grant No. NRF-2017M3C4A7078548), and partly by Institute for Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)
Revised: This work was supported partly by Next-Generation Information Computing Development Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (MSIT; Grant No. NRF-2017M3C4A7078548), and partly by Institute for Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)

Original: (No.2019-0-01367, Infant-Mimic Neurocognitive Developmental Machine Learning from Interaction Experience with Real World (BabyMind))
Revised: (No.2019-0-01367, Infant-Mimic Neurocognitive Developmental Machine Learning from Interaction Experience with Real World (BabyMind))

Original: A PPENDIX A D ATA P REPROCESSING
Revised: A PPENDIX A D ATA P REPROCESSING

Original: Image processing : All video clips downloaded from the AVSpeech dataset were resampled to be 25FPS.
Revised: Image processing : All video clips downloaded from the AVSpeech dataset were resampled to be 25FPS.

Original: If more than one face are detected in a frame, a face closer to the coordinates of the target speaker in the ﬁrst frame was selected.
Revised: If more than one face are detected in a frame, a face closer to the coordinates of the target speaker in the ﬁrst frame was selected.

Original: Note that the coordinates of the target speakers are provided by the AVSpeech dataset.
Revised: Note that the coordinates of the target speakers are provided by the AVSpeech dataset.

Original: Because the size of the speaker’s face on the screen varies from video to video, the image was resized to maintain interocular distance as 55 pixels, as described in Cole et al.
Revised: Because the size of the speaker’s face on the screen varies from video to video, the image was resized to maintain interocular distance as 55 pixels, as described in Cole et al.

Original: We used a publicly available software, Dlib (King, 2009), to detect and crop the face images.
Revised: We used a publicly available software, Dlib (King, 2009), to detect and crop the face images.

Original: The images were cropped to the size of 224 × 224 and resized to 128 × 128 for training.
Revised: The images were cropped to the size of 224 × 224 and resized to 128 × 128 for training.

Original: We additionally applied horizontal ﬂip method as a data augmentation strategy.
Revised: We additionally applied horizontal ﬂip method as a data augmentation strategy.

Original: These procedures of cropping, resize, and ﬂipping are adapted to all images in VGGFace and VoxCeleb, too.
Revised: These procedures of cropping, resize, and ﬂipping are adapted to all images in VGGFace and VoxCeleb, too.

Original: Audio processing : All audio samples were resampled to 16kHz, and stereo audio samples were converted to mono.
Revised: Audio processing : All audio samples were resampled to 16kHz, and stereo audio samples were converted to mono.

Original: We used 6 seconds of audio in both the inference and generation model.
Revised: We used 6 seconds of audio in both the inference and generation model.

Original: If the audio is longer than 6 seconds, the audio was randomly truncated.
Revised: If the audio is longer than 6 seconds, the audio was randomly truncated.

Original: If the audio is shorter than 6 seconds, the entire audio was duplicated until it becomes longer than 6 seconds.
Revised: If the audio is shorter than 6 seconds, the entire audio was duplicated until it becomes longer than 6 seconds.

Original: After then the duplicated audio sample was randomly truncated to be 6 seconds.
Revised: After then the duplicated audio sample was randomly truncated to be 6 seconds.

Original: We applied root-mean-square normalization to make the overall amplitude of speech signals to be consistent.
Revised: We applied root-mean-square normalization to make the overall amplitude of speech signals to be consistent.

Original: The reference level was selected as 0.01.
Revised: The reference level was selected as 0.01.

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: We used stochastic gradient descent (SGD) with the momentum of 0.9 and weight decay of 5 e −to optimize inference networks (speech encoder and face encoder).
Revised: We used stochastic gradient descent (SGD) with the momentum of 0.9 and weight decay of 5 e −to optimize inference networks (speech encoder and face encoder).

Original: The learning rate was initialized to 0.001 and decayed by the factor of 10 if validation loss was not decreased for 1 epoch.
Revised: The learning rate was initialized to 0.001 and decayed by the factor of 10 if validation loss was not decreased for 1 epoch.

Original: Training was stopped if learning rate decay occurred three times.
Revised: Training was stopped if learning rate decay occurred three times.

Original: Minibatch size was ﬁxed to 32 and 12 for
Revised: Minibatch size was ﬁxed to 32 and 12 for

Original: V-F and F-V training conﬁguration, respectively.
Revised: V-F and F-V training conﬁguration, respectively.

Original: For both training and test phase, negative samples were randomly selected for every step while pre-deﬁned negative samples for each positive sample were used for validation phase to ensure stable learning rate decay scheduling.
Revised: For both training and test phase, negative samples were randomly selected for every step while pre-deﬁned negative samples for each positive sample were used for validation phase to ensure stable learning rate decay scheduling.

Original: To train cGANs, we used Adam optimizer using β 1 and β 1 of 0.9 and 0.9, respectively.
Revised: To train cGANs, we used Adam optimizer using β 1 and β 1 of 0.9 and 0.9, respectively.

Original: The learning rate of generator and discriminator was ﬁxed to 0.0001 and 0.00005 during training, respectively.
Revised: The learning rate of generator and discriminator was ﬁxed to 0.0001 and 0.00005 during training, respectively.

Original: Each time the discriminator was updated twice, the generator was updated once.
Revised: Each time the discriminator was updated twice, the generator was updated once.

Original: Batch size is 

Revised: Batch size is

Original: and we trained the model about 500,000 iterations.
Revised: and we trained the model about 500,000 iterations.

Original: Note that we adopted the truncation trick of Brock et al.
Revised: Note that we adopted the truncation trick of Brock et al.

Original: (2019) in subsection 4.2.
Revised: (2019) in subsection 4.2.

Original: In all experiments except QTA 1 and QTA 2, where the truncation trick was not adopted, the truncation threshold was set to 1.0.
Revised: In all experiments except QTA 1 and QTA 2, where the truncation trick was not adopted, the truncation threshold was set to 1.0.

Original: C N ETWORK S TRUCTURES
Revised: C N ETWORK S TRUCTURES

Original: Inference network consists of a speech encoder and a face encoder.
Revised: Inference network consists of a speech encoder and a face encoder.

Original: The network structure of the speech encoder is based on the problem agnostic speech encoder (PASE) (Pascual et al., 2019) followed by an additional time pooling layer and a fully-connected layer (FC).
Revised: The network structure of the speech encoder is based on the problem agnostic speech encoder (PASE) (Pascual et al., 2019) followed by an additional time pooling layer and a fully-connected layer (FC).

Original: PASE consists of
Revised: PASE consists of

Original: SincNet and 7 stacks convolutional-block (ConvBlock) composed of 1d-CNN, batch normalization, and multi-parametric rectiﬁed linear unit (PReLU) activation.
Revised: SincNet and 7 stacks convolutional-block (ConvBlock) composed of 1d-CNN, batch normalization, and multi-parametric rectiﬁed linear unit (PReLU) activation.

Original: Following PASE, average pooling on time dimension is applied to make the size of embedding time-invariant.
Revised: Following PASE, average pooling on time dimension is applied to make the size of embedding time-invariant.

Original: The FC layer was used as the last layer of the speech encoder.
Revised: The FC layer was used as the last layer of the speech encoder.

Original: The details of the speech encoder is depicted in Fig.
Revised: The details of the speech encoder is depicted in Fig.

Original: 7 For face encoder, we used 2d-CNN based residual block (ResBlock) in each layer.
Revised: 7 For face encoder, we used 2d-CNN based residual block (ResBlock) in each layer.

Original: Note that the network structure is similar to the discriminator network of (Miyato & Koyama, 2018).
Revised: Note that the network structure is similar to the discriminator network of (Miyato & Koyama, 2018).

Original: The details of the architecture is shown in Fig.
Revised: The details of the architecture is shown in Fig.

Original: For the generator, we followed the generator network structure of (Miyato & Koyama, 2018) with some modiﬁcations, such as concatenating z and c as an input and adopting adaptive instance normalization (AdaIN) for the direct conditioning method.
Revised: For the generator, we followed the generator network structure of (Miyato & Koyama, 2018) with some modiﬁcations, such as concatenating z and c as an input and adopting adaptive instance normalization (AdaIN) for the direct conditioning method.

Original: The details of the discriminator are almost the same as the face encoder except that it includes additional FC layer, projection layer, and sigmoid as activation function.
Revised: The details of the discriminator are almost the same as the face encoder except that it includes additional FC layer, projection layer, and sigmoid as activation function.

Original: The details of the discriminator and generator are shown in Fig.
Revised: The details of the discriminator and generator are shown in Fig.

Original: 8 and
Revised: 8 and

Original: Fig. 9, respectively.
Revised: Fig. 9, respectively.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: D S UPPLEMENTARY G ENERATED I MAGES
Revised: D S UPPLEMENTARY G ENERATED I MAGES

Original: Here we provide some additional uncurated generated face images.
Revised: Here we provide some additional uncurated generated face images.

Original: Fig.
Revised: Fig.

Original: 10 and Fig.
Revised: 10 and Fig.

Original: 11 show generated images with truncation threshold 0.5 and 1.0, respectively
Revised: 11 show generated images with truncation threshold 0.5 and 1.0, respectively

Original: E T HE E FFECTIVENESS OF T WO - STAGE T RAINING
Revised: E T HE E FFECTIVENESS OF T WO - STAGE T RAINING

Original: In this subsection, we would like to address concerns regarding the validity of the two-stage training method as proposed in Fig.
Revised: In this subsection, we would like to address concerns regarding the validity of the two-stage training method as proposed in Fig.

Original: Here the effect of the two-stage training method is demonstrated by showing the failure case of training when we skip the inference training stage.
Revised: Here the effect of the two-stage training method is demonstrated by showing the failure case of training when we skip the inference training stage.

Original: Fig.
Revised: Fig.

Original: 12 shows that the generator ignores the speech condition vector in the generation process and, consequently, the generated images are only dependent on the random vector z .
Revised: 12 shows that the generator ignores the speech condition vector in the generation process and, consequently, the generated images are only dependent on the random vector z .

Original: This shows that the learned representation using the proposed self-supervised method is effective and can be used as an important cue for the conditional generation.
Revised: This shows that the learned representation using the proposed self-supervised method is effective and can be used as an important cue for the conditional generation.