Original: L
Revised: L

Original: ATENT
Revised: ATENT

Original: K EYPOINT
Revised: K EYPOINT

Original: GAN: C ONTROLLING GAN S VIA
Revised: GAN: C ONTROLLING GAN S VIA

Original: L ATENT K EYPOINTS
Revised: L ATENT K EYPOINTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative adversarial networks (GANs) have attained photo-realistic quality in image generation.
Revised: Generative adversarial networks (GANs) have attained photo-realistic quality in image generation.

Original: However, how to best control the image content remains an open challenge.
Revised: However, how to best control the image content remains an open challenge.

Original: We introduce LatentKeypointGAN a two-stage GAN which is trained end-to-end on the classical GAN objective with internal conditioning on a set of space keypoints
Revised: We introduce LatentKeypointGAN a two-stage GAN which is trained end-to-end on the classical GAN objective with internal conditioning on a set of space keypoints

Original: These keypoints have associated appearance embed dings that respectively control the position and style of the generated objects and their parts
Revised: These keypoints have associated appearance embed dings that respectively control the position and style of the generated objects and their parts

Original: A major difﬁculty that we address with suitable network architectures and training schemes is disentangling the image into spatial and appearance factors without domain knowledge and supervision signals.
Revised: A major difﬁculty that we address with suitable network architectures and training schemes is disentangling the image into spatial and appearance factors without domain knowledge and supervision signals.

Original: We demonstrate that
Revised: We demonstrate that

Original: LatentKeypointGAN provides an interpretable latent space that can be used to re arrange the generated images by re positioning and exchanging keypoint embeddings, such as generating portraits by combining the eyes and mouth from different images
Revised: LatentKeypointGAN provides an interpretable latent space that can be used to re arrange the generated images by re positioning and exchanging keypoint embeddings, such as generating portraits by combining the eyes and mouth from different images

Original: In addition
Revised: In addition

Original: , the explicit generation of keypoints and matching images enables a new, GAN-based method for unsupervised keypoint detection.
Revised: , the explicit generation of keypoints and matching images enables a new, GAN-based method for unsupervised keypoint detection.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It is a long-standing goal to build generative models that depict the distribution of example images faithfully.
Revised: It is a long-standing goal to build generative models that depict the distribution of example images faithfully.

Original: While photo-realism is reached for well-constrained domains, such as faces, it remains challenging to make this image generation process interpretable and editable.
Revised: While photo-realism is reached for well-constrained domains, such as faces, it remains challenging to make this image generation process interpretable and editable.

Original: Desired is a latent space that disentangles an image into parts and their appearances, which would allow a user to recombine and re-imagine a generated face interactively and artistically.
Revised: Desired is a latent space that disentangles an image into parts and their appearances, which would allow a user to recombine and re-imagine a generated face interactively and artistically.

Original: There are promising attempts (Lorenz et al., 2019) that use spatial image transformations, such as the thin plate spline (TPS) method, to create pairs of real and deformed images and impose an equivariance loss to discover keypoints and object appearance embeddings as the bottleneck of an autoencoder.
Revised: There are promising attempts (Lorenz et al., 2019) that use spatial image transformations, such as the thin plate spline (TPS) method, to create pairs of real and deformed images and impose an equivariance loss to discover keypoints and object appearance embeddings as the bottleneck of an autoencoder.

Original: Thereby, one can edit the image by moving the keypoints or modifying the appearance embedding.
Revised: Thereby, one can edit the image by moving the keypoints or modifying the appearance embedding.

Original: However, Li et al.
Revised: However, Li et al.

Original: (2020) points out that the learned keypoints may be biased and only meaningful for the artiﬁcially introduced image transformations, the TPS transformation require careful parameter tuning for each domain, and Xu et al.
Revised: (2020) points out that the learned keypoints may be biased and only meaningful for the artiﬁcially introduced image transformations, the TPS transformation require careful parameter tuning for each domain, and Xu et al.

Original: (2020b) shows that some geometry and appearance entanglement remains.
Revised: (2020b) shows that some geometry and appearance entanglement remains.

Original: [Figure]
Revised: [Figure]

Original: To avoid these shortcomings of existing autoencoder-based techniques, we propose an alternative
Revised: To avoid these shortcomings of existing autoencoder-based techniques, we propose an alternative

Original: GAN-based approach that does not rely on paired images and TPS transformation, thereby alleviating their domain-speciﬁc tuning and bias.
Revised: GAN-based approach that does not rely on paired images and TPS transformation, thereby alleviating their domain-speciﬁc tuning and bias.

Original: Instead of an autoencoder, we use a GAN that includes keypoints as a latent embedding.
Revised: Instead of an autoencoder, we use a GAN that includes keypoints as a latent embedding.

Original: Figure 1 shows how this LatentKeypointGAN enables our main goal of editing the output image by changing the keypoint position, adding or removing points, and exchanging associated appearance embedding locally.
Revised: Figure 1 shows how this LatentKeypointGAN enables our main goal of editing the output image by changing the keypoint position, adding or removing points, and exchanging associated appearance embedding locally.

Original: Although entirely unsupervised, the learned keypoints meaningfully align with the image landmarks, such as a keypoint linked to the nose when generating images of faces.
Revised: Although entirely unsupervised, the learned keypoints meaningfully align with the image landmarks, such as a keypoint linked to the nose when generating images of faces.

Original: As a byproduct, we can learn a separate keypoint detector on generated image-keypoint pairs for unsupervised keypoint detection and to quantify localization accuracy.
Revised: As a byproduct, we can learn a separate keypoint detector on generated image-keypoint pairs for unsupervised keypoint detection and to quantify localization accuracy.

Original: The difﬁculty of such unsupervised learning approaches lies in ﬁnding the right implicit bias to facilitate the desired disentanglement Locatello et al.
Revised: The difﬁculty of such unsupervised learning approaches lies in ﬁnding the right implicit bias to facilitate the desired disentanglement Locatello et al.

Original: LatentKeypointGAN is designed as a two-stage GAN architecture that is trained end-to-end on the standard GAN objective.
Revised: LatentKeypointGAN is designed as a two-stage GAN architecture that is trained end-to-end on the standard GAN objective.

Original: In the ﬁrst step, a generator network turns the input values sampled from a normal distribution into 2D keypoint locations and their associated encoding.
Revised: In the ﬁrst step, a generator network turns the input values sampled from a normal distribution into 2D keypoint locations and their associated encoding.

Original: We ensure with suitable layer connectivity that some of the encodings are correlated while others remain independent.
Revised: We ensure with suitable layer connectivity that some of the encodings are correlated while others remain independent.

Original: These generated keypoints are then mapped to spatial heatmaps of increasing resolution.
Revised: These generated keypoints are then mapped to spatial heatmaps of increasing resolution.

Original: The heatmaps deﬁne the position of the keypoints and their support sets the inﬂuence range of their respective encodings.
Revised: The heatmaps deﬁne the position of the keypoints and their support sets the inﬂuence range of their respective encodings.

Original: In the second step, a SPADE-like (Park et al., 2019) image generator turns these spatial encodings into a complete and realistic image.
Revised: In the second step, a SPADE-like (Park et al., 2019) image generator turns these spatial encodings into a complete and realistic image.

Original: We summarize our contributions below:
Revised: We summarize our contributions below:

Original: Development of a GAN-based framework for image manipulation and keypoint detection; 2.
Revised: Development of a GAN-based framework for image manipulation and keypoint detection; 2.

Original: Design of a keypoint generator that models dependent and independent factors explicitly; 3.
Revised: Design of a keypoint generator that models dependent and independent factors explicitly; 3.

Original: Empirical study comparing different editing methods in terms of perceptual quality; 4.
Revised: Empirical study comparing different editing methods in terms of perceptual quality; 4.

Original: A new methodology for keypoint detection that contests established autoencoder methods.
Revised: A new methodology for keypoint detection that contests established autoencoder methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In the following, we discuss variants of deep generative models that learn to synthesize images from a collection of examples, focusing on methods modeling keypoints and those providing control on the image content.
Revised: In the following, we discuss variants of deep generative models that learn to synthesize images from a collection of examples, focusing on methods modeling keypoints and those providing control on the image content.

Original: GANs (Goodfellow et al., 2014) are trained to generate images from a distribution that resembles the training distributions.
Revised: GANs (Goodfellow et al., 2014) are trained to generate images from a distribution that resembles the training distributions.

Original: Recent approaches attain photo-realism for portrait images.
Revised: Recent approaches attain photo-realism for portrait images.

Original: Karras et al.
Revised: Karras et al.

Original: (2018) train a progressive growth of the GAN’s generator and discriminator.
Revised: (2018) train a progressive growth of the GAN’s generator and discriminator.

Original: After that, Karras et al.
Revised: After that, Karras et al.

Original: (2019; 2020) integrate multiplicative neural network layers to build StyleGAN and StyleGAN2.
Revised: (2019; 2020) integrate multiplicative neural network layers to build StyleGAN and StyleGAN2.

Original: StyleGAN gains some control on the generated high-resolution, high-quality faces, by adopting ideas from style transfer (Gatys et al., 2016; Huang & Belongie, 2017) and exchanging features of hidden layers between different samples.
Revised: StyleGAN gains some control on the generated high-resolution, high-quality faces, by adopting ideas from style transfer (Gatys et al., 2016; Huang & Belongie, 2017) and exchanging features of hidden layers between different samples.

Revised: We build upon these network architectures and equip them with additional structure.

Original:  More recently, efforts have been made on exploring the latent space of a pre-trained StyleGAN for image editing (Shen et al., 2020; Jahanian* et al., 2020).
Revised: Editing GANs globally. More recently, efforts have been made on exploring the latent space of a pre-trained StyleGAN for image editing (Shen et al., 2020; Jahanian* et al., 2020).

Original: Furthermore, to allow editing real-world images, various encoders (Zhu et al., 2020a; Abdal et al., 2019; 2020; 
Revised: To allow editing real-world images, various encoders (Zhu et al., 2020a; Abdal et al., 2019; 2020; Guan et al., 2020;

Original: Guan et al., 2020; Wulff & Torralba, 2020; Richardson et al., 2020) have been trained to project images into the latent space of StyleGANs.
Revised: Wulff & Torralba, 2020; Richardson et al., 2020) have been trained to project images into the latent space of StyleGANs.

Original: These methods provide control over the image synthesis process, such as for changing age, pose, and gender.
Revised: These methods provide control over the image synthesis process, such as for changing age, pose, and gender.

Original: To enable rig-like controls over semantic face parameters that are interpretable in 3D, such as illumination, Tewari et al.
Revised: To enable rig-like controls over semantic face parameters that are interpretable in 3D, such as illumination, Tewari et al.

Original: (2020b;a); Ghosh et al.
Revised: (2020b;a); Ghosh et al.

Original: ; Deng et al.
Revised: ; Deng et al.

Original: (2020) integrate 3D face models (Blanz & Vetter, 1999; Li et al., 2017) with GANs.
Revised: (2020) integrate 3D face models (Blanz & Vetter, 1999; Li et al., 2017) with GANs.

Original: Compared with these methods, our model focuses on detailed and local semantic controls.
Revised: Compared with these methods, our model focuses on detailed and local semantic controls.

Original: Instead of changing the face as a whole, our method is able to change a local patch without an obvious impact on other regions.
Revised: Instead of changing the face as a whole, our method is able to change a local patch without an obvious impact on other regions.

Original: Furthermore, our keypoints provide control handles for animation without manual rigging.
Revised: Furthermore, our keypoints provide control handles for animation without manual rigging.

Original: Therefore, LatentKeypointGAN can be applied to many different objects and image domains.
Revised: Therefore, LatentKeypointGAN can be applied to many different objects and image domains.

Revised: Editing GANs locally.

Revised: Instead of exchanging entire feature maps for style editing, local modiﬁcations are possible but require care to maintain consistent and artifact free synthesis.

Original: Conditional image synthesis methods synthesize images that resemble a given reference input, such as category labels (Mirza & Osindero, 2014; Odena et al., 2017), text (Zhang et al., 2017;
Revised: Collins et al. (2020) propose to cluster the middle-layer feature maps of StyleGAN and to use the embedding to edit local appearance but require human supervision to select suitable clusters. Instead of using existing feature maps, Alharbi & Wonka (2020) and Kim et al. (2021) generate spatially-variable maps by structured noise generators and then swap the pixel embedding of the maps to edit the images.

Revised: However, this requires to draw source and target regions by hand for every instance.

Revised: By contrast, our model can explicitly change the pose by modifying keypoint locations that are consistent across images.

Revised: Kwon & Ye (2021) manipulate attention maps to edit the pose and local shape, but they do not demonstrate local appearance editing.

Revised: While enabling different forms of editing, all of the methods above require some form of pixel-level selection of regions at test time or manual cluster selection out of hundreds of candidates.

Revised: In addition, they do not demonstrate on out-of-distribution editing, such as moving parts relative to each other and adding or removing parts, which requires stronger local disentanglement.

Revised: Zhan et al.

Revised: (2019) use a cycle consistency with a spatial transformation network to fuse a foreground object on top of a background reference.

Revised: They can add a part but only if separate datasets with and without that part are available (e.g., with and without glasses).

Revised: In sum, our keypoint-based editing is both more intuitive and enables additional editing capabilities.

Revised: Conditional image synthesis methods synthesize images that resemble a given reference input, such as category labels (Mirza & Osindero, 2014; Odena et al., 2017), text (Zhang et al., 2017;

Original: Reed et al., 2016), and layout (Zhao et al., 2019; Sun & Wu, 2019).
Revised: Reed et al., 2016), and layout (Zhao et al., 2019; Sun & Wu, 2019).

Original: A variant of the conditional
Revised: A variant of the conditional

Original: GANs can do image-to-image translation (Isola et al., 2017; Huang et al., 2018;
Revised: GANs can do image-to-image translation (Isola et al., 2017; Huang et al., 2018;

Original: Liu et al., 2017; Zhu et al., 2017b;a; Wang et al., 2018; Park et al., 2019).
Revised: Liu et al., 2017; Zhu et al., 2017b;a; Wang et al., 2018; Park et al., 2019).

Original: These approaches aim to transfer images from

Original: [Figure]

Original: toRGB toRGB toRGB

Original: [Figure]

Original: Spatial embedding layer Image generator SPA D E B l o ck SPA D E B l o ck concat one domain to another while preserving the image structure, such 
Revised: These approaches aim to transfer images from one domain to another while preserving the image structure, such as mapping from day to night.

Original: Park et al.
Revised: Park et al.

Original: (2019) pioneered using spatially-adaptive normalization to transfer segmentation masks to images, which we borrow and adapt to be conditioned on landmark position.
Revised: (2019) pioneered using spatially-adaptive normalization to transfer segmentation masks to images, which we borrow and adapt to be conditioned on landmark position.

Original: To control individual aspects of faces, such as changing eye or nose shape, recent works condition on segmentation masks (Lee et al., 2020; Yu et al., 2020; Richardson et al., 2020; Zhu et al., 2020b; Tan et al., 2020), rough sketches (Tseng et al., 2020; Chen et al., 2020), landmarks (Yang et al., 2019), or word labels such as with or w/o glasses (Cheng et al., 2020a), of faces as input. 
Revised: To control individual aspects of faces, such as changing eye or nose shape, recent works condition on segmentation masks (Lee et al., 2020; Yu et al., 2020; Richardson et al., 2020; Zhu et al., 2020b; Tan et al., 2020), rough sketches (Tseng et al., 2020; Chen et al., 2020), landmarks (Yang et al., 2019), or word labels such as with or w/o glasses (Cheng et al., 2020a), of faces as input. Hong et al. (2018) can superimpose a new object without requiring an exact mask, but still require one at training time.

Original: Compared with these methods, our model does not take any kind of supervision or other conditions at training time.
Revised: Compared with these methods, our model does not take any kind of supervision or other conditions at training time.

Original: It is trained in a totally unsupervised manner.
Revised: It is trained in a totally unsupervised manner.

Original: Still, our method allows the landmarks to learn a meaningful location and semantic  that can be controlled at test time.
Revised: Still, our method allows the landmarks to learn a meaningful location and semantic embedding that can be controlled at test time.

Original: Unsupervised landmark discovery approaches aim to detect the landmarks from images without supervision.
Revised: Unsupervised landmark discovery approaches aim to detect the landmarks from images without supervision.

Original: Most works train two-branch autoencoders, where shape and appearance are disentangled by training on pairs of images where one of the two is matching while the other factor varies.
Revised: Most works train two-branch autoencoders, where shape and appearance are disentangled by training on pairs of images where one of the two is matching while the other factor varies.

Original: The essence of these methods is to compare the pair of images to discover disentangled landmarks.
Revised: The essence of these methods is to compare the pair of images to discover disentangled landmarks.

Original: These pairs can stem from different views (Suwajanakorn et al., 2018;
Revised: These pairs can stem from different views (Suwajanakorn et al., 2018;

Original: Rhodin et al., 2019), and frames of the same video (Siarohin et al., 2019; Kulkarni et al., 2019; Minderer et al., 2019).
Revised: Rhodin et al., 2019), and frames of the same video (Siarohin et al., 2019; Kulkarni et al., 2019; Minderer et al., 2019).

Original: However, this additional motion information is not always available or is difﬁcult to capture.
Revised: However, this additional motion information is not always available or is difﬁcult to capture.

Original: Existing unsupervised methods trained on single images create pairs from spatial deformation and color shifts of a source image (Xing et al., 2018; Shu et al., 2018; Thewlis et al., 2019; Li et al., 2020; Cheng et al., 2020b; Dundar et al., 2020).
Revised: Existing unsupervised methods trained on single images create pairs from spatial deformation and color shifts of a source image (Xing et al., 2018; Shu et al., 2018; Thewlis et al., 2019; Li et al., 2020; Cheng et al., 2020b; Dundar et al., 2020).

Original: However, the parameters of augmentation strategies, such as the commonly used thin plate spline deformation model (Thewlis et al., 2017; Zhang et al., 2018; Jakab et al., 2018; Lorenz et al., 2019), are difﬁcult to calibrate and the learned keypoints may be biased to be only meaningful for the transformation (Li et al., 2020).
Revised: However, the parameters of augmentation strategies, such as the commonly used thin plate spline deformation model (Thewlis et al., 2017; Zhang et al., 2018; Jakab et al., 2018; Lorenz et al., 2019), are difﬁcult to calibrate and the learned keypoints may be biased to be only meaningful for the transformation (Li et al., 2020).

Original: Besides, the appearance information may be hidden in keypoint location (Xu et al., 2020b).
Revised: Besides, the appearance information may be hidden in keypoint location (Xu et al., 2020b).

Original: The cycle consistency-based methods (Wu et al., 2019; Xu et al., 2020b) cannot provide local editing ability.
Revised: The cycle consistency-based methods (Wu et al., 2019; Xu et al., 2020b) cannot provide local editing ability.

Original: We show that our generator can well disentangle the appearance and keypoint location and control the images locally.
Revised: We show that our generator can well disentangle the appearance and keypoint location and control the images locally.

Original: Furthermore, the underlying idea of these methods, no matter whether they depend on multi-views, videos, or thin plate splines, is creating pairs of images to compare to ﬁnd the clue of equivariant keypoints in comparison.
Revised: Furthermore, the underlying idea of these methods, no matter whether they depend on multi-views, videos, or thin plate splines, is creating pairs of images to compare to ﬁnd the clue of equivariant keypoints in comparison.

Original: On the other hand, our method does not need any paired correspondences.
Revised: On the other hand, our method does not need any paired correspondences.

Original: It is totally based on generation.
Revised: It is totally based on generation.

Original: Our GAN approach poses a viable alternative by training a keypoint detector on synthetic examples generated by LatentKeypointGAN.
Revised: Our GAN approach poses a viable alternative by training a keypoint detector on synthetic examples generated by LatentKeypointGAN.

Original:  L ATENT K EYPOINT GAN M ETHOD AND M ETHODOLOGY
Revised:  L ATENT K EYPOINT GAN M ETHOD AND M ETHODOLOGY

Original: The unsupervised spatial disentanglement of image content that we target has been shown to be impossible without an implicit bias imposed by the network architecture (Locatello et al., 2019).
Revised: The unsupervised spatial disentanglement of image content that we target has been shown to be impossible without an implicit bias imposed by the network architecture (Locatello et al., 2019).

Original: To this end, our proposed LatentKeypointGAN architecture encodes position explicitly and appearance locally with two carefully designed sub-networks.
Revised: To this end, our proposed LatentKeypointGAN architecture encodes position explicitly and appearance [Figure] toRGB toRGB toRGB [Figure]

Revised: embedding layer

Original:  First, the keypoint generator, K , deﬁnes the spatial arrangement of parts and their embedding.
Revised: Image generator SPA D E B l o ck SPA D E B l o ck concat locally with two carefully designed sub-networks. First, the keypoint generator, K , deﬁnes the spatial arrangement of parts and their embedding.

Original: Subsequently, a spatial embedding layer, S , turns these sparse estimates into dense feature maps, and the image generator G up-scales these into a highresolution image.
Revised: Subsequently, a spatial embedding layer, S , turns these sparse estimates into dense feature maps, and the image generator G up-scales these into a highresolution image.

Original: Figure 2 shows the entire architecture.
Revised: Figure 2 shows the entire architecture.

Original: In the following, we explain architectural details and how we can train these networks end-to-end on a standard GAN objective, without any supervision on keypoints or masks.
Revised: In the following, we explain architectural details and how we can train these networks end-to-end on a standard GAN objective, without any supervision on keypoints or masks.

Original: At inference time, the latent keypoints allow one to author the keypoint location and appearance interactively.
Revised: At inference time, the latent keypoints allow one to author the keypoint location and appearance interactively.

Original: Moreover, generated keypoint-image pairs can be used to train an independent keypoint detector E , which enables unsupervised keypoint localization.
Revised: Moreover, generated keypoint-image pairs can be used to train an independent keypoint detector E , which enables unsupervised keypoint localization.

Original: 3. K EYPOINT G ENERATOR
Revised: 3. K EYPOINT G ENERATOR

Original: The keypoint generator, K , learns the embeddings and spatial arrangement of image parts, such as eyes, nose, and mouth for describing a portrait image.
Revised: The keypoint generator, K , learns the embeddings and spatial arrangement of image parts, such as eyes, nose, and mouth for describing a portrait image.

Original: It takes three Gaussian noise vectors as input z kp pose , z kp app , z bg emb ∼ N ( 0 D noise , I D noise × D noise ) , where D noise is the dimension.
Revised: It takes three Gaussian noise vectors as input z kp pose , z kp app , z bg emb ∼ N ( 0 D noise , I D noise × D noise ) , where D noise is the dimension.

Original: Each vector is passed through a three-layer MLP to respectively generates the K keypoint coordinates k j ∈ [ − 1 , 1] 2 , j = 1 , ..., K , a global style vector w global ∈ R D embed , and a background embedding w bg . 
Revised: Each vector is passed through a three-layer MLP to respectively generate the K keypoint coordinates k j ∈ [ − 1 , 1] 2 , j = 1 , ..., K , a global style vector w global ∈ R D embed , and a background embedding w bg . Here K is a pre-deﬁned hyperparameter.

Original: Special is our keypoint embedding w j that combines local and global factors,
Revised: Special is our keypoint embedding w j that combines local and global factors,

Original: [Equation]
Revised: [Equation]

Original: with the elementwise product ⊗ .
Revised: with the elementwise product ⊗ .

Original: The constant embedding w j const ∈ R D embed is designed to encode the keypoint semantics, e.g., left or right eye.
Revised: The constant embedding w j const ∈ R D embed is designed to encode the keypoint semantics, e.g., left or right eye.

Original: They are updated during the training but ﬁxed during testing.
Revised: They are updated during the training but ﬁxed during testing.

Original: The global style vector w global ∈ R D embed can be regarded as a noise vector with learned distribution; it ensures that a different appearance is drawn for every generated image.
Revised: The global style vector w global ∈ R D embed can be regarded as a noise vector with learned distribution; it ensures that a different appearance is drawn for every generated image.

Original: We show in the evaluation that this factorization into dependent and independent factors is crucial and improves on straightforward variants that directly use an MLP.
Revised: We show in Appendix F that this factorization into dependent and independent factors is crucial and improves on straightforward variants that directly use an MLP.

Original: For instance, it facilitates learning the correlation of the two eyes of a face while still covering the wide variety of eye variations.
Revised: For instance, it facilitates learning the correlation of the two eyes of a face while still covering the wide variety of eye variations.

Original: 3. S PATIAL E MBEDDING L AYER
Revised: 3. S PATIAL E MBEDDING L AYER

Original: With keypoint coordinates and embeddings generated, we now turn these point-wise estimates into discrete style maps that are further processed by the convolutional generator.
Revised: With keypoint coordinates and embeddings generated, we now turn these point-wise estimates into discrete style maps that are further processed by the convolutional generator.

Original: The style map S j ∈ R D embed × H × W for each keypoint j is generated by broadcasting keypoint embedding w j ∈ R D embed at each pixel of the heatmap H j ∈ R H × W and rescaling w j by the heatmap value H j ( p ) , 
Revised: The style map S j ∈ R D embed × H × W for each keypoint j is generated by broadcasting keypoint embedding w j ∈ R D embed at each pixel of the heatmap H j ∈ R H × W and rescaling w j by the heatmap value H j ( p ) , S j ( p [Equation]

Original: S j ( p ) =
Revised: p [Equation]

Original: H j ( p ) w j , where H j ( p ) = exp (cid:0) −(cid:107) p − k j (cid:107) [-
-]
Revised: w j , where H j ( p ) = exp (cid:0) −(cid:107) p − k j (cid:107) [-
-]22 / [Equation]

Original: /τ (cid:1) (2) has Gaussian shape, is centered at the keypoint location k j , and τ controls the inﬂuence range.
Revised: (cid:1) [Equation] has Gaussian shape, is centered at the keypoint location k j , and τ controls the inﬂuence range.

Original: We also deﬁne a heatmap H bg for the background as the negative of all keypoint maps, H bg ( p )
Revised: We also deﬁne a heatmap H bg for the background as the negative of all keypoint maps, H bg ( p )

Original: = 1 − max j { H j ( p ) } Kj =1 .
Revised: = 1 − max j { H j ( p ) } Kj =1 .

Original: The background heatmap is multiplied with the independent noise vector w bg generated from z bg emb instead of keypoint embedding, but treated equally otherwise.
Revised: The background heatmap is multiplied with the independent noise vector w bg generated from z bg emb instead of keypoint embedding, but treated equally otherwise.

Original: Then we concatenate all K keypoint style maps { S j } Kj =

Revised: Then we concatenate all K keypoint style maps { S j } Kj =

Original: and the background style map S bg in the channel dimension to obtain the style map S ∈ R D embed × H × W × K +1 .
Revised: and the background style map S bg in the channel dimension to obtain the style map S ∈ R D embed × H × W × K +1 .

Original: How these style maps are used as conditional variables at different levels of the image generator is explained in the next section.
Revised: How these style maps are used as conditional variables at different levels of the image generator is explained in the next section.

Original: 3. I MAGE G ENERATOR
Revised: 3. I MAGE G ENERATOR

Original: The image generator G follows the progressively growing architecture of StyleGAN (Karras et al., 2019), combined with the idea of spatial normalization from SPADE (Park et al., 2019), which was designed to generate images conditioned on segmentation masks.
Revised: The image generator G follows the progressively growing architecture of StyleGAN (Karras et al., 2019), combined with the idea of spatial normalization from SPADE (Park et al., 2019), which was designed to generate images conditioned on segmentation masks.

Original: The supplemental document provides additional details.
Revised: Please refer to Appendix E for additional details.

Original: In the following, we explain how we replace the role of manually annotated segmentation masks in the original with learned keypoints.
Revised: In the following, we explain how we replace the role of manually annotated segmentation masks in the original with learned keypoints.

Original: Our generator starts from a learned 4 × 4 × 512 constant matrix and keeps applying convolutions and upsampling to obtain feature maps of increasing resolution.
Revised: Our generator starts from a learned 4 × 4 × 512 constant matrix and keeps applying convolutions and upsampling to obtain feature maps of increasing resolution.

Original: Following SPADE (Park et al., 2019), the original BatchNorm (Ioffe & Szegedy, 2015) layers are replaced with spatial adaptive normalization to control the content.
Revised: Following SPADE (Park et al., 2019), the original BatchNorm (Ioffe & Szegedy, 2015) layers are replaced with spatial adaptive normalization to control the content.

Original: By contrast to SPADE, we do not condition on annotated segmentation masks but instead on the learned feature maps introduced in Section 3.1, and we do not use the residual links (He et al., 2016) because we found it detrimental in combination with progressive training.
Revised: By contrast to SPADE, we do not condition on annotated segmentation masks but instead on the learned feature maps introduced in Section 3.1, and we do not use the residual links (He et al., 2016) because we found it detrimental in combination with progressive training.

Original: The generator increases resolution layer-by-layer with multiple adaptive normalization layers, requiring differently-sized feature maps.
Revised: The generator increases resolution layer-by-layer with multiple adaptive normalization layers, requiring differently-sized feature maps.

Original: To create feature maps that match the respective spatial resolution of the generator, we apply Equation 2 multiple times.
Revised: To create feature maps that match the respective spatial resolution of the generator, we apply Equation 2 multiple times.

Original: 3. L OSS F UNCTIONS
Revised: 3. L OSS F UNCTIONS

Original: Adversarial losses.
Revised: Adversarial losses.

Original: We found it crucial for LatentKeypointGAN to use the non-saturating loss (Goodfellow et al., 2014),
Revised: We found it crucial for LatentKeypointGAN to use the non-saturating loss (Goodfellow et al., 2014),

Original: [Equation]
Revised: [Equation]

Original:  for the generator, and logistic loss,
Revised: (3) for the generator, and logistic loss,

Original: [Equation]
Revised: [Equation]

Original:  for the discriminator, with gradient penalty (Mescheder et al., 2018) applied only on real data,
Revised: 4) for the discriminator, with gradient penalty (Mescheder et al., 2018) applied only on real data,

Original: [Equation]
Revised: [Equation]

Original: Other losses such as hinge loss Park et al.
Revised: Other losses such as hinge loss Park et al.

Original: (2019) failed.
Revised: (2019) failed.

Original: Background loss.
Revised: Background loss.

Original: To further disentangle the background and the keypoints, and stabilize the keypoint location, we introduce a background penalty,
Revised: To further disentangle the background and the keypoints, and stabilize the keypoint location, we introduce a background penalty,

Original: [Equation]
Revised: [Equation]

Original: where z 1 and z 2 share the same keypoint location and appearance input noise, and only differ at the background noise input.
Revised: where z 1 and z 2 share the same keypoint location and appearance input noise, and only differ at the background noise input.

Original: The H 1 and H 2 are the background heatmaps generated by z 1 and z 2 .
Revised: The H 1 and H 2 are the background heatmaps generated by z 1 and z 2 .

Original: With this penalty, we expect the keypoint location and appearance do not change with the background.
Revised: With this penalty, we expect the keypoint location and appearance do not change with the background.

Original: The ﬁnal loss for the discriminator and the generator are, respectively,
Revised: The ﬁnal loss for the discriminator and the generator are, respectively,

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the improved quality and editing operations that our unsupervised LatentKeypointGAN approach for learning disentangled representations brings about.
Revised: We evaluate the improved quality and editing operations that our unsupervised LatentKeypointGAN approach for learning disentangled representations brings about.

Original: We show the types of edits in
Revised: We show the types of edits in

Original: Section 4.3 and quantitatively and qualitatively evaluate our improved editing quality compared with existing unsupervised and supervised methods in Section 4.4 and appendix, respectively.
Revised: Section 4.3 and quantitatively and qualitatively evaluate our improved editing quality compared with existing unsupervised and supervised methods in Section 4.4 and appendix, respectively.

Original: To test the keypoint consistency, that keypoints stick to a particular object part, we compare to existing autoencoder frameworks for unsupervised landmark detection in Section 4.7.
Revised: To test the keypoint consistency, that keypoints stick to a particular object part, we compare to existing autoencoder frameworks for unsupervised landmark detection in Section 4.7.

Original: Even though not our core focus, we show that by further tailoring the proposed architecture for precise localization, this new GAN method reaches state-of-the-art accuracy on a keypoint detection benchmark.
Revised: Even though not our core focus, we show that by further tailoring the proposed architecture for precise localization, this new GAN method reaches state-of-the-art accuracy on a keypoint detection benchmark.

Original: Since our main goal of enhanced editing quality is hard to quantify, we also conducted a user study to further validate these improvements.
Revised: Since our main goal of enhanced editing quality is hard to quantify, we also conducted a user study to further validate these improvements.

Original: The appendix contains exhaustive ablation tests on network architecture, hyperparameters, and training strategies to quantify the contributions of each individual component of our approach.
Revised: The appendix contains exhaustive ablation tests on network architecture, hyperparameters, and training strategies to quantify the contributions of each individual component of our approach.

Original: The supplemental material contains additional video results.
Revised: The supplemental material contains additional video results.

Original: Hyperparameters. We use 512 × 512 images for face editing, 256 × 256 for FID calculation on face datasets, and human pose experiments, and 128 × 128 for experiments using the bedrooms dataset.
Revised: M o v e e y e s

Original: Unless speciﬁed otherwise, we set τ = 0 .
Revised: E d i t h a i r l o c a t i o n S c a li n g

Original: 01 and use 10 keypoints.
Revised: K e y p

Original: For editing experiments on FFHQ and detection experiments on CelebA, we augment by randomly cropping the training images to a size of 70-100%.
Revised: o i n t e x c h a n g e

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: CelebA (Liu et al., 2015) contains 200k celebrity faces.
Revised: CelebA (Liu et al., 2015) contains 200k celebrity faces.

Original: We use this dataset to test our model’s ability to discover the keypoints unsupervised.
Revised: We use this dataset to test our model’s ability to discover the keypoints unsupervised.

Original: Following Thewlis et al.
Revised: Following Thewlis et al.

Original: , we use three subsets of this dataset: CelebA training set without MAFL (160k images), MAFL training set (19k images),
Revised: , we use three subsets of this dataset: CelebA training set without MAFL (160k images), MAFL training set (19k images),

Original: MAFL test set (1k images).
Revised: MAFL test set (1k images).

Original:  More details can be found in Section 4.7. 
Revised: MAFL (Zhang et al., 2014) is a subset of CelebA. More details can be found in Section 4.7. CelebA-HQ (Liu et al., 2015) contains 30k celebrity face images selected from CelebA. Following Zhu et al.

Revised: (2020b), we use the ﬁrst 28k images as training set and the rest 2k images as testing set.

Original: FlickrFaces-HQ (FFHQ) (Karras et al., 2019) consists of 70k high-quality portrait images, with more variation than CelebA (Liu et al., 2015).
Revised: FlickrFaces-HQ (FFHQ) (Karras et al., 2019) consists of 70k high-quality portrait images, with more variation than CelebA (Liu et al., 2015).

Original: Therefore, we use this dataset to test our model’s ability to disentangle the local representations of images.
Revised: Therefore, we use this dataset to test our model’s ability to disentangle the local representations of images.

Original: BBC Pose (Charles et al., 2013) consists of 20 videos of different sign-language signers with various backgrounds.
Revised: BBC Pose (Charles et al., 2013) consists of 20 videos of different sign-language signers with various backgrounds.

Original: We use this dataset to test our model’s ability to edit human appearance.M o v e e y e s E d i t h a i r l o c a t i o n S c a li n g
Revised: We use this dataset to test our model’s ability to edit human appearance.

Original: K e y p o i n t e x c h a n g e LSUN bedroom (Yu et al., 2015) consists of more than 3 million images of bedrooms, with very diverse spatial structure.
Revised: LSUN bedroom (Yu et al., 2015) consists of more than 3 million images of bedrooms, with very diverse spatial structure.

Original: We use this dataset to test our model’s generalization ability to edit entire indoor scenes.
Revised: We use this dataset to test our model’s generalization ability to edit entire indoor scenes.

Revised: Hyperparameters.

Revised: We use 512 × 512 images for face editing on FFHQ, 256 × 256 for FID calculation, comparison with SEAN (Zhu et al., 2020b) on CelebA-HQ, and human pose experiments on BBCPose, and 128 × 128 for experiments on LSUN Bedroom, and comparison with unsupervised keypoint based methods on CelebA. Unless speciﬁed otherwise, we set τ = 0 . 01 and usekeypoints.

Revised: For editing experiments on FFHQ and detection experiments on CelebA, we augment by randomly cropping the training images to a size of 70-100%.

Original: 4. I NTERACTIVE E DITING
Revised: 4. I NTERACTIVE E DITING

Revised: We show the capabilities of LatentKeypointGAN by changing the keypoint locations and exchanging the keypoint embeddings between different images.

Revised: As shown in Figure 3,

Revised: we can thereby

Original: [Figure]
Revised: [Figure]

Original: The background is changed while the faces are ﬁxed. We show the capabilities of LatentKeypointGAN by changing the keypoint locations and exchanging the keypoint embeddings between different images.
Revised: O r i g i n a l M o d i f i e d

Original: As shown in Figure 3, we can thereby edit the face direction, face size, and individual key parts by changing the keypoint locations.
Revised: The background is changed while the faces are ﬁxed. edit the face direction, face size, and individual key parts by changing the keypoint locations.

Original: If only a subset of the keypoint embeddings is changed, the other parts are not signiﬁcantly affected.
Revised: If only a subset of the keypoint embeddings is changed, the other parts are not signiﬁcantly affected.

Original: Figure 5 shows a heatmap of the area of interest.
Revised: Figure 5 shows a heatmap of the area of interest.

Original: Since the GAN learns to generate a coherent face from its parts, global effects remain mostly unchanged, for instance, hairstyle and color.
Revised: Since the GAN learns to generate a coherent face from its parts, global effects remain mostly unchanged, for instance, hairstyle and color.

Original: We discuss in Section 5 to which degree this is desired and what limitations remain.
Revised: We discuss in Section 5 to which degree this is desired and what limitations remain.

Original: Additional ablation studies, editing, and interpolation examples are shown in the appendix document and  supplemental video.
Revised: Additional ablation studies, editing, and interpolation examples are shown in Appendix C F and the supplemental video.

Original: 4. D ISENTANGLED R EPRESENTATIONS
Revised: 4. D ISENTANGLED R EPRESENTATIONS

Original: We ﬁrst analyze how background and keypoint embeddings are disentangled and used for editing portrait images.
Revised: We ﬁrst analyze how background and keypoint embeddings are disentangled and used for editing portrait images.

Original: Further examples are analyzed in the user study.
Revised: Further examples are analyzed in the user study.

Original: Disentangled keypoint embeddings.
Revised: Disentangled keypoint embeddings.

Original: Figure 5 shows editing operations of independent facial regions.
Revised: Figure 5 shows editing operations of independent facial regions.

Original: We ﬁx the background noise, z bg emb , and change some of the keypoint embeddings.
Revised: We ﬁx the background noise, z bg emb , and change some of the keypoint embeddings.

Original: This enables exchanging of eyes, mouth, or nose between persons.
Revised: This enables exchanging of eyes, mouth, or nose between persons.

Original: Figure 5 includes heatmaps that visualize the difference between original and interpolated images.
Revised: Figure 5 includes heatmaps that visualize the difference between original and interpolated images.

Original: Their local activation highlights the spatial disentanglement of individual keypoint features.
Revised: Their local activation highlights the spatial disentanglement of individual keypoint features.

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: Disentangled background.
Revised: Disentangled background.

Original: Figure 4 shows a faithful change of backgrounds while keeping the face ﬁxed.
Revised: Figure 4 shows a faithful change of backgrounds while keeping the face ﬁxed.

Original: To this end, we ﬁx the keypoint noise z kp pose , z kp app , and change only the background noise input, z bg emb .
Revised: To this end, we ﬁx the keypoint noise z kp pose , z kp app , and change only the background noise input, z bg emb .

Original: The local change in the three diverse examples shows that the background and keypoint encodings are disentangled well.
Revised: The local change in the three diverse examples shows that the background and keypoint encodings are disentangled well.

Original: The illumination and hair color is learned to be part of the background, which makes sense as a global feature cannot be attributed to individual keypoints.
Revised: The illumination and hair color is learned to be part of the background, which makes sense as a global feature cannot be attributed to individual keypoints.

Original: 4. I MAGE Q UALITY
Revised: 4. I MAGE Q UALITY

Original: We test our image quality on CelebA-HQ and compare it with an unsupervised autoencoder and segmentation mask-conditioned GANs, which require labeled training images.
Revised: We test our image quality on CelebA-HQ and compare it with an unsupervised autoencoder and segmentation mask-conditioned GANs, which require labeled training images.

Original: This comparison is biased in that the mask-conditioned GANs use the masks from the test set as input, while ours is unconditional.
Revised: This comparison is biased in that the mask-conditioned GANs use the masks from the test set as input, while ours is unconditional.

Original: To make an as fair as possible comparison, we add a version that uses a selfsupervised keypoint detector (see Sec. 4.7) to condition the latent keypoint locations on the same test images as the baselines do.
Revised: To make an as fair as possible comparison, we add a version that uses a selfsupervised keypoint detector (see Sec. 4.7) to condition the latent keypoint locations on the same test images as the baselines do.

Original: Scores are computed by using the Pytorch FID calculator (Seitzer, 2020).
Revised: Scores are computed by using the Pytorch FID calculator (Seitzer, 2020).

Original: We followed Zhu et al.
Revised: We followed Zhu et al.

Original: (2020b) to use the ﬁrst 28k images as the training set and the last 2k images as the test set.
Revised: (2020b) to use the ﬁrst 28k images as the training set and the last 2k images as the test set.

Original: We list the results in Table 1.
Revised: We list the results in Table 1.

Original: Our approach attains better FID scores than those GANs conditioning on segmentation masks.
Revised: Our approach attains better FID scores than those GANs conditioning on segmentation masks.

Original: It gets close to SEAN (Zhu et al., 2020b),  which has an auto-encoder architecture and their reported FID is on reconstructed images instead of randomly generated appearances.
Revised: It gets close to SEAN (Zhu et al., 2020b), [Table] [Table] which has an auto-encoder architecture and their reported FID is on reconstructed images instead of randomly generated appearances.

Original: In addition, the user study shows that on editing tasks, our method also exceeds the image editing quality of SEAN signiﬁcantly (preferred by 94.78% of all responses).
Revised: In addition, the user study shows that on editing tasks, our method also exceeds the image editing quality of SEAN signiﬁcantly (preferred by 94.78% of all responses).

Original: 4. G ENERALIZATION TO O THER D ATASETS
Revised: 4. G ENERALIZATION TO O THER D ATASETS

Original: LSUN Bedroom and BBC Pose.
Revised: LSUN Bedroom and BBC Pose.

Original: In Figure 6, we explore the editing ability of entire scenes on the LSUN bedroom dataset.
Revised: In Figure 6, we explore the editing ability of entire scenes on the LSUN bedroom dataset.

Original: No previous unsupervised part-based model has tried this difﬁcult task before.
Revised: No previous unsupervised part-based model has tried this difﬁcult task before.

Original: We successfully interpolate the local appearance by changing the corresponding keypoint embeddings and translating the local key parts (window, bed) by moving the corresponding keypoints.
Revised: We successfully interpolate the local appearance by changing the corresponding keypoint embeddings and translating the local key parts (window, bed) by moving the corresponding keypoints.

Original: We can also remove or add some parts by removing or adding peaks in the Gaussian
Revised: We can also remove or add some parts by removing or adding peaks in the Gaussian

Original: Heatmaps.
Revised: Heatmaps.

Original: Since our learned representation is two-dimensional, it is not possible to rotate objects entirely.
Revised: Since our learned representation is two-dimensional, it is not possible to rotate objects entirely.

Original: Figure 7 explores the editing of individuals.
Revised: Figure 7 explores the editing of individuals.

Original: Although artifacts remain due to the detailed background and motion blur in the datasets, pose and appearance can be exchanged separately.[Figure] Appearance Pose
Revised: Although artifacts remain due to the detailed background and motion blur in the datasets, pose and appearance can be exchanged separately.

Original: Background

Original: 4. A BLATION T ESTS
Revised: 4. A BLATION T ESTS

Original: Method

Original: L 1 error % ↓

Original: FID ↓ full model 5.85% 23.

Original: w/

Original: [Table]

Original: background 6.43% 25.

Original: w/o global style vector 6.76% 28.

Original: adding global style vector 5.29% 42.

Original: contrastive keypoint embedding 7.53% 28.

Original: w/o keypoint embedding 22.81% 32.

Original: w/o keypoint34.

Original: We test different variants of our architecture and demonstrate the effectiveness and necessity of our design.
Revised: We test different variants of our architecture and demonstrate the effectiveness and necessity of our design.

Original: We test 1) removing the background; 2) removing the global style vector; 3) using additive global style vector instead of multiplicative ones; 4) using contrastive learned keypoint embeddings instead of multiplicative ones; 5) removing the keypoint embedding; 6) removing keypoints
Revised: We test 1) removing the background; 2) removing the global style vector; 3) using additive global style vector instead of multiplicative ones; 4) using contrastive learned keypoint embeddings instead of multiplicative ones; 5) removing the keypoint embedding; 6) removing keypoints.

Original: We show the detection result on MAFL and FID score on FFHQ in Table 2.
Revised: We show the detection result on MAFL and FID score on FFHQ in Table 2.

Original: All contributions are crucial for the success and consistent for both detection and image quality tasks.
Revised: All contributions are crucial for the success and consistent for both detection and image quality tasks.

Original: The keypoint detection evaluation is explained in the subsequent Section 4.7.
Revised: The keypoint detection evaluation is explained in the subsequent Section 4.7.

Original: For simplicity, the FID is calculated at resolution 256 × 256 , between 50k generated images and the original dataset; slightly different from previous experiments.
Revised: For simplicity, the FID is calculated at resolution 256 × 256 , between 50k generated images and the original dataset; slightly different from previous experiments.

Original: 4. U NSUPERVISED K EYPOINTS D ISCOVERY
Revised: 4. U NSUPERVISED K EYPOINTS D ISCOVERY

Original:  We demonstrated that our keypoints are semantically meaningful by editing the image via embeddings.
Revised: [Figure] [Table] [Table] We demonstrated that our keypoints are semantically meaningful by editing the image via embeddings.

Original: In this part, we further demonstrate the consistency of our keypoints on the task of unsupervised keypoint detection as introduced by Thewlis et al.
Revised: In this part, we further demonstrate the consistency of our keypoints on the task of unsupervised keypoint detection as introduced by Thewlis et al.

Original: We test whether keypoints consistently align to the same image parts, without randomly moving on the image when other factors such as color or texture are changed.
Revised: We test whether keypoints consistently align to the same image parts, without randomly moving on the image when other factors such as color or texture are changed.

Original: To this end, we train a standard ResNet detector (Xiao et al., 2018) supervised on 200,000 image-keypoint pairs generated on-the-ﬂy by our LatentKeypointGAN.
Revised: To this end, we train a standard ResNet detector (Xiao et al., 2018) supervised on 200,000 image-keypoint pairs generated on-the-ﬂy by our LatentKeypointGAN.

Original: Note, the LatentKeypointGAN is trained on the same number of training images as the other detectors, hence, results are comparable.
Revised: Note, the LatentKeypointGAN is trained on the same number of training images as the other detectors, hence, results are comparable.

Original: For evaluation, we follow Thewlis et al.
Revised: For evaluation, we follow Thewlis et al.

Original: (2017) and subsequent methods: As the order and semantics of unsupervised keypoints are undeﬁned, a linear regressor from the predicted keypoints to the 5 ground truth keypoints is learned on the MAFL training set.
Revised: (2017) and subsequent methods: As the order and semantics of unsupervised keypoints are undeﬁned, a linear regressor from the predicted keypoints to the 5 ground truth keypoints is learned on the MAFL training set.

Original: All images are resized to 128 × 128 and the training and test set of the MAFL subset from CelebA are excluded when training the LatentKeypointGAN.
Revised: All images are resized to 128 × 128 and the training and test set of the MAFL subset from CelebA are excluded when training the LatentKeypointGAN.

Original: The test error is the Euclidean distance to the ground truth on the MAFL test set, normalized by the inter-ocular distance in percent.
Revised: The test error is the Euclidean distance to the ground truth on the MAFL test set, normalized by the inter-ocular distance in percent.

Original: High accuracy can therefore only be obtained when estimated keypoints move consistently with the human-annotated ones.
Revised: High accuracy can therefore only be obtained when estimated keypoints move consistently with the human-annotated ones.

Original: LatentKeypointGAN strikes a low error of 5.9%, which lies between the original 8.0% by Thewlis et al.
Revised: LatentKeypointGAN strikes a low error of 5.9%, which lies between the original 8.0% by Thewlis et al.

Original: (2017) and the most recent 2.8% by Dundar et al.
Revised: (2017) and the most recent 2.8% by Dundar et al.

Original: We further tuned our architecture for precise localization by enforcing stronger translation equivariance of the keypoints (see appendix B for details).
Revised: We further tuned our architecture for precise localization by enforcing stronger translation equivariance of the keypoints (see appendix B for details).

Original: This improved variant, LatentKeypointGAN-tuned, reaches 3.3%, which gets close to Dunbar et al.
Revised: This improved variant, LatentKeypointGAN-tuned, reaches 3.3%, which gets close to Dunbar et al.

Original: On the in-the-wild variant of CelebA (more diverse head orientation, position, and scale), which is signiﬁcantly more difﬁcult, we reach an error of 5 . 6% .
Revised: On the in-the-wild variant of CelebA (more diverse head orientation, position, and scale), which is signiﬁcantly more difﬁcult, we reach an error of 5 . 6% .

Original: Here, we outperform all existing methods reporting on this metric (8.7% by Jakab et al.
Revised: Here, we outperform all existing methods reporting on this metric (8.7% by Jakab et al.

Original: (2018) and 11.4% by Lorenz et al.
Revised: (2018) and 11.4% by Lorenz et al.

Original: We use the same hyperparameters as before, showing the strong robustness to new settings.
Revised: We use the same hyperparameters as before, showing the strong robustness to new settings.

Original: Thereby, although not our main goal, LatentKeypointGAN also provides an alternative methodology for unsupervised keypoint/landmark detection that avoids paired training and TPS deformation, which are notoriously difﬁcult to tune.
Revised: Thereby, although not our main goal, LatentKeypointGAN also provides an alternative methodology for unsupervised keypoint/landmark detection that avoids paired training and TPS deformation, which are notoriously difﬁcult to tune.

Original: 4. U SER S TUDY ON E DITING Q UALITY
Revised: 4. U SER S TUDY ON E DITING Q UALITY

Original: We conducted a user study to separately access image quality after editing, comparing against the unsupervised keypoint-based method (Zhang et al., 2018) on CelebA (Liu et al., 2015) and the maskconditioned method (Zhu et al., 2020b) on CelebA-HQ (Liu et al., 2015).
Revised: We conducted a user study to separately access image quality after editing, comparing against the unsupervised keypoint-based method (Zhang et al., 2018) on CelebA (Liu et al., 2015) and the maskconditioned method (Zhu et al., 2020b) on CelebA-HQ (Liu et al., 2015).

Original: The study conﬁrms the main results: While Zhang et al.
Revised: The study conﬁrms the main results: While Zhang et al.

Original: strike a high keypoint accuracy, its image quality (ours preferred by 92.17%) and editing results (our disentanglement preferred by 67.83%) are unsatisfactory.
Revised: strike a high keypoint accuracy, its image quality (ours preferred by 92.17%) and editing results (our disentanglement preferred by 67.83%) are unsatisfactory.

Original: While
Revised: While

Original: Zhu et al.
Revised: Zhu et al.

Original: reach a better FID score before editing, after editing, ours is the most realistic (preferred by 94.78%).
Revised: reach a better FID score before editing, after editing, ours is the most realistic (preferred by 94.78%).

Original: Moreover, our disentanglement comes close to the one of Zhu et al.
Revised: Moreover, our disentanglement comes close to the one of Zhu et al.

Original: (49.57% voted equal quality).
Revised: (49.57% voted equal quality).

Original: Zhu et al. have better shape preservation because their masks provide silhouettes explicitly, but this likely also leads to the lower editing quality as masks and embeddings can become incompatible.
Revised: Zhu et al. have better shape preservation because their masks provide silhouettes explicitly, but this likely also leads to the lower editing quality as masks and embeddings can become incompatible.

Original: For instance, the hair shape and strain direction need to be correlated.
Revised: For instance, the hair shape and strain direction need to be correlated.

Original: The full details are in the appendix.
Revised: The full details are in the appendix.

Original:  L IMITATIONS AND F UTURE W ORK
Revised:  L IMITATIONS AND F UTURE W ORK

Original: For portrait images, the hair can mix with the background encoding, such that it is considered part of the background.
Revised: For portrait images, the hair can mix with the background encoding, such that it is considered part of the background.

Original: Still, the hair can be changed by selecting a background embedding with the desired hairstyle.
Revised: Still, the hair can be changed by selecting a background embedding with the desired hairstyle.

Original: Moreover, the disentanglement into locally encoded features can lead to asymmetric faces, such as a pair of glasses with differently styled sides.
Revised: Moreover, the disentanglement into locally encoded features can lead to asymmetric faces, such as a pair of glasses with differently styled sides.

Original: For BBC Pose, the keypoints are not well localized.
Revised: For BBC Pose, the keypoints are not well localized.

Original: They are consistent across images with the same pose, which permits pose transfer, but are too inconsistent for keypoint detection.
Revised: They are consistent across images with the same pose, which permits pose transfer, but are too inconsistent for keypoint detection.

Original: Limitations could be overcome by linking keypoints hierarchically with a skeleton and by reasoning about masks on top of keypoints.
Revised: Limitations could be overcome by linking keypoints hierarchically with a skeleton and by reasoning about masks on top of keypoints.

Original: While the face orientation in portrait images can be controlled, we found that orientation changes on the bedroom images are not reliable.
Revised: While the face orientation in portrait images can be controlled, we found that orientation changes on the bedroom images are not reliable.

Original: The orientation is mostly baked into the appearance encoding.
Revised: The orientation is mostly baked into the appearance encoding.

Original: We believe that it will be necessary to learn a 3D representation and datasets with less-biased viewpoints.
Revised: We believe that it will be necessary to learn a 3D representation and datasets with less-biased viewpoints.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a GAN-based framework that is internally conditioned on keypoints and their appearance encoding, providing an interpretable hidden space that enables intuitive editing.
Revised: We present a GAN-based framework that is internally conditioned on keypoints and their appearance encoding, providing an interpretable hidden space that enables intuitive editing.

Original: We provide the idea and implementation to avoid issues with TPS and related deformation models that are common in existing autoencoders.
Revised: We provide the idea and implementation to avoid issues with TPS and related deformation models that are common in existing autoencoders.

Original: This LatentKeypointGAN also facilitates the generation of image-keypoint pairs, thereby providing a new methodology for unsupervised keypoint detection.
Revised: This LatentKeypointGAN also facilitates the generation of image-keypoint pairs, thereby providing a new methodology for unsupervised keypoint detection.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: This research provides a new unsupervised disentanglement and editing method.
Revised: This research provides a new unsupervised disentanglement and editing method.

Original: By contrast to existing supervised ones, e.g., those requiring manually annotated masks, ours can be trained on any image collection.
Revised: By contrast to existing supervised ones, e.g., those requiring manually annotated masks, ours can be trained on any image collection.

Original: This enables training on very diverse sets as well as on personalized models for a particular population and can thereby counter biases in the annotated datasets.
Revised: This enables training on very diverse sets as well as on personalized models for a particular population and can thereby counter biases in the annotated datasets.

Original: Photo-realistic editing tools have the risk of abuse via deep fakes .
Revised: Photo-realistic editing tools have the risk of abuse via deep fakes .

Original: A picture of a real person could be altered to express something not intended by the person.
Revised: A picture of a real person could be altered to express something not intended by the person.

Original: In that regard, our method has the advantage of only enabling the editing of generated images; it does not enable modifying real images; it only works in the synthetic to real direction.
Revised: In that regard, our method has the advantage of only enabling the editing of generated images; it does not enable modifying real images; it only works in the synthetic to real direction.

Original: However, future work could extend it with an appearance encoder, which bears some risk.
Revised: However, future work could extend it with an appearance encoder, which bears some risk.

Original: Another risk is that the network could memorize appearances from the training set and therefore re-produce unwanted deformations of the pictured subjects.
Revised: Another risk is that the network could memorize appearances from the training set and therefore reproduce unwanted deformations of the pictured subjects.

Original: Due to the nature of GANs, where the discriminator and generator learn how faces look but are notdesigned to memorize entire faces, our method has a low risk ofleaking such identity information unless the training set is very small.
Revised: While Brock et al. (2018) and Karras et al. (2018) argue that GANs do not memorize training datasets, recently Feng et al. (2021) empirically proved that whether GANs memorize training datasets depends on the complexity of the training datasets.

Original: However, we noted that for the BBCpose dataset, which only contains a handful of subjects, the source appearance is learned.
Revised: Therefore, our model has some risk of leaking such identity information if the training set is very small or the number of persons involved is limited, such as BBCPose.

Original: To mitigate the risk, we only use established and publicly available datasets, in particular those that collect pictures of celebrities or other public ﬁgures and also those not containing any person (bedroom dataset).
Revised: To mitigate the risk, we only use established and publicly available datasets, in particular those that collect pictures of celebrities or other public ﬁgures and also those not containing any person (bedroom dataset).

Original: Our IRB approved the survey and we collected explicit and informed consent at the beginning of the study.
Revised: Our IRB approved the survey and we collected explicit and informed consent at the beginning of the study.

Original: Results are stored on secure servers.
Revised: Results are stored on secure servers.

Original: This research did not cause any harm to any subject.
Revised: This research did not cause any harm to any subject.

Original: On the technical side, we disclose all limitations in both the main paper and the supplementary.
Revised: On the technical side, we disclose all limitations in both the main paper and the supplementary.

Original: This research does not have conﬂicts with others.
Revised: This research does not have conﬂicts with others.

Original: We credit the previous works in the Introduction (Section 1) and Related Work (Section 2).
Revised: We credit the previous works in the Introduction (Section 1) and Related Work (Section 2).

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: We not only described our algorithm in Section 3 but also provide detailed information of our architecture, training strategy, and hyperparameters in the supplementary E. All the datasets we used are publicly available and
Revised: We not only described our algorithm in Section 3 but also provide detailed information of our architecture, training strategy, and hyperparameters in the supplementary E. All the datasets we used are publicly available and

Original: the pre-processing is also described in Section 4.
Revised: the pre-processing is also described in Section 4.

Original: Furthermore, we provide the source code to facilitate future work in this direction of GAN-based generative image editing
Revised: Furthermore, we provide the source code to facilitate future work in this direction of GAN-based generative image editing

Original: A K EYPOINTS I LLUSTRATION
Revised: A K EYPOINTS I LLUSTRATION

Original: In Figure 8, we show the keypoints generated on all datasets.
Revised: Figure 8 shows the keypoints generated on all datasets.

Original: We can see that the keypoints are semantically meaningful and very consistent across different instances.
Revised: The keypoints are semantically meaningful and very consistent across different instances.

Original: [Figure]
Revised: [Figure]

Original: B T UNING L ATENT K EYPOINT GAN FOR K EYPOINTS D ETECTION
Revised: B T UNING L ATENT K EYPOINT GAN FOR K EYPOINTS D ETECTION

Original: We desire an architecture that encodes the location of parts solely in the keypoint locations to improve keypoint localization and the subsequent learning of a detector.
Revised: We desire an architecture that encodes the location of parts solely in the keypoint locations to improve keypoint localization and the subsequent learning of a detector.

Original: Eventhough the convolutional generator is translation invariant, additional care is necessary to prevent leakage of global position at the image boundary and from the starting tensor.
Revised: Even though the convolutional generator is translation invariant, additional care is necessary to prevent leakage of global position at the image boundary and from the starting tensor.

Original: All these tuning steps are explained below.
Revised: All these tuning steps are explained below.

Original: Padding Margins.
Revised: Padding Margins.

Original: As pointed out by Islam et al.
Revised: As pointed out by Islam et al.

Original: Alsallakh et al.
Revised: Alsallakh et al.

Original: ; Xu et al. (2020a);
Revised: ; Xu et al. (2020a);

Original: Kayhan & Gemert (2020), convolutions with zero padding are very good at implicitly encoding absolute grid position at deeper layers.
Revised: Kayhan & Gemert (2020), convolutions with zero padding are very good at implicitly encoding absolute grid position at deeper layers.

Original: To prevent this, we follow Karras et al.
Revised: To prevent this, we follow Karras et al.

Original: By maintaining a ﬁxed margin around the feature map and cropping after each upsampling, we effectively prevent the leaking of absolute position and the bias to the center because none of the generated pixels ever reaches a boundary condition.
Revised: By maintaining a ﬁxed margin around the feature map and cropping after each upsampling, we effectively prevent the leaking of absolute position and the bias to the center because none of the generated pixels ever reaches a boundary condition.

Original: We use a 10-pixel margin.
Revised: We use a 10-pixel margin.

Original: Note that such measures do not directly apply to autoencoders who are bound to the ﬁxed resolution of the input.
Revised: Note that such measures do not directly apply to autoencoders who are bound to the ﬁxed resolution of the input.

Original: Positional Encoded Starting Tensor.
Revised: Positional Encoded Starting Tensor.

Original: Wealso need to remove the 4 × 4 × 512 starting tensor because it can encode absolute grid position.
Revised: We remove the 4 × 4 × 512 starting tensor because it can encode absolute grid position.

Original: We replace it with the positional encoding M of difference between keypoints k 1 , ..., k K and the grid positions p ,
Revised: We replace it with the positional encoding M of difference between keypoints k 1 , ..., k K and the grid positions p ,

Original: [Equation]
Revised: [Equation]

Original: The underlying idea is to only encode relative distances to keypoints but not to the image boundary.
Revised: The underlying idea is to only encode relative distances to keypoints but not to the image boundary.

Original: Larger Starting Tensor.
Revised: Larger Starting Tensor.

Original: We found that starting the feature map from 32 × 32 instead of 4 ×improves keypoint localization accuracy.
Revised: We found that starting the feature map from 32 × 32 instead of 4 ×improves keypoint localization accuracy.

Original: Background Handling.
Revised: Background Handling.

Original: The background should be consistent across the whole image, but complete occlusion is inaccurate to model with the sparse keypoints and their limited support in the intermediate feature maps.
Revised: The background should be consistent across the whole image, but complete occlusion is inaccurate to model with the sparse keypoints and their limited support in the intermediate feature maps.

Original: Hence, we introduce the explicit notion of a foreground mask that blends in the background.
Revised: Hence, we introduce the explicit notion of a foreground mask that blends in the background.

Original: The mask is generated as one additional layer in the image generator.
Revised: The mask is generated as one additional layer in the image generator.

Original: To generate the background, we use a separate network that is of equivalent architecture to LatentKeypointGANtuned.
Revised: To generate the background, we use a separate network that is of equivalent architecture to LatentKeypointGANtuned.

Original: Note that in background generation we use AdaIN (Huang & Belongie, 2017) instead of
Revised: Note that in background generation we use AdaIN (Huang & Belongie, 2017) instead of

Original: SPADE because there is no spatial sensitive representations like keypoints.
Revised: SPADE because there is no spatial sensitive representation, such as keypoints.

Original: Foreground and background are then blended linearly based on the foreground mask.
Revised: Foreground and background are then blended linearly based on the foreground mask.

Original: We use an 1 × 1 convolution layer to generate the ﬁnal RGB images.
Revised: We use an 1 × 1 convolution layer to generate the ﬁnal RGB images.

Original: [Table]
Revised: [Table]

Original: Simpliﬁcation
Revised: Simpliﬁcation

Original: In addition, we remove the progressive training (Karras et al., 2018) and use the
Revised: In addition, we remove the progressive training (Karras et al., 2018) and use the

Original: ResBlocks as deﬁned by Park et al.
Revised: ResBlocks as deﬁned by Park et al.

Original: This step is for simpliﬁcation (quicker training results on medium-sized images) and does not provide  measurable change in keypoint accuracy.
Revised: This step is for simpliﬁcation (quicker training results on medium-sized images) and does not provide a measurable change in keypoint accuracy.

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: each step of improvement contributes signiﬁcantly to the keypoint accuracy and consistent improvement on prior works on the two in-the-wild settings.
Revised: each step of improvement contributes signiﬁcantly to the keypoint accuracy and consistent improvement on prior works on the two in-the-wild settings.

Original: Please see the main paper for the complete comparison.
Revised: Please see the main paper for the complete comparison.

Original: C I MAGE E DITING Q UALITY C OMPARISON
Revised: C I MAGE E DITING Q UALITY C OMPARISON

Original: Figure 9 validates the improved editing quality, showing comparative quality to conditional GAN (supervised by paired masks), and superior quality to unsupervised methods (greater detail in the hair and facial features).
Revised: Figure 9 validates the improved editing quality, showing comparative quality to conditional GAN (supervised by paired masks), and superior quality to unsupervised methods (greater detail in the hair and facial features).

Original: Note that for SEAN (Zhu et al., 2020b)
Revised: Note that for SEAN (Zhu et al., 2020b)

Original: we use the edited images (combine mask with different face appearance) instead of reconstructed images (combine mask with the corresponding face appearance) for fair comparison with other GAN methods.
Revised: we use the edited images (combine mask with different face appearance) instead of reconstructed images (combine mask with the corresponding face appearance) for fair comparison with other GAN methods.

Original: The qualitative improvement is further estimated in the subsequent user study.
Revised: The qualitative improvement is further estimated in the subsequent user study.

Original: The main paper reports FID scores for all those datasets where prior work does.
Revised: The main paper reports FID scores for all those datasets where prior work does.

Original: To aid future comparisons we also report the remaining FID scores: 17.88 on FFHQ,  30.53 on BBC Pose, and 18.89 on LSUN Bedroom.
Revised: To aid future comparisons we also report the remaining FID scores: 17.88 on FFHQ, 18.25 on CelebA, 30.53 on BBC Pose, and 18.89 on LSUN Bedroom.

Original: The FID is calculated by the generated 50k images and the resized original dataset.
Revised: The FID is calculated by the generated 50k images and the resized original dataset.

Revised: We also qualitatively compare our method with Lorenz et al.

Revised: (2019) on LSUN Bedroom.

Revised: As shown in Figure 10, their model generates trivial keypoints and fails to reconstruct the images.

Original: [Figure]
Revised: [Figure]

Original: D E DITING Q UALITY S URVEY
Revised: D E DITING Q UALITY S URVEY

Original: We designed comparisons in 4 different aspects to demonstrate our editing quality.
Revised: We designed comparisons in 4 different aspects to demonstrate our editing quality.

Original: We compare to two methods, one unsupervised keypoint-based method (Zhang et al., 2018) on CelebA (Liu et al., 
Revised: We compare to two methods, one unsupervised keypoint-based method (Zhang et al., 2018) on CelebA (Liu et al., 2015), and one mask-conditioned method (Zhu et al., 2020b) on CelebA-HQ (Liu et al., 2015). For both methods, we follow their experiment design in their papers to make a fair comparison. For each question, we ask the participants to chose from 3 options: 1) ours is better; 2) theirs is better; 3) both methods have equal quality.

Original: [Table]
Revised: The order of the pair to compare (ours in the ﬁrst or the second) in the questions is randomly generated by the function numpy.random.choice of the package Numpy in Python with random seed 1. The results are illustrated in Table 4. The entire study is in anonymous form in the supplemental document at survey/index.html.

Original: Study details.
Revised: Study details.

Original: We invited 23 participants answering 35 questions in total, with 5 for each of the categories above and an additional 5 comparing to randomly selected real images from the celebA training set.
Revised: We invited 23 participants answering 35 questions in total, with 5 for each of the categories above and an additional 5 comparing to randomly selected real images from the celebA training set.

Original: The generated images are selected as follows.
Revised: The generated images are selected as follows.

Original: We generate 32 images with LatentKeypointGAN.
Revised: We generate 32 images with LatentKeypointGAN.

Original: For the baselines we take the corresponding images from their paper (as implementations are not available).
Revised: For the baselines we take the corresponding images from their paper (as implementations are not available).

Original: We then ﬁnd our image (out of the 32) that best match the baseline images (in gender, pose, head size).
Revised: We then ﬁnd our image (out of the 32) that best match the baseline images (in gender, pose, head size).

Original: If multiple images are similar, we use the ﬁrst one.
Revised: If multiple images are similar, we use the ﬁrst one.

Original: Editing image quality.
Revised: Editing image quality.

Original: We edited the images by swapping the appearance embedding between different images.
Revised: We edited the images by swapping the appearance embedding between different images.

Original: In each question, we show one image of ours and one image of theirs.
Revised: In each question, we show one image of ours and one image of theirs.

Original: We ask the participants to compare the resulting image quality.
Revised: We ask the participants to compare the resulting image quality.

Original: Part disentanglement.
Revised: Part disentanglement.

Original: To compare with Zhang et al.
Revised: To compare with Zhang et al.

Original: (2018), we moved part of the keypoints, as they did in their paper.
Revised: (2018), we moved part of the keypoints, as they did in their paper.

Original: To compare with Zhu et al. (2020b), we exchange the part embedding between different images, as they did in their paper.
Revised: To compare with Zhu et al. (2020b), we exchange the part embedding between different images, as they did in their paper.

Original: In each question, we show one pair of images of ours and one pair of images of theirs.
Revised: In each question, we show one pair of images of ours and one pair of images of theirs.

Original: We ask the participants to choose the one with better spatial disentanglement regardless of the image quality.
Revised: We ask the participants to choose the one with better spatial disentanglement regardless of the image quality.

Original: Identity preservation while changing expression.
Revised: Identity preservation while changing expression.

Original: We compare identity preservation with Zhang et al.
Revised: We compare identity preservation with Zhang et al.

Original: Following their paper, we change part (or all) of the keypoints to change the expression of the face.
Revised: Following their paper, we change part (or all) of the keypoints to change the expression of the face.

Original: In each question, we show the two pairs of images.
Revised: In each question, we show the two pairs of images.

Original: Each pair contains two images, one original image, and one edited image.
Revised: Each pair contains two images, one original image, and one edited image.

Original: We ask the participants to choose the pair that can better preserve the identity of the face regardless of the image quality, as quality is assessed separately.
Revised: We ask the participants to choose the pair that can better preserve the identity of the face regardless of the image quality, as quality is assessed separately.

Original: Shape preservation while changing appearance.
Revised: Shape preservation while changing appearance.

Original: We compare the shape preservation with Zhu et al.
Revised: We compare the shape preservation with Zhu et al.

Original: (2020b).
Revised: (2020b).

Original: We edited the images by swapping the appearance embedding between different images.
Revised: We edited the images by swapping the appearance embedding between different images.

Original: In each question, we show the two triplets of images.
Revised: In each question, we show the two triplets of images.

Original: Each triplet contains three images, one shape source image and one appearance image, and one combined image.
Revised: Each triplet contains three images, one shape source image and one appearance image, and one combined image.

Original: We ask the participants to choose the triplet where the combined image has the more similar shape as the shape source image regardless of the image quality.
Revised: We ask the participants to choose the triplet where the combined image has the more similar shape as the shape source image regardless of the image quality.

Original: Interpretation - comparative.
Revised: Interpretation - comparative.

Original: This study conﬁrms the ﬁndings of the main paper.
Revised: This study conﬁrms the ﬁndings of the main paper.

Original: Our method outperforms Zhang et al.
Revised: Our method outperforms Zhang et al.

Original: in all metrics in Table 4.
Revised: in all metrics in Table 4.

Original: We also outperform SEAN in image editing quality.
Revised: We also outperform SEAN in image editing quality.

Original: This conﬁrms our claims of superior editing capability but may be surprising on the ﬁrst  glance since they attain a higher quality (better FID score) on unedited images.
Revised: This conﬁrms our claims of superior editing capability but may be surprising on the ﬁrst [Table] glance since they attain a higher quality (better FID score) on unedited images.

Original: However, it can be explained with the limited editing capabilities of the mask-based approaches discussed next.
Revised: However, it can be explained with the limited editing capabilities of the mask-based approaches discussed next.

Original: Participants give SEAN a higher shape preservation quality (47% in favour and 19% equal), which is expected since it conditions on pixel-accurate segmentation masks that explicitly encode the feature outline.
Revised: Participants give SEAN a higher shape preservation quality (47% in favour and 19% equal), which is expected since it conditions on pixel-accurate segmentation masks that explicitly encode the feature outline.

Original: However, the masks have the drawback that they dictate the part outline strictly, which leads to inconsistencies when exchanging appearance features across images.
Revised: However, the masks have the drawback that they dictate the part outline strictly, which leads to inconsistencies when exchanging appearance features across images.

Original: For instance, the strain direction of the hair and their outline must be correlated.
Revised: For instance, the strain direction of the hair and their outline must be correlated.

Original: This explains why our mask-free method attaining signiﬁcantly higher image quality after editing operations (95% preferred ours).
Revised: This explains why our mask-free method attaining signiﬁcantly higher image quality after editing operations (95% preferred ours).

Original: Hence, the preferred method depends on the use case.
Revised: Hence, the preferred method depends on the use case.

Original: E.g., for the ﬁne editing of outlines SEAN would be preferred while ours is better at combining appearances from different faces.
Revised: E [Table] g., for the ﬁne editing of outlines SEAN would be preferred while ours is better at combining appearances from different faces.

Original: An additional strong outcome is that our unsupervised approach has equal disentanglement scores compared to SEAN; 50% judge them equal, with 29% giving preference to ours and only 22% giving preference to SEAN.
Revised: An additional strong outcome is that our unsupervised approach has equal disentanglement scores compared to SEAN; 50% judge them equal, with 29% giving preference to ours and only 22% giving preference to SEAN.

Original: Validating that LatentKeypointGAN enables localized editing.
Revised: Validating that LatentKeypointGAN enables localized editing.

Original: Interpretation - realism.
Revised: Interpretation - realism.

Original: When comparing our GAN (without modifying keypoint location or appearance) to real images at resolution 128 × 128 of the training set, 42% rate them as equal.
Revised: When comparing our GAN (without modifying keypoint location or appearance) to real images at resolution 128 × 128 of the training set, 42% rate them as equal.

Original: Surprisingly 33% even prefer ours over the ground truth.
Revised: Surprisingly 33% even prefer ours over the ground truth.

Original: This preference may be because the ground truth images have artifacts in the background due to the forced alignment that are smoothed out in any of the generated ones.
Revised: This preference may be because the ground truth images have artifacts in the background due to the forced alignment that are smoothed out in any of the generated ones.

Original: Overall, these scores validate that the generated images come close to real images, even though minor artifacts remain at high resolutions.
Revised: Overall, these scores validate that the generated images come close to real images, even though minor artifacts remain at high resolutions.

Original: E E XPERIMENTS D ETAILS
Revised: E E XPERIMENTS D ETAILS

Original: SPADE (Park et al., 2019)
Revised: SPADE (Park et al., 2019)

Original: As shown in Figure 10, SPADE takes two inputs, feature map and style map, and use the style map to calculate the mean and standard deviation, which is used to denormalize the batch normalized feature map.
Revised: As shown in Figure 11, SPADE takes two inputs, feature map and style map, and use the style map to calculate the mean and standard deviation, which is used to denormalize the batch normalized feature map.

Original: Formally speaking, let F i ∈ R N × C
Revised: Formally speaking, let F i ∈ R N × C

Original: i × H
Revised: i × H

Original: i
Revised: i

Original: ×
Revised: ×

Original: W
Revised: W

Original: i be a i -th feature map in the network for a batch of N samples, where C i is the number of channels.
Revised: i be a i -th feature map in the network for a batch of N samples, where C i is the number of channels.

Original: Here we slightly abuse the notation to denote N batched style maps of size ( H i , W i ) as S i ∈
Revised: Here we slightly abuse the notation to denote N batched style maps of size ( H i , W i ) as S i ∈

Original: R N × ( K +1)
Revised: R N × ( K +1)

Original: D embed × H i × W i .
Revised: D embed × H i × W i .

Original: The same equation as for BatchNorm (Ioffe & Szegedy, 2015) is used to normalize the feature map, but now the denormalization coefﬁcients stem from the conditional map, which in our case is the processed style map.
Revised: The same equation as for BatchNorm (Ioffe & Szegedy, 2015) is used to normalize the feature map, but now the denormalization coefﬁcients stem from the conditional map, which in our case is the processed style map.

Original: Speciﬁcally, the resulting value of the spatial adaptive normalization is
Revised: Speciﬁcally, the resulting value of the spatial adaptive normalization is

Original: [Equation]
Revised: [Equation]

Original: where n ∈ { 1 , ..., N } is the index of the sample, c ∈ { 1 , ..., C } is the index of channels of the feature map, and ( y, x ) is the pixel index.
Revised: where n ∈ { 1 , ..., N } is the index of the sample, c ∈ { 1 , ..., C } is the index of channels of the feature map, and ( y, x ) is the pixel index.

Original: The µ ic and σ ic are the mean and standard deviation of channel c .
Revised: The µ ic and σ ic are the mean and standard deviation of channel c .

Original: The γ ic,y,x ( S in ) and β ic,y,x ( S in ) are the parameters to denormalize the feature map.
Revised: The γ i c,y,x ( S i n ) and β i c,y,x ( S i n ) are the parameters to denormalize the feature map.

Original: They are obtained by applying two convolution layers on the style map S in .[Figure]
Revised: They are obtained by applying two convolution layers on the style map S in .

Original: Learning rate and initialization.
Revised: Learning rate and initialization.

Original: We set the learning rate to 0.0001 and 0.0004 for the generator and discriminators, respectively.
Revised: We set the learning rate to 0.0001 and 0.0004 for the generator and discriminators, respectively.

Original: To let our model learn the coordinates reliably, we ﬁrst set the [Table] [Figure] [Table]
Revised: To let our model learn the coordinates reliably, we ﬁrst set the learning rate of the MLP, which generates keypoint coordinates to 0.05x the generator learning rate, i.e. 0.00005.

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: We use Kaiming initialization (He et al., 2015) for our network.

Original: [Table] [Figure] [Table] [Figure]
Revised: We initialize the

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: learning rate of the MLP, which generates keypoint coordinates to 0.05x the generator learning rate, i.e. 0.00005.

Original: We use Kaiming initialization (He et al., 2015) for our network.

Original: We initialize the weights of the last layer of the MLP that generates the keypoint coordinates to 0.05x the Kaiming initialization.
Revised: weights of the last layer of the MLP that generates the keypoint coordinates to 0.05x the Kaiming initialization.

Original: Progressive Growing Training we adopt progressive growing training (Karras et al., 2018) to allow larger batch size, which helps both on keypoint localization and local appearance learning.
Revised: Progressive Growing Training we adopt progressive growing training (Karras et al., 2018) to allow larger batch size, which helps both on keypoint localization and local appearance learning.

Original: This is likely due to the BatchNorm that is essential in SPADE.
Revised: This is likely due to the BatchNorm that is essential in SPADE.

Original: We also tried to replace the BatchNorm in SPADE with LayerNorm (Ba et al., 2016) and PixelNorm (Karras et al., 2019), but both of them cause mode collapse.
Revised: We also tried to replace the BatchNorm in SPADE with LayerNorm (Ba et al., 2016) and PixelNorm (Karras et al., 2019), but both of them cause mode collapse.

Original: We use a scheduled learning rate for the Keypoint Generator K .
Revised: We use a scheduled learning rate for the Keypoint Generator K .

Original: As illustrated in Figure 11, at each resolution stage, the training is divided into adapting period and non-adapting period.
Revised: As illustrated in Figure 12, at each resolution stage, the training is divided into adapting period and non-adapting period.

Original: We set the learning rate of K to zero in the adapting period and back to normal in the nonadapting period.
Revised: We set the learning rate of K to zero in the adapting period and back to normal in the nonadapting period.

Original: In the adapting period, the training follows PGGAN (Karras et al., 2018) where the feature map is a linear combination of the larger resolution RGB image and current resolution
Revised: In the adapting period, the training follows PGGAN (Karras et al., 2018) where the feature map is a linear combination of the larger resolution RGB image and current resolution

Original: RGB image.
Revised: RGB image.

Original: The coefﬁcient α gradually increases from 0 to 1.
Revised: The coefﬁcient α gradually increases from 0 to 1.

Original: At the end of the adapting period, the network is fully adapted to generate higher resolution images.
Revised: At the end of the adapting period, the network is fully adapted to generate higher resolution images.

Original: In the non-adapting period, the network generates high-resolution images without the linear combination.
Revised: In the non-adapting period, the network generates high-resolution images without the linear combination.

Original: Following StyleGAN (Karras et al., 2019), we start from a 4 × 4 × 512 learned constant matrix, which is optimized during training and ﬁxed during testing.
Revised: Following StyleGAN (Karras et al., 2019), we start from a 4 × 4 × 512 learned constant matrix, which is optimized during training and ﬁxed during testing.

Original: We use the keypoint-based ConvBlock 2 and bilinear upsampling to obtain feature maps with increasing resolutions.
Revised: We use the keypoint-based ConvBlock 2 and bilinear upsampling to obtain feature maps with increasing resolutions.

Original: Unlike PGGAN (Karras et al., 2018) and StyleGAN (Karras et al., 2019), who generating RGB images from feature maps of all resolutions (from 4 × 4 to 1024 × 1024 ), we start generating RGB images from the feature maps of at least 64 ×resolution.
Revised: Unlike PGGAN (Karras et al., 2018) and StyleGAN (Karras et al., 2019), who generating RGB images from feature maps of all resolutions (from 4 × 4 to 1024 × 1024 ), we start generating RGB images from the feature maps of at least 64 ×resolution.

Original: This is possible with the keypint generator and its spatially localized embeddings taking over the role of low feature maps.
Revised: This is possible with the keypint generator and its spatially localized embeddings taking over the role of low feature maps.

Original: It helps to locate the keypoints more accurately.
Revised: It helps to locate the keypoints more accurately.

Original: Generator.
Revised: Generator.

Original: We illustrate the LatentKeypointGAN generator in Figure 11.
Revised: We illustrate the LatentKeypointGAN generator in Figure 12.

Original: The output image is linearly combined by the output of toRGB block, where the weights depend on the training stage.
Revised: The output image is linearly combined by the output of toRGB block, where the weights depend on the training stage.

Original: Discriminator.
Revised: Discriminator.

Original: We illustrate the discriminator in Figure 11.
Revised: We illustrate the discriminator in Figure 12.

Original: For each resolution, we use two convolutions followed by Leaky ReLU (Maas et al., 2013).
Revised: For each resolution, we use two convolutions followed by Leaky ReLU (Maas et al., 2013).

Original: The ﬁrst convolution has a kernel size 
Revised: The ﬁrst convolution has a kernel size 4 × 4 and stride 2 to downsample the feature map to 0.5x.

Original: [Table]
Revised: The second convolutions have a kernel size 3 × 3 and stride 1 to extract features.

Original: Hyperparameter Setting For all experiments in image generation, we use leaky ReLU (Maas et al., 2013) with a slope 0.2 for negative values as our activation function.
Revised: Hyperparameter Setting For all experiments in image generation, we use leaky ReLU (Maas et al., 2013) with a slope 0.2 for negative values as our activation function.

Original: We use ADAM optimizer (Kingma & Ba, 2015) with β 1 = 0 .
Revised: We use ADAM optimizer (Kingma & Ba, 2015) with β 1 = 0 .

Original: 5 and β 2 = 0 .
Revised: 5 and β 2 = 0 .

Original: We set the learning rate to 0.0001 and 0.0004 for generator and discriminators, respectively (Heusel et al., 2017).
Revised: We set the learning rate to 0.0001 and 0.0004 for generator and discriminators, respectively (Heusel et al., 2017).

Original: We start from generating 64 × 64 images in the progressive training.
Revised: We start from generating 64 × 64 images in the progressive training.

Original: The batch size for 64 2 , 128 2 , 256 2 , 512 2 images are 128 , 64 , 32 , 8 , respectively.
Revised: The batch size for 64 2 , 128 2 , 256 2 , 512 2 images are 128 , 64 , 32 , 8 , respectively.

Original: We set λ gp = 10 and λ bg = 100 .
Revised: We set λ gp = 10 and λ bg = 100 .

Original: We set D noise =
Revised: We set D noise =

Original: 256 and D embed =for all experiments unless otherwise stated (in ablation tests). 
Revised: 256 and D embed =for all experiments unless otherwise stated (in ablation tests). [Table] [Figure] [Table]

Revised: [Figure]

Original: We lists the different τ s and different background setting for all experiments in Table 5.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: We lists the different τ s and different background setting for all experiments in Table 5.

Original: In CelebAHQ and FFHQ, the foreground is naturally disentangled from the background.
Revised: In CelebAHQ and FFHQ, the foreground is naturally disentangled from the background.

Original: The face can be freely moved on the image.
Revised: The face can be freely moved on the image.

Original: However, in the Bedroom dataset, all objects and their parts are strongly correlated.
Revised: However, in the Bedroom dataset, all objects and their parts are strongly correlated.

Original: For example, the bed cannot be moved to the ceiling, and the window cannot be moved to the ﬂoor.
Revised: For example, the bed cannot be moved to the ceiling, and the window cannot be moved to the ﬂoor.

Original: Therefore, we treat every object in the bedroom as a key part, even the ﬂoor, but the possible motion is restricted to plausible locations (see the supplementary video).
Revised: Therefore, we treat every object in the bedroom as a key part, even the ﬂoor, but the possible motion is restricted to plausible locations (see the supplementary video).

Original: A separate background embedding does not make sense.
Revised: A separate background embedding does not make sense.

Original: Therefore, we set the background ( H bg = 0 ) and the background loss λ bg = 0 for the experiments on the Bedroom dataset.
Revised: Therefore, we set the background ( H bg = 0 ) and the background loss λ bg = 0 for the experiments on the Bedroom dataset.

Original: F A BLATION T ESTS F. A BLATION T EST ON THE N EURAL N ETWORK A RCHITECTURE
Revised: F A BLATION T ESTS F. A BLATION T EST ON THE N EURAL N ETWORK A RCHITECTURE

Original: We show the detection result on MAFL and FID score on FFHQ in Table 2 for different architectures we tested below.
Revised: We show the detection result on MAFL and FID score on FFHQ in Table 2 for different architectures we tested below.

Original: The detection follows Section 4.7.
Revised: The detection follows Section 4.7.

Original: The FID is calculated at resolution 256 × 256 , by generated 50k images and the resized original dataset.
Revised: The FID is calculated at resolution 256 × 256 , by generated 50k images and the resized original dataset.

Original: Note that it is different from the main paper for simplicity since the goal of this part is ablation tests instead of comparing with others.
Revised: Note that it is different from the main paper for simplicity since the goal of this part is ablation tests instead of comparing with others.

Original: Removing background embedding.
Revised: Removing background embedding.

Original: We remove the background embeddings from our architecture ( z bg emb and w bg in Figure 2).
Revised: We remove the background embeddings from our architecture ( z bg emb and w bg in Figure 2).

Original: In this case, the keypoint embedding controls the whole appearance of the image.
Revised: In this case, the keypoint embedding controls the whole appear- ance of the image.

Original: In addition, as shown in Figure 13, the keypoints are not exactly located at the key parts, though they are still consistent among different views.
Revised: In addition, as shown in Figure 14, the keypoints are not exactly located at the key parts, though they are still consistent among different views.

Original: Removing global style vector.
Revised: Removing global style vector.

Original: We remove the global style vector w global .
Revised: We remove the global style vector w global .

Original: Therefore, all the keypoint embeddings are constant.
Revised: Therefore, all the keypoint embeddings are constant.

Original: Only keypoint location and background embedding are different among the images.
Revised: Only keypoint location and background embedding are different among the images.

Original: In this case, the keypoint embedding works equivalent to one-hot encoding, and cannot fully capture the variations on the key parts.
Revised: In this case, the keypoint embedding works equivalent to one-hot encoding, and cannot fully capture the variations on the key parts.

Original: Therefore, it leads to inaccurate keypoints, as shown in Figure 13.
Revised: Therefore, it leads to inaccurate keypoints, as shown in Figure 14.

Original: Furthermore, we observed that without w global , the network hides the appearance information in the keypoint location.
Revised: Furthermore, we observed that without w global , the network hides the appearance information in the keypoint location.

Original: Changing keypoint embedding generation.
Revised: Changing keypoint embedding generation.

Original: We change the keypoint embedding generation in two ways.
Revised: We change the keypoint embedding generation in two ways.

Original: The ﬁrst way is generating constant embedding w j const and global style vector w global just as before and then add them elementwisely instead of multiplying them.
Revised: The ﬁrst way is generating constant embedding w j const and global style vector w global just as before and then add them elementwisely instead of multiplying them.

Original: Formally speaking, for each keypoint j , its corresponding embedding is
Revised: Formally speaking, for each keypoint j , its corresponding embedding is

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: where ⊕ means elementwise addition.
Revised: where ⊕ means elementwise addition.

Original: This gives slightly higher detection accuracy but lower image quality.
Revised: This gives slightly higher detection accuracy but lower image quality.

Original: We observe that in this case, the background controls the foreground appearance.
Revised: We observe that in this case, the background controls the foreground appearance.

Original: However, different from Removing global style vector F.1, the appearance information is not hidden in keypoint locations.
Revised: However, different from Removing global style vector F.1, the appearance information is not hidden in keypoint locations.

Original: We believe this is because that w global works as noise to avoid the network from hiding foreground appearance information in keypoint location.
Revised: We believe this is because that w global works as noise to avoid the network from hiding foreground appearance information in keypoint location.

Original: As a result of good disentanglement of appearance and keypoint location, the keypoint detection accuracy slightly increases.
Revised: As a result of good disentanglement of appearance and keypoint location, the keypoint detection accuracy slightly increases.

Original: However, again, in this setting, the keypoint embedding cannot fully capture the variations of the key parts.
Revised: However, again, in this setting, the keypoint embedding cannot fully capture the variations of the key parts.

Original: Therefore, the background takes the control of appearance.
Revised: Therefore, the background takes the control of appearance.

Original: The second way is to generate [ w j ] Kj =1 together from z kp app using MLP.
Revised: The second way is to generate [ w j ] Kj =1 together from z kp app using MLP.

Original: In this case, there is no constant embeddings or global style vector.
Revised: In this case, there is no constant embeddings or global style vector.

Original: To force the embedding of the same keypoint to be similar, and the embedding for different keypoints to be different, we use Supervised Contrastive
Revised: To force the embedding of the same keypoint to be similar, and the embedding for different keypoints to be different, we use Supervised Contrastive

Original: Losses (Khosla et al., 2020),
Revised: Losses (Khosla et al., 2020),

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: A ( j ) =
Revised: A ( j ) =

Original: { i : w i , w j are in the same batch }
Revised: { i : w i , w j are in the same batch }

Original: K

Original: [Equation]
Revised: [Equation]

Original: w j belong to the same keypoint in the same batch } 
Revised: j belong to the same keypoint in the same batch } J

Original: [Equation]
Revised: [Equation]

Original:  indices of all keypoint embeddings in the same batch }
Revised: { indices of all keypoint embeddings in the same batch }

Original: As shown in Figure 13, the keypoints are neither on the key parts nor consistent.
Revised: As shown in Figure 14, the keypoints are neither on the key parts nor consistent.

Original: We further visualize the embeddings with T-SNE and PCA in Figure 12.
Revised: We further visualize the embeddings with T-SNE and PCA in Figure 13.

Original: Although the contrastive learned embedding has comparable T-SNE with our multiplicative design, the PCA shows that our multiplicative embedding is linearly separable while contrastive learned embedding is not.
Revised: Although the contrastive learned embedding has comparable T-SNE with our multiplicative design, the PCA shows that our multiplicative embedding is linearly separable while contrastive learned embedding is not.

Original: Hence, we demonstrate that our original design of elementwise production is simple and effective.
Revised: Hence, we demonstrate that our original design of elementwise production is simple and effective.

Original: Removing keypoint embedding.
Revised: Removing keypoint embedding.

Original: We remove the keypoint embedding w j entirely.
Revised: We remove the keypoint embedding w j entirely.

Original: In this case, we only have background embedding w bg
Revised: In this case, we only have background embedding w bg

Original: and the keypoint location.
Revised: and the keypoint location.

Original: Thus, instead of generating the style map S , we directly concatenate the keypoint heatmaps { H k } Kk =1 and the broadcasted background style map to generate the style map without keypoint embedding.
Revised: Thus, instead of generating the style map S , we directly concatenate the keypoint heatmaps { H k } Kk =1 and the broadcasted background style map to generate the style map without keypoint embedding.

Original: As shown in Figure 13, the keypoints are not meaningful or consistent.
Revised: As shown in Figure 14, the keypoints are not meaningful or consistent.

Original: The keypoint location hides the appearance and is entangled with the background severely.
Revised: The keypoint location hides the appearance and is entangled with the background severely.

Original: Removing keypoints.
Revised: Removing keypoints.

Original: If we remove the keypoints, then SPADE (Park et al., 2019) degenerates to AdaIN (Huang & Belongie, 2017).
Revised: If we remove the keypoints, then SPADE (Park et al., 2019) degenerates to AdaIN (Huang & Belongie, 2017).

Original: Instead of using a style map S (2D), we now use a style vector (1D), which is the background embedding.
Revised: Instead of using a style map S (2D), we now use a style vector (1D), which is the background embedding.

Original: In this case, we do not have the power of local controllability on the generated images.
Revised: In this case, we do not have the power of local controllability on the generated images.

Original: [Figure]
Revised: Ablation test on the dimension of embeddings. Different numbers of embedding dimensions make the expression power vary. As shown in

Original: [Table] [Table]
Revised: [Figure] [Figure]

Original:  A too-small value for τ does not inﬂuence the image and will cause artifacts as shown in Figure 14.
Revised: Ablation Test on τ A too-small value for τ does not inﬂuence the image and will cause artifacts as shown in Figure 15.

Original: A too-large value for τ will disable the background embedding and control the background.
Revised: A too-large value for τ will disable the background embedding and control the background.

Original: Ablation test on the number of keypoints.
Revised: Ablation test on the number of keypoints.

Original: By selecting different numbers of keypoints, we can achieve different levels of control.
Revised: By selecting different numbers of keypoints, we can achieve different levels of control.

Original: In the second row of Figure 14, we use 6 keypoints rather than the default 10.
Revised: In the second row of Figure 15, we use 6 keypoints rather than the default 10.

Original: Thereby, keypoints have a smaller area of effect.
Revised: Thereby, keypoints have a smaller area of effect.

Original: We observe that the background encoding then takes a larger role and contains the encoding of hair and beard, while the keypoints focus only on the main facial features (nose, mouth, and eyes).
Revised: We observe that the background encoding then takes a larger role and contains the encoding of hair and beard, while the keypoints focus only on the main facial features (nose, mouth, and eyes).

Original: Ablation test on the dimension of embeddings.
Revised: Ablation test on the combination of number of keypoints and τ .

Original: Different numbers of embedding dimensions make the expression power vary.
Revised: The impact of keypoints depends on the combination of number of keypoints and τ . We test the pairwise combination between K = 1 , 6 , 8 , 12 , 16 , 32 and τ = 0 . The FID is listed in Table 7 and the detecDimension of embeddings D embed

Original: As shown in Table 6, larger D embed leads to larger error on MAFL but lower (better) FID.
Revised: Keypoint detection error on MAFL ↓ FID score on FFHQ ↓ 4.61% 28. [Table] [Figure]

Revised: number

Original: We use D embed =128 in our main paper because the increase in error is small but the decrease of FID is signiﬁcant.
Revised: [Figure] keypoints [Figure] K =

Revised: K =

Revised: K =

Revised: K =

Revised: K =

Original: {+K =τ = 
+}
Revised: {+K =τ = 
+}

Revised: τ = 0 .18.

Revised: τ = 0 .19.

Original: {+τ = 
+}
Revised: {+τ = 
+}

Revised: tion error is listed in Table 8.

Revised: The image quality does not change much for different combinations.

Revised: We illustrate samples of keypoints of each combination in Figure 16 and editing results in Figureand Figure 18.

Revised: If both the number of keypoints and τ are small, e.g., K = 1 , and the τ = 0 .

Revised: 002 , then the background controls both foreground appearance and pose, and the keypoints are trivial.

Revised: If both, the number of keypoints and τ , are large, e.g., K = 32 , and τ = 0 .

Revised: 02 , then the keypoint appearance controls background and pose.

Revised: While the model degenerates in extreme cases, we found the model to be robust for a wide range of values, i.e., K = 8 , 12 , 16 and τ = 0 .

Revised: We summarize all the cases in Table 9.

Original: F. A BLATION T ESTS ON THE T RAINING S TRATEGY
Revised: F. A BLATION T ESTS ON THE T RAINING S TRATEGY

Original: Ablation Test on GAN Loss If we replace equation 3, 4 and 5 with the spectral norm (Miyato et al., 2018) and hinge loss (Miyato et al., 2018; Park et al., 2019) used in the original SPADE architecture, we get mostly static, meaningless latent keypoint coordinates.
Revised: Ablation Test on GAN Loss If we replace equation 3, 4 andwith the spectral norm (Miyato et al., 2018) and hinge loss (Miyato et al., 2018; Park et al., 2019) used in the original SPADE architecture, we get mostly static, meaningless latent keypoint coordinates.

Original: The object part location information is entangled with the key part appearance.
Revised: The object part location information is entangled with the key part appearance.

Original: The comparison is shown in Figure 15.
Revised: The comparison is shown in Figure 21.

Revised: Ablation Test on Background Loss If we remove the background loss in Eq.

Revised: 6, most keypoints are still at reasonable positions while some move to the background.

Revised: As shown in Figure 20, the yellow keypoint is on the background while all the others are still on the foreground.

Original: Removing Keypoint Scheduler.
Revised: Removing Keypoint Scheduler.

Original: If we move the keypoint scheduler, i.e., updating keypoint generator during resolution adaption, the keypoint locations diverge and the appearance collapses, as shown in Figure 15.
Revised: If we move the keypoint scheduler, i.e., updating keypoint generator during resolution adaption, the keypoint locations diverge and the appearance collapses, as shown in Figure 21. [Table] [Table] [Table]

Original: G F AILURE C ASES AND L IMITATIONS
Revised: E means entangled pose, appearance and background. (cid:88)(cid:88) means disentangled control and (cid:88) means inferior disentanglement, where one of the pairs { (pose, appearance), (pose, background), (appearance, background) is entangled. } . For a small keypoint size of τ = 0 . 0002 the model always gives trivial keypoints. With a large number of keypoints and a large keypoint size, i.e., K > 16 and τ > 0 .

Revised: 01 , our model gives entangled representations.

Revised: Our model is robust in the range of K ∈ [8 , 16] and τ ∈ [0 . 005 , 0 . 01] .

Revised: G F AILURE C ASES AND L IMITATIONS

Original: As described in the main text, our model sometimes generates asymmetric faces as shown in the ﬁrst two images in Figure 16.
Revised: As described in the main text, our model sometimes generates asymmetric faces as shown in the ﬁrst two images in Figure 22.

Original: In addition, the hair sometimes is entangled with the background, especially long hair, as shown in the right two images in Figure 16. [Figure] [Figure]
Revised: In addition, the hair sometimes is entangled with the background, especially long hair, as shown in the right two images in Figure 22.

Original: Reducing the learning rate after each progressive up-scaling step prevents mode collapse and enables high-resolution training.

Original: From the ablation tests in training strategy, we can see that this method can heavily depend on the state-of-the-art GAN training loss function and image-to-image translation architectures.
Revised: From the ablation tests in training strategy, we can see that this method can heavily depend on the state-of-the-art GAN training loss function and image-to-image translation architectures.

Original: In fact, we observed some image quality degeneration as the training goes on in the highest resolution ( 512 × 512 ).
Revised: In fact, we observed some image quality degeneration as the training goes on in the highest resolution ( 512 × 512 ).

Original: Therefore, we apply early stopping in the highest resolution.
Revised: Therefore, we apply early stopping in the highest resolution.

Original: We expect that researchers will push GAN and spatially adaptive image-to-image translation even further.
Revised: We expect that researchers will push GAN and spatially adaptive image-to-image translation even further.

Original: Then people can plug in our keypoint generator to receive higher image quality and more accurate keypoints.
Revised: Then people can plug in our keypoint generator to receive higher image quality and more accurate keypoints.

Original: In this paper, we only provide the idea and focus on unsupervised local image editing without using any loss on pairs of images.
Revised: In this paper, we only provide the idea and focus on unsupervised local image editing without using any loss on pairs of images.

Revised: K=16,32.

Revised: Extreme small τ ( τ = 0 . 0002 ) constantly gives trivial keypoints even if K is large ( K = 32 ).

Revised: When both, K and τ , are large ( K = 32 , τ > 0 . 01 ), the keypoint embeddings control the background and the pose.

Revised: We found the combinations of ( K = 16 , τ = 0 . 0005 ) and ( K = 32 , τ = 0 .

Revised: 005 ) give the best editing controllability.

Original: [Figure]
Revised: [Figure]

Revised: rate prevents enables