Original: L ATENT I MAGE A NIMATOR : L EARNING TO ANIMATE IMAGES VIA LATENT SPACE NAVIGATION
Revised: L ATENT I MAGE A NIMATOR : L EARNING TO ANIMATE IMAGES VIA LATENT SPACE NAVIGATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Due to the remarkable progress of Generative Adversarial Networks (GANs) and auto-encoder animating images has become increasingly efﬁcient, whereas associated results have become increasingly realistic.
Revised: Due to the remarkable progress of deep generative models, animating images has become increasingly efﬁcient, whereas associated results have become increasingly realistic.

Original: Current animation-approaches commonly exploit structure representation extracted from driving videos.
Revised: Current animation-approaches commonly exploit structure repre- sentation extracted from driving videos.

Original: Such structure representation( e.g., keypoints or regions) is instrumental in transferring motion from driving videos to still images.
Revised: Such structure representation is instrumental in transferring motion from driving videos to still images.

Original: However, such approaches fail in case that a source image and driving video encompass large appearance variation.
Revised: However, such approaches fail in case the source image and driving video encompass large appearance variation.

Original: Moreover, the extraction of structure information requires additional modules that endow the animation-model with increased complexity.
Revised: Moreover, the extraction of structure information requires ad- ditional modules that endow the animation-model with increased complexity.

Original: Deviating from such models, we here introduce the Latent Image Animator (LIA), a self-supervised autoencoder that evades need for structure representation.
Revised: Deviating from such models, we here introduce the Latent Image Animator (LIA), a self-supervised autoencoder that evades need for structure representation.

Original: LIA is streamlined to animate images by linear navigation in the latent space.
Revised: LIA is streamlined to animate images by linear navigation in the latent space.

Original: Speciﬁcally, motion in generated video is constructed by linear displacement of codes in the latent space.
Revised: Speciﬁ- cally, motion in generated video is constructed by linear displacement of codes in the latent space.

Original: Towards this, we learn a set of orthogonal motion directions simultaneously, and use their linear combination, in order to represent any displacement in the latent space.
Revised: Towards this, we learn a set of orthogonal motion directions si- multaneously, and use their linear combination, in order to represent any displacement in the latent space.

Original: Extensive quantitative and qualitative analysis suggests that our model systematically and signiﬁcantly outperforms state-of-art methods on VoxCeleb, Taichi and TED-talk datasets w.r.t.
Revised: Extensive quantitative and qualitative analysis suggests that our model systematically and signiﬁcantly outperforms state-of-art methods on VoxCeleb, Taichi and TED-talk datasets w.r.t.

Original: generated quality.
Revised: generated quality.

Revised: Source code and pre-trained models are publicly available 1 .

Revised: LIA-project/

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the series of science ﬁction books Harry Potter (Rowling et al., 2016; Rowling, 2019), wizards and witches were able to magically enchant portraits, bringing them to life .
Revised: In the series of science ﬁction books Harry Potter (Rowling et al., 2016; Rowling, 2019), wizards and witches were able to magically enchant portraits, bringing them to life .

Original: Remarkable progress of deep generative models has recently turned this vision into reality.
Revised: Remarkable progress of deep generative models has recently turned this vision into reality.

Original: This work examines the scenario where a framework animates a source image by motion representations learned from a driving video .
Revised: This work examines the scenario where a framework animates a source image by motion representations learned from a driving video .

Original: Existing approaches for image animation are classically related to computer graphics (Cao et al., 2014; Thies et al., 2016; 2019) or exploit explicit structure representations such as labels (Wang et al., 2020a), semantic maps (Pan et al., 2019; Wang et al., 2018; 2019), human key-points (Jang et al., 2018;
Revised: Existing approaches for image animation are classically related to computer graphics (Cao et al., 2014; Thies et al., 2016; 2019; Zhao et al., 2018) or exploit motion labels (Wang et al., 2020b) and structure representations such as semantic maps (Pan et al., 2019; Wang et al., 2018; 2019), human keypoints (Jang et al., 2018;

Original: Yang et al., 2018; Walker et al., 2017; Chan et al., 2019; Zakharov et al., 2019; Wang et al., 2019; Siarohin et al., 2019), 3D face mesh (Zhao et al., 2018) and optical ﬂow (Li et al., 2018; Ohnishi et al., 2018).
Revised: Yang et al., 2018; Walker et al., 2017; Chan et al., 2019; Zakharov et al., 2019; Wang et al., 2019; Siarohin et al., 2019), 3D meshes (Liu et al., 2019; Chen et al., 2021), and optical ﬂows (Li et al., 2018; Ohnishi et al., 2018).

Original: We note that the ground truth of such structure representations is being computed a-priori for the purpose of supervised training. This poses constraints on applications, where such representations of unseen testing images might be fragmentary or missing.
Revised: We note that the ground truth of such structure representations has been computed a-priori for the purpose of supervised training, which poses constraints on applications, where such representations of unseen testing images might be fragmentary or difﬁcult to access.

Original: Self-supervised motion transfer approaches (Wiles et al., 2018; Siarohin et al., 2019; 2021) accept raw videos as input and learn to reconstruct driving images by warping source image with predicted dense  ﬂow ﬁelds .
Revised: Self-supervised motion transfer approaches (Wiles et al., 2018; Siarohin et al., 2019; 2021) accept raw videos as input and learn to reconstruct driving images by warping source image with predicted dense optical ﬂow ﬁelds .

Original: While the need for domain knowledge or labeled ground truth data has been obviated (improving performance on in-the-wild testing images), such methods entail necessity of explicit structure representations as motion guidance.
Revised: While the need for domain knowledge or labeled ground truth data has been obviated, which improves performance on in-the-wild testing images, such methods entail necessity of explicit structure representations as motion guidance.

Original: Prior information such as keypoints  or regions  are learned in an end-to-end training manner by additional networks as intermediate features, in order to predict target ﬂow ﬁelds.
Revised: Prior information such as keypoints (Siarohin et al., 2019; Wang et al., 2021a) or regions (Siarohin et al., 2021) are learned in an end-to-end training manner by additional networks as intermediate features, in order to predict target ﬂow ﬁelds.

Original: Although online prediction of such representations is less tedious than the acquisition of ground truth labels, it still strains the complexity of networks.
Revised: Although online prediction of such representations is less tedious than the acquisition of ground truth labels, it still strains the complexity of networks.

Original: Deviating from such approaches, we here aim to fully eliminate the need of explicit structure representations by directly manipulating the latent space of a deep generative model.
Revised: Deviating from such approaches, we here aim to fully eliminate the need of explicit structure rep- resentations by directly manipulating the latent space of a deep generative model.

Original: To the best of our knowledge, this constitutes a new direction in the context of image animation .
Revised: To the best of our knowledge, this constitutes a new direction in the context of image animation .

Original: Our work is motivated by interpretation of GANs (Shen et al., 2020; Goetschalckx et al., 2019; Jahanian et al., 2020; Voynov & Babenko, 2020), showcasing that latent spaces of StyleGAN (Karras et al., 2019) and
Revised: Our work is motivated by interpretation of GANs (Shen et al., 2020; Goetschalckx et al., 2019; Jahanian et al., 2020; Voynov & Babenko, 2020), showcasing that latent spaces of StyleGAN (Karras et al., 2019; 2020b) and

Original: BigGAN (Brock et al., 2019) contain rich semantically meaningful directions.
Revised: BigGAN (Brock et al., 2019) contain rich semantically meaningful directions.

Original: Given that walking along such directions, basic visual transformations such as zooming and rotation can be induced in generated results.
Revised: Given that walking along such directions, basic visual transformations such as zooming and rotation can be induced in generated results.

Original: As in image animation, we have that motion between source and driving images can be considered as higher-level transformation, a natural question here arises: can we discover a set of directions in the latent spaceof an image animator that induces high-level  transformations collaboratively?
Revised: As in image animation, we have that motion between source and driving images can be considered as higher-level transformation, a natural question here arises: can we discover a set of directions in the latent space that induces high-level motion transformations collaboratively?

Original: Towardsanswering this question, we introduce  a novel Latent Image Animator(LIA), constituting of an auto-encoder for animating still images via latent space navigation.
Revised: Towards answering this question, we introduce LIA, a novel Latent Image Animator constituting of an autoencoder for animating still images via latent space navigation.

Original: LIA seeks to animate a source image via linearly navigating associated source latent code along a learnedlatent path to reach a target latent code, which represents the high-level transformation for animating the source image.
Revised: LIA seeks to animate a source image via linearly navigating associated source latent code along a learned path to reach a target latent code, which represents the high-level transformation for animating the source image.

Original: We introduce a Linear Motion Decomposition (LMD) approach aiming to represent a latent path via a linearly combination of a set of learned motion directions and associated magnitudes.
Revised: We introduce a Linear Motion Decomposition (LMD) approach aiming to represent a latent path via a linear combination of a set of learned motion directions and associated magnitudes.

Original: Speciﬁcally, we constrain the set as an orthogonal basis, where each vector indicates a basic visual transformation.
Revised: Speciﬁcally, we constrain the set as an orthogonal basis, where each vector indicates a basic visual transformation.

Original: By describing the whole motion space using such learned basis, LIA eliminates the need of explicit structure representations.
Revised: By describing the whole motion space using such learned basis, LIA eliminates the requirement of explicit structure representations.

Original: In addition, we design LIA to disentangle motion and appearance within a single encoder-generator architecture.
Revised: In addition, we design LIA to disentangle motion and appearance within a single encoder-generator architecture.

Original: Deviating from existing methods using separate networks to learn disentangled features, LIA integrates both, latent motion code, as well as appearance features in a single encoder , which highly reduces the model complexity and simpliﬁes training.
Revised: Deviating from existing methods using separate networks to learn disentangled features, LIA integrates both, latent motion code, as well as appearance features in a single encoder , which highly reduces the model complexity and simpliﬁes training.

Original: We provide evaluation on multiple datasets including VoxCeleb, TaichiHDand TED-talk.
Revised: We provide evaluation on multiple datasets including VoxCeleb (Chung et al., 2018), TaichiHD

Revised: (Siarohin et al., 2019) and TED-talk (Siarohin et al., 2021).

Original: In addition, we show that LIA outperforms the state-of-the-art in preserving the facial structure in generated videos in the setting of one-shot image animation on unseen dataset (FFHQ and GermanAudio).
Revised: In addition, we show that LIA outperforms the state-of-the-art in preserving the facial structure in generated videos in the setting of one-shot image animation on unseen datasets such as FFHQ (Karras et al., 2019) and GermanPublicTV (Thies et al., 2020).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Video generation Generative adversarial networks (GAN)-based video generation is aimed at mapping Gaussian noise to video, directly and in the absence of prior information (Vondrick et al., 2016;
Revised: Video generation GAN-based video generation is aimed at mapping Gaussian noise to video, di- rectly and in the absence of prior information (Vondrick et al., 2016;

Original: Saito et al., 2017; Tulyakov et al., 2018; Wang et al., 2020b).
Revised: Saito et al., 2017; Tulyakov et al., 2018; Wang et al., 2020a; Wang, 2021).

Original: Recently, with the progress of
Revised: Approaches based on deep probabilistic models (Den- ton & Birodkar, 2017; Li & Mandt, 2018; Bhagat et al., 2020; Xie et al., 2020) were also proposed to tackle this problem, however only show results on toy datasets with low resolution.

Original:  GANs in photo-realistic image generation (Brock et al., 2019; Karras et al., 2019; 2020), a series of works (Clark et al., 2019; Wang et al., 2021b) explored production of high-resolution videos by incorporating the architecture of an image generator into video GANs , trained jointly with RNNs.
Revised: Recently, with the progress of GANs in photo-realistic image generation (Brock et al., 2019; Karras et al., 2019; 2020a), a series of works (Clark et al., 2019; Wang et al., 2021c) explored production of highresolution videos by incorporating the architecture of an image generator into video GANs, trained jointly with RNNs.

Original: Tian et al.
Revised: Tian et al.

Original: (2021) directly leveraged the knowledge of a pre-trained StyleGAN to produce videos of resolution up to 1024 × 1024 .
Revised: (2021) directly leveraged the knowledge of a pre-trained StyleGAN to produce videos of resolution up to 1024 × 1024 .

Original: Unlike these approaches, which generate random videos based on noise vectors in an unconditional manner, in this paper, we focus on conditionally creating novel videos by transferring motion from driving videos to input images.
Revised: Unlike these approaches, which generate random videos based on noise vectors in an unconditional manner, in this paper, we focus on conditionally creating novel videos by transferring motion from driving videos to input images.

Original: Latent space editing In an effort to control generated images, recent works explored the discovery of semantically meaningful directions in the latent space of pre-trained GANs, where linear navigation corresponds to desired image manipulation.
Revised: Latent space editing In an effort to control generated images, recent works explored the discovery of semantically meaningful directions in the latent space of pre-trained GANs, where linear navigation corresponds to desired image manipulation.

Original: Supervised (Shen et al., 2020; Jahanian et al., 2020; Goetschalckx et al., 2019) and unsupervised (Voynov & Babenko, 2020; Peebles et al., 2020;
Revised: Supervised (Shen et al., 2020; Jahanian et al., 2020; Goetschalckx et al., 2019) and unsupervised (Voynov & Babenko, 2020; Peebles et al., 2020;

Original: Shen & Zhou, 2021) approaches were proposed to edit semantics such as facial attributes, colors and basic visual transformations ( e.g., rotation and zooming) in generated or inverted real images (Zhu et al., 2020; Abdal et al., 2020).
Revised: Shen & Zhou, 2021) approaches were proposed to edit semantics such as facial attributes, colors and basic visual transformations ( e.g., rotation and zooming) in generated or inverted real images (Zhu et al., 2020; Abdal et al., 2020).

Original: In this work, as opposed to ﬁnding directions corresponding to individual visual transformations, we seek to learn a set of directions that cooperatively allows for high-level visual transformations that can be beneﬁcial in image animation.
Revised: In this work, as opposed to ﬁnding directions corresponding to individual visual transformations, we seek to learn a set of directions that cooperatively allows for high-level visual transformations that can be beneﬁcial in image animation.

Original: Image animation Related approaches (Chan et al., 2019; Wang et al., 2018; Zakharov et al., 2019; Wang et al., 2019) in image animation required strong prior structure labels as motion guidance.
Revised: Image animation Related approaches (Chan et al., 2019; Wang et al., 2018; Zakharov et al., 2019; Wang et al., 2019; Yang et al., 2020) in image animation required strong prior structure labels as motion guidance.

Original: In particular, Chan et al.
Revised: In particular, Chan et al.

Original: (2019) and Wang et al.
Revised: (2019), Yang et al. (2020) and Wang et al.

Original: (2018) proposed to map representations such as human keypoints and facial landmarks to videos in the setting of image-to-image translation proposed by Isola et al.
Revised: (2018) proposed to map representations such as human keypoints and facial landmarks to videos in the setting of image-to-image translation proposed by Isola et al.

Original: However, such approaches were only able to learn an individual model for a single identity.
Revised: However, such approaches were only able to learn an individual model for a single identity.

Original: Transferring motion on new appearances requires retraining the entire model from scratch by using videos of target identities.
Revised: Transferring motion on new appearances requires retraining the entire model from scratch by using videos of target identities.

Original: Several recent works (Zakharov et al., 2019; Wang et al., 2019) explored meta learning in ﬁne-tuning models on target identities.
Revised: Several recent works (Zakharov et al., 2019; Wang et al., 2019) explored meta learning in ﬁne-tuning models on target identities.

Original: While only few images of target identities were required during inference time, it was still compulsory to input pre-computed structure representations in those approaches, which usually are hard to access in many real-world scenarios.
Revised: While only few images of target identities were required during inference time, it was still compulsory to input pre-computed structure representations in those approaches, which usually are hard to access in many real-world scenarios.

Original: Towardsaddressing this issue, very recent works (Siarohin et al., 2019; 2021; Wang et al., 2021a; Wiles et al., 2018) proposed to learn image animation in self-supervised manner, only relying on RGB videos for both, training and testing without any priors.
Revised: Towards addressing this issue, very recent works (Siarohin et al., 2019; 2021; Wang et al., 2021b; Wiles et al., 2018) proposed to learn image animation in self-supervised manner, only relying on RGB videos for both, training and testing without any priors.

Original: They ﬁrstly predicted dense ﬂow ﬁelds from input images, which were then utilized to warp source images, in order to obtain ﬁnal generated results.
Revised: They ﬁrstly predicted dense ﬂow ﬁelds from input images, which were then utilized to warp source images, in order to obtain ﬁnal generated results.

Original: Inference only required one image of a target identity without any ﬁne-tuning step on pretrained models.
Revised: Inference only required one image of a target identity without any ﬁne-tuning step on pre-trained models.

Original: While no priors were required, state-of-the-art methods still followed the idea of using explicit structure representations.
Revised: While no priors were required, state-of-the-art methods still followed the idea of using explicit structure representations.

Original: FOMM (Siarohin et al., 2019) proposed a ﬁrst order motion approach to predict keypoints and local transformations online to generate ﬂow ﬁelds.
Revised: FOMM (Siarohin et al., 2019) proposed a ﬁrst order motion approach to predict keypoints and local transformations online to generate ﬂow ﬁelds.

Original: Siarohin et al.
Revised: Siarohin et al.

Original: (2021) developed this idea to model articulated objects by replacing a keypoints predictor by a PCA-based region prediction module.
Revised: (2021) developed this idea to model articulated objects by replacing a keypoints predictor by a PCA-based region prediction module.

Original: Wang et al.
Revised: Wang et al.

Original: (2021a)
Revised: (2021b)

Original: extended FOMM by predicting 3D keypoints for view-free generation.
Revised: extended FOMM by predicting 3D keypoints for view-free generation.

Original: We note though that in all approaches, given that keypoints or regions are inadequately predicted, the quality of generated images drastically decreases.
Revised: We note though that in all approaches, given that keypoints or regions are inadequately predicted, the quality of generated images drastically decreases.

Original: In contrast to such approaches, our method does not require any explicit structure representations.
Revised: In contrast to such approaches, our method does not require any explicit structure representations.

Original: We dive into the latent space of the generator and self-learn to navigate motion codes in certain directions with the goal to reach target codes, which are then decoded to ﬂow ﬁelds for warping.
Revised: We dive into the latent space of the generator and self-learn to navigate motion codes in certain directions with the goal to reach target codes which are then decoded to ﬂow ﬁelds for warping.

Original:  M ETHOD
Revised:  M ETHOD

Original: Self-supervised image animation aims at learning to transfer motion from a subject of a driving video to a subject in a source image based on training with a large video dataset.
Revised: Self-supervised image animation aims at learning to transfer motion from a subject of a driving video to a subject in a source image based on training with a large video dataset.

Original: In this work, we propose to model such motion transformation via latent space navigation.
Revised: In this work, we propose to model such motion transformation via latent space navigation.

Original: The general pipeline is illustrated in Fig.
Revised: The general pipeline is illustrated in Fig.

Original: Speciﬁcally, for training, our model takes in a pair of source and driving images, randomly sampled from one video sequence.
Revised: Speciﬁcally, for training, our model takes in a pair of source and driving images, randomly sampled from one video sequence.

Original: These two images are encoded into a latent code which is used to represent motion transformation in the image space.
Revised: These two images are encoded into a latent code which is used to represent motion transformation in the image space.

Original: The training objective is to reconstruct the driving image by combining source image with learned motion transformation.
Revised: The training objective is to reconstruct the driving image by combining source image with learned motion transformation.

Original: For testing, frames of a driving video are sequentially processed with the source image to animate the source subject.
Revised: For testing, frames of a driving video are sequentially processed with the source image to animate the source subject.

Original: We provide an overview of the proposed model in Fig.
Revised: We provide an overview of the proposed model in Fig.

Original: Our model is an auto-encoder, consisting of two main networks, an encoder E and a generator G .
Revised: Our model is an autoencoder, consisting of two main networks, an encoder E and a generator G .

Original: In general, our model requires two steps to transfer motion.
Revised: In general, our model requires two steps to transfer motion.

Original: In the ﬁrst step, E encodes source and driving images x s , x d ∼ X ∈ R 3 × H × W into latent codes in the latent space.
Revised: In the ﬁrst step, E encodes source and driving images x s , x d ∼ X ∈ R 3 × H × W into latent codes in the latent space.

Original: The source code is then navigated into a target code, which is used to represent target motion transformation, along a learned latent path.
Revised: The source code is then navigated into a target code, which is used to represent target motion transformation, along a learned latent path.

Original: Based on proposed Linear
Revised: Based on proposed Linear

Original: Motion Decomposition (LMD), we represent such a path as a linear combination of a set of learned motion directions and associated magnitudes, which are learned from x d .
Revised: Motion Decomposition (LMD), we represent such a path as a linear combination of a set of learned motion directions and associated magnitudes, which are learned from x d .

Original: In the second step, once the target latent code is obtained, G decodes it as a dense ﬂow ﬁeld φ s → d ∼ Φ ∈ R 2 × H × W and uses φ s → d to warp x s and then to obtain the output image.
Revised: In the second step, once the target latent code is obtained, G decodes it as a dense ﬂow ﬁeld φ s → d ∼ Φ ∈ R 2 × H × W and uses φ s → d to warp x s and then to obtain the output image.

Original: In the following, we proceed to discuss the two steps in detail.
Revised: In the following, we proceed to discuss the two steps in detail.

Original: 3. L ATENT MOTION REPRESENTATION
Revised: 3. L ATENT MOTION REPRESENTATION

Original: Given a source image x s and a driving image x d , our ﬁrst step constitutes of learning a latent code z s → d ∼ Z ∈ R N to represent the motion transformation from x s to x d .
Revised: Given a source image x s and a driving image x d , our ﬁrst step constitutes of learning a latent code z s → d ∼ Z ∈ R N to represent the motion transformation from x s to x d .

Original: Due to the uncertainty of two images, directly learning z s → d puts forward a high requirement on the model to capture a complex distribution of motion.
Revised: Due to the uncertainty of two images, directly learning z s → d puts forward a high requirement on the model to capture a complex distribution of motion.

Original: Mathematically, it requires modeling directions and norms of the vector z s
Revised: Mathematically, it requires modeling directions and norms of the vector z s

Original: →
Revised: →

Original: d simultaneously, which is challenging.
Revised: d simultaneously, which is challenging.

Original: Therefore, instead of modeling motion transformation x s → x d , we assume there exists a reference image x r and motion transfer can be modeled as x s → x r → x d , where z s → d is learned in an indirect manner.
Revised: Therefore, instead of modeling motion transfor- mation x s → x d , we assume there exists a reference image x r and motion transfer can be modeled as x s → x r → x d , where z s → d is learned in an indirect manner.

Original: We model z s → d as a target point in the latent space, which can be reached by taking linear walks from a starting point z s → r along a linear path w r → d (see Fig. 2), given by
Revised: We model z s → d as a target point in the latent space, which can be reached by taking linear walks from a starting point z s → r along a linear path w r → d (see Fig. 2), given by

Original: [Equation]
Revised: [Equation]

Original: where z s → r and w r → d indicate the transformation x s → x r and x r → x d respectively.
Revised: where z s → r and w r → d indicate the transformation x s → x r and x r → x d respectively.

Original: Both z s → r and w r → d are learned independently and z s → r is obtained by passing x s through E .
Revised: Both z s → r and w r → d are learned independently and z s → r is obtained by passing x s through E .

Original: We learn w r → d via Linear Motion Decomposition (LMD).
Revised: We learn w r → d via Linear Motion Decomposition (LMD).

Original: Our idea is to learn a set of motion directions
Revised: Our idea is to learn a set of motion directions

Original: D m = { d 1 , ..., d M } to represent any path in the latent space.
Revised: D m = { d 1 , ..., d M } to represent any path in the latent space.

Original: We constrain D m as an orthogonal basis, where each vector indicates a motion direction d i .
Revised: We constrain D m as an orthogonal basis, where each vector indicates a motion direction d i .

Original: We then combine each vector in the basis with a vector A r → d = { a 1 , ..., a M } , where a i represents the magnitude of d i .
Revised: We then combine each vector in the basis with a vector A r → d = { a 1 , ..., a M } , where a i represents the magnitude of d i .

Original: Hence, any linear path in the latent space can be represented using a linear combination
Revised: Hence, any linear path in the latent space can be represented using a linear combination

Original: [Equation]
Revised: [Equation]

Original: where d i ∈ R N and a i ∈ R for all i ∈ { 1 , ..., M } .
Revised: where d i ∈ R N and a i ∈ R for all i ∈ { 1 , ..., M } .

Original: Semantically, each d i should represent a basic visual transformation and a i indicates the required steps to walk in d i towards achieving w r → d .
Revised: Semantically, each d i should represent a basic visual transformation and a i indicates the required steps to walk in d i towards achieving w r → d .

Original: Due to D m entailing an orthogonal basis, any two directions d i , d j follow the constrain
Revised: Due to D m entailing an orthogonal basis, any two directions d i , d j follow the constrain

Original: [Equation]
Revised: [Equation]

Original: We implement D m as a learnable matrix and apply the Gram-Schmidt process during each forward pass, in order to meet the requirement of orthogonality.
Revised: We implement D m as a learnable matrix and apply the Gram-Schmidt process during each forward pass, in order to meet the requirement of orthogonality.

Original: A r → d is obtained by mapping z d → r , which is the output of x d after E , through a 5-layer MLP.
Revised: A r → d is obtained by mapping z d → r , which is the output of x d after E , through a 5-layer MLP.

Original: The ﬁnal formulation of latent motion representation for each x s and x d is thus given as
Revised: The ﬁnal formulation of latent motion represen- tation for each x s and x d is thus given as

Original: [Equation]
Revised: [Equation]

Original: 3. L ATENT CODE DRIVEN IMAGE ANIMATION
Revised: 3. L ATENT CODE DRIVEN IMAGE ANIMATION

Original: Once we obtain z s → d , in our second step, we use G to decode a ﬂow ﬁeld φ s → d and warp x s .
Revised: Once we obtain z s → d , in our second step, we use G to decode a ﬂow ﬁeld φ s → d and warp x s .

Original: Our G consists of two components, a ﬂow ﬁeld generator G f and a reﬁnement network G r (we provide details in Appendix A).
Revised: Our G consists of two components, a ﬂow ﬁeld generator G f and a reﬁnement network G r (we provide details in App. A).

Original: Towards learning multi-scale features, G is designed as a residual network containing N models to produce a pyramid of ﬂow ﬁelds φ s → d = { φ i } N 1 in different layers of G f .
Revised: Towards learning multi-scale features, G is designed as a residual network containing N models to produce a pyramid of ﬂow ﬁelds φ s → d = { φ i } N 1 in different layers of G f .

Original: Multi-scale source features x encs =
Revised: Multi-scale source features x encs =

Original: { x enci } N 1 are obtained from E and are warped in G f .
Revised: { x enci } N 1 are obtained from E and are warped in G f .

Original: However, as pointed out by Siarohin et al.
Revised: However, as pointed out by Siarohin et al.

Original: (2019), only relying on φ s → d to warp source features is insufﬁcient to precisely reconstruct driving images due to the existing occlusions in some positions of x s .
Revised: (2019), only relying on φ s → d to warp source features is insufﬁcient to precisely reconstruct driving images due to the existing occlusions in some positions of x s .

Original: In order to predict pixels in those positions, the network is required to inpaint the warped feature maps.
Revised: In order to predict pixels in those positions, the network is required to inpaint the warped feature maps.

Original: Therefore, we predict multi-scale masks { m i } N 1 along with { φ i } N 1 in G f to mask out the regions required to be inpainted.
Revised: Therefore, we predict multi-scale masks { m i } N 1 along with { φ i } N 1 in G f to mask out the regions required to be inpainted.

Original: In each residual module, we have
Revised: In each residual module, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes the Hadamard product and T denotes warping operation,
Revised: where (cid:12) denotes the Hadamard product and T denotes warping operation,

Original: whereas x (cid:48)
Revised: whereas x (cid:48)

Original: i signiﬁes the masked features.
Revised: i signiﬁes the masked features.

Original: We generate both dense ﬂow ﬁelds, as well as masks by letting each residual module output a 3-channel feature map in which the ﬁrst two channels represent φ i and the last channel m i .
Revised: We generate both dense ﬂow ﬁelds, as well as masks by letting each residual module output a 3-channel feature map in which the ﬁrst two channels represent φ i and the last channel m i .

Original: Based on an inpainted feature map f ( x (cid:48) i ) , as well as an upsampled image g
Revised: Based on an inpainted feature map f ( x (cid:48) i ) , as well as an upsampled image g

Original: ( x i − 1 ) provided by the previous module in G r , the RGB image from each module is given by
Revised: ( x i − 1 ) provided by the previous module in G r , the RGB image from each module is given by

Original: [Equation]
Revised: [Equation]

Original:  where f and g denote the inpainting and upsampling layers, respectively.
Revised: (6) where f and g denote the inpainting and upsampling layers, respectively.

Original: The output image o N from the last module constitutes the ﬁnal generated image x s → d = o N .
Revised: The output image o N from the last module constitutes the ﬁnal generated image x s → d = o N .

Original: 3. L EARNING
Revised: 3. L EARNING

Original: We trainthe LIA- framework in a self-supervised manner to reconstruct x d using three losses, i.e., a reconstruction loss L recon , a perceptual loss L vgg (Johnson et al., 2016) and an adversarial loss L adv .
Revised: We train LIA in a self-supervised manner to reconstruct x d using three losses, i.e. , a reconstruction loss L recon , a perceptual loss L vgg (Johnson et al., 2016) and an adversarial loss L adv .

Original: We use L recon to minimize the pixel-wise L 1 distance between x d and x s → d , calculated as
Revised: We use L recon to minimize the pixel-wise L 1 distance between x d and x s → d , calculated as

Original: [Equation]
Revised: [Equation]

Original: Towards minimizing the perceptual distance, we apply a VGG19-based L vgg on multi-scale feature maps between real and generated images, written as
Revised: Towards minimizing the perceptual distance, we apply a VGG19-based L vgg on multi-scale feature maps between real and generated images, written as

Original: [Equation]
Revised: [Equation]

Original: where F n denotes the n th layer in a pre-trained VGG19 (Simonyan & Zisserman, 2015).
Revised: where F n denotes the n th layer in a pre-trained VGG19 (Simonyan & Zisserman, 2015).

Original: In practice, towards penalizing real and generated images in multi-scale images, we use a pyramid of four resolutions, namely 256 × 256 , 128 × 128 , 64 × 64 and 32 × 32 as inputs of VGG19.
Revised: In practice, towards penalizing real and generated images in multi-scale images, we use a pyramid of four resolutions, namely 256 × 256 , 128 × 128 , 64 × 64 and 32 × 32 as inputs of VGG19.

Original: The ﬁnal perceptual loss is the addition of perceptual losses in four resolutions.
Revised: The ﬁnal perceptual loss is the addition of perceptual losses in four resolutions.

Original: Further, towards generating photo-realistic results, we incorporate a non-saturating adversarial loss L adv on x s → d , which is calculated as
Revised: Further, towards generating photo-realistic results, we incorporate a non-saturating adversarial loss L adv on x s → d , which is calculated as

Original: [Equation]
Revised: [Equation]

Original: where D is a discriminator, aimed at distinguishing reconstructed  from  original images.
Revised: where D is a discriminator, aimed at distinguishing reconstructed images from the original ones.

Original: Our full loss function is the combination of three losses with λ as a balanced hyperparameter
Revised: Our full loss function is the combination of three losses with λ as a balanced hyperparameter

Original: [Equation]
Revised: [Equation]

Original: 3. I NFERENCE
Revised: 3. I NFERENCE

Original: In inference stage, given a driving video sequence V d = { x t } T 1 , we aim to transfer motion from V d to x s , in order to generate a novel video V
Revised: In inference stage, given a driving video sequence V d = { x t } T 1 , we aim to transfer motion from V d to x s , in order to generate a novel video V

Original: d
Revised: d

Original: → s = { x t → s } T 1 .
Revised: → s = { x t → s } T 1 .

Original: If V d and x s stem from the same video sequence, i.e. , x s = x 1 , our task comprises of reconstructing the entire original video sequence.
Revised: If V d and x s stem from the same video sequence, i.e. , x s = x 1 , our task comprises of reconstructing the entire original video sequence.

Original: Therefore, we construct the latent motion representation of each frame using absolute transfer , which follows the training process, given as
Revised: Therefore, we construct the latent motion representation of each frame using absolute transfer , which follows the training process, given as

Original: [Equation]
Revised: [Equation]

Original: [Equation] However, in real world applications, interest is rather placed on the scenario, where motion transfer between x s and V d , the latter stemming from different identities, i.e., x s (cid:54) = x 1 .
Revised: However, in real world applications, interest is rather placed on the scenario, where motion transfer between x s and V d , the latter stemming from different identities, i.e. , x s (cid:54) = x 1 .

Original: Taking a talking head video as an example, in this setting, beyond identity, x 1 and x s might also differ in pose and expression.
Revised: Taking a talking head video as an example, in this setting, beyond identity, x 1 and x s might also differ in pose and expression.

Original: Therefore, we propose relative transfer to eliminate the motion impact of w r → 1 and involve motion of w r → s in the full generated video sequence.
Revised: Therefore, we propose relative transfer to eliminate the motion impact of w r → 1 and involve motion of w r → s in the full generated video sequence.

Original: Owing to a linear representation of the latent path, we can easily represent z s → t for each frame as
Revised: Owing to a linear representation of the latent path, we can easily represent z s → t for each frame as

Original: [Equation]
Revised: [Equation]

Original: In the ﬁrst part of Eq.
Revised: The ﬁrst term in Eq.

Original: (12), z s → s indicates the reconstruction of x s , while the second part ( w r → t − w r → 1 ) represents the motion from x 1 to x t .
Revised: (12), z s → s indicates the reconstruction of x s , while the second term ( w r → t − w r → 1 ) represents the motion from x 1 to x t .

Original: This equation indicates that the original pose is preserved in x s , at the same time motion is transferred from V d .
Revised: This equation indicates that the original pose is preserved in x s , at the same time motion is transferred from V d .

Original: We note that in order to completely replicate the position and pose in V d , relative motion transfer requires x s and x 1 to contain similar poses, else face will have as starting image the pose from x s rather than x 1 .
Revised: We note that in order to completely replicate the position and pose in V d , it requires x s and x 1 to contain similar poses in relative motion transfer.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrstly describe our experimental setup including implementation details and datasets.
Revised: In this section, we ﬁrstly describe our experimental setup including implementation details and datasets.

Original: Secondly, we qualitatively demonstrate generated results based on testing datasets.
Revised: Secondly, we qualitatively demonstrate generated results based on testing datasets.

Original: Then, we provide quantitative evaluation w.r.t.
Revised: Then, we provide quantitative evaluation w.r.t.

Original: image quality on (a) same-identity reconstruction, (b) crossvideo motion transfer, presenting (c) a user study.
Revised: image quality on (a) same-identity reconstruction, (b) crossvideo motion transfer, presenting (c) a user study.

Original: Next, we conduct an ablation study that demonstrates (d) the effectiveness of our proposed motion dictionary, as well as (e) associated size.
Revised: Next, we conduct an ablation study that demon- strates (d) the effectiveness of our proposed motion dictionary, as well as (e) associated size.

Original: Finally, we provide an in-depth analysis of our (f) latent codes and (g) motion dictionary to interpret their semantic meanings.
Revised: Finally, we provide an in-depth analysis of our (f) latent codes and (g) motion dictionary to interpret their semantic meanings.

Original: Datasets Our model is trained on the datasets VoxCeleb (Chung et al., 2018), TaichiHD(Siarohin et al., 2019) and TED-talk (Siarohin et al., 2021).
Revised: Datasets Our model is trained on the datasets VoxCeleb, TaichiHD and TED-talk.

Original: We follow the pre-processing method in (Siarohin et al., 2019) to crop frames into 256 × 256 resolution for quantitative evaluation.
Revised: We follow the pre-processing method in (Siarohin et al., 2019) to crop frames into 256 × 256 resolution for quantitative evaluation.

Original: In addition, we process a 512 × 512 version of VoxCeleb towards qualitatively demonstrating the capacity of method on high resolution generation (see Appendix B.1).

Original: Implementation details Our model is implemented in PyTorch (Paszke et al., 2019).
Revised: Implementation details Our model is implemented in PyTorch (Paszke et al., 2019).

Original: All models are trained on four 16G NVIDIA V100 GPUs.
Revised: All models are trained on four 16G NVIDIA V100 GPUs.

Original: The total batch size is 32 with 8 images per GPU.
Revised: The total batch size is 32 with 8 images per GPU.

Original: We use a learning rate of 0.002 to train our model with the Adam optimizer (Kingma & Ba, 2014).
Revised: We use a learning rate of 0.002 to train our model with the Adam optimizer (Kingma & Ba, 2014).

Original: The dimension of all latent codes, as well as directions in D m are set to be 512.
Revised: The dimension of all latent codes, as well as directions in D m is set to be 512.

Original: In our loss function, we use λ = 10 in order to penalize more on the perceptual loss.
Revised: In our loss function, we use λ = 10 in order to penalize more on the perceptual loss.

Original: It takes around 150 hours to fully train our framework.
Revised: It takes around 150 hours to fully train our framework.

Original: Evaluation metrics We evaluate our model w.r.t.
Revised: Evaluation metrics We evaluate our model w.r.t.

Original: (i) reconstruction faithfulness using L 1 , LPIPS, (ii) generated video quality using video FID, as well as (iii) semantic consistency using average keypoint distance (AKD), missing keypoint rate (MKR) and average euclidean distance (AED).
Revised: (i) reconstruction faithfulness using L 1 , LPIPS, (ii) generated video quality using video FID, as well as (iii) semantic consistency using average keypoint distance (AKD), missing keypoint rate (MKR) and average euclidean distance (AED).

Original: Details are available in Appendix B.2.
Revised: Details are available in App. B.2.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: Firstly, we evaluate the ability of LIA to generate realistic videos and compare related results with four state-of-the-art methods.
Revised: Firstly, we evaluate the ability of LIA to generate realistic videos and compare related results with four state-of-the-art methods.

Original: For TaichiHD and TED-talk datasets, we conduct an experiment related to cross-video generation.
Revised: For TaichiHD and TED-talk datasets, we conduct an experiment related to cross-video generation.

Original: Corresponding results (see Fig. 4) conﬁrm that our method is able to correctly transfer motion on articulated human bodies, in the absence of explicit structure representations.
Revised: Corresponding results (see Fig. 4) conﬁrm that our method is able to correctly transfer motion on articulated human bodies, in the absence of explicit structure representations.

Original: For the VoxCeleb dataset, we conduct a cross-dataset generation-experiment, where we transfer motion from VoxCeleb to images of the FFHQ dataset (Karras et al., 2019).
Revised: For the VoxCeleb dataset, we conduct a cross-dataset generation-experiment, where we transfer motion from VoxCeleb to images of the FFHQ dataset.

Original: We observe that our method outperforms FOMM and MRAA w.r.t.
Revised: We observe that our method outperforms FOMM and MRAA w.r.t.

Original: image quality, as both approaches  deform the shape of  original faces visibly.
Revised: image quality, as both approaches visibly deform the shape of the original faces.

Original: This is speciﬁcally notable in the case that source and driving images entail large pose variations.
Revised: This is speciﬁcally notable in the case that source and driving images entail large pose variations.

Original: At the same time, LIA is able to successfully tackle this challenge and no similar deformations are visible.
Revised: At the same time, LIA is able to successfully tackle this challenge and no similar deformations are visible.

Original: 4. C OMPARISON WITH S TATE - OF - THE - ART M ETHODS
Revised: 4. C OMPARISON WITH S TATE - OF - THE - ART M ETHODS

Original: We quantitatively compare our method with the state-of-art approaches X2Face, Monkey-Net, FOMM and MRAA on two tasks, namely (a) same-identity reconstruction and (b) cross-video motion transfer.
Revised: We quantitatively compare our method with the state-of-the-art approaches X2Face, Monkey-Net, FOMM and MRAA on two tasks, namely (a) same-identity reconstruction and (b) cross-video motion transfer.

Original: Additionally, we conduct a (c) user study.
Revised: Additionally, we conduct a (c) user study.

Original: (a) Same-identity reconstruction We here evaluate the reconstruction ability of our method.
Revised: (a) Same-identity reconstruction We here evaluate the reconstruction ability of our method

Original: Speciﬁcally, we reconstruct each testing video by using the ﬁrst frame as x s and the remaining frames as x d . Results on three datasets are reported in Table 1.
Revised: Speciﬁcally we reconstruct each testing video by using the ﬁrst frame as x s and the remain- ing frames as x d . Results on three datasets are reported in Table 1.

Original: Focusing on foregroundreconstruction, our method outperforms the other approaches w.r.t.
Revised: Focusing on foreground- reconstruction, our method outperforms the other approaches w.r.t.

Original: all metrics.
Revised: all metrics.

Original: More results are presented in Appendix B.3, discussing background-reconstruction.
Revised: More results are presented in App. B.3, discussing background-reconstruction.

Original: [Table] ) Cross-video motion transfer Next, we conduct experiments, where source images and driving videos stem from different video sequences.
Revised: (b) Cross-video motion transfer Next, we conduct experiments, where source images and driving videos stem from different video sequences.

Original: In this context, we mainly focus on evaluating talking [Table] [Table] [Table]
Revised: In this context, we mainly focus on evaluating talking head videos and explore two different cases.

Original: [Table]

Original: head videos and explore two different cases.

Original: In the ﬁrst case, we generate videos using the VoxCeleb testing set to conduct motion transfer .
Revised: In the ﬁrst case, we generate videos using the VoxCeleb testing set to conduct motion transfer .

Original: In the second case, source images are from an unseen dataset, namely the GermanAudio dataset (Thies et al., 2020), as we conduct cross-dataset motion transfer .
Revised: In the second case, source images are from an unseen dataset, namely the GermanPublicTV dataset, as we conduct cross-dataset motion transfer .

Original: In both experiments, we randomly construct source and driving pairs and transfer motion from driving videos to source images to generate a novel manipulated dataset.
Revised: In both experiments, we randomly construct source and driving pairs and transfer motion from driv- ing videos to source images to generate a novel manipulated dataset.

Original: Since ground truth data for our generated videos is not available, we use video FID (as initialized by Wang et al.
Revised: Since ground truth data for our generated videos is not available, we use video FID (as initialized by Wang et al.

Original: (2020b))
Revised: (2020a))

Revised: to

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: to compute the distance between generated and real data distributions.
Revised: compute the distance between generated and real data distributions.

Original: As shown in Tab. 2, our method outperforms all other approaches w.r.t.
Revised: As shown in Tab. 2, our method outperforms all other approaches w.r.t.

Original: video FID, indicating the best generated video quality.
Revised: video FID, indicating the best generated video quality.

Original: (c) User study We conduct a user study to evaluatesubjective video quality.
Revised: (c) User study We conduct a user study to evaluate video quality.

Original: Towards this, we displayed paired videos and asked 20 human raters ‘which clip is more realistic’.
Revised: Towards this, we displayed paired videos and asked 20 human raters ‘which clip is more realistic?’.

Original: Each videopair contains a generated video from our method, as well as a video generated from FOMM or MRAA.
Revised: Each video-pair contains a generated video from our method, as well as a video generated from FOMM or MRAA.

Original: Results suggest that our results are more realistic in comparison to FOMM and MRAA across all three datasets (see Tab. 3).
Revised: Results suggest that our results are more realistic in comparison to FOMM and MRAA across all three datasets (see Tab. 3).

Original: Hence, the obtained human preference is in accordance with our quantitative evaluation. 
Revised: Hence, the obtained human preference is in accordance with our quantitative evaluation. [Table] [Table]

Original: 4. A BLATION STUDY
Revised: 4. A BLATION STUDY

Original: We here analyze our proposed motion dictionary and focus on answering following two questions.
Revised: We here analyze our proposed motion dictionary and focus on answering following two questions.

Original: (d) Is the motion dictionary D m beneﬁcial?
Revised: (d) Is the motion dictionary D m beneﬁcial?

Original: We here explore the impact of proposed D m , by training our model without D m .
Revised: We here explore the impact of proposed D m , by training our model without D m .

Original: Speciﬁcally, we output w r → d directly from MLPs, without using LMD to learn an orthogonal basis.
Revised: Speciﬁcally, we output w r → d directly from MLP, without using LMD to learn an orthogonal basis.

Original: From the evaluation results reported in Tab. 4, we observe that in the absence of D m , LIA fails to generate high-quality images, which proves the effectiveness of D m , consistently on all datasets.We show qualitative results in Appendix B.7.
Revised: From the evaluation results reported in Tab. 4 and qualitative results in App. B.5, we observe that in the absence of D m , model fails to generate high-quality images, which proves the effectiveness of D m , consistently on all datasets.

Original: (e) How many directions are required in D m ?
Revised: (e) How many directions are required in D m ?

Original: Towards ﬁnding an effective size of D m , we empirically test three different M , viz.
Revised: Towards ﬁnding an effective size of D m , we empirically test three different M , viz.

Original: 5, 10, 20, 40 and 100.
Revised: 5, 10, 20, 40 and 100.

Original: Quantitative results in Tab.
Revised: Quantitative results in Tab.

Original: 5 show that when using 20 directions, we obtain the best reconstruction results.
Revised: 5 show that when using 20 directions, the model achieves the best reconstruction results.

Original: 4. F URTHER ANALYSIS
Revised: 4. F URTHER ANALYSIS

Original: (f) Latent code analysis.
Revised: (f) Latent code analysis.

Original: While our method successfully transfers motion via latent space navigation, we here aim at answering the question — what does x r represent ?
Revised: While our method successfully transfers motion via latent space navi- gation, we here aim at answering the question — what does x r represent ?

Original: Towards answering this question, we proceed to visualize x r .
Revised: Towards answering this question, we proceed to visualize x r .

Original: We ﬁrstly decode z s → r into a dense ﬂow ﬁeld φ s → r , which  then warps xs .
Revised: We ﬁrstly decode z s → r into a dense ﬂow ﬁeld φ s → r , which is then used to warp x

Revised: s (we show details in App.

Revised: B.4).

Original: Fig.
Revised: Fig.

Original: 5 shows examples of x s and x r .
Revised: 5 shows examples of x s and x r .

Original: Interestingly, we observe that x r represents the canonical pose of x s , regardless of original poses of the subjects.
Revised: Interestingly, we observe that x r represents the canonical pose of x s , regardless of original poses of the subjects.

Original: And for all datasets, reference images resemble each other w.r.t.
Revised: And for all datasets, reference images resemble each other w.r.t.

Original: pose and scale.
Revised: pose and scale.

Original: As such reference images can be considered as a normalized form of x s , learning transformations between x s and x d using x s → x r → x d is considerably more efﬁcient than x s → x d , once x r is ﬁxed.
Revised: As such reference images can be considered as a normalized form of x s , learning transformations between x s and x d using x s → x r → x d is considerably more efﬁcient than x s → x d , once x r is ﬁxed.

Original: Noteworthy, we found a similar idea of learning a ‘reference image’  explored by Siarohin et al.
Revised: Noteworthy, we found the similar idea of learning a ‘reference image’ has also been explored by Siarohin et al.

Original: (2019) (FOMM) andby Wiles et al.
Revised: (2019) (FOMM) and Wiles et al.

Original: (X2Face).
Revised: (X2Face).

Original: However, deviating from our visualized ‘reference image’, in FOMMit refers to a non-visualized and abstract concept.
Revised: However, deviating from our visualized ‘reference image’, the ’reference image ’in FOMM refers to a non-visualized and ab- stract concept.

Original: In contrast to X2Face, we only require a latent code z s → r , rather than its explicit form φ s → r and x r during both, training and 
Revised: In addition, LIA only requires a latent code z s → r , rather than the ’reference image’ for both, training and testing, which is contrast to X2Face.

Original: (g) Motion dictionary interpretation.
Revised: (g) Motion dictionary interpretation.

Original: Towards further interpretation of directions in D m , we conduct linear manipulations on each d i .
Revised: Towards further interpretation of directions in D m , we conduct linear manipulations on each d i .

Original: Details of the analysis are provided in Appendix B.5.

Original: Images pertained to manipulating four motion directions are depicted in Fig.
Revised: Images pertained to manipulating four motion directions are depicted in Fig.

Original: The results suggest that the directions in D m are semantically meaningful, as they represent basic visual transformations such as head nodding ( d 8 ), eye movement ( d 6 ) and facial expressions ( d 19 and d 7 ).
Revised: The results suggest that the directions in D m are semantically meaningful, as they represent basic visual transformations such as head nodding ( d 8 ), eye movement ( d 6 ) and facial expressions ( d 19 and d 7 ).

Revised: More results can be found on our project webpage 2 .

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we presented a novel self-supervised auto-encoder LIA, aimed at animating images via latent space navigation.
Revised: In this paper, we presented a novel self-supervised autoencoder LIA, aimed at animating images via latent space navigation.

Original: By the proposed Linear Motion Decomposition (LMD), we were able to formulate the task of transferring motion from driving videos to source images as learning linear transformations in the latent space.
Revised: By the proposed Linear Motion Decomposition (LMD), we were able to formulate the task of transferring motion from driving videos to source images as learning lin- ear transformations in the latent space.

Original: We evaluated proposed method on real-world videos and demonstrated that our approach is able to successfully animate still images, while eliminating the necessity of explicit structure representations .
Revised: We evaluated proposed method on real-world videos and demonstrated that our approach is able to successfully animate still images, while eliminating the necessity of explicit structure representations .

Original: In addition, we showed that the incorporated motion dictionary is interpretable and contains directions pertaining to basic visual transformations.
Revised: In addition, we showed that the incorporated motion dictionary is interpretable and contains directions pertaining to basic visual transformations.

Original: Both quantitative and qualitative evaluations showed that LIA outperforms state-of-art algorithms on all benchmarks.
Revised: Both quantitative and qualitative evaluations showed that LIA outperforms state-of-art algorithms on all benchmarks.

Original: We postulate that LIA opens a new door in design of interpretable generative models for video generation.
Revised: We postulate that LIA opens a new door in design of interpretable generative models for video generation.

Revised: E THIC S TATEMENT

Revised: In this work we aim to synthesize high-quality videos by transferring motion on still images.

Revised: Our approach can be used for movie production, making video games, online education, generating synthetic data for other computer vision tasks etc.

Original: Public release.
Revised: We note that our framework mainly focuses on learning how to model motion distribution rather than directly model appearance, therefore it is not biased towards any speciﬁc gender, race, region, or social class. It works equally well irrespective of the difference in subjects. R EPRODUCIBILITY S TATEMENT We assure that all the results shown in the paper and supplemental materials can be reproduced.

Original: We intend to release oursource code, as well as trained models.
Revised: We intend to open-source our code, as well as trained models.

Original: Moreover, we will release a dataset of videos, generated by LIA, in order to facilitate research on deepfake detection.
Revised: This work was granted access to the HPC resources of IDRIS under the allocation AD011011627R1. It was supported by the French Government, by the National Research Agency (ANR) under Grant ANR-18-CE92-0024, project RESPECT and through the 3IA C ˆ ote d’Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19P3IA-0002.

Original: A D ETAILS OF MODEL ARCHITECTURE
Revised: A D ETAILS OF MODEL ARCHITECTURE

Original: We proceed to describe the model architecture in this section.
Revised: We proceed to describe the model architecture in this section.

Original: Fig.
Revised: Fig.

Original: 7 shows details of our E .
Revised: 7 shows details of our E .

Original: In each ResBlock in E , spatial size of input feature maps are downsampled.
Revised: In each ResBlock in E , spatial size of input feature maps are downsampled.

Original: We take feature maps of spatial sizes from 8 × 8 to 256 × 256 as our appearance features x enci .
Revised: We take feature maps of spatial sizes from 8 × 8 to 256 × 256 as our appearance features x enci .

Original: We use a 5-layer MLP to predict a magnitude vector A d → r from z d → r . Fig.
Revised: We use a 5-layer MLP to predict a magnitude vector A d → r from z d → r . Fig.

Original: 8 (a) shows the general architecture of our G , which consists of two components, a ﬂow ﬁeld generator G f and a reﬁnement network G r .
Revised: 8 (a) shows the general architecture of our G , which consists of two components, a ﬂow ﬁeld generator G f and a reﬁnement network G r .

Original: We apply StyleConv (Upsample + Conv 3 × 3 ), which is proposed by StyleGAN2, in G f .
Revised: We apply StyleConv (Upsample + Conv 3 × 3 ), which is proposed by StyleGAN2, in G f .

Original: StyleConv takes latent representation z s → t as style code and generates ﬂow ﬁeld φ i and corresponding mask
Revised: StyleConv takes latent representation z s → t as style code and generates ﬂow ﬁeld φ i and corresponding mask

Original: m i .
Revised: m i .

Original: G r uses UpConv (Conv 1 × 1 + Upsample) to upsample and reﬁne inpainted feature maps to target resolution.
Revised: G r uses UpConv (Conv 1 × 1 + Upsample) to upsample and reﬁne inpainted feature maps to target resolution.

Original: We show details pertaining to G block in Fig.
Revised: We show details pertaining to G block in Fig.

Original: 8 (b).
Revised: 8 (b).

Original: Each G block is used to upsample × 2 the previous resolution.
Revised: Each G block is used to upsample × 2 the previous resolution.

Original: We stack 6 blocks towards producing 256 resolution images.
Revised: We stack 6 blocks towards producing 256 resolution images.

Original: B E XPERIMENTS
Revised: B E XPERIMENTS

Original: We proceed to introduce details of datasets and evaluation metrics used in our experiments.
Revised: We proceed to introduce details of datasets and evaluation metrics used in our experiments

Original:  B. D ATASETS
Revised: . B. D ATASETS

Original: VoxCeleb (Nagrani et al., 2019) consists of a large amount of interview videos of different celebrities.
Revised: VoxCeleb (Nagrani et al., 2019) consists of a large amount of interview videos of different celebri- ties.

Original: Following the process of FOMM (Siarohin et al., 2019), we extract frames and crop them into 256 × 256 resolution.
Revised: Following the process of FOMM (Siarohin et al., 2019), we extract frames and crop them into 256 × 256 resolution.

Original: In total, VoxCeleb contains a training set of 17928 videos and a test set ofvideos.
Revised: In total, VoxCeleb contains a training set of 17928 videos and a test set ofvideos.

Original: TaiChiHD (Siarohin et al., 2019) consists of videos of full human bodies performing Tai Chi actions.
Revised: TaiChiHD (Siarohin et al., 2019) consists of videos of full human bodies performing Tai Chi actions.

Original: We follow the original pre-processing of FOMM (Siarohin et al., 2019) and utilize its 256 ×version.
Revised: We follow the original pre-processing of FOMM (Siarohin et al., 2019) and utilize its 256 ×version.

Original: TaiChiHD contains 1096 training videos and 115 testing videos.
Revised: TaiChiHD contains 1096 training videos and 115 testing videos.

Original: TED-talk is a new dataset proposed in MRAA (Siarohin et al., 2021).
Revised: TED-talk is a new dataset proposed in MRAA (Siarohin et al., 2021).

Original: It comprises a number of TED-talk videos, where the main subjects have been cropped out.
Revised: It comprises a number of TED-talk videos, where the main subjects have been cropped out.

Original: We resize the original version into 256 × 256 resolution to train our model.
Revised: We resize the original version into 256 × 256 resolution to train our model.

Original: This dataset includes 1124 training videos andtesting videos.
Revised: This dataset includes 1124 training videos andtesting videos.

Original: B. E VALUATION METRICS
Revised: B. E VALUATION METRICS

Original: We use ﬁve different metrics to evaluate our experimental results, namely L 1 , LPIPS, AKD, MKR and AED that quantify the reconstructed results.
Revised: We use ﬁve different metrics to evaluate our experimental results, namely L 1 , LPIPS, AKD, MKR and AED that quantify the reconstructed results.

Original: In addition, we compute video FID to evaluate video quality in motion transferring tasks.
Revised: In addition, we compute video FID to evaluate video quality in motion transferring tasks.

Original: Lrepresents the mean absolute pixel difference between reconstructed and real videos.
Revised: L 1 represents the mean absolute pixel difference between reconstructed and real videos.

Original: LPIPS (Zhang et al., 2018) aims at measuring the perceptual similarity between reconstructed and real images by leveraging the deep features from AlexNet (Krizhevsky et al., 2012).
Revised: LPIPS (Zhang et al., 2018) aims at measuring the perceptual similarity between reconstructed and real images by leveraging the deep features from AlexNet (Krizhevsky et al., 2012).

Original: Video FID is a modiﬁed version of  original FID (Heusel et al., 2017).
Revised: Video FID is a modiﬁed version of the original FID (Heusel et al., 2017).

Original: We here follow the same implementation as Wang et al.
Revised: We here follow the same implementation as Wang et al.

Original: (2020b)
Revised: (2020a)

Original: and utilize a pre-trained ResNext101 (Hara et al., 2018) to extract spatio-temporal features to compute the distance between real and generated videos distributions.
Revised: and utilize a pre-trained ResNext101 (Hara et al., 2018) to extract spatio-temporal features to compute the distance between real and generated videos distributions.

Original: We take the ﬁrst 100 frames of each video as input of the feature-extractor to compute the ﬁnal scores.
Revised: We take the ﬁrst 100 frames of each video as input of the feature-extractor to compute the ﬁnal scores.

Original: Average keypoint distance (AKD) and missing keypoint rate (MKR) evaluate the difference between keypoints of reconstructed and ground truth videos.
Revised: Average keypoint distance (AKD) and missing keypoint rate (MKR) evaluate the difference between keypoints of reconstructed and ground truth videos.

Original: We extract landmarks using the face alignment approach of (Bulat & Tzimiropoulos, 2017) and extract body poses for both TaiChiHD and TED-talks using OpenPose (Cao et al., 2019).
Revised: We extract landmarks using the face alignment approach of (Bulat & Tzimiropoulos, 2017) and extract body poses for both TaiChiHD and TED-talks using OpenPose (Cao et al., 2019).

Original: AKD is computed as the average distance between corresponding keypoints, whereas MKR is the proportion of keypoints present in the groundtruth that are missing in a reconstructed video.
Revised: AKD is computed as the average distance be- tween corresponding keypoints, whereas MKR is the proportion of keypoints present in the groundtruth that are missing in a reconstructed video.

Original: Average Euclidean distance (AED) measures the ability of preserving identity in reconstructed video.
Revised: Average Euclidean distance (AED) measures the ability of preserving identity in reconstructed video.

Original: We use a person re-identiﬁcation pretrained model (Zheng et al., 2020) for measuring human bodies (TaichiHD and TED-talk) and OpenFace (Amos et al., 2016) for faces to extract identity embeddings from reconstructed and ground truth frame pairs, then we compute MSE of their difference for all pairs.
Revised: We use a person re-identiﬁcation pretrained model (Zheng et al., 2020) for measuring human bodies (TaichiHD and TED-talk) and OpenFace (Amos et al., 2016) for faces to extract identity embeddings from reconstructed and ground truth frame pairs, then we compute MSE of their difference for all pairs.

Original: B. C OMPARISON WITH FULL MRAA
Revised: B. C OMPARISON WITH FULL MRAA

Original: We show quantitative evaluation results with the full MRAA model in Tab. 6.
Revised: We show quantitative evaluation results with the full MRAA model in Tab. 6.

Original: We observe that our method achieves competitive results in reconstruction and keypoint evaluation.
Revised: We observe that our method achieves competitive results in reconstruction and keypoint evaluation.

Original: W.r.t. the TaichiHD dataset, while we do not explicitly predict keypoints,  interestingly we outperform MRAA in both, AKD and MKR.
Revised: While we do not explicitly predict keypoints, w.r.t. the TaichiHD dataset, interestingly we outperform MRAA in both, AKD and MKR.

Original: Such results showcase the effectiveness of our proposed method on modeling articulated human structures.
Revised: Such results showcase the effectiveness of our proposed method on modeling articulated human structures.

Original: However, reconstruction evaluation cannot provide a completely fair comparison on how well the main subjects ( e.g., faces and human bodies) are generated in videos.
Revised: However, reconstruction evaluation cannot provide a completely fair comparison on how well the main subjects ( e.g., faces and human bodies) are generated in videos.

Original: This is in particular the case for TaichiHD and TED-talk, where backgrounds have large contributions to the ﬁnal scores.
Revised: This is in particular the case for TaichiHD and TED-talk, where backgrounds have large contribu- tions to the ﬁnal scores.

Original: [Table]
Revised: [Table]

Original: To produce x r , we use G to decode z s → r into the ﬂow ﬁeld φ s → r .

Original: Reference image x r is obtained by warping x s using φ s → r .

Original: The entire process is shown in Fig.

Original: B. M ANIPULATION IN MOTION DICTIONARY
Revised: B. R EFERENCE IMAGE GENERATION .

Original: Towards interpreting D m , we ﬁrstly obtain z s → r and A r → s from the encoder by reconstructing x s . From Eq. (4), we obtain [Equation]
Revised: To produce x r

Original: [Equation]

Original: Our interpretation is based on linearly manipulating each d

Original: i in D m .

Original: We compute a min and a max for each d i for reconstructing the training data and use them as a manipulation range [ a min , a max ] .

Original: The manipulation step size is computed as

Original: [Equation]
Revised: [Table]

Original: where n indicates the total steps. The manipulated magnitude is then given by [Equation] We use a (cid:48) to replace a i .
Revised: we use G to decode z s → r into the ﬂow ﬁeld φ s → r .

Original: Hence the latent code for the the manipulated image  is given by [Equation] [Equation] [Equation]
Revised: Reference image x r is obtained by warping x s using φ s → r .

Original: When manipulating each d i , we eliminate the contributions of other directions by setting

Original: [Equation]

Original: We show more manipulation results in Fig.
Revised: The entire process is shown in Fig.

Original: B. Q UALITATIVE RESULTS ON HIGH RESOLUTION GENERATION

Original: We show generated results on 512 × 512 resolution for talking head images.

Original: We train LIA using

Original: VoxCeleb, with resolution 512 × 512 and show motion transfer results using a real-world driving video in Fig.

Original: B. Q UALITATIVE RESULTS ON EFFECTIVENESS OF USING MOTION DICTIONARY
Revised: B. Q UALITATIVE RESULTS ON EFFECTIVENESS OF USING MOTION DICTIONARY

Original: Fig.
Revised: Fig.

Original: 12 illustrates the generated results on transferring motion from VoxCeleb to GermanAudio with and without motion dictionary.
Revised: 10 illustrates the generated results on transferring motion from VoxCeleb to GermanPublicTV with and without motion dictionary.

Original: We observe that without the motion dictionary, appearance information is undesirably transferred from driving videos to generated videos
Revised: We observe that without the motion dictionary, appearance information is undesirably transferred from driving videos to generated videos.

Original: . B. O RTHOGONALITY ANALYSIS IN MOTION DICTIONARY .

Original: We provided comparison in Tab.

Original: 7 to analyze the effectiveness of orthogonality in the motion dictionary.

Original: In non-orthogonality experiments, we randomly initialize 20 directions without using GramSchmidt process.

Original: Tab.

Original: 7 shows that models using orthogonal basis perform better than the ones using non-orthogonal basis.

Original: To further analyze the correlation between each direction after training, we also computed angles between direction pairs in the motion dictionary for three datasets and show results in Fig.

Original: 13, Fig.

Original: 14 and Fig.

Original: We observe that most of the directions are nearly orthogonal after training and angles are 90 ◦ ± 10 ◦ .

Original: Therefore we consider on current three training datasets that employing an orthogonal basis is effective.

Original: However, there are other factors that may affect such choice, e.g., complexity of the datasets and latent code dimension.

Original: It remains an open question whether orthogonality is a general solution for all cases or just in our context.

Original: B. L IMITATIONS
Revised: B. L IMITATIONS

Original: For human body, one limitation of our method is dealing with body occlusion.
Revised: For human body, one limitation of our method is dealing with body occlusion.

Original: We observe in Fig.
Revised: We observe in Fig.

Original: 16 that in taichi videos, in case of occlusion cause by legs and arms, motion is not transferred successfully.
Revised: 11 that in taichi videos, in case of occlusion cause by legs and arms, motion is not transferred successfully.

Original: In addition, in TED-talks, transferring hand motion is challenging, as hands are of small size, articulated and sometimes occluded by human bodies.[Table] C E THIC S TATEMENT
Revised: In addition, in TED-talks, transferring hand motion is challenging, as hands are of small size, articulated and sometimes occluded by human bodies.

Original: In this work, we aim to synthesize high-quality videos by transferring motion on still images.

Original: Our approach can be used for movie production, making video games, online education, generating synthetic data for other computer vision tasks, etc.

Original: We note that our framework mainly focuses on learning how to model motion distribution rather than directly model appearance, therefore it is not biased towards any speciﬁc gender, race, region, or social class.

Original: It works equally well irrespective of the difference in subjects.