Original: H UMAN M OTION F ORMER : T RANSFERRING H UMAN M OTIONS WITH V ISION T RANSFORMERSAnonymous authors
Revised: H UMAN M OTION F ORMER : T RANSFERRING H UMAN M OTIONS WITH V ISION T RANSFORMERS

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Human motion transfer aims to transfer motions from a target dynamic person to a source static one for motion synthesis.
Revised: Human motion transfer aims to transfer motions from a target dynamic person to a source static one for motion synthesis.

Original: An accurate matching between the source person and the target motion in both large and subtle motion changes is vital for improving the transferred motion quality.
Revised: An accurate matching between the source person and the target motion in both large and subtle motion changes is vital for improving the transferred motion quality.

Original: In this paper, we propose Human MotionFormer, a hierarchical ViT framework that leverages global and local perceptions to capture large and subtle motion matching, respectively.
Revised: In this paper, we propose Human MotionFormer, a hierarchical ViT framework that leverages global and local perceptions to capture large and subtle motion matching, respectively.

Original: It consists of two ViT encoders to extract input features (i e., a target motion image and a source human image) and a ViT decoder with several cascaded blocks for feature matching and motion transfer.
Revised: It consists of two ViT encoders to extract input features (i.e., a target motion image and a source human image) and a ViT decoder with several cascaded blocks for feature matching and motion transfer.

Original: In each block, we set the target motion feature as Query and the source person as Key and Value, calculating the cross-attention maps to conduct a global feature matching.
Revised: In each block, we set the target motion feature as Query and the source person as Key and Value, calculating the crossattention maps to conduct a global feature matching.

Original: Further, we introduce a convolutional layer to improve the local perception after the global cross-attention computations.
Revised: Further, we introduce a convolutional layer to improve the local perception after the global cross-attention computations.

Original: This matching process is implemented in both warping and generation branches to guide the motion transfer.
Revised: This matching process is implemented in both warping and generation branches to guide the motion transfer.

Original: During training, we propose a mutual learning loss to enable the co-supervision between warping and generation branches for better motion representations.
Revised: During training, we propose a mutual learning loss to enable the co-supervision between warping and generation branches for better motion representations.

Original: Experiments show that our Human MotionFormer sets the new state-of-the-art performance both qualitatively and quantitatively.
Revised: Experiments show that our Human MotionFormer sets the new state-of-the-art performance both qualitatively and quantitatively.

Revised: Project page: https://github.com/KumapowerLIU/

Revised: Human-MotionFormer

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Human Motion Transfer, which transfers the motion from a target person’s video to a source person, has grown rapidly in recent years due to its substantial entertaining applications for novel content generation Wang et al.
Revised: Human Motion Transfer, which transfers the motion from a target person’s video to a source person, has grown rapidly in recent years due to its substantial entertaining applications for novel content generation Wang et al.

Original: ; Chan et al.
Revised: ; Chan et al.

Original: For example, a dancing target automatically animates multiple static source people for efficient short video editing.
Revised: For example, a dancing target automatically animates multiple static source people for efficient short video editing.

Original: Professional actions can be transferred to celebrities to produce educational, charitable, or advertising videos for a wide range of broadcasting.
Revised: Professional actions can be transferred to celebrities to produce educational, charitable, or advertising videos for a wide range of broadcasting.

Original: Bringing static people alive suits short video creation and receives growing attention on social media platforms.
Revised: Bringing static people alive suits short video creation and receives growing attention on social media platforms.

Original: During motion transfer, we expect the source person to redo the same action as the target person.
Revised: During motion transfer, we expect the source person to redo the same action as the target person.

Original: To achieve this purpose, we need to establish an accurate matching between the target pose and the source person (i.e., each body part skeleton in a pose image matches its corresponding body part in a source image), and use this matching to drive the source person with target pose (i.e., if the hand skeleton in the target pose is raised, the hand in the source person should also be raised).
Revised: To achieve this purpose, we need to establish an accurate matching between the target pose and the source person (i.e., each body part skeleton in a pose image matches its corresponding body part in a source image), and use this matching to drive the source person with target pose (i.e., if the hand skeleton in the target pose is raised, the hand in the source person should also be raised).

Original: According to the degree of difference between the target pose and the source person pose, this matching can be divided into two types: global and local .
Revised: According to the degree of difference between the target pose and the source person pose, this matching can be divided into two types: global and local .

Original: When the degree is large, there is a large motion change between the target pose and the source person, and the target pose shall match a distant region in the source image (e.g., the arm skeleton of the target pose is distant from the source man arm region in Fig. 1(b)).
Revised: When the degree is large, there is a large motion change between the target pose and the source person, and the target pose shall match a distant region in the source image (e.g., the arm skeleton of the target pose is distant from the source man arm region in Fig. 1(b)).

Original: When the degree is small, there are only subtle motion changes, and the target pose shall match its local region in the source image (e.g., the arm skeleton of the target pose is close to the source woman arm region in Fig. 1(b)).
Revised: When the degree is small, there are only subtle motion changes, and the target pose shall match its local region in the source image (e.g., the arm skeleton of the target pose is close to the source woman arm region in Fig. 1(b)).

Original: As the human body moves non-rigidly, large and subtle  motion changes usually appear simultaneously.
Revised: As the human body moves non-rigidly, large and subtle [Figure] motion changes usually appear simultaneously.

Original: The local and global matching shall be conducted simultaneously to ensure high-quality human motion transfer.
Revised: The local and global matching shall be conducted simultaneously to ensure high-quality human motion transfer.

Original: Existing studies Ren et al.
Revised: Existing studies Ren et al.

Original: Tao et al.
Revised: Tao et al.

Original: ; Zhao & Zhang (2022) leverage 2D human body keypoints Cao et al.
Revised: ; Zhao & Zhang (2022) leverage 2D human body keypoints Cao et al.

Original: ; Xiu et al.
Revised: ; Xiu et al.

Original: as the initial pose representations and introduce image-to-image translation Isola et al.
Revised: as the initial pose representations and introduce image-to-image translation Isola et al.

Original: (2017) or predictable warping fields (e.g., optical flow, affine transformations) for geometric matching.
Revised: (2017) or predictable warping fields (e.g., optical flow, affine transformations) for geometric matching.

Original: Since the 2D human body keypoints are sparse and the[Figure] geometric matching is captured by CNNs, it is hard to capture global matching as the CNN receptive field is limited.
Revised: Since the 2D human body keypoints are sparse and the geometric matching is captured by CNNs, it is hard to capture global matching as the CNN receptive field is limited.

Original: As a result, artifacts occur and details are missing in the transferred results when facing large motion changes between the target pose and source person.
Revised: As a result, artifacts occur and details are missing in the transferred results when facing large motion changes between the target pose and source person.

Original: The alternative 3D methods Neverova et al.
Revised: The alternative 3D methods Neverova et al.

Original: ; Gafni et al.
Revised: ; Gafni et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021a) introduce DensePose Alp G ¨ uler et al.
Revised: (2021a) introduce DensePose Alp G ¨ uler et al.

Original: or parametric body mesh Loper et al.
Revised: or parametric body mesh Loper et al.

Original: (2015) as human representations to perform pixelwise matching.
Revised: (2015) as human representations to perform pixelwise matching.

Original: They can globally match target pose and source images by aligning two humans into one 3D model.
Revised: They can globally match target pose and source images by aligning two humans into one 3D model.

Original: However, the off-the-shelf Densepose and 3D models suffer from background interference and partial occlusion, limiting the image alignment quality for dense pixel-wise matching.
Revised: However, the off-the-shelf Densepose and 3D models suffer from background interference and partial occlusion, limiting the image alignment quality for dense pixel-wise matching.

Original: Is there a proper way to simultaneously utilize robust 2D human body keypoints and build global and local matching?
Revised: Is there a proper way to simultaneously utilize robust 2D human body keypoints and build global and local matching?

Original: Fortunately, recent methods Dosovitskiy et al.
Revised: Fortunately, recent methods Dosovitskiy et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Revised: (2021c; 2022);

Original: Dong et al.
Revised: Dong et al.

Original: (2022) demonstrate that the Vision Transformers (ViTs) can capture global dependencies in visual recognition.
Revised: (2022) demonstrate that the Vision Transformers (ViTs) can capture global dependencies in visual recognition.

Original: Inspired by this design, we combine the advantages of CNNs and Transformers and propose a Vision-Transformers-based framework named MotionFormer to model the visual correspondence between the source person image and the target pose image.
Revised: Inspired by this design, we combine the advantages of CNNs and Transformers and propose a Vision-Transformers-based framework named MotionFormer to model the visual correspondence between the source person image and the target pose image.

Original: As shown in Figure 2, our MotionFormer consists of two encoders and one decoder.
Revised: As shown in Figure 2, our MotionFormer consists of two encoders and one decoder.

Original: The encoders extract feature pyramids from the source person image and the target pose image, respectively.
Revised: The encoders extract feature pyramids from the source person image and the target pose image, respectively.

Original: These feature pyramids are sent to the decoder for accurate matching and motion transfer.
Revised: These feature pyramids are sent to the decoder for accurate matching and motion transfer.

Original: In the decoder, there are several decoder blocks and one fusion block.
Revised: In the decoder, there are several decoder blocks and one fusion block.

Original: Each block consists of two parallel branches (i.e., the warping and generation branches).
Revised: Each block consists of two parallel branches (i.e., the warping and generation branches).

Original: In the warping branch, we predict a flow field to warp features from the source person image, which preserves the information of the source image to achieve high-fidelity motion transfer.
Revised: In the warping branch, we predict a flow field to warp features from the source person image, which preserves the information of the source image to achieve high-fidelity motion transfer.

Original: Meanwhile, the generation branch produces novel content that cannot be directly borrowed from the source appearance to improve photorealism further.
Revised: Meanwhile, the generation branch produces novel content that cannot be directly borrowed from the source appearance to improve photorealism further.

Original: Afterward, we use a fusion block to convert the feature output of these two branches into the final transferred image.
Revised: Afterward, we use a fusion block to convert the feature output of these two branches into the final transferred image.

Original: The feature matching result dominates the flow field and generation content.
Revised: The feature matching result dominates the flow field and generation content.

Original: Specifically, we implement feature matching from two input images via convolutions and cross-attentions in each branch.
Revised: Specifically, we implement feature matching from two input images via convolutions and cross-attentions in each branch.

Original: The tokens from the source image features are mapped into Key and Value, and the tokens from the target pose image features are mapped into Query.
Revised: The tokens from the source image features are mapped into Key and Value, and the tokens from the target pose image features are mapped into Query.

Original: We compute the cross-attention map based on the
Revised: We compute the cross-attention map based on the

Original: Key and Query.
Revised: Key and Query.

Original: This map reflects the global correlations between two input images.
Revised: This map reflects the global correlations between two input images.

Original: Then, we send the output of cross-attention process into a convolution to capture locally matched results.
Revised: Then, we send the output of cross-attention process into a convolution to capture locally matched results.

Original: Thanks to the accurate global and local feature matching, our MotionFormer can improve the performance of both the warping and generation processes.
Revised: Thanks to the accurate global and local feature matching, our MotionFormer can improve the performance of both the warping and generation processes.

Original: Moreover, we propose a mutual learning loss to enable these two branches to supervise each other during training, which helps them to benefit from each other and facilitates the final fusion for high-quality motion transfer.
Revised: Moreover, we propose a mutual learning loss to enable these two branches to supervise each other during training, which helps them to benefit from each other and facilitates the final fusion for high-quality motion transfer.

Original: In the test phase, our method generates a motion transfer video on the fly given a single source image without training a personspecific model or fine-tuning.
Revised: In the test phase, our method generates a motion transfer video on the fly given a single source image without training a personspecific model or fine-tuning.

Original: Some results generated by our method can be found in Fig.
Revised: Some results generated by our method can be found in Fig.

Original: 1, and we show more video results in the supplementary files.
Revised: 1, and we show more video results in the supplementary files.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Human Motion Transfer.
Revised: Human Motion Transfer.

Original: Most human motion transfer works are built upon an image-to-image
Revised: Most human motion transfer works are built upon an image-to-image

Original: Isola et al.
Revised: Isola et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2018b) or video-to-video Wang et al.
Revised: (2018b) or video-to-video Wang et al.

Original: (2018a; 2019) translation framework.
Revised: (2018a; 2019) translation framework.

Original: With this framework, some methods Chan et al.
Revised: With this framework, some methods Chan et al.

Original: Esser et al.
Revised: Esser et al.

Original: Dong et al.
Revised: Dong et al.

Original: ; Han et al.

Original: (2019a)

Original: ; Ma et al.
Revised: ; Ma et al.

Original: ; Ren et al.
Revised: ; Ren et al.

Original: (2021; 2020) set the off-the-shelf 2D body keypoints Cao et al.
Revised: (2021; 2020) set the off-the-shelf 2D body keypoints Cao et al.

Original: ; Xiu et al.
Revised: ; Xiu et al.

Original: (2018) as a condition to animate the source person image.
Revised: (2018) as a condition to animate the source person image.

Original: And some methods Siarohin et al.
Revised: And some methods Siarohin et al.

Original: (2019a;b; 2021) use the unsupervised 2D body keypoints to extract motion representations.
Revised: (2019a;b; 2021) use the unsupervised 2D body keypoints to extract motion representations.

Original: Therefore, these methods generalize well to a wider range of objects(i.e., animals).
Revised: Therefore, these methods generalize well to a wider range of objects(i.e., animals).

Original: The 2D body keypoints can represent the body motion correctly, but this representation is sparse and the CNN models used by these methods have limited receptive fields, which leads to inaccurate global visual correspondence between this motion representation and the source image.
Revised: The 2D body keypoints can represent the body motion correctly, but this representation is sparse and the CNN models used by these methods have limited receptive fields, which leads to inaccurate global visual correspondence between this motion representation and the source image.

Original: To overcome this limitation, recent approachesNeverova et al.
Revised: To overcome this limitation, recent approachesNeverova et al.

Original: ; Grigorev et al.
Revised: ; Grigorev et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021a)
Revised: (2021a);

Original: ; Shysheya et al.
Revised: Shysheya et al.

Original:  Liu et al. ; Ma et al.
Revised: ; Liu et al. (2019b); Ma et al.

Original: (2018) project the source person and the target person into a unified 3D space (i.e., DensePose Alp G¨uler et al.
Revised: (2018) project the source person and the target person into a unified 3D space (i.e., DensePose Alp G ¨ uler et al.

Original: (2018) and SMPL Loper et al.
Revised: and SMPL Loper et al.

Original:  to capture pixel-level correspondences, which help render the appearance from source to target in global perspective.
Revised: (2015)) to capture pixel-level correspondences, which help render the appearance from source to target in global perspective.

Revised: And they use the inpainting methodsLiu et al.

Revised: (2019a; 2020; 2021a;b) to restore the background image.

Original: However, compared to 2D keypoint representations, DensePose and SMPL models are less accurate and may produce large misalignments in complex scenes.
Revised: However, compared to 2D keypoint representations, DensePose and SMPL models are less accurate and may produce large misalignments in complex scenes.

Original: In this paper, we design the first Vision-Transformer-based generation framework for human motion transfer.
Revised: In this paper, we design the first Vision-Transformer-based generation framework for human motion transfer.

Original: With merely 2D keypoints guiding the target motion, we capture large motion deformations with globally attended appearance and motion features and yield state-of-theart performance.
Revised: With merely 2D keypoints guiding the target motion, we capture large motion deformations with globally attended appearance and motion features and yield state-of-the-art performance.

Original: Unlike many previous methods that trained on a single video Chan et al.
Revised: Unlike many previous methods that trained on a single video Chan et al.

Original:  Shysheya et al.
Revised: ; Shysheya et al.

Original: or need fine-tuning Huang et al.
Revised: or need fine-tuning Huang et al.

Original: (2021a);
Revised: (2021a)

Original:  Liu et al.
Revised: ; Liu et al.

Revised: (2019b)

Original: ; Zakharov et al.
Revised: ; Zakharov et al.

Original: (2019) to achieve higher perceptual quality, our method works in a one-shot fashion that directly generalizes to unseen identities.
Revised: to achieve higher perceptual quality, our method works in a one-shot fashion that directly generalizes to unseen identities.

Original: Vision Transformer.
Revised: Vision Transformer.

Original: Transformer Vaswani et al.
Revised: Transformer Vaswani et al.

Original: (2017) has become increasingly popular in solving the computer vision problems, such as object detection Carion et al.
Revised: (2017) has become increasingly popular in solving the computer vision problems, such as object detection Carion et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Revised: (2021c);

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021), segmentation Wang et al.
Revised: (2021), segmentation Wang et al.

Original: (2021a)
Revised: (2021a)

Original: ; Zheng et al.
Revised: ; Zheng et al.

Original: ; Cao et al.
Revised: ; Cao et al.

Original: (2021b), inpainting Peng et al.
Revised: (2021b), inpainting Peng et al.

Original: ; Wan et al.
Revised: ; Wan et al.

Original: (2021), image generation Cao et al.
Revised: (2021), image generation Cao et al.

Original: (2021a)
Revised: (2021a)

Original: ; Lee et al.
Revised: ; Lee et al.

Original: ; Jiang et al.
Revised: Jiang et al.

Original: (2021); Esser et al.
Revised: (2021); Esser et al.

Original: (2021), restoration Chen et al.
Revised: (2021), restoration Chen et al.

Original: (2021b)
Revised: (2021b)

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2021b);
Revised: (2021b)

Original:  Liang et al.
Revised: ; Liang et al.

Revised: ; Zhu et al.

Original:  image classification Wang et al.
Revised: , image classification Wang et al.

Original: (2021a)
Revised: (2021a)

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021b)
Revised: (2021b)

Original: ; Ramachandran et al.
Revised: ; Ge et al.

Original:  Liu et al.
Revised: ; Liu et al.

Revised: (2021c)

Original: ; Wu et al.
Revised: ; Wu et al.

Original: ; Dong et al.
Revised: ; Dong et al.

Original: ; Wu et al. Srinivas et al.
Revised: ; Liang et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021a), and 3D human texture estimation Xu & Loy (2021).
Revised: (2022; 2021a), and 3D human texture estimation Xu & Loy (2021).

Original: Due to the powerful global information modeling ability, these Transformer-based methods achieve significant performance gain compared with CNNs that focus on local information.
Revised: Due to the powerful global information modeling ability, these Transformer-based methods achieve significant performance gain compared with CNNs that focus on local information.

Original: In this paper, we successfully manage to take advantages of Vision Transformers for motion transfer with a warping and generation two branch architecture.
Revised: In this paper, we successfully manage to take advantages of Vision Transformers for motion transfer with a warping and generation two branch architecture.

Original: The two branches employ cross-attention and convolution to enrich generation quality from both global and local viewpoints.
Revised: The two branches employ cross-attention and convolution to enrich generation quality from both global and local viewpoints.

Original: Moreover, we propose a novel mutual learning loss, which regularizes the two branches to learn the advantages of each other, thus effectively increasing photorealism.
Revised: Besides, we propose a novel mutual learning loss to regularize two branches to learn from each other, which effectively increases photorealism.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: Fig.
Revised: Fig.

Original: 2 shows an overview of MotionFormer.
Revised: 2 shows an overview of MotionFormer.

Original: It consists of two Transformer encoders and one Transformer decoder.
Revised: It consists of two Transformer encoders and one Transformer decoder.

Original: The two Transformer encoders first extract the features of source image I s and target pose image P t , respectively.
Revised: The two Transformer encoders first extract the features of source image I s and target pose image P t , respectively.

Original: Then the Transformer decoder builds the relationship between
Revised: Then the Transformer decoder builds the relationship between

Original: I s and P t with two-branch decoder blocks hierarchically.
Revised: I s and P t with two-branch decoder blocks hierarchically.

Original: Finally, a fusion block predicts the reconstructed person image I out .
Revised: Finally, a fusion block predicts the reconstructed person image I out .

Original: The network is trained end-to-end with the proposed mutual learning loss.
Revised: The network is trained end-to-end with the proposed mutual learning loss.

Original: We utilize the Cross-Shaped Window Self-Attention (CSWin Attention)
Revised: We utilize the Cross-Shaped Window Self-Attention (CSWin Attention)

Original: Dong et al.
Revised: Dong et al.

Revised: [Figure]

Revised: T

Revised: [Figure]

Revised: f

Revised: [Figure]

Original: as our Attention mechanism in the encoder and decoder.
Revised: ff [Figure] I [Figure]

Revised: P t

Revised: [Figure]

Revised: as our Attention mechanism in the encoder and decoder.

Original: The CSWin Attention calculates attention in the horizontal and vertical stripes in parallel to ensure the performance and efficiency.
Revised: The CSWin Attention calculates attention in the horizontal and vertical stripes in parallel to ensure the performance and efficiency.

Original: In our method, we assume a fixed background and simultaneously estimate a foreground mask M out to merge an inpainted background with I out in the testing phase.
Revised: In our method, we assume a fixed background and simultaneously estimate a foreground mask M out to merge an inpainted background with I out in the testing phase.

Original: We introduce the Transformer encoder and decoder in Sec.
Revised: We introduce the Transformer encoder and decoder in Sec.

Original: 3.1 and Sec. 3.2 respectively.
Revised: 3.1 and Sec. 3.2 respectively.

Original: The mutual learning loss ispresented in Sec. 3.3.
Revised: The mutual learning loss is in Sec. 3.3.

Original: [Figure]

Original: 3. T RANSFORMER E NCODER
Revised: 3. T RANSFORMER E NCODER

Original: The structure of the two Transformer encoders is the same.
Revised: The structure of the two Transformer encoders is the same.

Original: Each encoder consists of three stages with different spatial size.
Revised: Each encoder consists of three stages with different spatial size.

Original: Each stage has multiple encoder blocks and we adopt the CSwin Transformer Block Dong et al.
Revised: Each stage has multiple encoder blocks and we adopt the CSwin Transformer Block Dong et al.

Original: as our encoder block.
Revised: as our encoder block.

Original: Our Transformer encoder captures hierarchical representations of the source image I s and the target pose image P t .
Revised: Our Transformer encoder captures hierarchical representations of the source image I s and the target pose image P t .

Original: S i and T i ( i = 1 , 2 , 3 ) denote the output of the i -th stage for I s and P t , respectively, as shown in Fig.
Revised: S i and T i ( i = 1 , 2 , 3 ) denote the output of the i -th stage for I s and P t , respectively, as shown in Fig.

Original: We follow Dong et al.
Revised: We follow Dong et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2021) to utilize a convolutional layer between different stages for token reduction and channel increasing.
Revised: (2021) to utilize a convolutional layer between different stages for token reduction and channel increasing.

Original: We show more details of the encoder in the appendix.
Revised: We show more details of the encoder in the appendix.

Original: 3. T RANSFORMER D ECODER
Revised: 3. T RANSFORMER D ECODER

Original: There are three stages in our Transformer decoder.
Revised: There are three stages in our Transformer decoder.

Original: The number of the decoder block in each stage is 2, 4, and 12, respectively.
Revised: The number of the decoder block in each stage is 2, 4, and 12, respectively.

Original: We concatenate the output of each stage and the corresponding target pose feature by skip-connections.
Revised: We concatenate the output of each stage and the corresponding target pose feature by skip-connections.

Original: The concatenated results are sent to the second and third stages.
Revised: The concatenated results are sent to the second and third stages.

Original: Similar to the encoder, we set the convolutional layer between different stages to increase token numbers and decrease channel dimensions.
Revised: Similar to the encoder, we set the convolutional layer between different stages to increase token numbers and decrease channel dimensions.

Original: 3.2. D ECODER BLOCK .
Revised: 3.2. D ECODER BLOCK .

Original: As shown in Fig 3, the decoder block has warping and generation branches.
Revised: As shown in Fig 3, the decoder block has warping and generation branches.

Original: In each branch, there is a cross-attention process and a convolutional layer to capture the global and local correspondence respectively.
Revised: In each branch, there is a cross-attention process and a convolutional layer to capture the global and local correspondence respectively.

Original: Let X lde denote the output of l -th decoder block ( l > 1 ) or the output of precedent stage ( l = 1 ).
Revised: Let X lde denote the output of l -th decoder block ( l > 1 ) or the output of precedent stage ( l = 1 ).

Original: For the first decoder stage, we set the T 3 as input so the X 1 de = T 3 .
Revised: For the first decoder stage, we set the T 3 as input so the X 1 de = T 3 .

Original: The decoder block first extracts ˆ X lde from X l − 1 de with a Multi-Head Self-Attention process.
Revised: The decoder block first extracts ˆ X lde from X l − 1 de with a Multi-Head Self-Attention process.

Original: Then we feed ˆ X lde to the warping branch and generation branch as Query ( Q ), and we use the feature of source encoder S i as Key ( K ) and Value ( V ) to calculate the cross-attention map similar to Vaswani et al.
Revised: Then we feed ˆ X lde to the warping branch and generation branch as Query ( Q ), and we use the feature of source encoder S i as Key ( K ) and Value ( V ) to calculate the cross-attention map similar to Vaswani et al.

Original: (2017) with Multi-Head
Revised: (2017) with Multi-Head

Original: Cross-Attention process.
Revised: Cross-Attention process.

Original: The cross-attention map helps us build the global correspondence between the target pose and the source image.
Revised: The cross-attention map helps us build the global correspondence between the target pose and the source image.

Original: Finally, we send the output of the Multi-Head Cross-Attention to a convolutional layer to extract the local correspondence.
Revised: Finally, we send the output of the Multi-Head Cross-Attention to a convolutional layer to extract the local correspondence.

Original: The warping branch predicts a flow field to deform the source feature conditioned on the target pose, which helps the generation of regions that are visible in the source image.
Revised: The warping branch predicts a flow field to deform the source feature conditioned on the target pose, which helps the generation of regions that are visible in the source image.

Original: While for the invisible parts, the generation branch synthesizesnovel content with the contextual information mined from the source feature.
Revised: While for the invisible parts, the generation branch synthesizes

Original: We combine the advantages of these two branches in each decoder block to improve the generation quality. Warping branch. The warping branch aims to generate a flow field to warp the source feature S i . Specifically, the Multi-Head Cross-Attention outputs the feature with the produced Q , K , V , and we feed the output to a convolution to inference the flow field.
Revised: [Figure]

Original: Inspired by recent approaches that gradually refine the estimation of optical flow Hui et al. ; Han et al. (2019b), we estimate a
Revised: I

Original: [Figure]
Revised: [Figure]

Original: C NN C NN
Revised: C NN C NN

Original: C NN
Revised: C NN

Original: W a r p
Revised: W a r p

Original: M u lti - H
Revised: M u lti - H

Original: ea d
Revised: ea d

Original: S e l f-
Revised: S e l f-

Original: A tt e n ti on
Revised: A tt e n ti on

Original: A dd & N o r m
Revised: A dd & N o r m

Original: C NN
Revised: C NN

Original: M u lti - H
Revised: M u lti - H

Original: ea d
Revised: ea d

Original: C r o ss - A tt e n ti on v q k
Revised: C r o ss - A tt e n ti on v q k

Original: C NN
Revised: C NN

Original: W a r p
Revised: W a r p

Original: C NN
Revised: C NN

Original: A dd & N o r m
Revised: A dd & N o r m

Original: M L P k q q v k v f iWarpingBranch
Revised: M L P k q q v k v f iWarpingBranch

Original: [Figure]
Revised: [Figure]

Original: M u lti - H
Revised: M u lti - H

Original: ea d
Revised: ea d

Revised: C r o ss - A tt e n ti

Original: [Figure]
Revised: [Figure]

Original: e n
Revised: novel content with the contextual information mined from the source feature. We combine the advantages of these two branches in each decoder block to improve the generation quality. Warping branch. The warping branch aims to generate a flow field to warp the source feature S i .

Original: [Figure]
Revised: Specifically, the Multi-Head Cross-Attention outputs the feature with the produced Q , K , V , and we feed the output to a convolution to inference the flow field. Inspired by recent approaches that gradually refine the estimation of optical flow Hui et al. ; Han et al.

Original:  residual flow to refine the estimation of the previous stage.
Revised: (2019), we estimate a residual flow to refine the estimation of the previous stage.

Original: Next, we warp the feature map S
Revised: Next, we warp the feature map S

Original: i ac- cording to the flow field using bilinear interpolation.
Revised: i ac- cording to the flow field using bilinear interpolation.

Original: Formally, the whole process can be formulated as follows:
Revised: Formally, the whole process can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Revised: = W V

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: ( Multi-Head Cross

Original: [Equation]
Revised: [Equation]

Revised: Attention (

Revised: [Equation]

Revised: f l = Up ( f i − 1 ) + f l

Original:  where W Q , W K , W V are the learnable projection heads in the self-attention module, the O lw denotes the output of the warping branch in l -th block, the Up is a × 2 nearest-neighbor upsampling, and Warp denotes warping feature map S i according to flow f l using grid sampling Jaderberg et al.
Revised: [Equation] if l = 1 and i > [Equation] where W Q , W K , W V are the learnable projection heads in the self-attention module, the O lw denotes the output of the warping branch in l -th block, the Up is a × 2 nearest-neighbor upsampling, and Warp denotes warping feature map S i according to flow f l using grid sampling Jaderberg et al.

Original: For the i -th decoder stage, the flow predicted by the last decoder block is treated as f i and then refined by the subsequent blocks.
Revised: For the i -th decoder stage, the flow predicted by the last decoder block is treated as f i and then refined by the subsequent blocks.

Original: Generation branch.
Revised: Generation branch.

Original: The architecture of the generation branch is similar to the warping branch.
Revised: The architecture of the generation branch is similar to the warping branch.

Original: The attention outputs the feature with the produced Q , K , V , and then we feed the output to a convolution to infer the final prediction O lg :
Revised: The attention outputs the feature with the produced Q , K , V , and then we feed the output to a convolution to infer the final prediction O lg :

Original: [Equation]
Revised: [Equation]

Original: where W Q , W K , W V are the learnable projection heads in the self-attention module.
Revised: where W Q , W K , W V are the learnable projection heads in the self-attention module.

Original: The generation branch can generate novel content based on the global information of source feature S i .
Revised: The generation branch can generate novel content based on the global information of source feature S i .

Original: Therefore, it is complementary to the warping branch when the flow field is inaccurate or there is no explicit reference in the source feature.
Revised: Therefore, it is complementary to the warping branch when the flow field is inaccurate or there is no explicit reference in the source feature.

Original: Finally, we concatenate the output of warping and generation branch and reduce the dimension with a convolutional layer followed by an MLP and a residual connection:
Revised: Finally, we concatenate the output of warping and generation branch and reduce the dimension with a convolutional layer followed by an MLP and a residual connection:

Original: [Equation]
Revised: [Equation]

Original: where the ¯ X l de is the combination of warping and generation branches in the l -th decoder block.
Revised: where the ¯ X lde is the combination of warping and generation branches in the l -th decoder block.

Original: 3.2. F USION BLOCK .
Revised: 3.2. F USION BLOCK .

Original: The fusion block takes the decoder output to predict the final result.
Revised: The fusion block takes the decoder output to predict the final result.

Original: The fusion block has a warping branch and generation branch at the pixel level.
Revised: The fusion block has a warping branch and generation branch at the pixel level.

Original: The warping branch refines the last decoder flow field f 3 and estimates a final flow f f .
Revised: The warping branch refines the last decoder flow field f 3 and estimates a final flow f f .

Original: And the generation branch synthesizes the RGB value I f .
Revised: And the generation branch synthesizes the RGB value I f .

Original: At the same time, a fusion mask M f is predicted to merge the output of these two branches:
Revised: At the same time, a fusion mask M f is predicted to merge the output of these two branches:

Original: [Equation]
Revised: [Equation]

Original: where the O de is the output of decoder, ⊙ is the element-wise multiplication, and I out is the final prediction.
Revised: where the O de is the output of decoder, ⊙ is the element-wise multiplication, and I out is the final prediction.

Original: 3. M UTUAL LEARNING LOSS
Revised: 3. M UTUAL LEARNING LOSS

Original: The generation and warping branches have their own advantages as mentioned above.
Revised: The generation and warping branches have their own advantages as mentioned above.

Original: Intuitively, we concatenate the output of these two branches followed by a convolution layer and an MLP as shown in Fig. 2, but we empirically find the convolution layer and MLP cannot combine these advantages well (see Sec. 5).
Revised: Intuitively, we concatenate the output of these two branches followed by a convolution layer and an MLP as shown in Fig. 2, but we empirically find the convolution layer and MLP cannot combine these advantages well (see Sec. 5).

Original: To address this limitation and ensure the results have both advantages of these two branches, we propose a novel mutual learning loss to enforce these two branches to learn the advantages of each other.
Revised: To address this limitation and ensure the results have both advantages of these two branches, we propose a novel mutual learning loss to enforce these two branches to learn the advantages of each other.

Original: Specifically, the mutual learning loss enables these two branches to supervise each other within each decoder block, let O kw , O
Revised: Specifically, the mutual learning loss enables these two branches to supervise each other within each decoder block, let O kw , O

Original: kg ∈ R ( H × W ) ×
Revised: kg ∈ R ( H × W ) ×

Original: C denote the reshaped outputs of the last warping and generation branch at the k -th decoder stage (see Eqs.
Revised: C denote the reshaped outputs of the last warping and generation branch at the k -th decoder stage (see Eqs.

Original: (1) and (2) for their definition).
Revised: (1) and (2) for their definition).

Original: If we calculate the similarity between the feature vector O kw,i ∈ R C at the spatial location i of O kw and all feature vectors O kg,j ∈ R C ( j = 1 , 2 , . .
Revised: If we calculate the similarity between the feature vector O kw,i ∈ R C at the spatial location i of O kw and all feature vectors O kg,j ∈ R C ( j = 1 , 2 , . .

Original: . , HW ) in O kg , we argue that the most similar vector to O kw,i should be O kg,i , which is at the same position in O kg .
Revised: . , HW ) in O kg , we argue that the most similar vector to O kw,i should be O kg,i , which is at the same position in O kg .

Original: In another word, we would like to enforce i =
Revised: In another word, we would like to enforce i =

Original: arg max j
Revised: arg max j

Original: Cos ( O kw,i , O kg,j ) , where Cos ( · , · ) is the cosine similarity.
Revised: Cos ( O kw,i , O kg,j ) , where Cos ( · , · ) is the cosine similarity.

Original: This is achieved by the following mutual learning loss:
Revised: This is achieved by the following mutual learning loss:

Original: [Equation]
Revised: [Equation]

Original: where the SoftArgMax is a differentiable version of arg max that returns the spatial location of the maximum value.
Revised: where the SoftArgMax is a differentiable version of arg max that returns the spatial location of the maximum value.

Original: The mutual learning loss constrain the two branches to have high correlations at the same location, enhancing the complementariness of warping and generation.
Revised: The mutual learning loss constrain the two branches to have high correlations at the same location, enhancing the complementariness of warping and generation.

Original: In addition to the perceptual diversity loss, we follow the Ren et al.
Revised: In addition to the perceptual diversity loss, we follow the Ren et al.

Original: and
Revised: and

Original: Huang et al.
Revised: Huang et al.

Original: (2021a) utilize the reconstruction loss Johnson et al.
Revised: (2021a) utilize the reconstruction loss Johnson et al.

Original: feature matching loss Wang et al.
Revised: feature matching loss Wang et al.

Original: (2018c), hinge adversarial loss Lim & Ye (2017), style loss Gatys et al.
Revised: (2018c), hinge adversarial loss Lim & Ye (2017), style loss Gatys et al.

Original: (2015), total variation loss Johnson et al.
Revised: (2015), total variation loss Johnson et al.

Original: (2016) and mask loss Huang et al.
Revised: (2016) and mask loss Huang et al.

Original: (2021a) to optimize our network.
Revised: (2021a) to optimize our network.

Original: We show the details inthe appendix.
Revised: Details are in appendix.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Datasets.
Revised: Datasets.

Original: We use the solo dance YouTube videos collected by Huang et al.
Revised: We use the solo dance YouTube videos collected by Huang et al.

Original: (2021a)
Revised: (2021a)

Original: and iPer Liu et al.
Revised: and iPer Liu et al.

Original: (2019) datasets.
Revised: (2019b) datasets.

Original: These videos contain nearly static backgrounds and subjects that vary in gender, body shape, hairstyle, and clothes.
Revised: These videos contain nearly static backgrounds and subjects that vary in gender, body shape, hairstyle, and clothes.

Original: All the frames are center cropped and resized to 256 × 256 .
Revised: All the frames are center cropped and resized to 256 × 256 .

Original: We train a separate model on each dataset to fairly compare with other methods.
Revised: We train a separate model on each dataset to fairly compare with other methods.

Original: Implementation details.
Revised: Implementation details.

Original: We use OpenPose Cao et al.
Revised: We use OpenPose Cao et al.

Original: (2017) to detect 25 body joints for each frame.
Revised: (2017) to detect 25 body joints for each frame.

Original: These joints are then connected to create a target pose stick image P t , which has 26 channels and  each channel indicating one stick of the body.
Revised: These joints are then connected to create a target pose stick image P t , which has 26 channels and [Figure] each channel indicating one stick of the body.

Original: We use M gt to separate foreground person image I gt from the original frame.
Revised: We use M gt to separate foreground person image I gt from the original frame.

Original: Our model is optimized using Adam optimizer with β 1 = 0 . 0 , β 2 = 0 . 99 , and initial learning rate of 10 − 4 .
Revised: Our model is optimized using Adam optimizer with β 1 = 0 . 0 , β 2 = 0 . 99 , and initial learning rate of 10 − 4 .

Original: We utilize the TTUR strategy Heusel et al.
Revised: We utilize the TTUR strategy Heusel et al.

Original: (2017) to train our model.
Revised: (2017) to train our model.

Original: During the inference phase, we select one image from one video as the source image.
Revised: During the inference phase, we select one image from one video as the source image.

Original: The target background is generated by inpainting the source image background with LAMA Suvorov[Figure] [Figure] et al.
Revised: The target background is generated by inpainting the source image background with LAMA Suvorov et al.

Original: When the target person has different body shapes ( e .
Revised: When the target person has different body shapes ( e .

Original: g ., heights, limb lengths) or the target person and source person are at different distances from the camera, we use the strategy in
Revised: g ., heights, limb lengths) or the target person and source person are at different distances from the camera, we use the strategy in

Original: Chan et al.
Revised: Chan et al.

Original: (2019) to normalize the pose of the target human.
Revised: (2019) to normalize the pose of the target human.

Original: Baselines.
Revised: Baselines.

Original: We compare MotionFormer with the state-of-the-art human motion transfer approaches:
Revised: We compare MotionFormer with the state-of-the-art human motion transfer approaches:

Original: LWG Liu et al.
Revised: LWG Liu et al.

Revised: (2019b),

Original: , GTM Huang et al.
Revised: GTM Huang et al.

Original: (2021a),
Revised: (2021a),

Original: MRAA Siarohin et al.
Revised: MRAA Siarohin et al.

Original: and DIST Ren et al.
Revised: and DIST Ren et al.

Original: For LWG, we test it on iPer dataset with the released pre-trained model, and we train
Revised: For LWG, we test it on iPer dataset with the released pre-trained model, and we train

Original: LWG on YouTube videos with its source code.
Revised: LWG on YouTube videos with its source code.

Original: At test time, we fine-tune LWG on the source image as official implementation (fine-tuning is called “personalize” in the source code).
Revised: At test time, we fine-tune LWG on the source image as official implementation (fine-tuning is called “personalize” in the source code).

Original: For GTM, we utilize the pre-trained model on the YouTube videos dataset provided by the authors and retrain the model on iPer with the source code.
Revised: For GTM, we utilize the pre-trained model on the YouTube videos dataset provided by the authors and retrain the model on iPer with the source code.

Original: As GTM supports testing with multiple source images, we use 20 frames in the source video and fine-tune the pre-trained network as described in the original paper Huang et al.
Revised: As GTM supports testing with multiple source images, we use 20 frames in the source video and fine-tune the pre-trained network as described in the original paper Huang et al.

Original: (2021a).
Revised: (2021a).

Original: For MRAA, we use the source code provided by the authors to train the model.
Revised: For MRAA, we use the source code provided by the authors to train the model.

Original: For DIST Ren et al.
Revised: For DIST Ren et al.

Original: (2020), we compare with it using the pre-trained model on iPer dataset.
Revised: (2020), we compare with it using the pre-trained model on iPer dataset.

Original: For synthesizing a 1,000 frame video, the average per frame time costs 
Revised: For synthesizing a 1,000 frame video, the average per frame time costs of MRAA, LWG,

Original: [Table]
Revised: GTM, DIST, and our method are 0.021s, 1.242s, 1.773s, 0.088s, and 0.94s, respectively.

Original: [Table]
Revised: Meanwhile, MotionFormer does not require an online fine-tuning, while LWG and GTM do.

Original: 4. Q UALITATIVE C OMPARISONS
Revised: 4. Q UALITATIVE C OMPARISONS

Original: Qualitatively comparisons are given in Fig.
Revised: Qualitatively comparisons are given in Fig.

Original: 4 and Fig.
Revised: 4 and Fig.

Original: Although LWG Liu et al. (2019) can maintain the overall shape of the human body, it fails to reconstruct complicated human parts ( e .
Revised: Although LWG Liu et al. (2019b) can maintain the overall shape of the human body, it fails to reconstruct complicated human parts ( e .

Original: g ., long hair and shoes in Fig. 4) of the source person and synthesis image with a large body motion ( e .
Revised: g ., long hair and shoes in Fig. 4) of the source person and synthesis image with a large body motion ( e .

Original: g ., squat in the red box of Fig. 4), which leads to visual artifacts and missing details.
Revised: g ., squat in the red box of Fig. 4), which leads to visual artifacts and missing details.

Original: This is because
Revised: This is because

Original: LWG relies on the 3D mesh predicted by HMR Kanazawa et al.
Revised: LWG relies on the 3D mesh predicted by HMR Kanazawa et al.

Original: (2018), which is unable to model detailed shape information.
Revised: (2018), which is unable to model detailed shape information.

Original: In contrast, GTM Huang et al.
Revised: In contrast, GTM Huang et al.

Original: (2021a) reconstructs better the body shape as it uses multiple inputs to optimize personalized geometry and texture.
Revised: (2021a) reconstructs better the body shape as it uses multiple inputs to optimize personalized geometry and texture.

Original: However, the geometry cannot handle the correspondence between the source image and the target pose.
Revised: However, the geometry cannot handle the correspondence between the source image and the target pose.

Original: The synthesized texture also presents severe artifacts, especially for invisible regions in the source images.
Revised: The synthesized texture also presents severe artifacts, especially for invisible regions in the source images.

Original: As an unsupervised method, MRAA Siarohin et al.
Revised: As an unsupervised method, MRAA Siarohin et al.

Original: (2021) implicitly models the relationship between source and target images.
Revised: (2021) implicitly models the relationship between source and target images.

Original: Without any prior information about the human body, MRAA generates unrealistic human images.
Revised: Without any prior information about the human body, MRAA generates unrealistic human images.

Original: DIST Huang et al.
Revised: DIST Huang et al.

Original: (2021a) does not model correct visual correspondence ( e .
Revised: (2021a) does not model correct visual correspondence ( e .

Original: g ., the coat buttons are missing in the last example) and suffers from overfitting ( e .
Revised: g ., the coat buttons are missing in the last example) and suffers from overfitting ( e .

Original: g ., the coat color becomes dark blue in the third example).
Revised: g ., the coat color becomes dark blue in the third example).

Original: Compared to existing methods, MotionFormer renders more realistic and natural images by effectively modeling long-range correspondence and local details. 
Revised: Compared to existing methods, MotionFormer renders more realistic and natural images by effectively modeling long-range correspondence and local details. [Figure] [Table]

Original: 4. Q UANTITATIVE C OMPARISONS
Revised: 4. Q UANTITATIVE C OMPARISONS

Original: We use SSIM Wang et al.
Revised: We use SSIM Wang et al.

Original: (2004), PSNR, FID Heusel et al.
Revised: (2004), PSNR, FID Heusel et al.

Original: (2017), and LPIPS Zhang et al.
Revised: (2017), and LPIPS Zhang et al.

Original: as numerical evaluation metrics.
Revised: as numerical evaluation metrics.

Original: The quantitative results are reported in Table 1 and Table 2.
Revised: The quantitative results are reported in Table 1 and Table 2.

Original: We observe that our method outperforms existing methods by large margins across all metrics.
Revised: We observe that our method outperforms existing methods by large margins across all metrics.

Original: Additionally, we perform a human subjective evaluation.
Revised: Additionally, we perform a human subjective evaluation.

Original: We generate the motion transfer videos of these different methods by randomly selecting 3-second video clips in the test set.
Revised: We generate the motion transfer videos of these different methods by randomly selecting 3-second video clips in the test set.

Original: On each trial, a volunteer is given compared results on the same video clip and is then asked to select the one with the highest generation quality.
Revised: On each trial, a volunteer is given compared results on the same video clip and is then asked to select the one with the highest generation quality.

Original: We tally the votes and show the statistics in the last column of Tableand Table 2.
Revised: We tally the votes and show the statistics in the last column of Tableand Table 2.

Original: We can find that our method is favored in most of the trials.
Revised: We can find that our method is favored in most of the trials.

Original:  A BLATION S TUDY
Revised:  A BLATION S TUDY

Original: Attention Mechanism.
Revised: Attention Mechanism.

Original: To evaluate the effects of the cross-attention module, we delete the crossattention in both the warping and generation branch.
Revised: To evaluate the effects of the cross-attention module, we delete the crossattention in both the warping and generation branch.

Original: Instead, we concatenate the source feature S i and Query directly in the Transformer decoder, followed by a convolution layer constructing their local relationship (this experiment is named Ours w/o Attention).
Revised: Instead, we concatenate the source feature S i and Query directly in the Transformer decoder, followed by a convolution layer constructing their local relationship (this experiment is named Ours w/o Attention).

Original: As shown in Fig. 6(c), without modeling the long-range relationship between the source and target, Ours w/o Attention achieves worse results ( e .
Revised: As shown in Fig. 6(c), without modeling the long-range relationship between the source and target, Ours w/o Attention achieves worse results ( e .

Original: g ., distorted skirt, limbs, and shoes).
Revised: g ., distorted skirt, limbs, and shoes).

Original: The numerical comparison shown in Tableis consistent with the visual observation.[Figure]
Revised: The numerical comparison shown in Tableis consistent with the visual observation.

Original: Generation and warping branches.
Revised: Generation and warping branches.

Original: We show the contributions of the generation branch and warping branch in the decoder block by removing them individually ( i .
Revised: We show the contributions of the generation branch and warping branch in the decoder block by removing them individually ( i .

Original: e .
Revised: e .

Original: , Ours w/o warping, Ours w/o generation).
Revised: , Ours w/o warping, Ours w/o generation).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6(d), without the warping branch, the generated clothing contains unnatural green and black regions in the man’s T-shirt and woman’s skirt, respectively.
Revised: 6(d), without the warping branch, the generated clothing contains unnatural green and black regions in the man’s T-shirt and woman’s skirt, respectively.

Original: This phenomenon reflects that a single generation branch is prone to over-fitting.
Revised: This phenomenon reflects that a single generation branch is prone to over-fitting.

Original: On the other hand, the warping branch can avoid over-fitting as shown in Fig. 6(e).
Revised: On the other hand, the warping branch can avoid over-fitting as shown in Fig. 6(e).

Original: However, the results still lack realism as the warping branch cannot generate novel appearances which are invisible in the source image ( e .
Revised: However, the results still lack realism as the warping branch cannot generate novel appearances which are invisible in the source image ( e .

Original: g ., the shoes of the man and the hair of the woman are incomplete).
Revised: g ., the shoes of the man and the hair of the woman are incomplete).

Original: Our full model combines  the advantages of these two branches and produces better results in Fig. 6(g).
Revised: Our full model combines [Table] [Figure] the advantages of these two branches and produces better results in Fig. 6(g).

Original: We also report the numerical results in Table 3, in which our full method achieves the best performance.
Revised: We also report the numerical results in Table 3, in which our full method achieves the best performance.

Original: Mutual learning loss.
Revised: Mutual learning loss.

Original: We analyze the importance of mutual learning loss (Eq.
Revised: We analyze the importance of mutual learning loss (Eq.

Original: (5)) by removing it during training (Ours w/o mutual).
Revised: (5)) by removing it during training (Ours w/o mutual).

Original: Fig. 6(f) shows the prediction combining the advantages of both warping and generation branches without using the mutual learning loss, which still produces noticeable visual artifacts.
Revised: Fig. 6(f) shows the prediction combining the advantages of both warping and generation branches without using the mutual learning loss, which still produces noticeable visual artifacts.

Original: The proposed mutual learning loss aligns the output features from these two branches and improves the performance.
Revised: The proposed mutual learning loss aligns the output features from these two branches and improves the performance.

Original: The numerical evaluation in Table 3 also indicates that mutual learning loss improves the generated image quality.
Revised: The numerical evaluation in Table 3 also indicates that mutual learning loss improves the generated image quality.

Original: The other loss terms have been demonstrated effective in Balakrishnan et al.; Han et al.
Revised: The other loss terms have been demonstrated effective in Balakrishnan et al.

Original: (2019b)

Original:  Wang et al.
Revised: ; Wang et al.

Original: (2018b)
Revised: (2018b)

Original:  Liu et al. (2019) with sufficient studies, so we do not include them in the ablation studies.
Revised: ; Liu et al. (2019b) with sufficient studies, so we do not include them in the ablation studies.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUDING R EMARKS

Original: In this paper [Table] we introduce MotionFormer, a Transformers-based framework for realistic human motion transfer.
Revised: In this paper, we introduce MotionFormer, a Transformers-based framework for realistic human motion transfer.

Original: MotionFormer captures the global and local relationship between the source appearance and target pose with carefully designed Transformer-based decoder blocks, synthesizing promising results.
Revised: MotionFormer captures the global and local relationship between the source appearance and target pose with carefully designed Transformer-based decoder blocks, synthesizing promising results.

Original: At the core of each block lies a warping branch to deform the source feature and a generation branch to synthesize novel content.
Revised: At the core of each block lies a warping branch to deform the source feature and a generation branch to synthesize novel content.

Original: By minimizing a mutual learning loss, these two branches supervise each other to learn better representations and improve generation quality.
Revised: By minimizing a mutual learning loss, these two branches supervise each other to learn better representations and improve generation quality.

Original: Experiments on a dancing video dataset verify the effectiveness of MotionFormer.
Revised: Experiments on a dancing video dataset verify the effectiveness of MotionFormer.

Original: E THIC D ISCUSSIONS
Revised: E THIC D ISCUSSIONS

Original: This work introduces a motion transfer method that can transfer motions from one subject to another.
Revised: This work introduces a motion transfer method that can transfer motions from one subject to another.

Original: It may raise the potential ethical concern that malicious actions can be transferred to anyone ( e .
Revised: It may raise the potential ethical concern that malicious actions can be transferred to anyone ( e .

Original: g ., celebrities).
Revised: g ., celebrities).

Original: To prevent action retargeting on celebrities, we may insert a watermark to the human motion videos.
Revised: To prevent action retargeting on celebrities, we may insert a watermark to the human motion videos.

Original: The watermark contains the original motion source, which may differentiate the celebrity movement.
Revised: The watermark contains the original motion source, which may differentiate the celebrity movement.

Original: Meanwhile, we can construct a celebrity set.
Revised: Meanwhile, we can construct a celebrity set.

Original: We first conduct face recognition on the source person; if that person falls into this set, we will not perform human motion transfer.
Revised: We first conduct face recognition on the source person; if that person falls into this set, we will not perform human motion transfer.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The MotionFormer is trained for 10 epochs, and the learning rate decays linearly after the 5-th epoch.
Revised: The MotionFormer is trained for 10 epochs, and the learning rate decays linearly after the 5-th epoch.

Original: We provide the pseudo-code of the training process in Algorithm 1.
Revised: We provide the pseudo-code of the training process in Algorithm 1.

Original: We denote the Transformer encoder of the source images as En s , the Transformer encoder of the target pose as En t , the
Revised: We denote the Transformer encoder of the source images as En s , the Transformer encoder of the target pose as En t , the

Original: Transformer decoder as De , and the discriminator as D .
Revised: Transformer decoder as De , and the discriminator as D .

Original: We set the batchsize as 4 and step max is obtained by dividing the image numbers of dataset by batchsize.
Revised: We set the batchsize as 4 and step max is obtained by dividing the image numbers of dataset by batchsize.

Original: Meanwhile, we show the details of the model architecture and loss function in the appendix, this information is useful for the reproduce process.
Revised: Meanwhile, we show the details of the model architecture and loss function in the appendix, this information is useful for the reproduce process.

Original: Algorithm 1 Training Process
Revised: Algorithm 1 Training Process

Original: Require: A set of source images I s , target pose images P t , and person mask images M gt .
Revised: Require: A set of source images I s , target pose images P t , and person mask images M gt .

Original: for Epoch = 1, 2, 3, ..., 10 do for Step = 1, 2, 3, ..., step max do
Revised: for Epoch = 1, 2, 3, ..., 10 do for Step = 1, 2, 3, ..., step max do

Original: Sample a batch of source images I s , target pose P t , and person mask M gt .
Revised: Sample a batch of source images I s , target pose P t , and person mask M gt .

Original: Get S 1 , S 2 , S 

Revised: Get S 1 , S 2 , S

Original: =
Revised: =

Original: En s ( I s ) , T 1 , T 2 , T 

Revised: En s ( I s ) , T 1 , T 2 , T

Original: =
Revised: =

Original: En t ( P t ) ;
Revised: En t ( P t ) ;

Original: Get I out , M out , f f =
Revised: Get I out , M out , f f =

Original: De ([ S 1 , S 2 , S 3 ] , [ T 1 , T 2 , T 3 ]) ;
Revised: De ([ S 1 , S 2 , S 3 ] , [ T 1 , T 2 , T 3 ]) ;

Original: Calculate the adversarial loss in Equation (11) in the main paper;
Revised: Calculate the adversarial loss in Equation (11) in the main paper;

Original: Update D ;
Revised: Update D ;

Original: Calculate the loss in Equation (14) in the main paper;
Revised: Calculate the loss in Equation (14) in the main paper;

Original: Update De , En s , En t ; end for if Epoch ≥ 5 thenUpdate learning rate
Revised: Update De , En s , En t ; end for if Epoch ≥ 5 then

Original: A A PPENDIX A. D ETAILS OF A TTENTION P ROCESS AND E NCODER

Original: As mentioned in the main paper, we utilize the Cross-Shaped Window Self-Attention (CSWin Attention)

Original: Dong et al.

Original: as our Attention mechanism in the encoder and decoder block.

Original: The CSwin

Original: Attention is based on the standard N heads attention of the original Transformer layer Vaswani et al.

Original: The main difference is that the CSWin Attention calculates attention in the horizontal and vertical stripes in parallel.

Original: For the attention in horizontal stripes at the n -th head, the query

Original: Q ∈ R ( H × W ) × d , key K ∈ R ( H × W ) × d , and value V ∈ R ( H × W ) ×

Original: d are evenly split into M nonoverlapping stripes of equal width sw ( i .

Original: e .,

Original: sw = H/M ).

Original: Then, it computes the standard attention ( Softmax ( QK T / √ d ) V + B ) separately for each stripe, where B is the locally-enhanced positional encoding defined in Dong et al.

Original: Meanwhile, it adopts the same operation on the vertical stripes.

Original: Finally, the CSWin Attention concatenates the output of the horizontal (H-Attention) and vertical (V-Attention) to predict final results:

Original: [Equation]

Original: where W Qn , W Kn , W Vn ∈ R C

Original: ×

Original: d , d = C/N .

Original: Next, an MLP with GELU non-linearity between them is used for further feature transformations.

Original: The LayerNorm (LN) and residual connection are applied both before the Attention and MLP.

Original: The whole encoder block is then formulated as follows:

Original: [Equation]

Original: where X l ∈ R ( H × W ) ×

Original: C denotes the output of the l -th encoder block ( l > 1 ) or the precedent convolutional layer for the first encoder block of each stage ( l = 1 ).

Original: We set numbers of encoder block to 1 , 2 , 21 for the three encoding stages.

Original: S i and T i ( i = 1 , 2 , 3 ) denote the output of the final encoder block of the i -th stage for I s and P t , respectively, as shown in Fig.

Original: A. D ETAILS OF L OSS F UNCTION

Original: In addition to the proposed mutual learning loss, we use losses below to optimize our model.

Original: Reconstruction loss.

Original: We utilize the perceptual loss as the reconstruction loss, which minimize the feature map distance in an ImageNet-pretrained VGG-19 network Simonyan & Zisserman (2015):

Original: [Equation]

Original: where F i is the feature map of the i -th layer of the VGG-19 network.

Original: In our work, F i corresponds to the activation maps from layers ReLU1 1, ReLU2 1, ReLU3 1, ReLU4 1, and ReLU5 1.

Original: Feature matching loss.

Original: The feature matching loss L fm compares the activation maps in the intermediate layers of the discriminator to stabilize training:

Original: [Equation]

Original: where D i is the activation of the i -th layer in the discriminator.

Original: Style loss.

Original: We utilize the style loss to refine the texture of I out .

Original: We write the style loss as:

Original: [Equation]

Original: where G Fi is a C

Original: i × C

Original: i Gram matrix computed given the feature map F i , and M i is the number of elements in G Fi .

Original: These feature maps are the same as those used in the perceptual loss as illustrated above.

Original: Hinge adversarial loss.

Original: We adopt the hinge adversarial loss to train our discriminator D and generator G :

Original: [Equation]

Original: where h ( t ) = min (0 , − 1 + t ) is a hinge function used to regularize the discriminator.

Original: Total variation loss.

Original: This loss regularizes the flow field f f in the fusion block to be smooth, which is defined as

Original: [Equation]

Original: Mask loss.

Original: We feed the output of the decoder to a convolutional layer to predict a person mask

Original: M out , which aims to combine the background and I out n.

Original: We utilize the L 1 loss as the mask loss which can be written as:

Original: [Equation]

Original: 13) where the M gt is the ground-truth person mask, which is obtained with an off-the-shelf person segmentation model based on DeepLab V3 Chen et al.

Original: Total losses.

Original: The total loss function can be written as:

Original: [Equation]

Original: where λ re , λ fm , λ adv , λ mu , λ tv , λ mask , and λ style are the scalars controlling the influence of each loss term.

Original: Following the practice in Han et al.

Original: (2019b)

Original: ; Wang et al.

Original: (2018b), we set λ re = 10 , λ fm = 10 , λ adv = 1 , λ mu = 1 , λ tv = 0 .

Original: 5 , λ mask = 1 , and λ style = 10 .

Original: [Equation]

Original: [Figure]

Original: A RCHITECTURE

Original: Fig.

Original: 7, Fig. 8 and Fig.

Original: 9 show the architectures of our Transformer encoder, decoder and discriminator in the training step.

Original: ic denotes the number of input channels and oc is the number of output channels.

Original: In the Transformer encoder, ic = 26 for the target pose and ic = 3 for the source person image.

Original: The Flatten and UnFlatten are reshaping operations to make the shape of features suitable for the attention and convolutional layers, respectively.

Original: The Upsample operation doubles the spatial size and halves the number of channels of the input feature.

Original: The convolutional layers in the fusion block and mask prediction have the same architecture as this Upsample operation but without the concatenation.

Original: [Figure]

Original: [Figure]

Original: B A NALYSIS OF NUMBER OF THE DECODER BLOCKS

Original: We half the numbers of the decoder blocks to further analyze the performance of the decoder (this experiment is named Ours half).

Original: We set the number of each stage as 1, 2, and 6, respectively.

Original: As shown in Fig.

Original: 10, the results contain some artifacts when we reduce the number of decoder blocks.

Original: The numerical comparison shown in Table 4 is consistent with the visual observation.

Original: Please refer to the supplementary video for more results.

Original: Conv(ic, 64, 4, 2, 2)

Original: Conv(64, 128, 4, 2, 2)

Original: Conv(128, 256, 4, 2, 2)

Original: LeakyReLU

Original: InstanceNorm

Original: LeakyReLU

Original: InstanceNorm LeakyReLU Conv(128, 256, 4, 2, 2) InstanceNorm
Revised: Update learning rate; end if end for

Original: LeakyReLU

Original: Conv(256, 512, 4, 2, 2)

Original: InstanceNorm

Original: LeakyReLU

Original: Conv(512, 1, 4, 1, 2)

Original: [Figure]

Original: Method

Original: [Table]

Original: C M ORE V ISUAL C OMPARISONS

Original: We show more comparisons with LWG Liu et al.

Original: , GTM Huang et al.

Original: (2021a),

Original: MRAA Siarohin et al.

Original: (2021), DIST Ren et al.

Original: (2020) in Fig.

Original: 12 and Fig.

Original: The LWG fails to reconstruct the body with complicated human motion ( e .

Original: g ., the squat in the red box of the first row in Fig.

Original: In contrast, the GTM can synthesize the body shape better, but the textures are blurry.

Original: The MRAA captures the motion in an unsupervised manner, and the performance is constrained by the precision of motion prediction.

Original: DIST does not capture the correct relationship between the source image and target pose ( e .

Original: g ., the arm in the last example in Fig. 11) and suffers from overfitting ( e .

Original: g ., the shoes become red in the second example in Fig. 11).

Original: Overall, our method can effectively synthesize images with more accurate poses and finer texture details.

Original: We further provide video comparisons in the supplementary video.

Original: And we find our method can synthesize more visually pleasing and temporally coherent results.

Original: Our proposed framework generates images with the highest visual quality.