Paper: Towards 4D Human Video Stylization (ICLR-2024)

Reviewer comment (Experiments):
While there are plenty of video stylizatio works, in the experiment section, the authors only compare the proposed method with 3 exsisting methods, which seems to be not adequate.

Author response:
**[Q1] Comparison with more methods.**

We present 2 more methods in Table 3 & Figure 6. We use 50 videos on two datasets and collect about 5000 votes for user study. Visual comparison with state-of-the-art is shown in the video [01:20-03:29].

Reviewer comment (Evaluation):
In both qualitative comparison (table2) and user study (Figure 5), the advantage of proposed method over previous seems not very obvious. Especially in user study, the proposed method only have obvious advantage over CCPL but received similar evaluation compared to LST and AdaAttN.

Author response:
**[Q2] In Table 2 & Figure 5, the advantage seems not very obvious.**

**The advantage of our method is not obvious compared with certain methods because** the inputs of the proposed method and others are different. 2D video stylization methods use the original video as the input. Our method uses NeRFs to reconstruct the input and then stylizes the reconstructed video. The reconstruction would bring inconsistency because training frames are very sparse. However, despite this limitation, the proposed method still performs better than state-of-the-art. 

**The advantage of our method is quite obvious when all methods use the same input**, which can be verified in the ablation studies (Table 5). By replacing the input of the 2D stylization methods with the **reconstructed videos by our method**, our unified framework can generate more consistent results. To further show our superior performance on novel views and animated humans, we render another three videos and show the results in Table 3 and Figure 6. Here all methods use the same inputs, which are predicted by the proposed method. We can see that the proposed method strikes a superior balance between stylized textures and temporal coherence. Visual examples can be found in the supplementary video [01:20-03:29].

Reviewer comment (Novelty):
The authors are putting "4D" in the title, trying to emphesize the contribution on video stylizaiton. Personally I don't think it's proper to regard the video stylization as a main contribution of this paper, as the proposed method directly apply stylization on the NeRF rendered feature space and doesn't have any mechanism for inter-frame stablizaiton. Also, the novel view synthesis / human animation function and stylization seems to have little connection.

Author response:
**[Q3] "4D" in the title and inter-frame stabilization. Connection between the stylization in the rendered feature map, synthesis/human animation function and the proposed geometry-guided tri-plane representation.**

We adopt "4D" to highlight our contribution: the first method capable of stylizing entire scenes (human & background), under novel views and poses aside from the original videos. This has not yet explored in existing methods.

The inter-frame stabilization is ensured by two-step training. We first train NeRFs without stylization and decoder modules to reconstruct the input. Then we fix the density branch to optimize the stylization and decoder modules. Experiments in Table 2 & 3 show that our method generates better consistency than state-of-the-art. 

**The connection between stylization in the feature map and novel view synthesis/animation is built on efficacy via a unified framework.** Typically, novel view synthesis/human animation and stylization can be performed separately—first by employing NeRFs to predict novel views or animate humans, and then by applying 2D video stylization methods. However, this separate approach tends to result in inconsistency compared to our unified framework. This is verified by ablation studies in Table 5.

**The connection between stylization in the feature map and tri-plane representation is built on the same goal-efficiency.** There are two ways to achieve stylization with NeRFs: stylization on the 3D points or on the rendered map. We opt for the second way for the first one is computationally intensive. In addition, tri-plane is utilized for efficiency compared to MLPs when numerous points are sampled. Both stylization strategy and tri-plane representation contributes to efficient learning.

Reviewer comment (Novelty):
The stylization is directly applied on the NeRF rendered feature space, and seems to have littile connection to the proposed geometry-guided tri-plane representation and two branch NeRF for human and background scene. Also, in the METHODOLOGY section, the authors are repeatedly mentioning that many components of the framework are motived by previous methods. How could authors persuade reviewers the proposed the framework is innocative instead of an incremental work with the combination of previous works?

Author response:
**[Q4] Novelty.** 

While some components are known in the field, it requires **meticulous algorithmic design** to develop an efficient and effective model to synthesize humans in **novel views**, **novel poses**, and **different styles** with **few samples**. As shown in Table 1 and ablation studies (Section 4.3, Figure 2-3, Table 5-6), all the modules are of great importance to achive state-of-the-art results. We also show that a straighforward combination of some modules would not lead to the desirable results in the ablation studies. It is worth emphasizing that prior works can only achieve **one** or **two** effects (Table 1). We hope that reviewers appreciate more on the improtance of advancing the state-of-the-art with the proposed method (algorithm and implementation).

Our first novelty is that our 4D human video stylization method is the first one that is able to stylize the whole scene **under novel views and poses**, which has not been explored in the existing methods.

Our second novelty is that the proposed geometry-guided triplane can encode features more effective than the vanilla tri-plane. The vanilla tri-plane is sub-optimal when given **sparse views** as the input and will generate blurred prediction and inconsistency among frames. To solve this problem, we encode coordinates by U-Nets to generate more smooth details. Visual comparisons of both tri-planes in Figure 2 and supplementary video [04:29-04:47] shows that our method not only yields better consistency but also superior visual quality. This is further verified by the quantitative results in Table 6, which demonstrates the better consistency of our method.

Reviewer comment (Novelty):
Limited technical novelty. The NeRF reconstruction part is almost identical to NeuMan except that it uses a tri-plane representation, which has also been exploited widely. This choice is straightforward. The style transfer part follows AdaAttn without any modification. So the main paper is a straightforward combination of two existing works.

Author response:
We note that while some components are known in the field, it requires **meticulous algorithmic design** to develop an efficient and effective model to synthesize humans in **novel views**, **novel poses**, and **different styles** with **few samples (sparse background views)**. As shown in Table 1 and demonstrated in the ablation studies (Section 4.3, Figure 2-3, Table 5-6), all the modules in the proposed method are of great importance to achive state-of-the-art results, accomplishing all tasks (novel view, animation, stylization, and efficiency). We also show that a straighforward combination of some modules would not lead to the desirable results in the ablation studies. It is worth emphasizing that prior works can only achieve **one** or **two** effects (Table 1). We hope that reviewers appreciate more on the improtance of advancing the state-of-the-art with the proposed method (algorithm and implementation).

Our first novelty is that our 4D human video stylization method is the first method that is able to stylize the whole scene (human and background) **under novel views and poses**, which has not been explored in the existing methods.

Our second novelty is that the proposed geometry-guided triplane is able to encode the features more **effective** than the vanilla tri-plane [1]. The vanilla tri-plane has been widely explored for its **efficiency** compared to the MLPs in NeRFs. However, the existing tri-plane based methods such as [1] optimize the features of three planes directly, which is sub-optimal when given **sparse views** as the input and will generate noisy prediction and inconsistency among frames. To solve this problem, we propose the geometry-guided tri-plane that takes the voxel coordinates transformed by the positional encoding as the input. Three U-Nets are utilized to encode the triplane features, which utilize nearby pixels on the planes to represent the scenes and can generate more smooth details compared to the existing tri-plane structure. The visual comparisons of the proposed tri-plane and the existing tri-plane [1] on the **reconstructed background** is shown in Figure 2. It can be seen that the proposed tri-plane can recover more clear background texture (1st row) and sharper contours (2nd row).  For a more comprehensive set of visual evidence pertaining to reconstruction and stylization, please refer to the supplementary video [04:29-04:47]. Our method not only yields better consistency but also superior visual quality. This is further verified by the quantitative results presented in Table 6, which demonstrates the better consistency of our approach.

[1] Fridovich-Keil et. al. K-planes: Explicit radiance fields in space, time, and appearance, CVPR2023.

Reviewer comment (Experiments):
The overall visual quality is low, as shown in the paper and supplement material. Editing viewpoint and human pose is not new as this part is almost identical to the NeuMan. But for stylization, compared to existing baselines, the qualitative experiments are not enough, just less than 5 style transfer results are shown.

Author response:
We present additional visual examples that showcase 9 more styles in the supplementary video on novel views and poses [01:20-03:29]. Compared to the existing methods, our method demonstrates better consistency and visual quality. We note that other state-of-the-art methods could not achieve **all** effects achieved by our model.