Paper: HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR-2024)

Reviewer comment (Experiments):
An important baseline that finetunes on HumanVerse images without structure outputs using the SoTA network architecture is missing. It is not clear to me how much improvement the structure prediction brings to the performance. i.e. do we really need to predict the structure information or we can just fine-tune the general-purpose T2I model to the human images dataset?

Author response:
**A2:** Yes, we totally agree that such a baseline experiment is very important to show the effectiveness of our method. We would like to modestly point out that we have already done this ablation experiment in the "Denoise RGB" setting of Table 2, main paper. This ablation setting means that we only denoise the RGB as target while not simultaneously denoising any structural maps, which just means finetuning on HumanVerse without structure outputs. The performance gain ($20.76$\% in $\text{FID}$ and $23.66$\% in $\text{FID}_\text{CLIP}$) from "Denoise RGB" to the full model ("HyperHuman (Ours)") shows the effectiveness of our proposed method. Besides, such framework also supports more functionalities by jointly outputting the spatially aligned RGB, depth, and surface-normal maps.

Reviewer comment (Reproducibility):
The curated HumanVerse dataset is claimed to be one of the contributions, but the author did not mention if they will open-source the dataset. I hope the authors can clarify this as it can hurt the contribution of this paper if the dataset is kept internal.

Author response:
**A3:** Thanks again for acknowledging that our dataset could benefit future research! We definitely would love to release the dataset, but are not able to commit to exact timelines at this point.

Reviewer comment (Experiments):
It is not clear to me why to use independent noise for modeling the different modalities, have you tried modeling using a single noise?

Author response:
**A4:** Yes, at the early stage of our experiment, we have tried to use a single noise for different modalities. However, the model fails to generate reasonable structural maps and RGB images at the inference stage. The key reason is that sampling a single noise gives the model a shortcut to trivially learn all the modalities. Specifically, when we sample a single noise for different modalities, the required outputs for different branches are exactly the same. In this way, the model can cheat to only learn the distribution of a single modality and just replicating the predicted noise to other branches. Though seemingly converged at the training stage, it fails to capture the joint distribution of all the modalities, which performs poorly in simultaneously generating aligned results.

Reviewer comment (Theory):
Depth, normal etc have different distributions to RGB images, how do you ensure the VAE can capture its semantic in the latent space?

Author response:
**A5: 1)** As also shown in **A1**, even for the depth and surface-normal maps of different distributions to RGB images, they can still be well reconstructed both quantitatively and visually. This suggests that we faithfully compress structural maps like depth and surface-normal into latent space for diffusion learning.

**2)** A faithful reconstruction means there is little compression loss in the structural map encoding process. Such accurate reconstruction emcompasses both the low-level image details and rich latent space semantics.

Reviewer comment (Experiments):
The label of the HumanVerse dataset has been created based on other pre-trained models, such as depth and pose estimation models. However, the quality of the dataset does not guarantee accurate labelling (especially in crowded scene or extreme lighting condition), and its reusability for other tasks remains uncertain.

Author response:
> **Q1: "The label of the HumanVerse dataset has been created based on other pre-trained models, such as depth and pose estimation models. However, the quality of the dataset does not guarantee accurate labelling (especially in crowded scene or extreme lighting condition), and its reusability for other tasks remains uncertain."**

**A1: 1)** It’s a common practice for existing studies to use other pre-trained models for estimation and labeling. For example, SDv2-depth (https://huggingface.co/stabilityai/stable-diffusion-2-depth) also uses MiDaS [a] for depth labeling; MonoSDF [b] also uses Omnidata [c] for surface-normal estimation as supervision and structural guidance; ControlNet, T2I-Adapter, and HumanSD also use OpenPose [d] or ViTPose [e] for body keypoint detection.

**2)** We have tried our best to make the annotations more accurate. For example, when choosing the backbone for body skeleton annotation, we use ViTPose-H, which performs the best over several pose estimation benchmarks. Besides, we also propose to outpaint each image for a more holistic, which gives more accurate annotations. Please kindly refer to the "Outpaint for Accurate Annotations" paragraph of main paper Section 4 for how we improve annotation pipelines for more accurate labeling.

**3)** We admit that certain estimation error indeed exists due to the limitation of current state-of-the-art estimators. However, training on the large-scale dataset with noisy labelings can still contribute to a robust model. For example, the image captions of current text-to-image dataset (*e.g.*, LAION and COYO) are quite noisy, with some irrelevant information like HTTP tags or random emojis. In spite of this, robust text-to-image models can be well trained with unprecedented performance and satisfactory text-image alignment.

Reviewer comment (Evaluation):
The paper's objective is to address incoherent parts and unnatural poses. However, it falls short in terms of providing quantitative metrics to evaluate the effectiveness of the proposed method in addressing these issues.

Author response:
> **Q3: "About evaluating the incoherent parts and unnatural poses."**

**A3: 1)** The incoherent parts and unnatural poses appear as human image artifacts in the pixel space. Since the image quality metrics of $\text{FID}$, $\text{KID}$, and $\text{FID}_\text{CLIP}$ calculate the distribution distance between the real data and generated data, they can capture those generated artifacts that have gap with real data distribution. In this way, the incoherent parts and unnatural poses can be reflected with poor image quality metrics. The superior performance of our method on image quality metrics can show the effectiveness for solving these unnatural cases.

**2)** Though incoherent parts and unnatural poses could be hard to quantitatively evaluated with automatic pipeline, we have conducted a comprehensive user study with extensive visual comparisons. As human evaluators are highly sensitive to the human image artifacts of unreasonable structures, the better human preference of our model shows our effectiveness. Besides, from the extensive qualitative comparisons as shown in the main paper Figure 1 and Appendix Figure 13, 14, 15, 16, we surpass recent state-of-the-art T2I models with more coherent and natural human generation.

Reviewer comment (Experiments):
This paper lacks a fair comparison on two fronts. Firstly, it compares itself to other non-pose conditional guided methods (such as SD and DeepFloy-IF). Secondly, the method discussed in the paper was trained on the HumanVerse dataset, which exclusively contains the rich human class, while other methods are trained on a broader set of classes.

Author response:
> **Q4: "Comparison with some non-pose conditional guided methods, and the fact that not all existing works are trained on the exact same dataset."**

**A4: 1)** We would like to modestly re-emphasize that our focused setting is **pose-conditioned human generation**. Human generation is one of the most challenging sub-domains in current text-to-image studies, which fails to be well addressed in general T2I models. Therefore, we consider it as an individual research problem.

* Based on this, we follow the comparison settings of previous controllable text-to-human studies to compare with both general T2I models and controllable methods with pose condition, and use the officially released model for all the baselines. We would like to politely point out that the amount of resources and computation to re-train all existing large-scale foundation image models on the same dataset would be too huge for us to afford. We sincerely hope that you could kindly understand this.

* For comprehensive comparisons, we have grouped comparison models into two categories (the "Comparison Methods" paragraph of main paper Section 5): **a)** General T2I models, including SDv1.5, SDv2.0, SDv2.1, SDXL, and DeepFloyd-IF; **b)** Controllable methods with pose condition, including ControlNet, T2I-Adapter, and HumanSD. To further highlight the difference between these two types of comparison methods, we have modified the table by adding a horizontal line to make them more clearly separated. Please kindly refer to Table 1 of the main paper for the revision. We would like to kindly mention that the reason we include the results from non-posed conditional models like the Stable Diffusion series is to provide *a detailed and comprehensive analysis* between various approaches.

**2)** All the quantitative and qualitative evaluations are conducted on the zero-shot MS-COCO 2014 validation dataset, which is unseen to all the tested models. This can further guarantee the fairness of comparisons.

Reviewer comment (Reproducibility):
According to Figure 2, the pose skeleton is encoded before being concatenated with noise. What is the encoder used for the pose, and how does the author process the pose input before encoding it?

Author response:
> **Q5: "According to Figure 2, the pose skeleton is encoded before being concatenated with noise. What is the encoder used for the pose, and how does the author process the pose input before encoding it?"**

**A5:** Thank you for pointing out this question! We have polished to make clarifications in the Appendix Section A.13 "More Details on Pose Processing and Encoding".

**1)** The encoder used for pose is the pretrained VAE encoder of Stable Diffusion, which is the same as the encoder used for RGB, depth, and surface-normal maps.

**2)** Before pose encoding, we visualize the body keypoints on a black canvas to form a skeleton map, similar to previous controllable methods with pose condition. Specifically, we use exactly the same pose drawing method as HumanSD and T2I-Adapter to ensure fairness.

Reviewer comment (Evaluation):
What about inference requirements (GPU memory, time)?

Author response:
> **Q6: "What about inference requirements (GPU memory, time)?"**

**A6:** We run inference on a single 40GB NVIDIA A100 GPU. The first-stage Latent Structural Diffusion Model uses 2 seconds for 50-step DDIM sampling, which is roughly similar to SDv2.0. The second-stage Structure-Guided Refiner uses 10 seconds for 50-step DDIM sampling, which is roughly similar to SDXL. We have included this in the "Implementation Details" paragraph of the main paper Section 5.

Reviewer comment (Evaluation):
How do we ensure the input pose and caption are aligned with the predicted depth, map, and surface normal? Is there any evaluation on this?

Author response:
> **Q7: "How do we ensure the input pose and caption are aligned with the predicted depth, map, and surface normal? Is there any evaluation on this?"**

**A7: 1)** We have evaluated the spatial alignment between the predicted RGB and structural maps (depth and surface-normal maps) in ablation study. Specifically, we extract the depth and surface-normal from the predicted RGB image by off-the-shelf estimator, then calculate the $\mathcal{L}_2$-error with the predicted depth and surface-normal as a proxy for alignment. The ablation study results are shown in Table 2 of the main paper, which shows that our structural expert branch design can generate aligned RGB images, depth and surface-normal maps. Besides, the pose-images and RGB images are aligned as validated by Pose Accuracy metrics, and the image and captions are aligned as validated by text-image alignment metrics of CLIP score. Therefore, we can conclude that the input pose, text prompt (caption), synthesized RGB image, predicted depth and surface-normal maps are all aligned with each other.

**2)** Since we can use the predicted depth and surface-normal to generate the aligned image with input pose and caption, this indirectly shows that they are aligned. Otherwise, if there exists misalignment, the generated image will be unaligned to input pose or caption, leading to generation conflicts. Extensive visualization samples also show that the predicted depth and surface-normal maps are aligned with input pose and caption.

Reviewer comment (Presentation):
In the qualitative result, the comparison among other SOTA methods does not include the pose skeleton figure, I suggest adding it for better visualization.

Author response:
> **Q8: "In the qualitative result, the comparison among other SOTA methods does not include the pose skeleton figure, I suggest adding it for better visualization."**

**A8:** Thanks for your precious advice! The Figure 1 of the main paper shows the corresponding conditioning pose for each generation sample. We have further included the pose skeleton figures for all the qualitative comparisons in the Figure 13, 14, 15, 16 of the Appendix Section A.13 "More Details on Pose Processing and Encoding".

Reviewer comment (Presentation):
The description of Figure 2 lacks clarity, particularly when the authors refer to the colours "purple" and "blue." However, the figure itself makes it confusing to recognize which part they mention. I suggest the authors should redraw it.

Author response:
> **Q9: "Improve the description of Figure 2."**

**A9:** Many thanks for your advice! We have polished the pipeline illustration in Figure 2 of the main paper to make it easier recognizable.

Reviewer comment (Experiments):
I suggest adding the ablation study between joint denoising the targets and individually denoising the targets to show the effectiveness of joint denoising. This is the main technical contribution beside the new dataset.

Author response:
> **Q10: "I suggest adding the ablation study between joint denoising the targets and individually denoising the targets to show the effectiveness of joint denoising."**

**A10:** Thank you so much for the insightful suggestion!

**1)** Actually, we have tried to individually denoise each target at the early stage of this work, where we separately finetune the text2depth, text2normal, and text2image models. However, we find that due to the high-variance and stochaticity of diffusion model, the individually predicted RGB, depth, and surface-normal are totally unaligned with each other. Even with the help of same prompt, same random seed, and same initial noise $\mathbf{x}_T$, they are still mismatched. This makes them unable to be used for the second-stage Structure-Guided Refiner of conditional generation. 

**2)** Due to the current resource and time limit, we could not conduct this ablation experiment right now. But we will definitely add the experiment results in the final version. Thanks in advance for your kind understanding to this situation!

Reviewer comment (Experiments):
In Table 7: Additional Ablation Results for Structure-Guided Refiner, no ablation study on conditioning on predicted low-resolution image x (from the first module).

Author response:
> **Q11: "In Table 7: Additional Ablation Results for Structure-Guided Refiner, no ablation study on conditioning on predicted low-resolution image x (from the first module)."**

**A11:** In the current pipeline, we do not take the low-resolution image $\mathbf{x}$ as a condition for the second-stage Structure-Guided Refiner, mainly due to two considerations:

**1)** It’s hard to take a coarse-version RGB image as condition at the training stage, since it is too time consuming to use the diffusion-based inference to get the coarse images from the first-stage model. An efficient choice is to get a low-resolution image simply by downsampling (directly resize to low-resolution images). However, this will make the second-stage model degrade to a super-resolution model. The disadvantage is that the generation diversity is largely limited, since we can only create a super-resolution view of the original image, while the appearances can not be flexibly changed. 

**2)** Taking the predicted structural maps rather than the low-resolution images as conditions strike a good balance between the structural guidance and generation freedom: **a)** On the one hand, they give the fine-grained explicit structural guidance for generation of better quality. **b)** On the other hand, unlike the coarse image condition that restricts the appearance in synthesis, it still leaves much freedom for diverse results. With such conditioning design, our second-stage Structure-Guided Refiner can be inserted into any pretrained diffusion models in a plug-and-play manner.

Reviewer comment (Experiments):
The curated dataset is filtered with the criteria: only those images containing 1 to 3 human bounding boxes are retained; people should be visible with an area ratio exceeding 15%; plus rule out samples of poor aesthetics (< 4.5) or low resolution (< 200 × 200). I wonder if these rules reduce the diversity of the images that the model could produce, e.g., group/family photo or small faces? This is also observed in some visualizations in the supplementary materials where it does seem like the model improves the visual quality at the cost of diversity. I wonder if the authors could comment more on this? Besides, the CLIP alignment score is also a bit lower than SDXL, is it related to this diversity issue?

Author response:
> **Q1: "How the data filtering criteria relates to the generation diversity. The comparison with SDXL for CLIP score."**

**A1: 1)** Filtering out low-resolution or poor-aesthetic images for training is a common practice in prevalent diffusion text-to-image pipelines, *e.g.*, SDv1.5 and SDv2.0.

**2)** We filter out too-many persons and over-small human images mainly due to below reasons:

* For those small human images, the prompts are very likely to be uncorrelated with human, since people is not the main subject of this image. This could confuse the model training. 

* For the too-many person images, there are two issues with its caption and image quality: **a)** The noisy image captions mostly fail to describe the detailed information of each person, forcing the model to learn an one-to-many mapping from prompts like “a group of people” to *arbitrary number* of *arbitrary appearance* humans, which is an ill-mannered problem and hard to learn. **b)** Most multi-person images are confronted with the camera focus problem, where camera focusing on the nearer or further humans will make other human instances highly blurry of bad quality. 

* The focused problem of this work is pose-conditioned human generation, which allows users control body skeleton according to their needs. Therefore, we follow the experiment settings of controllable T2I baselines like ControlNet, T2I-Adapter and HumanSD to filter an image subset that could be accurately annotated by off-the-shelf pose estimators. Notably, our dataset domain is much more **diverse** than baselines, including various backgrounds and partial human regions such as clothing and limbs.

**3)** The image quality evaluation metrics of $\text{FID}$, $\text{KID}$, and $\text{FID}_\text{CLIP}$ calculate the distribution gap between the real data and generated data, which can reflect both aspects of image quality and **diversity** [a]. The superior performance on all the image quality metrics can reflect the diversity of our method. As also recognized by **Reviewer zFqJ, 1Ma5, jWKR**, we can generate high-quality human images in **diverse** scenarios.

**4)** As shown in Figure 13, 15, 16, 17, 18 in the Appendix, we manage to generate multiple humans of high realism and image quality. Besides, throughout all the qualitative results and visual comparisons, we have demonstrated diverse generation of various layouts under diverse scenarios, *e.g.*, different age groups of baby, child, young people, middle-aged people, and old persons; different contexts of canteen, indoor scenes, in-the-wild roads, snowy mountains, and streetview, etc. How to enhance realistic multi-person generation of high diversity remains an open question. Thank you so much for pointing out this important problem! We will explore this in future work.

**5)** We would like to modestly point out that CLIP score focuses more on text-image alignment, rather than image diversity. Besides, as explained in the "Quantitative Analysis" paragraph of main paper Section 5.1, SDXL uses two text encoders with $3\times$ larger UNet of more cross-attention layers, leading to superior text-image alignment. In spite of this, we still obtain an on-par CLIP score and surpass all the other baselines that have similar text encoder parameters.

Reviewer comment (Experiments):
Since providing accurate poses in real applications is hard, it would be very useful if the model also produces pleasant unconditional results. So I am also curious about the unconditional results. To be specific, how about images synthesized with just the text input without the poses?

Author response:
> **Q2: "How about images synthesized with just the text input without the poses?"**

**A2:** Thank you so much for pointing out this! We have included the unconditional generation results and analysis in the Appendix Section A.9 "Model Performance without Input Pose". Thanks to our framework design of robust conditioning scheme, the model is trained to predict reasonable denoising results, even when the conditions are dropout or masked. Therefore, we manage to create realistic human images with superior performance even without the pose skeleton as input.

Reviewer comment (Evaluation):
Regarding the results in Table 1, I am not sure why SDXL is much worse than SD? Could the authors further clarify on this? In addition, it also looks a bit strange to have DeepFloyd-IF 1024 images downsampled. Is it possible to directly compare the 1024 resolution with the proposed method?

Author response:
> **Q5: "Why SDXL is much worse than SD in Table 1? DeepFloyd-IF 1024 images downsampled are strange. Is it possible to directly compare the 1024 resolution with the proposed method?"**

**A5: 1)** As reported in Appendix Section F of the SDXL paper [b], its FID score is indeed worse than SDv1.5 and SDv2.1 on zero-shot MS-COCO dataset. Besides, we surpass all the baseline models including SDXL and SD series by a clear margin, in terms of both image quality metrics and the subjective user study. This can validate the superiority of our proposed method. 

**2)** Since the Ground Truth images of MS-COCO dataset are in resolution of 512x512, we have to downsample all the generated images to 512x512 for image-quality metrics evaluation. Note that the $\text{FID}$, $\text{KID}$, and $\text{FID}_\text{CLIP}$ are incomparable for images of different resolution, because they have to encode the same-resolution image features for distribution distance calculation. 

**3)** In terms of the comparisons in 1024x1024 resolution, we have already shown in the main paper with two aspects: **a)** All the presented visual comparisons against baselines are shown in 1024x1024 resolution. **b)** The user study is also conducted on generated images of 1024x1024 resolution. It can be clearly seen that our results are better than Deep-Floyd IF and multiple state-of-the-art baselines, generating more realistic images under high resolution.

Reviewer comment (Experiments):
Since the numerical results are mainly without the second-stage refiner plus the second-stage dataset is internal, it would be helpful to show visual results with only the first stage.

Author response:
> **Q6: "About visual results from the first stage."**

**A6:** Thank you so much for the precious advice! We have included more first-stage visual results in the Appendix Section A.10 "More First-Stage Generation Results". It can be seen that although the generation results are not as high-quality and realistic as those from the final two-stage pipeline, it still generates plausible humans with coherent structures.

Reviewer comment (Writing):
I am not sure if this intuition is well explained. Maybe the authors could further clarify “Such monotonous images may leak low-frequency signals like the mean of each channel during training; The zero terminal SNR (αT = 0, σT = 1) is further enforced to eliminate structure map’s low-frequency information.” in subsection “Noise Schedule for Joint Learning”.

Author response:
> **Q7: "Clarification of the intuition ‘Such monotonous images … low-frequency information.’."**

**A7:** Thank you for pointing out this problem! We have polished to include the below clarifications in the Appendix Section A.12 "Detailed Intuition of Updated Noise Schedule":

**1)** It is hard to finetune the Stable Diffusion to generate pure-color images. As shown in Figure 3(h) of the paper [c], we can not even overfit to a single solid-black image with the text prompt of "Solid black background". The main reason is that common diffusion noise schedules are flawed, which corrupts image incompletely when sampling $t=T$ at the training phase: $\mathbf{x}_T = \alpha_T \cdot \mathbf{x}_0 +  \sigma_T \cdot \mathbf{\epsilon}$, but $\alpha_T \neq 0, \sigma_T \neq 1$. Due to this reason, a small amount of signal is still included, which leaks the lowest frequency information such as the overall mean of each channel. In contrast, at the inference stage, the sampling starts from a pure Gaussian noise, which has a zero mean. Such train-test gap hinders SD from generating pure-color images.

**2)** Similar to pure color images, the depth and surface-normal maps are visualized based on certain scheme, where its color and patterns are highly constrained. For example, the depth map is grey-scale image without colorful textures, and current estimators tend to infer similar depth values for each local patch. Therefore, the low frequency information of per-channel mean and standard deviation could be misused by network as shortcut for denoising, which harms the joint learning of multiple modalities (RGB, depth, and surface-normal). Motivated by this, we propose to enforce the zero-terminal SNR ($\mathbf{x}_T = 0.0 \cdot \mathbf{x}_0 + 1.0 \cdot \mathbf{\epsilon}$, that is, $\alpha_T = 0, \sigma_T = 1$) to fully eliminate low-frequency information at the training stage, so that we manage generate both RGB images and structural maps of high quality at the inference stage.

Reviewer comment (Reproducibility):
Will the dataset be released for both stages?

Author response:
> **Q8: "About the datasets."**

**A8:** Thanks again for acknowledging that our dataset could benefit future research! We definitely would love to release the dataset, but are not able to commit to exact timelines at this point.

Reviewer comment (Experiments):
Does the input pose also contain the face landmarks or hand poses? Or just the body skeleton? If it's just skeleton, would adding hand poses helps with the generated details?

Author response:
> **Q9: "Does the input pose also contain the face landmarks or hand poses? Or just the body skeleton? If it's just skeleton, would adding hand poses helps with the generated details?"**

**A9: 1)** No, the current input pose is just the body skeleton without face or hand keypoints. We choose such skeleton mainly for the fair comparison consideration: three controllable T2I baselines all support the body skeleton as condition, while some of them can not be directly adapted to more fine-grained keypoints like face and hands. Besides, body skeleton is comparatively easier to obtain than face and hand keypoints, which allows more flexible user control. 

**2)** You are totally correct that adding more detailed poses helps with the generation! At the early stage of experiment, we verified the effectiveness of giving additional face and hand keypoints as conditions, which could indeed facilitate better image quality with more fine-grained face and hand generation. But in the later experiments, since we have to align our setting with baselines, we do not include them as conditions. We sincerely hope that you could kindly understand this.

Reviewer comment (Evaluation):
It would be insightful to understand how the model generalizes to unseen poses and whether there are specific pose complexities that present challenges.

Author response:
**A2: 1)** Thank you so much for the insightful suggestions! We further verify the model’s robustness and generalization ability on unseen and challenging pose of acrobatic pose. Specifically, we choose an acrobatic-related image from the Human-Art dataset [f], which is a highly challenging and rare pose unseen from the common human-centric images. The generation results and discussions are included in Appendix Section A.7 "Model Robustness on Unseen and Challenging Pose". It can be seen from Figure 6 that our model generalizes well to unseen poses. We also agree that generalization to highly rare/challenging poses is an open problem. We will explore this in future work.

**2)** The MS-COCO 2014 validation dataset itself is zero-shot, which means all the images and human poses are unseen at the training phase. Besides, all the quantitative evaluations, qualitative results and visual comparisons are generated from the zero-shot MS-COCO, which shows that we can generalize well to unseen poses and prompts.

**3)** Our dataset establishment process naturally guarantees the pose diversity of HumanVerse. Specifically, various human samples are curated, including different age groups, appearance, contexts (background scenes), poses, wearings, etc. As elaborated in the "Dataset Preprocessing" paragraph of main paper Section 4, even the partial human images such as clothing and limbs are preserved. This enables us to train an in-the-wild human generation model that is robust to unseen poses.

Reviewer comment (Experiments):
The computational requirements for annotation and training are significant, given the use of multiple GPUs. Are there potential optimizations or simplifications that could maintain performance while reducing resource demands?

Author response:
**A3:** Thank you for pointing out this interesting problem, which we believe can help our method adapt to different computational settings! We provide potential optimizations and simplifications from the training and annotation perspectives. All the discussions and experiment results are added to Appendix Section A.8 "Potential Optimization to Reduce Computation Cost".

**1)** From the perspective of optimizing training:

* We can change our models into a smaller diffusion backbone to save the training and memory cost, *e.g.*, Small SD (https://huggingface.co/segmind/small-sd) and Tiny SD (https://huggingface.co/segmind/tiny-sd) [a, b], which achieve on-par performance with Stable Diffusion, but lighter and faster in training and inference. 

* We can leverage some efficient parameter fine-tuning techniques like LoRA [c] and Adapter [d] to finetune the shared backbone with fewer parameters. 

* We can adopt some common engineering tricks to reduce memory consumption, *e.g.*, gradient checkpointing, gradient accumulation with smaller batch size, deepspeed model parallelism, lower floating point precision like fp16, efficient_xformers, etc.

**2)** From the perspective of optimizing annotation:

* Our efficient architecture design (only adding lightweight branches) can actually produce reasonable results with smaller dataset scale and fewer training iterations, capturing the joint distribution of RGB, depth, and surface-normal. Before the large-scale training, we first verify method effectiveness on a small-scale 1M subset, which is less than 3% of the HumanVerse full-set scale. In spite of this, we can still obtain good results with only 8 40GB A100 within one day, generating spatially aligned results for each modality. A generation sample is shown in Figure 7 of the Appendix Section A.8 "Potential Optimization to Reduce Computation Cost". Note that since this is an early-stage experiment, the pose conditioning and visualization are little bit different from the final version we have used. In spite of this, we manage to achieve simultaneous denoising of multiple modalities with a much smaller dataset scale.

* The annotation overhead mostly comes from the diffusion-based image outpainting process (the "Outpaint for Accurate Annotations" paragraph of main paper Section 4), while the cost for depth and normal estimation is relatively low. Though facilitating more accurate pose annotations, it is not a mandatory step. Moreover, in the final evaluation process, we use the raw human pose without the help of outpainting, but can still achieve superior performance.

Reviewer comment (Theory):
The model can generate photo-realistic and stylistic images. How does the model balance realism with artistic style variations, and could there be a more detailed explanation of how this balance is achieved?

Author response:
**A4: 1)** Currently, the balance between the photo-realism and artistic style variations comes from two aspects:

* If we want to enforce the generation of either photorealistic or artistic styles, we can use text prompts as explicit guidance, *e.g.*, the prefix of "artistic renderings of …", "a DSLR photo of …", etc.

* The finetune dataset domain influences the realistic-artistic balance a lot. For example, HumanSD [e] is trained on Human-Art dataset [f], which mostly consists of stylistic images. Therefore, they struggle to generate realistic images at the inference stage. In contrast, our dataset filtering process only requires the existence of partial human body, preserving both the photo-realistic and stylistic images for model training.

**2)** This is a very insightful question ignored by most existing studies! We come up with a potential idea thanks to your inspiration:

* We can label each image with a binary attribute of whether it is a photo-realistic one or a stylistic one. Such a condition can be taken as model input, similar to how we add the size conditioning in this work (the "Implementation Details" paragraph of main paper Section 5). In this way, we are free to adjust which style to generate at the inference stage. If such labeling is the extent of how photo-realistic or stylistic the image is, *i.e.*, a continuous value, we can even smoothly control the extent of photo-realism and artistic style variations. We will explore this interesting problem in future work.