Paper: 3D Interacting Hands Diffusion Model (ICLR-2024)

Reviewer comment (Experiments):
The DDIM sampling steps are set to 1000, which makes the fitting extremely slow and brings marginal improvement to vertex error.

Author response:
**1. Sampling speed**

Like many diffusion-based methods, we agree that sampling speed could be one of our weaknesses.
We will clarify this in the final version.
We observed that even with such a high number of iterations, thanks to our low-dimensional representation of two-hand (201 dimension), our sampling speed is affordable.
We observed that it takes about 25 seconds for 1000 iterations, shorter than 1 minute of Tevet et. al. (MDM).

Reviewer comment (Novelty):
The novelty is quite limited. The architecture is not different from the current 3D hand mesh reconstruction methods where cross-attention from different hands is widely used. What makes these three applications useful?

Author response:
**2. Novelty**

We think our biggest novelty is using image-independent generative model to **address the domain gap between in-the-lab datasets (e.g., InterHand2.6M) and in-the-wild images.** As discussed in the Introduction section, 3D annotations are available only for images captured from a lab environment, such as InterHand2.6M dataset, where those images have very different appearances (e.g., background, color space, and illuminations) from those of in-the-wild images. **Hence, simply training networks, including Zuo et al. ICCV 2023 and IntagHand, on such in-the-lab datasets fail to produce good results on in-the-wild images.** We could not experiment with Zuo et al.'s method as their code and pre-trained checkpoint are not available. **IntagHand gives 55.62 mm vertex error, 1.23 collision, and 3.57 contact, which is worse than the InterWild baseline and our IHDiff.** We used their official code and checkpoint. Please note that our numbers of IntagHand could be different from those from InterWild paper as we calculated the vertex error considering 3D relative translation between two hands, while the numbers in InterWild ignored it. Such worse results than InterWild are already verified in InterWild's paper. We think Zou et al. would suffer from the same problem as it does nothing to address the domain gap. On the other hand, our prior distribution is not conditioned on any data including images; hence, **even if it is trained only on in-the-lab datasets, it can be used to improve 3D interacting hands quality from in-the-wild images.** Table 3 and Figure 7 show that our IHDiff improves InterWild in physics-based metrics while preserving the vertex error.

Reviewer comment (Novelty):
Predicting the original clean sample instead of predicting added noise for denoising is a common practice in various diffusion model applications.

Author response:
**3. Predicting the original clean sample**

We agree and we do not argue this is our main contribution.
Please check out the above rebuttal (2. Novelty) for our main contribution.

Reviewer comment (Novelty):
Any network can model the prior distributions of hand meshes. What benefits does the method offer, aside from modeling through a diffusion model?

Author response:
**4. What benefits does the method offer?**

Please refer the the above rebuttal (2. Novelty).

Reviewer comment (Novelty):
Why is generating interacting hand meshes considered an important task?

Author response:
**5. Why is generating interacting hands meshes considered an important task?**

There could be two benefits.
First, being able to generate means we successfully modeled the prior distribution of 3D interacting hands. By successfully modeling the prior distribution, we could bridge the domain gap between in-the-lab and in-the-wild environment, as described in (2. Novelty).
Second, it can be used for many action decision problems, including human-like robots. Let's say human-like robots are doing some jobs or having social interaction with humans. For every timestep, robots should decide what to do next based on current/past observations and status. Robots should not do any unexpected or unnatural behaviors. In this case, we can regularize the next 3D hand pose of robots be included in our prior distribution so that robots do some plausible and meaningful interaction between two hands. Without such a regularizer, robots can do very unnatural hands, for example, collisions between two hands, which can destroy the robots.

Reviewer comment (Experiments):
Only the VAE part of MLD is used for comparison. More diffusion model-based baselines are needed for comparison.

Author response:
**6. More comparison to diffusion-based methods**

Unfortunately, there are no diffusion models for 3D interacting hands.
Instead, in Table 2, we compared ours with IHDiff (SA only). 
**In Section 3.2 and page 8, we described that the IHDiff (SA only) has almost the same network architecture as Tevet et. al. (MDM).**
Table 2 shows that our IHDiff produces much better results than IHDiff (SA only).
We will clarify more about this in the final version.

Reviewer comment (Reproducibility):
The methodology for integrating the generative prior with the regressor in an end-to-end manner remains ambiguous. It would be beneficial if the authors could delve deeper into this aspect, either in the appendix or the section dedicated to future work.

Author response:
Once our IHDiff is trained, we can design a regressor that takes a single image and estimates the latent code.
Then, the latent code can be passed to our pre-trained IHDiff to decode it to 3D interacting hands.
By estimating additional 3D global orientation and translation of the right hand, we can put the decoded 3D interacting hands in the camera-centered coordinate system.
Then, we can supervise the 3D interacting hands with various targets, such as 2D joint coordinates.
While training the regressor, we think there could be two options: fixing the pre-trained IHDiff or fine-tuning it.
We think this is a research problem.
If we fix IHDiff, it can preserve useful prior knowledge, learned in the studio data; however, there could be a small domain gap to the in-the-wild datasets.
On the other hand, if we fine-tune IHDiff, we can further reduce the domain gap; however useful prior knowledge could be diminished.
We think both directions make sense, and different research contributions should be made for those two directions.

Reviewer comment (Evaluation):
The model's capacity for enhanced generalization, especially concerning varying hand sizes and shapes, is not explicitly addressed. A clearer exposition on this topic would provide valuable insights into the model's adaptability and robustness.

Author response:
Thanks for pointing out such a valuable comment.
We actually agree with the reviewer.
In the current manuscript, what we did for the hand shape is regularizing it with our collision avoidance loss.
This should encourage our IHDiff to give hand shape parameters that do not introduce collisions given 3D poses; however, it is not fully investigated in terms of *diverse* hand shapes.
For example, the extremely thick hand could still introduce collision between two hands.
For this research direction, we think we first need to collect a 3D hand dataset with diverse hand shapes as currently available ones do not have enough diversity.
We'd like to push that research direction in the future.

Reviewer comment (Novelty):
The method seems to be a trivial modification of diffusion process with limited novelty.

Author response:
**1. Novelty**

We think our biggest novelty is using image-independent generative model to **address the domain gap between in-the-lab datasets (e.g., InterHand2.6M) and in-the-wild images.** As discussed in the Introduction section, 3D annotations are available only for images captured from a lab environment, such as InterHand2.6M dataset, where those images have very different appearances (e.g., background, color space, and illuminations) from those of in-the-wild images. **Hence, simply training networks, including Zuo et al. ICCV 2023 and IntagHand, on such in-the-lab datasets fail to produce good results on in-the-wild images.** We could not experiment with Zuo et al.'s method as their code and pre-trained checkpoint are not available. **IntagHand gives 55.62 mm vertex error, 1.23 collision, and 3.57 contact, which is worse than the InterWild baseline and our IHDiff.** We used their official code and checkpoint. Please note that our numbers of IntagHand could be different from those from InterWild paper as we calculated the vertex error considering 3D relative translation between two hands, while the numbers in InterWild ignored it. Such worse results than InterWild are already verified in InterWild's paper. We think Zou et al. would suffer from the same problem as it does nothing to address the domain gap. On the other hand, our prior distribution is not conditioned on any data including images; hence, **even if it is trained only on in-the-lab datasets, it can be used to improve 3D interacting hands quality from in-the-wild images.** Table 3 and Figure 7 show that our IHDiff improves InterWild in physics-based metrics while preserving the vertex error.

Reviewer comment (Experiments):
The paper mainly compared with InterWild (Moon et.al. CVPR 2023) in Table.3 on the HIC dataset. Why not use the interhand2.6M dataset for comparison?

Author response:
**2. Why not use InterHand2.6M for Table 3?**

As described in the paragraph right above Section 6, networks trained on InterHand2.6M already perform well on test images of InterHand2.6M as training and testing sets of InterHand2.6M look almost the same. As we described in the above rebuttal (1. Novelty), we are focusing on more real-world applications by **addressing the domain gap between in-the-lab datasets and in-the-wild images**. Hence, we used the HIC dataset (Table 3) and Figure 7, which consists of images captured in a daily environment, not from a studio.

Reviewer comment (Experiments):
The authors cited "Reconstructing Interacting Hands with Interaction Prior from Monocular Images" (Zuo et.al. ICCV 2023), which is the new state-of-the-art. It would be better to compare with it as it also claims an interaction prior. Although Zuo et.al. handled a more narrow task (image-based hand reconstruction), it is valuable to discuss the difference between the two priors for image-based hand reconstruction.

Author response:
**1. Novelty**

We think our biggest novelty is using image-independent generative model to **address the domain gap between in-the-lab datasets (e.g., InterHand2.6M) and in-the-wild images.** As discussed in the Introduction section, 3D annotations are available only for images captured from a lab environment, such as InterHand2.6M dataset, where those images have very different appearances (e.g., background, color space, and illuminations) from those of in-the-wild images. **Hence, simply training networks, including Zuo et al. ICCV 2023 and IntagHand, on such in-the-lab datasets fail to produce good results on in-the-wild images.** We could not experiment with Zuo et al.'s method as their code and pre-trained checkpoint are not available. **IntagHand gives 55.62 mm vertex error, 1.23 collision, and 3.57 contact, which is worse than the InterWild baseline and our IHDiff.** We used their official code and checkpoint. Please note that our numbers of IntagHand could be different from those from InterWild paper as we calculated the vertex error considering 3D relative translation between two hands, while the numbers in InterWild ignored it. Such worse results than InterWild are already verified in InterWild's paper. We think Zou et al. would suffer from the same problem as it does nothing to address the domain gap. On the other hand, our prior distribution is not conditioned on any data including images; hence, **even if it is trained only on in-the-lab datasets, it can be used to improve 3D interacting hands quality from in-the-wild images.** Table 3 and Figure 7 show that our IHDiff improves InterWild in physics-based metrics while preserving the vertex error.

Reviewer comment (Experiments):
I would like to see a direct comparison with IntagHand (Li et.al. CVPR 2022) for single-image two-hand recovery. In my opinion, IntagHand focused on capturing the interaction part of hands using attention, which might be more suitable than InterWild as a baseline.

Author response:
**1. Novelty**

We think our biggest novelty is using image-independent generative model to **address the domain gap between in-the-lab datasets (e.g., InterHand2.6M) and in-the-wild images.** As discussed in the Introduction section, 3D annotations are available only for images captured from a lab environment, such as InterHand2.6M dataset, where those images have very different appearances (e.g., background, color space, and illuminations) from those of in-the-wild images. **Hence, simply training networks, including Zuo et al. ICCV 2023 and IntagHand, on such in-the-lab datasets fail to produce good results on in-the-wild images.** We could not experiment with Zuo et al.'s method as their code and pre-trained checkpoint are not available. **IntagHand gives 55.62 mm vertex error, 1.23 collision, and 3.57 contact, which is worse than the InterWild baseline and our IHDiff.** We used their official code and checkpoint. Please note that our numbers of IntagHand could be different from those from InterWild paper as we calculated the vertex error considering 3D relative translation between two hands, while the numbers in InterWild ignored it. Such worse results than InterWild are already verified in InterWild's paper. We think Zou et al. would suffer from the same problem as it does nothing to address the domain gap. On the other hand, our prior distribution is not conditioned on any data including images; hence, **even if it is trained only on in-the-lab datasets, it can be used to improve 3D interacting hands quality from in-the-wild images.** Table 3 and Figure 7 show that our IHDiff improves InterWild in physics-based metrics while preserving the vertex error.