Paper: Learning Implicit Representation for Reconstructing Articulated Objects (ICLR-2024)

Reviewer comment (Reproducibility):
The presented pipeline is overly complicated, with the iterative SIOS2 algorithm coupled with optical flow estimation to constrain the prediction of shape skeleton based on consistency across different frames of the video. And so, the method may not be easily extendable to potential new projects in this direction. Plus, I have concerns about the reproducibility of the method.

Author response:
1a. Complexity: As shown in Alg.1 from the Appendix, LIMR iteratively updates the reconstruction model (shape, time-varying parameters) and the skeleton, and the skeleton is updated according to the 2 intuitive physical constraints. We will try to make LIMR more easy to follow and not seem over-complicated to the readers.

1b. Extensibility: Regarding extensibility to new projects, we demonstrate LIMR's scalability to NMR-based methods (e.g.,[5]) and NeRF-based methods (e.g.,[3]), showing it can adapt to new tasks that use blend skinning for object pose manipulation with video input. Our method eliminates the need for ground truth skeletons required by existing approaches and does not require any additional annotations. Moreover, LIMR is able to obtain a skeleton comparable to that obtained by RigNet [8], which is trained with massive 3D data (mesh, skeleton, and skinning weights). We will include comparisons in the paper and examples are here https://drive.google.com/file/d/18evWJ-uBl7U7NoHH0PN1tKiz_7FAFX1U/view for immediate reference. Therefore, LIMR can be easily extended to methods like [2] that leverage RigNet [8] for the skeleton.

1c. Reproducibility: To ensure reproducibility, we have provided all implementation details of LIMR in both NMR-based and NeRF-based contexts in the Appendix (Section A.3). Additionally, we have described each loss function utilized in our study in detail (Section A.2). Following publication of our paper, we will immediately release the code for public access and reference.

Reviewer comment (Experiments):
Another major concern I have is the lack of diversity of shape samples used to demonstrate the superiority of the reconstruction/rendering results from LIMR (the paper’s approach). Specifically, Fig 3 shows limited diversity of shape samples (a Camel and a Horse). And I have not seen much diversity in the Appendix either (Fox, Zebra and Elephant; I would consider a Zebra equivalent to a Horse). Is this because PlanetZoo and DAVIS datasets only have these classes of animals?

Author response:
2. Diversity of Shape Samples: The diversity of the PlanetZoo and DAVIS datasets is limited. The structural, appearance, and size differences captured are indicated by the differences among, e.g., horse, dinosaur, giraffe, elephant, and tiger. We will include additional results for more animals, e.g., bear, cow, giraffe, dinosaur and tiger in the paper.

Reviewer comment (Presentation):
The rendering results are poor in quality, see Fig 3 and 7. It is difficult to evaluate the goodness of the results from these visualizations. It will help the paper if the renderings are improved.

Author response:
3. Rendering Image Quality: We will surely provide higher-quality renderings. In the meantime, we believe that Figures 3 and 7 already illustrate the improvements made by LIMR, particularly in challenging image regions within the red boxes. We will show video results on the project pages; meanwhile, we have added some examples at https://drive.google.com/drive/folders/1IUrSi-1VcBtr4FbIRrRSt9PuVrKYFE14 for immediate reference.

Reviewer comment (Evaluation):
The discussion in Section 4.3 are good. A similar discussion will be good for “Quantitative Comparison” paragraph that compares LIMR with LASR and ViSER. This is missing and the “Quantitative Comparison” paragraph is simply a plain run of Table1. Some intuition on the performance comparison is missing, perhaps due to the page limit?

Author response:
4. More Discussion about Quantitative Performance: We agree that an additional, Sec 4.3-like discussion about quantitative comparisons will be good to include, but we had to limit these details in Section 4.2 due to space constraints. We will add this to the Appendix.

Reviewer comment (Evaluation):
Having just CD as the eval metric for reconstruction task does not provide a full picture. Additional evaluation metrics such as EMD, F-Score, Normal Consistency and even IoU will provide a good picture of the quality of reconstructions.

Author response:
5. Additional Evaluation Metrics: We will add evaluation metrics of F-score and IoU. F-score clearly brings out our improvements over BANMo. Our IoU results are less pronounced due to the dominance of the silhouette loss component within reconstruction loss (In certain instances, LASR [5] may incorrectly deform the shape, but still yield a favorable IoU score. For example, rather than positioning the legs correctly, it erroneously stretches the area beneath the belly without correspondingly hurting the IoU score.

Reviewer comment (Writing):
I am not sure if the skeleton representation should be termed an implicit representation. It is intrinsic/internal to the shape. In today’s context of neural fields, using “implicit rep” to refer to skeletonization is misleading.

Author response:
1. Terminology: We agree and will replace our term 'implicit' with 'latent'.

Reviewer comment (Experiments):
The minimum number of videos that can be input to the method is 1. What is the max?

Author response:
2. Video Length: There's no maximum limit to video number and length. LIMR can produce satisfactory results with approximately 15 frames, but we can assure accurate reconstruction for only the viewed angles. We have not tested performance with fewer frames as they may not fully contain an action. Exploring the minimal frame limit is a potential avenue for future work.

Reviewer comment (Experiments):
Is there a limit on the video length? What is the minimum number of frames in the input video that should contain the articulating living being for LIMR to work? Can it work if there are just two frames that contain the living being? Just one frame is not possible I believe?

Author response:
2. Video Length: There's no maximum limit to video number and length. LIMR can produce satisfactory results with approximately 15 frames, but we can assure accurate reconstruction for only the viewed angles. We have not tested performance with fewer frames as they may not fully contain an action. Exploring the minimal frame limit is a potential avenue for future work.

Reviewer comment (Experiments):
What is the resolution of the video input?

Author response:
3. Video Resolution: For NMR-based LIMR, the training image resolution is 1920x1080 same as in [5]. For NeRF-based LIMR, images are downsampled to 64x64, matching the settings used in [3]. We will add these details to the paper.

Reviewer comment (Reproducibility):
What is the resolution of video frames processed by the LIMR framework? Same as video input or do you downsize?

Author response:
3. Video Resolution: For NMR-based LIMR, the training image resolution is 1920x1080 same as in [5]. For NeRF-based LIMR, images are downsampled to 64x64, matching the settings used in [3]. We will add these details to the paper.

Reviewer comment (Experiments):
Can LIMR work with grayscale videos?

Author response:
4. Grayscale Videos: Although not included in the paper, we had performed experiments showing that removing texture (color) related losses has a negligible effect on the resulting mesh quality (likely because texture information is already minimal for some animals which are of almost a single color, e.g., camels). This suggests that LIMR will be negligibly affected if we use gray-scale videos instead of color videos.

Reviewer comment (Experiments):
What is the range of number of bones and bone joints (max and min) that this method can work on?

Author response:
5. Bone Number Range: Initially, we worked with about 150-200 bones; later we could reduce this number to around 10 (e.g., for a tiger) while maintaining good performance.

Reviewer comment (Experiments):
Can videos containing multiple objects be used as input? If so, can all the objects be recovered in terms of their shapes? If not, which object will be selected in such a case? I guess the method will need an additional pre-processing step to detect object classes and/or add a saliency module. Can you comment on these? These are all important considerations that need to be discussed in the limitations section.

Author response:
6. Multiple Objects: We use GroundingDINO [6] and SAM [7] to obtain a mask for the target object. When multiple objects are present, GroundDINO and SAM can track and segment the target object using textual prompts.  LIMR reconstructs the object corresponding to the mask. Thus, LIMR works for one object at a time, selected by the user, from among the multiple objects detected and tracked. We will add these details and provide code in the supplementary materials or on GitHub.

Reviewer comment (Writing):
The usage of certain terms made it difficult to follow. For example: "semi-rigid part assignment" can be replaced by "skinning weights", which is a term that already exists in graphics and vision literature.

Author response:
(a) We will replace our term "semi-rigid part decomposition" with "skinning weights," which is a more conventional term within the graphics and vision literature.

Reviewer comment (Writing):
The term "implicit motion representation" is potentially misleading. It seems to be an explicit representation, given that it is actually a set of rigid transformations that are explicitly defined. "Internal / latent" representation may better describe skeleton movements.

Author response:
(b) We will replace our term "implicit motion representation" with "latent representation," to avoid confusion with explicit transformations; we agree that "latent" better captures the intended underlying skeletal movements that are not directly observable.

Reviewer comment (Writing):
The term "inverse blend skinning" (IBS) is unconventional and needs more explanation and highlighting. The naming "inverse of blend skinning" might not be accurate. Blend skinning maps a set of points X, rigid transformations G, and skinning weights W to another set of points X' = WGX. The inverse could be finding G and W from X'. My understanding is that the paper uses IBS to map vertex properties (e.g., 2D flow) to bones with an existing W. Is this correct?

Author response:
(c) We will replace our term "inverse blend skinning" (IBS) with simply "blend skinning" to align with standard practices and better convey its true role; we will also provide a more detailed explanation of it in mapping vertex properties to bones with an existing set of skinning weights.

Reviewer comment (Presentation):
The method section could be better structured with some "pointer" sentences. For example, bone motion direction is introduced in Eq (4) but it was not mentioned why it needs to be computed until Sec 3.3. This could be improved by either moving IBS to Sec 3.3 or pointing the reader to Sec 3.3 when motivating Eq (4).

Author response:
2. We will add pointer sentences to guide the reader through our methodology section. For instance, we will clearly state and explain the relevance, with reference to Section 3, of the rationale behind computing bone motion direction introduced in Eq (4).

Reviewer comment (Presentation):
It is not immediately clear what purpose Sec 3.2 serves. The rest of the paper describes the shape representation as meshes, but this section talks about the pros and cons of mesh vs neural fields. If the goal is to show that the method can be applied to neural fields as well, I feel this can be moved to the implementation details.

Author response:
3. Purpose of Section 3.2: We agree that the discussion in this section (about our method's ability to integrate with NeRF-based and NMR-based approaches) may distract the reader from the primary focus of our method, and will be better located as part of implementation details; accordingly, we will move this discussion to the implementation section.

Reviewer comment (Novelty):
Some related works are not discussed, such as: [A]-[B] find skeleton from video, [C]-[D] find skeleton from 3D point/mesh sequences, [E] finds skeleton from image features.

Author response:
(a) We thank 2K9U for pointing us to additional related works. We find these useful, will refer to them, and contrast our approach with these methods, particularly highlighting the differences in requirements for prior information and applicability in in-the-wild scenarios.

Reviewer comment (Novelty):
Since [A]-[D] also deal with motion data, some method-level discussion or even comparison is needed. [A] in particular is related as they also search for the skeleton structure and optimize shape in an alternating fashion.

Author response:
(e) General Discussion on Our Method vs Related Works: Methods like [2], [6], and [10] require a ground truth skeleton or pre-trained model like Rignet to obtain the skeleton. [1] gives decent results given multiple videos captured from different views and ground truth camera poses. In contrast, as discussed in sections  A.5 (NeRF-based methods VS NMR-based methods), such methods ([1], [2], [3], [4], [6]) struggle to obtain decent results given a short monocular video. Among current methods, Hi-Lassie [7] shows the best results in our setting (short monocular video without extra annotations and camera pose), and we will add a comparison with it in our paper; illustrative results we have obtained are at the link:  https://drive.google.com/file/d/1VRdZeV0GK3pELwL8Y7u8OYHuXW9xWAoI/view

In contrast, we believe that our method's ability to learn a satisfactory skeleton structure from a short monocular video without any prior skeletal, shape information, or camera poses is a significant contribution. We will revise our paper to emphasize this distinction, include the limitations of other methods in in-the-wild scenarios, and provide a visual comparison.

Reviewer comment (Experiments):
To strengthen the experiments, the bone discovery results could be separately evaluated and compared against some existing work, such as Rignet or Morig [D].

Author response:
(b) We agree that a comparison of our work with the approaches mentioned by 2K9U, particularly with Rignet [8] will strengthen the paper. We will include these in our discussion and also note the differences in that they require extensive ground truth 3D annotations such as high-quality mesh, skeleton, and skinning weights, which are expensive and challenging to obtain. An example of a comparison between RigNet and LIMR can be found at https://drive.google.com/file/d/18evWJ-uBl7U7NoHH0PN1tKiz_7FAFX1U/view?usp=sharing. These results show that our LIMR yields skeletons comparable to those given by RigNet, even though we do not require any 3D supervision.

Reviewer comment (Experiments):
For bone merging, one alternative is to use the similarity of SE(3) of bones. The effect of "2D flow" vs "3D rigid transform" could be ablated through an experiment.

Author response:
(d) Regarding alternative bone merging strategies: We have explored the use of the similarity of SE(3) transformations for bone merging. However, we found this approach to be not ideal due to the non-sparsity of skinning weights, which makes the motion of vertices (parts) depend on multiple bones, especially in the early stages with around 200 bones. Therefore, representation of the motion of each part by the SE(3) of one bone will lead to unreliable motion similarity assessments. Upon further exploration, we discovered that applying optical flow warp techniques significantly enhances the accuracy and logical coherence of part motions. We will add this discussion in the updated paper.

Reviewer comment (Writing):
Fig 1 caption: Is the rigidity coefficient a parameter? I thought R could be directly computed from D.

Author response:
(a) Fig 1 Caption: The rigidity coefficient is indeed derived from skinning weights; we will clarify this in the revised caption.

Reviewer comment (Experiments):
It would be more convincing to include video results and comparisons.

Author response:
(b) Video Results: We plan to show the video results on the project pages; we have included some examples that can be found here:https://drive.google.com/drive/folders/1IUrSi-1VcBtr4FbIRrRSt9PuVrKYFE14?usp=sharing for immediate reference.

Reviewer comment (Evaluation):
BANMo's result in Fig 3 appears particularly bad. Is there any explanation? The results on Swing also appear coarse. What is the resolution of the marching cubes? Also, overlaying the skeleton with the mesh similar to Fig 4 would make the results more compelling.

Author response:
(c) BANMo's Results: As discussed in our section A.5 (NeRF-based methods VS NMR-based methods), the NeRF-based methods such as ([1], [2], [3], [4], [6]) require many videos with from diverse viewpoints as input to obtain reasonable results. Given a single monocular video, they are not able to provide good results, which is also noted in CASA [11], Fig.6. And we will provide more results from NeRF-based methods in the paper.

Reviewer comment (Presentation):
Fig 6 is nice as it shows the initial configuration of the skeleton. It would be nicer to show the progression of the skeleton over iterations.

Author response:
(d) We will provide the progression of the skeleton over iterations in the paper.

Reviewer comment (Experiments):
Missing ablation of considering optical flow visibility (Eq. 4) and Laplacian contraction.

Author response:
(1) Ablation of considering optical flow visibility (Eq. 4) and Laplacian contraction: Thanks for pointing these out. We agree that this ablation will be useful. Our experiments reveal that removing visibility leads to instability and inaccuracy in calculating 2D bone motion when mapping 2D flow to 3D space. This is because typically each ray intersects the mesh at two points, with the 2D flow corresponding to the visible point. The non-visible point, often from a different part with an entirely different motion, causes instability when visibility is disregarded, resulting in an unreasonable skeleton. We will add this ablation to the main paper. We will also supplement it with ablation studies of the hyper-parameters of the mesh contraction module, including visual comparisons.

Reviewer comment (Evaluation):
The effectiveness of the rigidity coefficient/dynamic rigid does not seem substantial from Fig. 9 and Table 7.

Author response:
(2) Effectiveness of Dynamic Rigidity (DR): The 2D key point transfer accuracy we measure is primarily from visible views, and this does not effectively assess the physical plausibility of the entire mesh. We agree that Figure 9 could be enhanced by including more views to show the mesh differences (e.g., the unreasonable depressions in the body in the 'without DR' experiment); accordingly, we will add more views and animal comparisons to better demonstrate the improvements brought by DR.

Reviewer comment (Experiments):
The most recent method, MagicPony, also employs implicit and explicit representation and strongly relates to the proposed work, yet a direct comparison in the experiment is missing.

Author response:
(3) Comparison with MagicPony [4], WIM [1], CASA [8], and other Related Works: CAMM [2], and RAC [6] are similar as they all require ground truth skeletons (or pre-trained RigNet [9] to obtain skeleton) as input and leverage implicit fields (NeRF-based). As discussed in our Section A.5 (NeRF-based methods vs. NMR-based methods), such NeRF-based methods ([1],[2],[3],[4],[6]) place significant requirements on input videos in terms of the numbers of viewing angles and frame counts (over 1000 frames), as well as on ground truth camera poses (or use of pre-trained PoseNet [3] to obtain camera pose). When the input is a short monocular video, these methods fail to achieve satisfactory results (as seen from our Fig. 3 (b)). We will show more results from NeRF-based methods in the Appendix. In addition, we have found another recent work, Hi-lassie [7], which is similar to Magic Pony [4] but is based on NMR and yields acceptable results on monocular videos; our method, however, outperforms it as shown in https://drive.google.com/file/d/1VRdZeV0GK3pELwL8Y7u8OYHuXW9xWAoI/view. Further, we attempted to run CASA [8] but did not succeed in doing so since the authors did not release the ground truth skeleton for each animal, which is required to run the code as they use both predefined templates and skeletons. This makes a direct comparison difficult. We reached out to the authors multiple times via email but received no response.  

We will include a comparative discussion of all these methods in the paper.

Reviewer comment (Novelty):
Although as a post-processing step, WIM [1] also infers the skeleton with a variable number of joints for articulated targets and the relation could be discussed.

Author response:
We agree with your point that providing a comparison with WIM would make the paper stronger. We attempted to run WIM and compare it. However, the dataset we used does not have the ground truth camera pose, which is required in WIM. Additionally, our experiments focus on testing single monocular videos, while WIM is a method based on NeRF. We anticipate that without accurate camera poses and with a limited number of videos, it is highly likely that WIM may not be able to yield satisfactory results. In the alternative, we decided to compare with RigNet, which is trained with ground truth 3D supervision (mesh, skinning, and skeleton) to show the effectiveness of our method.

Reviewer comment (Experiments):
How would the result change without considering optical flow visibility and Laplacian contraction?

Author response:
Thank you for taking the time to read our rebuttal!

As you mentioned, two ablation experiments (Visibility and Laplacian contraction) are currently in progress (partly done). 

(1) We have already obtained results for key point transfer accuracy without visibility on five animal videos in the DAVIS dataset. The average is 77.3, which, although better than LASR (71.9) and ViSER (74.1), is lower than the results using visibility: 80.2. The PlanetZoo experiment is ongoing. The skeleton of DAVIS-camel without visibility is shown in the link: [https://drive.google.com/file/d/1a4HuUg-tJ8UF1_8c563tMQGZt9Fk1Trz/view?usp=sharing], which seems decent but not as good as using visibility. 

(2) By modifying parameters in Laplacian contraction, we found that the initial skeleton results are as shown in the link [https://drive.google.com/file/d/1UjaS_vkGyUx8QJj01OBluXdl-B6bo7QO/view?usp=sharing]. Although the number of bone points included is different, the overall structure is similar. We found out that all three types of initial skeletons can ultimately achieve results similar to those in the paper.

We will update the manuscript as soon as possible after completing the additional ongoing experiments.

Reviewer comment (Evaluation):
How should we interpret the improvement by rigid dynamics from Fig.9?

Author response:
(4) How should we interpret the improvement by Dynamic Rigidity (DR): As shown in Figure 9, using ARAP terms leads to unnatural and uneven depressions in the reconstruction. We believe this is because the ARAP term uniformly encourages distance preservation between all connected vertices in the mesh, which is unreasonable because certain areas (junctions of different parts) should have the freedom to deform, and only distances within semi-rigid parts should be restricted. Changing ARAP to DR allows deformations to occur more at the junctions of parts rather than within semi-rigid parts, thus maintaining the shape integrity of each part. In contrast, using ARAP, deformations might occur within a part, resulting in unreasonable depressions or uneven shapes. We will add this explanation to the paper.

Reviewer comment (Experiments):
Is there any reason more recent methods like CASA/MagicPony are not compared with the proposed work?

Author response:
(3) Comparison with MagicPony [4], WIM [1], CASA [8], and other Related Works: CAMM [2], and RAC [6] are similar as they all require ground truth skeletons (or pre-trained RigNet [9] to obtain skeleton) as input and leverage implicit fields (NeRF-based). As discussed in our Section A.5 (NeRF-based methods vs. NMR-based methods), such NeRF-based methods ([1],[2],[3],[4],[6]) place significant requirements on input videos in terms of the numbers of viewing angles and frame counts (over 1000 frames), as well as on ground truth camera poses (or use of pre-trained PoseNet [3] to obtain camera pose). When the input is a short monocular video, these methods fail to achieve satisfactory results (as seen from our Fig. 3 (b)). We will show more results from NeRF-based methods in the Appendix. In addition, we have found another recent work, Hi-lassie [7], which is similar to Magic Pony [4] but is based on NMR and yields acceptable results on monocular videos; our method, however, outperforms it as shown in https://drive.google.com/file/d/1VRdZeV0GK3pELwL8Y7u8OYHuXW9xWAoI/view. Further, we attempted to run CASA [8] but did not succeed in doing so since the authors did not release the ground truth skeleton for each animal, which is required to run the code as they use both predefined templates and skeletons. This makes a direct comparison difficult. We reached out to the authors multiple times via email but received no response.  

We will include a comparative discussion of all these methods in the paper.

Reviewer comment (Novelty):
The main concern lies in the necessity and value of the task in the current literature. There are two aspects to argue this: 1.) The reviewer guesses that given the current SoTA and technology in the community, the best way to model semi-nonrigid objects presented in this paper is to use template-based models. This paper presents animals and humans, which already have good template models. Only when the object motion structure differs a lot, and lacks a good template model, do we need some “unsupervised” algorithm to find the structure from videos. However, the paper never presents an example of such a case. 2.) The other way to recover the articulated object from the RGB (no depth) video is to first totally forget articulation and treat the scene as general dynamic 4D functions. In this way, given the current advanced dynamic rendering and reconstruction from monocular video, one can easily get the geometry as well as long-term correspondence first and then segment and easily extract the articulated object. Given these facts, I currently don’t believe this paper’s direction makes a real contribution. But as I write in the strengths, the idea of differentially learning the motion structure is interesting but needs to be carefully put into the right context and motivated nicely.

Author response:
We are thankful to dR3E for noting the interestingness of the ideas of learning skinning and differentiable skeleton our method uses, and that these may inspire other representations and general dynamic scene modeling and greatly appreciate dR3E for the weaknesses noted and the questions they have raised. We outline below our responses and how we plan to use these comments to improve the quality of our paper.

We disagree with the statement made by the reviewer that the model-based methods are the SOTA.
Despite the use of predefined models in model-based methods that endow them with a wealth of prior knowledge, intuitively, one would expect their performance to surpass that of template-free methods. However, experimental results often contradict this assumption. For instance, references [2] (Fig 1) and [1] (Fig 6) illustrate that the outcomes of model-based approaches ([3],[4],[5],[6]) are suboptimal despite employing predefined templates. Several reasons account for this:

1. Model-based methods assume that the shape template is sufficiently accurate, significantly restrict the shape's deformability, and permit only simple transformations such as stretching the length of each bone. They do so because (1) when mesh deformation is unconstrained, the mesh often loses detail and transitions into a crude shape in the first several stages, resulting in the complete loss of the original prior information, and (2) even if the shape were allowed to undergo substantial changes, the overall scale or scale of each part might vary, potentially creating new parts, which prevent the mesh from aligning with the skeleton template. The adverse impact of this restriction is seen in recent works such as [12] and [11], where the results closely resemble the initial shape templates, focusing primarily on learning pose without reconstructing the unique detailed shapes of each instance. For example, in [12], since the video of a cow used a buffalo template, the resulting model still retained horns, and the A-CSM [5] reconstructed a camel with only one hump, despite the video featuring a two-humped camel (shown in [2] Fig 1)

2. Although there are templates for many animal species, individual variances, even within the same species, are vast. For example, the camel and horse templates used in A-CSM [5] and SMALify [13] differ greatly from actual camel instances, and often, these instance shapes do not exist within the deformation space allowed by these template-based methods. Hence, even with the correct pose, the rendered silhouette may significantly differ from the ground truth silhouette.

3. In real-world applications, out-of-distribution (OOD) objects are inevitable, and articulated objects are not limited to quadrupeds and humans. For instance, our tests (Table 1) included dinosaurs (Tyrannosaurus Rex) that cannot be accommodated by any existing quadruped/human templates. Such cases highlight the need for more flexible and adaptable reconstruction techniques.

2) Ignoring the articulation and directly learning general dynamic 4D functions often yields unsatisfactory results because such methods require a large number of videos, captured from multiple viewing angles, and ground truth camera poses. Also, it leads to a large number of parameters, entailing lengthy training periods (e.g., D-NeRF [7] needs over two days as it learns per-point motion without blend skinning), and making it difficult to converge (mentioned in [2]). Also, they rely on accurate camera pose (mentioned in [8], [9]) and have difficulties handling large motions (mentioned in [9]). They always leverage COLMAP to obtain the accurate camera pose, whose performance is known to be bad when the input video is short and the object has a large motion. These requirements are not met by our, more challenging scenario of a single monocular video without camera poses. Hence, directly getting the geometry and then long-term correspondence is not an option in our case.

Reviewer comment (Experiments):
The method is somehow complicated and lacks principles when presenting, making it not very easy to follow. Given a complex algorithm like this, I can hardly tell which component is really important, although some ablations are provided, given the complexity of the model, these may not be enough.

Author response:
3) Importances of Components: To the best of our understanding and experience, we feel that in a challenging task like the one we have attempted in this paper (single, monocular, short video without prior template), it is not easy to depend on one or a few sources of information to estimate the shape, joint structure, and articulated motion. Because of the highly compressed, convoluted, and time-varying interdependence of the sources, we think any estimation algorithm will have to make joint use of one or more of the information sources captured in our formulation and loss function, their relative contributions possibly varying from time to time. We therefore think that there is no single component in the framework that is consistently important to reach the end goal. This distributed nature of information is confirmed by the related previous papers [1-1], which also do not quantify the relative contributions of the different sources beyond presenting the ablations.  In our paper Sec. 4.3, we have presented the following ablation studies to demonstrate the effectiveness and analyze the reasons for the effectiveness of the various modules: 
(a) Physical Skeleton (ours) vs Virtual Bones
(b) Refined Skeleton (ours) vs Initial Skeleton (Mesh Contraction) 
(c) Different Thresholds for Skeleton Refinement 
(d) Impact of Video Content on Skeleton
(e) Efficacy of Dynamic Rigidity and Part Refinement

Although Reviewers Z1Zw and gF7P have expressed satisfaction with the range of these ablations, we would be grateful if you (dR3E) could help us with which other ablations we could add; or how else could we specify module importances (given that there is no single consistently dominant module). We look forward to your suggestions about any additional experiments, analyses, etc. that we could include to improve our paper.
In addition to a-e above, we agree with your suggestion and plan to provide ablation studies on different parameters of the mesh contraction module.

Reviewer comment (Presentation):
The quality of the reconstruction shown in the figure is not appealing.

Author response:
4) We would be interested to know in what ways we could improve the quality of reconstruction to make it more appealing. The results we have shown are analogous to those presented in SOTA papers; we have shown that our results are superior to those of the SOTA methods [2], under the same conditions, and even better than those of the methods that use additional information (e.g., Banmo [8], that uses DensePose to predict camera pose). Despite our challenging experimental setup (single short monocular video without any shape/skeleton prior and without ground truth camera), LIMR has shown impressive results (mesh and skeleton). We will provide analogous comparisons of our mesh results with a more recent SOTA method [10] at   https://drive.google.com/file/d/1VRdZeV0GK3pELwL8Y7u8OYHuXW9xWAoI/view; we will add these to the paper.

Reviewer comment (Novelty):
The main question is the motivation for studying such a task in the way this paper presented.

Author response:
We are thankful to dR3E for noting the interestingness of the ideas of learning skinning and differentiable skeleton our method uses, and that these may inspire other representations and general dynamic scene modeling and greatly appreciate dR3E for the weaknesses noted and the questions they have raised. We outline below our responses and how we plan to use these comments to improve the quality of our paper.

We disagree with the statement made by the reviewer that the model-based methods are the SOTA.
Despite the use of predefined models in model-based methods that endow them with a wealth of prior knowledge, intuitively, one would expect their performance to surpass that of template-free methods. However, experimental results often contradict this assumption. For instance, references [2] (Fig 1) and [1] (Fig 6) illustrate that the outcomes of model-based approaches ([3],[4],[5],[6]) are suboptimal despite employing predefined templates. Several reasons account for this:

1. Model-based methods assume that the shape template is sufficiently accurate, significantly restrict the shape's deformability, and permit only simple transformations such as stretching the length of each bone. They do so because (1) when mesh deformation is unconstrained, the mesh often loses detail and transitions into a crude shape in the first several stages, resulting in the complete loss of the original prior information, and (2) even if the shape were allowed to undergo substantial changes, the overall scale or scale of each part might vary, potentially creating new parts, which prevent the mesh from aligning with the skeleton template. The adverse impact of this restriction is seen in recent works such as [12] and [11], where the results closely resemble the initial shape templates, focusing primarily on learning pose without reconstructing the unique detailed shapes of each instance. For example, in [12], since the video of a cow used a buffalo template, the resulting model still retained horns, and the A-CSM [5] reconstructed a camel with only one hump, despite the video featuring a two-humped camel (shown in [2] Fig 1)

2. Although there are templates for many animal species, individual variances, even within the same species, are vast. For example, the camel and horse templates used in A-CSM [5] and SMALify [13] differ greatly from actual camel instances, and often, these instance shapes do not exist within the deformation space allowed by these template-based methods. Hence, even with the correct pose, the rendered silhouette may significantly differ from the ground truth silhouette.

3. In real-world applications, out-of-distribution (OOD) objects are inevitable, and articulated objects are not limited to quadrupeds and humans. For instance, our tests (Table 1) included dinosaurs (Tyrannosaurus Rex) that cannot be accommodated by any existing quadruped/human templates. Such cases highlight the need for more flexible and adaptable reconstruction techniques.

2) Ignoring the articulation and directly learning general dynamic 4D functions often yields unsatisfactory results because such methods require a large number of videos, captured from multiple viewing angles, and ground truth camera poses. Also, it leads to a large number of parameters, entailing lengthy training periods (e.g., D-NeRF [7] needs over two days as it learns per-point motion without blend skinning), and making it difficult to converge (mentioned in [2]). Also, they rely on accurate camera pose (mentioned in [8], [9]) and have difficulties handling large motions (mentioned in [9]). They always leverage COLMAP to obtain the accurate camera pose, whose performance is known to be bad when the input video is short and the object has a large motion. These requirements are not met by our, more challenging scenario of a single monocular video without camera poses. Hence, directly getting the geometry and then long-term correspondence is not an option in our case.

Reviewer comment (Presentation):
More visualizations such as video comparisons like those shown in LASR and BANMo would be more intuitive and straightforward to show the object in the move/motion.

Author response:
1. Visual Comparisons: We will show videos of moving objects on the project page; we are including some examples here: https://drive.google.com/drive/folders/1IUrSi-1VcBtr4FbIRrRSt9PuVrKYFE14.

Reviewer comment (Experiments):
The proposed system includes multiple hyper-parameters and multiple separate but interdependent steps. Given the complications of the current system, it is unclear how robust this method is, eg, with regard to initialization, hyper-parameter settings, input video contents, etc.

Author response:
2. Hyperparameters and Robustness: The impact of video content and threshold hyperparameters related to skeleton updates on the reconstruction process is discussed in the experiments section. Further, we will: (a) elucidate the effects of skeleton initialization using different mesh construction parameters; (b) stability of results by comparing the results from multiple runs and the standard deviations; and (c) demonstrate the impact of input video contents by showing results for varying viewing directions.

Reviewer comment (Presentation):
Currently, the figures are not placed in an ascending order (Figure 7 before Figure 5) and the references to the figures in the text jump back and forth.

Author response:
3. Ordering of Figures and References: We acknowledge this weakness and will rectify it by renaming the figures.

Reviewer comment (Writing):
Some notifications are not well explained, eg, N_i in Eq. (3).

Author response:
4. Explaining Notation: We will add the clarification that \( N_i \) denotes the set of neighbor vertices of vertex \( i \), as well as look for any other such places for improvement.

Reviewer comment (Writing):
Typo: "(3)" in the first paragraph of appendix A: "quantitative"-->"qualitative".

Author response:
5. Typographical Errors: Thank you for noting this typographical error. We will replace the incorrect reference to "quantitative" in the Appendix with "qualitative".