Paper: A  Multi-resolution Dataset of Self-consistent Cloth Drapes for Physics-based Upsampling (ICLR-2024)

Reviewer comment (Novelty):
A primary concern regarding this paper is its relevance in the current landscape of cloth simulation and rendering. The field has seen a significant shift towards self-supervised learning approaches, as evidenced by recent papers such as SNUG (Santesteban et al., CVPR 2022), HOOD (Grigorev et al., CVPR 2023), and DrapeNet (De Luigi et al., CVPR 2023). These works have moved away from traditional supervised learning methods, which stands in contrast to the claim made in Section 2 of the paper that “Most recent approaches for neural garment deformations rely on supervised learning”. This discrepancy raises questions about the paper's alignment with current trends and best practices in the field.

Author response:
We value the recognition of our dataset's comprehensiveness and diversity and appreciate the acknowledgment of our objective as a dataset and benchmark paper to aid in the development and evaluation of existing upsampling methods as our contribution.

However, we would like to clarify an important aspect that was overlooked in this review: **the primary application of our dataset is specifically focused on cloth detail upsampling**. This task is uniquely defined as the direct enhancement of low-resolution, coarse cloth drapes (which are usually cheap and fast to compute/generate) into high-resolution, detailed, physically realistic versions, which inherently requires a multi-scale hierarchy (as defined in the abstract and introduction and detailed in Section 5 of our paper). **It is crucial to distinguish this cloth upsampling task from neural cloth simulation, a related yet completely different application** where the aim is to train a neural network that does forward time stepping, predicting realistic cloth motion (dynamics) with regression on a single scale. **These two are fundamentally different tasks, and our dataset is tailored to the former, not the latter**. We note that all of the three referenced papers here — SNUG by Santesteban et al. (CVPR 2022), HOOD by Grigorev et al. (CVPR 2023), and DrapeNet by De Luigi et al. (CVPR 2023) — all fall into the second category, which is not the task of this benchmark.

In terms of future trends, since **the task we are tackling is a fundamentally different one from learning garment motions**, it remains to be seen whether self-supervised approaches will be the dominant strategy for cloth detail upsampling. At the same time, we note that, for the state of the art, supervised learning methods, e.g., U-DNN (included in our benchmark), are currently in use. Likewise, benchmarks serve a dual purpose: they are not just tools enabling the development of new methods, but also for evaluating and comparing new and existing methods. It is commonly accepted that the results produced from current neural approaches, including those for learning dynamic garment motions still significantly lag in quality compared to SOTA high-fidelity cloth simulation outputs. Our analysis of existing upsampling methods with the high-fidelity cloth models in our PCS-generated dataset already exposes severe limitations in existing upsampling methods (as demonstrated in Table 1 and Figures 4, 7 and 8). These methods, while claiming various capabilities to generalize, when evaluated on the new dataset, demonstrate underperformance by missing the physically realistic and intricate folding behaviors formed through precise contact resolution across a wide range of real-world materials. This underperformance clearly demonstrates that the quality, complexity and diversity of the new dataset extends well beyond the original scope of the training data with which these methods were initially developed, and so demonstrates the need for both new research to bridge this gap and the utility of our current proposed benchmark.

Reviewer comment (Experiments):
The paper introduces a dataset for training and testing deep learning models, but there are concerns regarding its generalizability. The dataset includes a limited number of collider objects (100), which may not be sufficient for ensuring that models trained on this data can generalize well to unseen geometries, as indicated by the results in Table 3.

Author response:
In our network design, the PointNet encoder has been pre-trained on a large collection of shapes for learning occupancy fields  [Mescheder et al. 2019] (see Appendix F for details). This network design allows our network to be aware of the collider object geometry. We note that it is expected that a trained network performs better on in-distribution data (in our case, seen collider objects as in Table 1) and has performance degradation on out-of-distribution data (in our case, unseen collider objects as in Table 3). As shown in Table 3, while all methods suffer from performance drops on unseen collider objects, our learning-based baseline method still outperforms all competing methods by a large margin (at least 13% improvement on the Euclidean distance metric d_corr over U-DNN and 18% over Neural Subdivision). We look forward to further enriching our dataset with additional collision geometries, considering it an exciting avenue for future work.

Reviewer comment (Experiments):
The dataset and task are focused solely on static clothes subjected to gravity and resting on collider objects. This narrow scope may limit the utility of the dataset, as it does not reflect the dynamic nature of cloth in real-world scenarios.

Author response:
We reiterate that our focus is on cloth detail upsampling, a task that is fundamentally different from learning neural dynamic garment motions. 

Please see more discussion on the dataset's suitability for cloth detail upsampling within the current scope here (https://openreview.net/forum?id=aAhgJ1fQ4V&noteId=rqISyiyGnu).

Reviewer comment (Novelty):
The technical contributions of the paper appear to be modest. The proposed network architecture, consisting of a PointNet encoder and Neural Subdivision decoder, is relatively simple.

Author response:
We note that **the newly proposed baseline method for cloth upsampling is not the central emphasis of our work** – our paper's primary contributions, as a dataset and benchmark paper (as marked in the primary area section), lie in collecting high-quality data (currently no similar public datasets exist), benchmarking existing methods for this task and developing evaluation protocols for future research. Moreover, our detailed analysis of existing upsampling methods has unveiled a crucial insight: the diversity and complexity of the cloth drapes in our dataset expose severe limitations in existing upsampling methods. To demonstrate the potential for addressing this gap, we have further developed a learning-based baseline method for cloth upsampling that improves existing upsampling methods. This addition, not claimed as a novelty by us, serves as a valuable contribution we offer. As shown in Figures 4, 7 and 8, none of the existing upsampling methods can predict reasonable results. Our simple yet effective approach of incorporating collision geometries and material properties into the neural network has yielded plausible outcomes featuring detailed wrinkling and fewer intersections. We hope this can inspire future research to dive deeper and make further advancements.

Reviewer comment (Evaluation):
The use of different curvature computations as metrics for assessing mesh similarity is not clearly justified or explained, leaving the reader uncertain about the soundness and efficacy of these measures.

Author response:
The details of these curvatures were well explained in the supplemental of the PCS paper. Please see section 1 here (https://pcs-sim.github.io/pcs-supplement.pdf). We are happy to include them in our appendix if considered necessary. We will release the code that computes these metrics.

Reviewer comment (Novelty):
Why are supervised methods still relevant nowadays for cloth draping? This directly questions the relevance of a dataset for training such methods.

Author response:
We value the recognition of our dataset's comprehensiveness and diversity and appreciate the acknowledgment of our objective as a dataset and benchmark paper to aid in the development and evaluation of existing upsampling methods as our contribution.

However, we would like to clarify an important aspect that was overlooked in this review: **the primary application of our dataset is specifically focused on cloth detail upsampling**. This task is uniquely defined as the direct enhancement of low-resolution, coarse cloth drapes (which are usually cheap and fast to compute/generate) into high-resolution, detailed, physically realistic versions, which inherently requires a multi-scale hierarchy (as defined in the abstract and introduction and detailed in Section 5 of our paper). **It is crucial to distinguish this cloth upsampling task from neural cloth simulation, a related yet completely different application** where the aim is to train a neural network that does forward time stepping, predicting realistic cloth motion (dynamics) with regression on a single scale. **These two are fundamentally different tasks, and our dataset is tailored to the former, not the latter**. We note that all of the three referenced papers here — SNUG by Santesteban et al. (CVPR 2022), HOOD by Grigorev et al. (CVPR 2023), and DrapeNet by De Luigi et al. (CVPR 2023) — all fall into the second category, which is not the task of this benchmark.

In terms of future trends, since **the task we are tackling is a fundamentally different one from learning garment motions**, it remains to be seen whether self-supervised approaches will be the dominant strategy for cloth detail upsampling. At the same time, we note that, for the state of the art, supervised learning methods, e.g., U-DNN (included in our benchmark), are currently in use. Likewise, benchmarks serve a dual purpose: they are not just tools enabling the development of new methods, but also for evaluating and comparing new and existing methods. It is commonly accepted that the results produced from current neural approaches, including those for learning dynamic garment motions still significantly lag in quality compared to SOTA high-fidelity cloth simulation outputs. Our analysis of existing upsampling methods with the high-fidelity cloth models in our PCS-generated dataset already exposes severe limitations in existing upsampling methods (as demonstrated in Table 1 and Figures 4, 7 and 8). These methods, while claiming various capabilities to generalize, when evaluated on the new dataset, demonstrate underperformance by missing the physically realistic and intricate folding behaviors formed through precise contact resolution across a wide range of real-world materials. This underperformance clearly demonstrates that the quality, complexity and diversity of the new dataset extends well beyond the original scope of the training data with which these methods were initially developed, and so demonstrates the need for both new research to bridge this gap and the utility of our current proposed benchmark.

Reviewer comment (Experiments):
The dataset is said to contain “over a million” samples: how many exactly?

Author response:
There are 100 collider shapes, 3 collider size scalings, 2 cloth shapes, 3 cloth scalings, 24 cloth materials, 5 cloth initial rotations and 5 resolutions. This amounts to 100 x 3 x 2 x 3 x 24 x 5 x 5 = 1,080,000 samples.

Reviewer comment (Reproducibility):
How to split the coarse mesh into more detailed one?

Author response:
As mentioned in the problem formulation paragraph in Section 5, we use midpoint subdivision to divide a triangle into four triangles.

Reviewer comment (Novelty):
Some related work on the cloth dataset is missing: Towards Multi-Layered 3D Garments Animation, ICCV'23.

Author response:
Thank you for pointing this paper out. We note that this paper is published at ICCV 2023, which is **after** the ICLR paper submission deadline. We have now cited this paper in the cloth dataset paragraph of Section 2.

Reviewer comment (Experiments):
My major concern with the proposed dataset is the limited variety of cloth drape that is covered within it. It is true that the authors have put efforts into ensuring variation in factors such as different material types and collider objects, but it still spans a quite limited space of possible cloth drapes. For example, the dataset only includes cloth of square and round shapes, while in the real world, many interesting cloth examples do not belong to either of them, especially garments draped on human bodies which are of natural importance due to their applications. As another example, the final state of the draped cloth is significantly influenced by its initialization. For example, a cloth initialized as flat will be draped completely differently from being initialized evenly folded. Let alone all the infinite possibilities of manual interaction, for example in Fig. 1 (teaser) of the PCS paper (Zhang et al. 2022).

Author response:
We appreciate the recognition of the importance of a high-quality multi-resolution, cloth-drape dataset and benchmark evaluation protocols.

Here we'll focus on the important raised question regarding sufficient variety in our dataset:

Of course, we agreed that for the cloth upsampling task (and likewise generally) further increasing variety is always of benefit. Indeed we are excited to increase the dataset variety moving forward. At the same time, we'd argue that datasets are inherently constrained; a benchmark, in practice, can never comprehensively encompass every conceivable scenario within the environment. A dataset, at its core, is a substantial and well-defined collection of data, carefully scoped for the purposes of training and evaluation. The possibility of "better" should not come at the cost of the improvement for SOTA that this dataset provides.

Meanwhile, the natural draping of cloth itself, independent of material, boundary shape, human body draping and seamed garment panels, already produces detailed and complicated deformations. We'd like to disambiguate this important deformation complexity (for which existing learning methods already have significant trouble capturing) from the additional complexities posed by even-richer clothed human scenarios. To be very clear, we do not have confidence nor evidence that any of the evaluated upsampling methods will be applicable in such contexts. However, we want to highlight that **a predicate for new upsampling methods to work in such human-based scenarios is for them to first succeed in the current proposed direct-draping upsampling task for the current benchmark**. As standard in many other grand-challenge learning tasks, we begin with a simpler yet still well-scoped and well-beyond current SOTA benchmark task here. (As we demonstrate in our evaluation, this is a task that current learning-based upsampling methods are still far from able to address, similar to how early image recognition models were tested using MNIST and CIFAR-10 before advancing to more complex ones.) With successful methods developed, we look forward to enhancing the cloth upsampling dataset with even more complex draping and human interaction scenarios. Of course, this comes with significant further investment in time and computation resources. We note that this process is familiar in public datasets like PartNet (Mo et al. 2019) (evolved from ShapeNet (Chang et al. 2015)), Objaverse-XL (Deitke et al. 2023a) (originally Objaverse (Deitke et al. 2023a)), ImageNet (Russakovsky et al. 2015), and PASCAL VOC (Everingham et al. 2009), to name just a few – all of these started with constrained features and underwent significant expansion following their initial publication. Therefore, taking this initial step is not merely important but crucial for continued advancement in the field.

Reviewer comment (Experiments):
Due to the above limitation, I am concerned about the ability of this dataset to benchmark various upsampling algorithms under **complicated** deformation, let alone training machine learning models for complicated deformation. Notice that the perspective of judging a dataset is different from the algorithm that generates it. In the original PCS paper, even though relatively simple cloth shape and initialization have been demonstrated, it is less of a concern because as a physics-based algorithm, we generally have more confidence that it will work similarly for cloth of other shapes and from different initialization. In contrast, as a dataset intended for learning algorithms generated by this algorithm, its range of samples directly determines its scope of applicability because the common sense in machine learning is that a model cannot learn patterns outside its training distribution. In my own experience, the scenario where we are interested in a **square** cloth draped on an object without additional interaction is extremely rare.

Author response:
Yes! To capture a range of initializations, we vary the initial orientation of the cloth by different degrees, which indeed leads to distinct draping behaviors (detailed in Section 4 of our paper). Additionally, we emphasize the absence of a large-scale, high-quality cloth drape dataset and the substantial effort and resources needed to create one. Though initializing the cloth differently, such as by even folding, could further diversify our dataset, we are content with its current state and leave exploiting these variations for future work.

Reviewer comment (Writing):
On page 4, the paper writes that "researchers end up repeatedly and unnecessarily generating data". It may be good to avoid saying "unnecessary" because its tone may sound less respectful to the authors of the above work.

Author response:
We have updated the paper as suggested.

Reviewer comment (Experiments):
If I understand correctly, each level of upsampling in the dataset follows the midpoint subdivision as in Loop subdivision. Therefore, is it true that all the methods benchmarked in Table 1 should lead to the same topology? Is it possible to, or how can it be used for upsampling schemes that may not produce the same topologies?

Author response:
Yes, each level of upsampling follows the midpoint subdivision as in Loop subdivision. All methods in Table 1 will lead to the same topology (genus).

Yes! As long as the method's output is given, we can compute the metrics between them.

Reviewer comment (Experiments):
In Sec. 6.2, it is made clear that the training/test split is performed across cloth material and collision objects. How is the split performed for experiments in Sec 6.1?

Author response:
We shuffle the dataset and randomly split the dataset into the training (80%), validation (10%) and test (10%) sets.

Reviewer comment (Evaluation):
How does the proposed upsampling method compare with the original PCS method in terms of runtime performance etc.? It should be good to provide an analysis.

Author response:
See the table below. The unit is second.

|              |      PCS | Ours |
| :--- | ---: | ---: |
| Level 0  |    16.40 |       - |
| Level 1  |      7.96 |  0.69 |
| Level 2  |      6.92 |  0.75 |
| Level 3  |  281.61 |  0.81 |
| Level 4  |    18.34 |  1.15 |
| Others   |    33.37 |  1.43 |