Paper: SoftPhy: Soft-Body Physical Concept  Learning  and Reasoning from Videos (ICLR-2024)

Reviewer comment (Writing):
In Intro – Second Paragraph, the examples given for humans are very irrelevant to the “soft body,” the topic of this paper. For the liquid example, it can demonstrate the density, but for the pulley example, which physics parameters are the humans trying to distinguish? Moreover, the fact that humans can distinguish these physics properties does not mean AI models can, so there is a logic leap between the human examples and the statement “However, it remains an …”

Author response:
Thank you for the constructive comments!

**Q1(a). About Pulley**

> [P1S1] a. In Intro – Second Paragraph. The examples given for humans are very irrelevant to the “soft body”, the topic of this paper. For the liquid example, it can demonstrate the density, but for the pulley example, which physics parameters are the humans trying to distinguish? 

We are sorry for the potential confusion regarding the liquid and rope (pulley) example. Generally, in this work, we emphasize both the parametric side and the behavioral side of the soft-bodies. In detail, we address this question from two perspectives.

**[Why we choose pulley system as the basis of *rope* scenario?]** We believe, as for human perception, rope is very different from liquid in that rope is 1-dimensional. When we think of rope, we mainly consider the connectivity and length constraints in ropes as well as the derivative dynamics. Viewing liquid dynamics is quite different when we may focus more on the physical properties and its global contour behaviors. In the *rope(aka. pulley)* dataset, we attempt to guide models to learn the embedded physical mechanics and constraints behind a variety of objects. To some extent, lift-up and push-down, cooperating with pulleys, are the most common use of rope in both everyday life(e.g. gym) and industry(e.g. crane). For a simple case, humans could easily perceive the motion of another side of the rope if we see one side of the rope lifted up or pushed down, which is proven diffcult for many VLMs due to dataset gap. As for a more complex case, we know that the dynamic pulley in Fig. 1 (c) of the paper could reduce the tension between ropes compared with static pulley, helping us to pull the cargo up more easily. Hence, we think it's crucial to equip vision language models with a conceptual perception of these phenomena, which is more related to physical dynamics reasoning, as opposed to physical parameter or concept reasoning. For each scenario, we consciously include both reasoning sides.

**[What physical parameters relevant to "soft-body" we are trying to distinguish?]** Typical physical parameters of rope are like elastic coefficient and bending stiffness. Since these parameters are already displayed in the scenario of *cloth* or *ball*, where the parameters are shown in a more comprehensive and complicated way, thus considering the diversity of physical parameters in our overall dataset, in the *rope* scenario, we mainly focus on other physical characteristics of this kind of material. One is the relative weight of load attached to the rope, indicating the external force exerted to the rope terminals, the other is the relative tension force within the rope segments. We hope these two physical parameters can be ideal testing criteria for the conceptual/parametric perception of rope.

**Q1(b). About Logic Leap**

> [P1S1] Moreover, humans can distinguish these physics properties do no means AI models can, so there is a logic leap between the human examples to “However, it remains an …”

Thank you for your concern. In this paragraph, we express concerns about whether current AI models have the capability to accurately identify distinctions in the physical properties and dynamics of soft-body objects, which humans can intuitively discern. To improve our writing, we show the revised paragraph below and have incorporated it in our revision of paper.

*We are able to know that the clear liquid in Fig.1(a) at the bottom has a higher density than the yellow liquid on the top; we know that the dynamic pulley in Fig.1(c) could reduce the tension between ropes compared with static pulley, helping us to pull the cargo up more easily. These innate human skills raise an intriguing question：can current AI models have the physical common sense to infer soft objects' physical properties and predict their corresponding dynamics?*

Reviewer comment (Theory):
From this paragraph on, the paper has a confusing meaning of “soft body.” How is liquid a type of soft body? According to Wikipedia (Soft-body dynamics - Wikipedia), it should be a solid object at least. A simple checkup on Wikipedia could avoid such conceptual mistakes.

Author response:
Thank you for your insightful suggestions! We agree with your idea that physical concepts should be clearer and more rigorous. Therefore, we've broadened our work's description **from 'soft body' to 'continuum'** to more accurately reflect its scope. 

In both physics and computer graphics, 'continuum' encompasses various bodies such as liquids, soft materials (e.g., soft balls, cloth, and ropes), rigid bodies (e.g., cubes, pillars, plates, and spheres), and articulated bodies (e.g., pulleys). Our dataset comprehensively encompasses fluids, soft bodies, rigid bodies, and articulated bodies (e.g., in the rope scenario). This inclusion leads to our utilization of the continuum concept, enhancing the breadth and relevance of our study.

Moreover, we consciously include both physical dynamics reasoning (e.g., interactions between fluids, soft bodies and rigid bodies), and physical parameter or concept reasoning (e.g., density for fluids; tension, elasticity for soft bodies; mass for rigid bodies). For instance, our rope and pulley scenario involves elements of rope, rigid bodies, and articulated bodies; the fluid scenario includes liquids; the cloth scenario covers both cloth and rigid bodies; and the ball scenario focuses on soft balls. This extensive coverage ensures that our dataset provides a comprehensive understanding of the interactions and couplings within these various types of continua, capturing the complexity and diversity of real-world physical phenomena.

In light of this, we have renamed the dataset **from SoPhy to ContPhy**. We have added an entire section in **Appendix A.4** to elaborate on the concept of the continuum. We have also **updated the entire paper accordingly**. 

We hope this clarification addresses all your concerns. We are looking forward to your further reply and suggestions. **If you find our response satisfactory, we would be grateful if you could consider revising your score.**

Reviewer comment (Theory):
According to Section 3 Dataset, the paper mentions physical properties: mass, friction, elasticity, density, deformability, and stretchiness. There is a difference between the physical properties a physics simulator can simulate and a property with genuine physics meaning. Sometimes the physics simulator combines many underlying physics processes and exposes some high-level properties for developers or artists to control. Physically speaking, elasticity and stretchiness are both types of deformability, so putting them at the same level is questionable. Besides, stretchiness and deformability are parameters without corresponding basic physics meaning, making them unmeasurable in the real world. It is unclear how the authors would measure the stretchiness of cloth from reality and whether the method of measurement is coherent with what’s inside the physics engine. Additionally, for soft body objects, there are more physics properties influencing deformation, such as plasticity and viscosity.

Author response:
**Q3. About Physical Parameters**

**[Q3(a). Why are different levels of concepts put together? Some are high-level / APIs.]** 

> [P1S1] c. According to Section 3 Dataset, the paper mentions physical properties: mass, friction, elasticity, density, deformability, and stretchiness. First, there’s a difference between the physical properties a physics simulator can simulate and a property with genuine physics meaning. Sometimes the physics simulator just combine many underlying physics process and expose some high-level properties for the game developer or artist to control. Here is exactly the case. For example, physically speaking, elasticity and stretchiness are both two types of deformability. I wonder how they can be put at the same level. 

We appreciate your opinions. They are meaningful thoughts from the view of serious physics. However, our thoughts are as follows.

1. **Concept's familiarity to common humans.** The reason why we choose those properties/concepts as the target concepts lies in the motivation of this research. We want to discuss those concepts that are on the intersection of both serious physics and psychology. That is, we explore the perception of common physical concepts on both human beings and machine models, which means human beings should be familiar with these concepts. Otherwise the human study would be undoable. We understand your concern that some concepts discussed in this work are only high-level abstract or integration of one or more basic physical parameters, such as stretchiness and elasticity. But we are afraid common people, especially those without physics background, are able to distinguish some of the most basic physical concepts such as Young's modulus, Cauchy stress, Lamé parameter, and so forth. To be specific, we list some relative concepts about different materials as a table below:

| Material  | Common Physical Concept      | Description | Physical or High-Level | (For common human) Familiarity | Is Included |
| --------- | ---------------------------- | ----------- | ---------------------- | ------------------------------ | ----------- |
| Rope      | Mass                         | Property    | **Physical**           | **Familiar**                   | Y           |
| Rope      | Tension                      | Force       | **Physical**           | **Familiar**                   | Y           |
| Fluid     | Density                      | Property    | **Physical**           | **Familiar**                   | Y           |
| Fluid     | Viscosity                    | Property    | **Physical**           | **Familiar**                   | N |
| Cloth     | Streching Stiffness          | Property    | **High-Level**         | **Familiar**                   | Y           |
| Cloth     | Bending Stiffness            | Property    | **High-Level**         | **Familiar**                   | Y           |
| Cloth     | Friction                     | Force       | **High-Level**         | **Familiar**                   | Y           |
| Cloth     | Dynamic Friction Coefficient | Property    | **Physical**           | **Not Sure**                   | **N**       |
| Cloth     | Young's Modulus              | Property    | **Physical**           | **Unfamiliar**                 | **N**       |
| Soft Ball | Elasticity                   | Property    | **High-Level**         | **Familiar**                   | Y           |
| Soft Ball | Plasticity                   | Property    | **High-Level**         | **Familiar**                   | Y           |
| Soft Ball | Young's Modulus              | Property    | **Physical**           | **Unfamiliar**                 | **N**       |
| Soft Ball | Poisson's Ratio              | Property    | **Physical**           | **Unfamiliar**                 | **N**       |
| Soft Ball | Yield Limit                  | Property    | **Physical**           | **Unfamiliar**                 | **N**       |

It's shown that we include the human-familiar concepts to fit this cognitive AI research motivation.

Reviewer comment (Presentation):
The pie charts of Fig 3 and Fig 5-7 do not give precise ratio numbers. The labels of the sections of the pie charts are not clear enough. In Fig 3, for example, it has “Mass,” “Mass Change,” and “Mass Goal,” but lacks “Shape,” “Shape Change,” or “Shape Goal.” If shape is not a physical property, then why are there also no labels like “Tension,” “Tension Change,” “Tension Goal”? A similar confusion goes for Fig 5-7.

Author response:
**Q4. About Writing**

> [P1S1] On the writing: The pie charts of Fig 3 and Fig 5-7 do not give precise ratio numbers. The labels of the sections of the pie charts are not clear enough. Take Fig 3 for example, it has “Mass”, “Mass Change”, and “Mass Goal”, then why it does not have “Shape”, “Shape Change”, or “Shape Goal”, if the shape is not a physical property, then why no “Tension”, “Tension Change”, “Tension Goal”. A similar confusion goes for Fig 5-7.

Thank you for all these suggestions. 

**[Q4(a). Improvement of Pie Charts]** We have updated all pie charts in our paper, incorporating a nested style, precise ratio numbers, and enhanced visual aesthetics. Since rebuttal comments here cannot display images, we kindly ask the reviewer to refer to Figures 3, 5, 6, and 7 in our revised paper.

**[Q4(b). Names of Questions]** The names assigned to questions are arbitrary and are not related to question designs. We use names as symbols to differentiate each question type. In the pulley scenario, mass is the primary factor that diverses these videos. Rope tension results from object mass, and the objects' shape does not influence the motion and dynamics of the videos. Therefore, these two factors do not contribute to dynamics, making it impractical to design dynamic questions, accounting for why we only design factual questions for them. In general, names are solely for identification purposes. We suggest the reviewer to check Table 3-6 for how we design each question type.

Reviewer comment (Experiments):
The number of videos in the dataset is not large. A potential reason is the lack of variance in the scene setup. The soft body can deform in infinite ways, so a 500-video dataset may not be sufficient to cover the dynamics.

Author response:
**Q5. About Dataset**

> [P1S1] On the dataset: The number of videos is not large. A potential reason is the lack of variance in the scene setup. The soft body can deform in infinite ways, how can a 500-video dataset satisfy the coverage of dynamics?

Actually, we do not want to build a training set to encompass all similar situations in the test set. Instead, our goal is to encourage AI models to generalize to unseen situations like human from **limited example demonstrations**. However, we can scale up the data size if necessary due to the dataset's synthetic nature.

Reviewer comment (Evaluation):
In Section 4.2 Paragraph "Dynamics", only the result of ALPRO and HCRN is discussed, and why other models do not work well is missing.

Author response:
**[Q1 (b). Other Baselines]** Traditional pure neural networks have difficulty understanding the physical scenarios and in capturing the physical laws from videos and question-answer pairs. Thus, they perform worse than their previous performance on our benchmark.

Reviewer comment (Experiments):
In Section 4.2 Paragraph "Scenario Analysis", only cloth and rope scenarios are discussed.

Author response:
**[Q1 \(c\). Other Scenarios]** ALPRO performs well in the rope and cloth scenario. Except for the reason that the cloth and rope scenarios share some similarities, another important reason is the fewer question types in the rope and cloth scenario than those in the fluid and ball scenarios. Specifically, the rope scenario has counterfactual and goal-driven, and the cloth scenario has predictive. Conversely, in the fluid and ball scenarios, we incorporated all four problem types, thereby making situations much more complicated. To effectively address these scenarios, models must tackle four distinct question types, each focusing different aspects of physical dynamics. Consequently, no baseline models can gain an absolute advantage in these scenarios. This indicates that our four proposed question types well evaluate different dimensions of physical reasoning, making the fluid and ball scenarios particularly challenging for AI models.

Reviewer comment (Writing):
The writing could benefit from some improvements. In Section 4.2 Paragraph 1, "We summarize the performance of all baselines in Table 1." should be Table 2.

Author response:
Thanks for the reminder. We have already fixed this and checked the whole paper again to improve our writing.

Reviewer comment (Evaluation):
Some statements are not supported well. In Section 4.2 Paragraph "Evaluation Conclusion", the authors concluded that "Machine models results show that even state-of-the-art models struggle with answering physical questions based on the visual input.". However, the relationship between "soft body physics reasoning capability" and "answering physical questions based on visual input" is not clear. For example, AI models may understand soft body physics well, but be unable to understand the questions, as the semantic information is not recognized by the model.

Author response:
Admittedly, semantic information can influence results. However, using QA formats to evaluate physical reasoning abilities has been widely acknowledged and shown effectiveness in previous works, such as ComPhy [A] and CLEVRER [B]. QA gives the flexibility to estimate different kinds of physical reasoning abilities such as understanding different physical properties like mass, density, viscosity, and predict the corresponding dynamics based on variance of such physical properties. To correctly answer these questions, AI models need to have both the physical common sense and the ability to understand natural language queries and answers. Since the question-answer pairs in our benchmark are synthesized by templates, the major challenge of AI models is from physical reasoning. Thus, we claim that our SoPhy benchmark mainly evaluates "soft-body physical reasoning capability". 

[A] Chen, Zhenfang, et al. "ComPhy: Compositional physical reasoning of objects and events from videos." NeurIPS (2022).

[B] Yi, Kexin, et al. "CLEVRER: Collision events for video representation and reasoning." ICLR (2020).

Reviewer comment (Novelty):
In the conclusion, the authors stated that "Despite progress, our evaluation of AI models revealed an ongoing challenge: they struggle to perform well on our benchmark, highlighting their limited physical commonsense for soft objects.", but there are already articles (i.e. [1]) concluding that AI models lack physical reasoning capability; if this capability is missing, the model should also lack physical reasoning capability for soft objects.

Author response:
Thank you very much for your suggestions. We appreciate the conclusion from the paper [C] that dynamics-based models may not accurately predict and AI models lack physical reasoning capability. However, on the other hand, several methods [D, E, F, G] have successfully tackled previous benchmarks like Physion [A] and CLEVRER [B]. In paper [D], VQA accuracy for CLEVRER is 96.5%, and for Physion is 67.1%, which is very close to human performance. These papers have successfully validated the performance of state-of-the-art AI models. Motivated by their significant contribution, we aim to extend this success further by evaluating dynamics and soft-body objects. **Additionally, we have discussed this paper [C] in the related work section of the revised paper.**

[A] Bear, Daniel M., et al. "Physion: Evaluating physical prediction from vision in humans and machines." NeurIPS (2023).

[B] Yi, Kexin, et al. "CLEVRER: Collision events for video representation and reasoning." ICLR (2020).

[C] Li, Shiqian, et al. "On the Learning Mechanisms in Physical Reasoning." NeurIPS (2022).

[D] Wu, Ziyi, et al. "Slotformer: Unsupervised visual dynamics simulation with object-centric models." ICLR (2023).

[E] Ding, David, et al. "Attention over learned object embeddings enables complex visual reasoning." NeurIPS (2021).

[F] Ding, Mingyu, et al. "Dynamic visual reasoning by learning differentiable physics models from video and language." NeurIPS (2021).

[G] Lu, Haoyu, et al. "VDT: General-purpose Video Diffusion Transformers via Mask Modeling." Arxiv (2023).

Reviewer comment (Experiments):
the task family is limited to the designed four types. Also the questions are generated from pre-defined sets of templates. These restrict the general use of the benchmark for other tasks, environments, and questions. Could the authors comment on how is it possible to extend the framework for other tasks?

Author response:
**Q1. About Extending the Framework for Other Tasks**

> [t1H7] the task family is limited to the designed four types. Also the questions are generated from pre-defined sets of templates. These restrict the general use of the benchmark for other tasks, environments, and questions. Could the authors comment on how is it possible to extend the framework for other tasks?

We have developed various templates for each question type across four distinct scenarios. Our experiments reveal that current AI models fail in understanding soft-body physical dynamics, leading to poor performance on our dataset, independent of template design. Additionally, our dataset generation framework is capable of producing different question types and video types, such video annotations for segmentation and depth. This framework can support a range of common vision tasks, such as video segmentation, object detection, text-video retrieval and video grounding. Since our primary objective is to evaluate physical reasoning abilities in the form of question answering, we will not prioritize other vision tasks.

Reviewer comment (Novelty):
the authors claimed that previous benchmarks cannot change mass and friction, but as many of them are also based on physical simulators, it's unclear why they couldn't do that.

Author response:
**Q2. About Previous Benchmarks**

> [t1H7] the authors claimed that previous benchmarks cannot change mass and friction, but as many of them are also based on physical simulators, it's unclear why they couldn't do that.

We agree with the reviewer that many prior benchmarks, such as Physion [A] and ComPhy [B], are also based on physical simulators. However, their benchmarks fall short in thoroughly assessing whether machine models have human-like visual reasoning abilities, particularly in understanding physical object properties and dynamics. These physical benchmarks do not adequately cover dynamics influenced by varying physical object properties across diverse scenarios, notably in the case of soft objects. Some of them lack the variance of object parameters, such as Physion [A], in which solids share the same mass and water maintains a consistent density. Some of them limit their scope to simple primitives and scenario, such as ComPhy [B], in which they only consider mass and electric charge in a basic solid collision scenario of cubes and spheres. In contrast, we are the only comprehensive benchmark with object-specific properties in a wide range of scenarios, which even encompasses soft-body objects and unique language-based questions about dynamics in counterfactual and goal-planning scenarios. Our dataset addresses the gap in existing benchmarks, which often **overlook complex intrinsic properties** and **corresponding dynamics** in diverse scenarios, a key aspect our SoPhy emphasizes.

[A] Bear, Daniel M., et al. "Physion: Evaluating physical prediction from vision in humans and machines." NeurIPS (2023).

[B] Chen, Zhenfang, et al. "ComPhy: Compositional physical reasoning of objects and events from videos." NeurIPS (2022).

Reviewer comment (Novelty):
the paper doesn't propose a solution to improve the performance based on the findings.

Author response:
**Q3. About Proposed Solution**

> [t1H7] the paper doesn't propose a solution to improve the performance based on the findings.

We thank the reviewer for this suggestion. We have presented a proposed solution for the fluid scenairo to better investigate the characterstics of the current benchmark in **G2** of the **General Response**. Please kindly refer to it and we are looking forward to your further feedbacks!