Paper: Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video (ICLR-2024)

Reviewer comment (Evaluation):
Since this is a probabilistic model, the predicted dynamics may be different for different sampled feature $\mathbf{z}_t$. Is there any reference to show the variations of predicted particle states? For example, the model would predict one sequence for multiple times and calculate the standard deviation compared with the ground truth.

Author response:
Good question! It is crucial to understand the diversity in probabilistic dynamics prediction. Therefore, in Tables 1~4 in the revised paper, we predict each sequence 10 times with different $\mathbf z_t$ drawn from the prior module, and calculate the standard deviation of the errors. The results are shown below:

Our results for novel scene simulation ($t\in [2,60]$), added to Table 1:
| Setting      | $\rho=2000,\nu=0.065$ | $\rho=1000,\nu=0.08$ | $\rho=500,\nu=0.2$  |
| ------------ | --------------------- | -------------------- | ------------------- |
| New geometry | 34.54 $\pm$ 0.55 | 33.11 $\pm$ 0.50  | 39.03 $\pm$ 0.79 |
| New boundary | 39.86 $\pm$ 0.90 | 37.79 $\pm$ 0.84  | 47.25 $\pm$ 1.26 |

Our future prediction results for the observed scenes ($t\in [51,60]$):
| $\rho=2000,\nu=0.065$ | $\rho=1000,\nu=0.08$                         | $\rho=500,\nu=0.2$ |
| ---------------------- | -------------------------------------------- | ------------------ |
| 32.41 $\pm$ 0.17    | 32.97 $\pm$ 0.71 |         41.15 $\pm$ 0.71           |

Our results on particle datasets, added to Table 3: 
| $d_{t+1}$                | $d_{t+2}$            | Average error over 190 steps |
| ------------------ | ----------------- | ---------------------------- |
| 0.31 $\pm$ 0.003 | 0.94 $\pm$ 0.011 | 38.37 $\pm$ 0.860             |

Our results of prediction error on generalization to unseen fluid dynamics, added to Table 4: 
|                 | Observed               | Unseen |
| ------------------ | ----------------- | ---------------------------- |
| Ours | 36.03 $\pm$ 0.80 | 44.25 $\pm$ 1.36            |

As we can see from the updated results, the predictions of our model yield larger variances as the prediction time horizon extends.

Note: In this context, we conduct multiple simulations to compute the standard deviation for each individual sequence. This differs from our evaluation setup for the subsequent question, where we "calculate the standard deviation of prediction errors across multiple sequences".

Reviewer comment (Evaluation):
What's the standard deviation of the prediction errors in Table 2? For example, one scene should include several sequences. The errors may vary from sequence to seqence. What's the variation of the errors?

Author response:
First, we want to clarify that Table 2 showcases the future prediction errors in observed scenes, where only videos of **a single sequence** are provided for each physical parameter. 

Instead, in Table 1, we do evaluate our method on 50 different sequences with diverse unseen fluid geometries and boundary conditions for each physical parameter. Below, we present the standard deviation of the simulation errors across these sequences.

We should note that there are substantial variations in simulation difficulty between these samples, resulting from diverse initial conditions (including initial positions and velocities) and boundary conditions. Consequently, we observe large standard deviations for all the compared methods. Despite these challenges, our model consistently presents significantly lower average errors and the lowest inter-sample standard deviation as well. This demonstrates its robustness in effectively handling diverse and challenging fluid scenes.

|            | $\rho=2000,\nu=0.065$ (Geom. / Bdy.) | $\rho=1000,\nu=0.08$ (Geom. / Bdy.)   | $\rho=500,\nu=0.2$ (Geom. / Bdy.)     |
| ---------- | ------------------------------------- | ------------------------------------- | ------------------------------------- |
| CConv      | 52.49 $\pm$ 29.83 / 64.29 $\pm$ 56.95 | 51.33 $\pm$ 33.47 / 56.33 $\pm$ 44.12 | 40.67 $\pm$ 21.12 / 53.28 $\pm$ 30.74 |
| NeuroFluid | 65.01 $\pm$ 45.16 / 73.55 $\pm$ 64.95 | 59.79 $\pm$ 46.17 / 60.46 $\pm$ 48.35 | 40.88 $\pm$ 21.20 / 50.73 $\pm$ 27.94 |
| PAC-NeRF   | 51.10 $\pm$ 31.39 / 59.61 $\pm$ 43.80 | 51.33 $\pm$ 26.78 / 56.84 $\pm$ 37.11 | 40.97 $\pm$ 22.07 / 62.05 $\pm$ 45.85 |
| Ours       | 32.88 $\pm$  7.30 / 39.86 $\pm$ 12.32 | 32.60 $\pm$ 7.58 / 37.62 $\pm$ 14.60  | 39.29 $\pm$ 14.70 / 47.31 $\pm$ 23.58 |

Reviewer comment (Presentation):
In Fig 4, is it an accurate reference of the results for the phrase "row 3-6". It seems like the results start from row 2.

Author response:
Thank you for your careful review. It's a typo and we have corrected it to "Rows 2-6" in the revision.

Reviewer comment (Novelty):
I am not sure about the claim of latent intuitive physics being proposed by them first, especially when similar ideas have existed in the literature on causal representation learning and other intuitive physics methods, albeit in different formats. I think this claim can be softened to say this is the first method to perform this for fluids using 3D exemplars.

Author response:
We appreciate your suggestion, and in the revised paper, we have refined the introduction to more precisely highlight the technical contributions. Specifically, the novel aspects of the proposed "latent intuitive physics" approach now include:
- It presents a novel learning-based approach for fluid simulation, which infers the hidden properties of fluids from 3D exemplars and transfers this knowledge to a fluid simulator.
- Our approach includes the first probabilistic network for particle-based fluid simulation.

Reviewer comment (Reproducibility):
Is the zero initial state velocity a general assumption for all experiments performed? Does this correspond to $(z_{t=1}, \tilde z_{t=1})$ being zero-initialized?

Author response:
(1) No, assuming zero initial velocity ($v_{t=1}^{i=1:N}$) is not a universal condition for all experiments. As shown below, we only adopt this assumption for the observed visual scene, while applying non-zero initial velocities for the novel scenes (Table 1, Table 4, and Table 5), as well as for the particle datasets (Table 3).

(2) For the latent states $(\mathbf{z}\_{t}, \tilde{\mathbf{z}}\_{t})$ in all experiments, we do initialize $\mathbf{z}\_{t=1}$ and $\tilde{\mathbf{z}}\_{t=1}$ with zero values, which is similar to the common practice for initializing the hidden states of RNNs. Notably, there is no direct connection between the initialization approaches for latent states and the velocity states.

| Zero-initialized?|Observed visual scenes| Novel scenes |Particle experiments|
|-|-|-|-|
| $v_{t=1}^{i=1:N}$| Yes| No| No|
| $\mathbf{z}\_{t=1}, \tilde{\mathbf{z}}\_{t=1}$ | Yes | Yes| Yes|

Reviewer comment (Theory):
Why is there a difference in the type of posterior used in the pre-training and the inference stages, being particle-based and visual, respectively?

Author response:
As illustrated below, both particle-based posteriors ${\mathbf{z}}\_t$ and visual posteriors $\hat{\mathbf{z}}$ are employed to offer effective training supervision for the prior learner. 
- **Pretraining stage:** The particle-based posterior module $q_\xi(\mathbf{z}\_t \mid \mathbf{x}\_{1: t}, \mathbf{z}\_{t-1})$ conditions on the prediction target $\mathbf{x}\_t$ and the previous particle states along with latent features $(\mathbf{x}\_{1:t-1}, \mathbf{z}\_{1:t-1})$. This setup enables the prior learner $p_\psi(\tilde{\mathbf{z}}\_t \mid \mathbf{x}\_{1:t-1}, \tilde{\mathbf{z}}\_{t-1})$ to generate meaningful latent features that capture hidden properties essential for accurate predictions.
- **Visual inference stage:** In this phase, true particle states are unavailable. Instead, we rely on visual observations and optimize a new set of variables known as visual posteriors ($\hat{\mathbf{z}}$). These visual posteriors are refined by backpropagating image rendering errors. They serve as adaptation targets for the prior learner, facilitating efficient adaptation to the specific observed fluid.

|| Particle-based posterior| Visual posterior|
| ------------- | -| -|
| Formulation   | $q_\xi(\mathbf{z}\_t \mid \mathbf{x}\_{1: t}, \mathbf{z}\_{t-1})$                    | $q(\hat{\mathbf{z}} \mid I_{1:T})$|
| Training stage  | Stage A: Particle-based pretraining                                               | Stage B: Visual inference|
| Data | Given particle states| Given RGB images when particle states are inaccessible|
| Optimization      | Optimize the parameters of both the posterior module and the prior module to minimize the KL divergence | Directly optimize $\hat{\mathbf{z}}$ to minimize the image rendering loss |
| Function |     Facilitate prior learner training when ground truth particle states are available|  Provide adaptation targets for the prior learner in the subsequent Stage C |

Reviewer comment (Theory):
Why is the prior learning less prone to overfitting?

Author response:
In Section 4.3 of the original manuscript, we claimed that "*Instead of finetuning the entire transition network as NeuroFluid (Guan et al., 2022) does, we only finetune the prior learner module, which is more efficient and less prone to overfitting.*" What we intended to convey is that tuning all parameters in the transition model in the visual scene (as NeuroFluid does) can lead to overfitting problems. Due to the unavailability of the ground truth supervision signal in particle space, the transition model may: 
- Forget the pre-learned knowledge of feasible dynamics; 
- Learn physically implausible particle transitions (as the learnable transition network has a wide variety of ways to move the input particles to the expected geometric positions, all of which are sufficient to minimize the image rendering loss).

The evidence is that NeuroFluid shows superior performance compared to our approach for the observed scene with $\rho=500,\nu=0.2$ (as presented in Table 1). However, it underperforms for corresponding novel scenes with new initial and boundary conditions (as detailed in Table 2).

|            | Observed scene, $t \in [51,60]$ | Novel scene, $t \in [2,60]$ (Geom./Bdy.) |
| ---------- | ------------------------------ | ------------------------------- | 
| NeuroFluid | 33.22                           | 40.88 / 50.73                            |
| Ours       | 41.15 $\pm$ 0.71                     | 39.03 $\pm$ 0.79 / 47.25 $\pm$ 1.26  |

Unlike NeuroFluid, our approach adapts the physical prior learner to visual scenes, without training a probabilistic physics engine. By leveraging knowledge from the pretraining stage, the transition model is less prone to overfitting on the observed scene. This not only enhances the generalization ability but also significantly reduces the training burden.

We have provided additional clarification on this statement to avoid misunderstanding in the revised paper.