Paper: DiffSound: Differentiable Modal Sound Simulation for Inverse Reasoning (ICLR-2024)

Reviewer comment (Experiments):
How sensitive is the approach in terms of placement of the sensor? If the microphone is too far away, does environmental effects influence the results (e.g., reverberation or other material properties that might affect reflectance, etc.)?

Author response:
Indeed, if the microphone is positioned too far, the model may be susceptible to noise interference. However, it's noteworthy that reverberation or reflectance should not impact the results since our model exclusively relies on frequency information in the sound, and these phenomena do not alter the frequency of modes.

Reviewer comment (Experiments):
How much does the complexity of the shape influence reconstruction? For example, an intricate and non-convex shape that impedes direct path to the sensor?

Author response:
In the context of shape reconstruction, we specifically leverage the eigenmodes of objects, making the sensor placement immaterial to the results. Notably, as objects of different shapes can produce remarkably similar sounds, a challenge arises when numerous shapes yield similar sounds to the ground truth, potentially leading to optimization convergence towards an incorrect shape.

Reviewer comment (Theory):
For Equation (10), why are both terms required? One is just a compressed form of the other?

Author response:
The absence of logarithmic transformation results in rapid signal decay, causing the model to focus solely on the initial spectrogram frames. The use of log spectrograms proves beneficial in capturing damping effects over time. However, it is essential to note that for modes with substantial damping factors, relying solely on log spectrograms might overlook their contribution. Hence, a combination of both approaches is preferable.

Reviewer comment (Theory):
Eq. (5) to (6) — I could follow the derivation of Eq. (6) from (5). Do you use $\partial {\bf u}_i=0$ or some transformation of ${\bf u}_i ^T {\bf M} {\bf u}_i$?

Author response:
In the general eigendecomposition definition, $u_i^TMu_i = I$.

Reviewer comment (Writing):
What do you mean by *hybrid loss*? Is it hybrid because linear and logarithm error is combined as in Eq. (10)? Or does this mean the use of $\ell_1$ loss and OT-based loss?

Author response:
The term "hybrid" denotes the utilization of both $\ell_1$ loss and OT-based loss.

Reviewer comment (Experiments):
One thing that confuses me is the ground truth eigenvalues. How do you obtain the ground truth eigenvalue? Without supervision on the eigenvalues, the optimization problem becomes much more challenging. Is it possible to optimize with only the audio loss?

Author response:
The experiments concerning shape geometry reasoning are conducted on a synthetic dataset, where ground truth eigenvalues are computed using the standard modal analysis process. For material parameters reasoning, real-world datasets are employed, relying solely on audio loss for optimization.

Reviewer comment (Experiments):
From my own experiences, the Wasserstein distance is indeed helpful in bridging the ground truth and predicted spectrograms. However, the switch timing between Wasserstein loss and L1/L2 spectrogram loss is undefined. How do you determine 'sufficient convergence'?

Author response:
Currently, we arbitrarily define 1000 epochs as a point of 'sufficient convergence.'

Reviewer comment (Experiments):
The current pipeline is split between a neural network approach and FEM simulator. Could both steps be modeled with a neural networks?

Author response:
While it is possible to model both steps using neural networks, achieving comparable accuracy to a classic solver is challenging.

Reviewer comment (Novelty):
How would the work compare with impact sound generation from videos through diffusion model?  
Su, Kun, et al. "Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Author response:
Our paper specifically addresses the inverse problem of inferring physical parameters from impact sound. In contrast, the referenced paper by Su, Kun, et al. focuses on predicting sound from visual features.

Reviewer comment (Theory):
What is the computational complexity of the pipeline?

Author response:
The computational complexity of the pipeline can be approximated as O(MN^2), where N represents the number of vertices and M is the number of modes. This estimation accounts for the most time-consuming aspect, which is the eigen decomposition process.