Paper: Disentangled Acoustic Fields For Multimodal Physical Scene Understanding (ICLR-2024)

Reviewer comment (Experiments):
The task of this paper is closely related to sound source localization. If the authors justify the effectiveness of the proposed method in terms of position errors of sound sources, experimental comparisons with several previous methods for this task would be mandatory. As presented in [Grumiaux+ JASA 2021 https://arxiv.org/abs/2109.03465], lots of previous methods have already been proposed for this purpose. I understand that many of those previous methods typically require multi-channel audio inputs and the proposed method employs a single-channel or binaural audio clip. However, you will find several techniques that work well only with single-channel audio, since single-channel sound source localization has been one of the typical tasks in audio signal processing before the deep learning era.

Author response:
In response, we choose two strong methods after reading the survey on sound source localization methods [4] and implement them with pytorch. One feeds the STFT of the audio into U-Net[5], and the other feeds the original waveform into CNN [6]. To evaluate the performance of our model in sound source localization, we employed the REALIMPACT dataset[3]. The experimental results, presented in the table below, clearly demonstrate that our proposed model outperforms the established baseline methods. **We appreciate the guidance to substantiate our claims through comparative analysis and believe these additional results strengthen the effectiveness of our proposed approach.**

| Method   | Angle Acc. | Height Acc. | Distance Acc.|
|---------|---------|----------|-------|
| Chance  | 0.100    | 0.067    | 0.250|
| Unet + STFT [5] |  0.825    |  0.902  | 0.972  |
| CNN + Waveform [6]  |  0.671    |   0.755   |  0.802|
| Resnet + STFT [3] | 0.758    | 0.881    | 0.983|
| Ours  | **0.900**    | **0.960**    | **0.994**|

Reviewer comment (Experiments):
If we want to simply predict acoustic scenes from audio signals, we do not have to introduce decoders that generate audio signals from object properties. In this sense, the ablation study that compares the full proposed method and the one without decoders should be presented.

Author response:
We agree that the comparison of methods with and without the decoder component is crucial for evaluating the efficacy of the decoder in our model. But we would like to clarify that in Section 4.2 of our paper, **we have already conducted an analysis comparing the performance of our proposed method with a variant that does not include the decoder**. Experiments revealed that while the Success Rate (SR) remained comparable, there was a noticeable decrease in Shortest Path Length (SPL) and Sound Navigation Accuracy (SNA) metrics without the decoder.

To address your concerns, we will enhance the clarity of this comparison in our manuscript, better articulating the critical role the decoder plays in the overall performance of our method.

Reviewer comment (Experiments):
If the main objective of this paper is a proposal of a novel construction of neural acoustic fields, the proposed method should be compared with the original NAFs in the task of audio synthesis.

Author response:
In response to reviewer RUoW's suggestion for a comparative evaluation with Neural Acoustic Fields (NAF) [1] in audio synthesis task, we collaborated with the NAF authors to create a baseline model equivalent to NAF. We then conducted a comparative analysis of our approach and NAF using 15,000 audio files from the REALIMPACT dataset[3]. This analysis focused on the accuracy of audio simulation, measured by comparing the Mean Squared Error (MSE) of the reconstructed Power Spectral Density (PSD) against the true PSD. The results, detailed in the accompanying table, clearly demonstrate our method's superior performance in audio simulation tasks compared to NAF, effectively addressing the reviewers' concerns.


| Model   | NAF | Ours | 
|---------|---------|----------|
| PSD MSE  | 0.0264    | 0.0052    |

Reviewer comment (Novelty):
The title “physical scene understanding” could be overclaimed the contribution since the audio perception is limited to constrained scenarios (e.g., fallen objects).

Author response:
We appreciate your suggestion regarding the title "physical scene understanding", given that our work primarily focuses on using audio information from objects impacting surfaces (which may not always be falling) to infer physical properties like location, size, and material. To more accurately represent the scope of our research, we propose modifying the title to "Disentangled Acoustic Field for **Physical Property Understanding"** This change better aligns the title with the goal and contributions of our work.

Reviewer comment (Novelty):
Though the DAFs or (the predict-generate) is novel in audio modality, it is not such innovative and is close to use the cycle-consistency to ensure robustness in vision modality. I would suggest this work more like an ICRA paper instead of ICLR.

Author response:
Our utilization of the analysis-by-synthesis framework is fundamentally geared towards enhancing the model's comprehension of physical properties through audio perception, rather than focusing on robustness in vision. This approach is distinct from cycle-consistency in vision, as it specifically targets the underconstrained task of inferring and understanding acoustic scenes and their physical characteristics.

Reviewer comment (Novelty):
The title “physical scene understanding” could be overclaimed the contribution since the audio perception is limited to constrained scenarios (e.g., fallen objects).

Author response:
We appreciate your suggestion regarding the title "physical scene understanding", given that our work primarily focuses on using audio information from objects impacting surfaces (which may not always be falling) to infer physical properties like location, size, and material. To more accurately represent the scope of our research, we propose modifying the title to "Disentangled Acoustic Field for **Physical Property Understanding"** This change better aligns the title with the goal and contributions of our work.

Reviewer comment (Novelty):
Though the DAFs or (the predict-generate) is novel in audio modality, it is not such innovative and is close to use the cycle-consistency to ensure robustness in vision modality. I would suggest this work more like an ICRA paper instead of ICLR.

Author response:
Our utilization of the analysis-by-synthesis framework is fundamentally geared towards enhancing the model's comprehension of physical properties through audio perception, rather than focusing on robustness in vision. This approach is distinct from cycle-consistency in vision, as it specifically targets the underconstrained task of inferring and understanding acoustic scenes and their physical characteristics.

Reviewer comment (Presentation):
A video demo would be very helpful for us to understand the model's performance on the localization of fallen objects in the real world.

Author response:
We strongly agree with the reviewer's suggestion for a video demonstration. We've created a video that showcases our model in action, with RGB images and semantic segmentation results during navigation. This website also includes a top-down view of the loss map and navigational trajectory, helping to illustrate how our model processes and interprets environmental data. We believe this video will effectively demonstrate the capabilities of our model in audio-visual navigation and object localization. For a detailed visual representation, we have provided the video link: [***https://sites.google.com/view/disentangled-acoustic-fields***](https://sites.google.com/view/disentangled-acoustic-fields).

Reviewer comment (Experiments):
Currently, all of the experiments are conducted on synthetic datasets. It would be interesting to see how the model generalizes to real-world data.

Author response:
Thanks for your kind suggestion. We want to clarify that our study **has already used real-world data (REALIMPACT dataset [3]) for the object property task**. As detailed in Section 4.1 of our paper, the REALIMPACT dataset [3] contains 150,000 sound recordings from 50 different object categories. This provided a wide range of **real-world acoustic scenarios**. The results, shown in Table 2, demonstrate our method's improved accuracy in predicting physical properties like angle, height, and distance over baseline methods. This shows that our method generalizes well in real-world scenarios.

To further clarify this point in our manuscript, we will augment this section with more explicit and detailed description, thereby addressing the concerns raised.

Reviewer comment (Experiments):
The proposed method requires full labels to train DAFs. It would be beneficial to develop a self-supervised learning approach to avoid using many labels during model training.

Author response:
We deeply appreciate the reviewer's insightful suggestion regarding the implementation of a self-supervised learning approach for training our DAF. We agree that reducing the dependency on fully labeled data is a valuable and important direction for advancing the field.

We would like to note that our system is supervised using the PSD, which is derived from the input. Our system is indeed self-supervised in this sense. 

However, we recognize the limitations this approach may have in terms of scalability in more diverse real-world scenarios. Moving forward, we are interested in exploring addition self-supervised learning paradigms. We believe that such an approach could significantly enhance the versatility and practicality of our method, enabling it to learn from a wider array of data with minimal labeling requirements.

Reviewer comment (Novelty):
The proposed work lacks novelty. The main contribution of this work is the introduction of DAFs, which are applied to the generative frameworks and multitask learning to address this task. Although it shows promising performance, it lacks some degree of innovation.

Author response:
We would like to clarify that the relationship between our proposed DAF (Disentangled Acoustic Field) and NAF (Neural Acoustic Field) [1], as they tackle fundamentally different tasks. While it's true that DAF builds upon the foundational aspects of NAF, they are distinct in their application and capabilities. NAF primarily focuses on predicting audio propagation characteristics within a scene. In contrast, DAF extends this premise to a more complex task of inferring physical properties such as the location, material, and size of objects across unseen novel scenes, solely from audio information. This advancement represents a substantial shift in the application of acoustic modeling, enabling a broader scope of physical scene understanding beyond what NAF offers. 

Thus, while DAF and NAF share some conceptual lineage, their respective applications and the complexity of the tasks they address are fundamentally different, underscoring the innovative leap our work represents in the realm of audio-based property analysis.

Reviewer comment (Experiments):
The paper mentions that NAFs lack generalization ability for new scenarios, while DAFs can effectively solve this problem. However, no relevant comparative experiments are observed in the experimental section.

Author response:
We appreciate the reviewer's observation regarding the generalization ability tests. In response, we conduct a comparative evaluation with NAF [1] in audio synthesis task. we collaborated with the NAF authors to create a baseline model equivalent to NAF. We then made a comparative analysis of our approach and NAF using 15,000 audio files from the REALIMPACT dataset[3]. This analysis focused on the accuracy of audio simulation, measured by comparing the Mean Squared Error (MSE) of the reconstructed Power Spectral Density (PSD) against the true PSD. The results, detailed in the accompanying table, clearly demonstrate our method's superior performance in audio simulation tasks compared to NAF, effectively addressing the reviewers' concerns.




| Model   | NAF | Ours | 
|---------|---------|----------|
| PSD MSE  | 0.0264    | 0.0052    |

Reviewer comment (Experiments):
The paper mentions that DAFs can address the generalization issue for different new scenarios, but the experimental section only demonstrates generalization for two different scenarios, kitchen and study room.

Author response:
We would like to clarify that our method not only generalizes across two scenarios. As detailed in the experimental setup of Section 4.2 in our study, we employed Find Fallen Challenge [2], a comprehensive dataset encompassesing 30 distinct physical object types situated within **64 uniquely configured rooms**. Importantly, these rooms are categorized into two primary types: study rooms and kitchens. However, it is crucial to note that each category itself comprises 32 different room variations, offering a wide array of acoustic environments. Each scenario presents its unique set of acoustic challenges, thereby providing a robust testbed for evaluating the generalization capabilities of our DAF. The revised manuscript will include a more detailed explanation of these diverse experimental conditions.

Moreover, our experiments follows [2] to utilize the same test split to assess the baseline algorithm's generalization performance. We believe that this extensive and varied dataset, encompassing a broad range of scenarios, adequately demonstrates the generalization prowess of DAF.

Reviewer comment (Experiments):
The authors claim that their method is generalizable, why the experiments are using only two different scenarios, kitchen and study room?

Author response:
We would like to clarify that our method not only generalizes across two scenarios. As detailed in the experimental setup of Section 4.2 in our study, we employed Find Fallen Challenge [2], a comprehensive dataset encompassesing 30 distinct physical object types situated within **64 uniquely configured rooms**. Importantly, these rooms are categorized into two primary types: study rooms and kitchens. However, it is crucial to note that each category itself comprises 32 different room variations, offering a wide array of acoustic environments. Each scenario presents its unique set of acoustic challenges, thereby providing a robust testbed for evaluating the generalization capabilities of our DAF. The revised manuscript will include a more detailed explanation of these diverse experimental conditions.

Moreover, our experiments follows [2] to utilize the same test split to assess the baseline algorithm's generalization performance. We believe that this extensive and varied dataset, encompassing a broad range of scenarios, adequately demonstrates the generalization prowess of DAF.