Paper: MapLearn: Indoor Mapping using Audio (ICLR-2024)

Reviewer comment (Reproducibility):
How is the envelope of h calculated?

Author response:
Response: Thanks, we should have explained this better. 
We use Hilbert transform. We will make it clearer in the paper.

Reviewer comment (Writing):
What does "patch" mean for the patch GAN discriminator?

Author response:
Response: We should have clarified this. PatchGAN is from the “Image-to-Image Translation with Conditional Adversarial Networks” paper in the reference. Instead of outputting a final real/fake value directly for the entire image, PatchGAN crops the image into patches and outputs a real/fake for each patch. A learnable function that fuses small patch results together is utilized for final real/fake decisions. This method has proven to be effective for recovering local details inside images.

Reviewer comment (Presentation):
Could the authors add the precision and recall values of each estimated floor plan map to the third and fourth rows of Figure 10?

Author response:
Response: 
We report the P(40) and R(40) values for all the six floor plan visualizations here (we will add this table if the paper is accepted). The P and R values are all around 0.6, showing promising floor plan estimation result. For the full MapLearn pipeline, due to the fewer fake walls predicted, the P(40) value is overall better than that of HintMap only solution.

|          | FP1         | FP2         | FP3         | FP4         | FP5         | FP6         |
|----------|-------------|-------------|-------------|-------------|-------------|-------------|
|          | P(40)/R(40) | P(40)/R(40) | P(40)/R(40) | P(40)/R(40) | P(40)/R(40) | P(40)/R(40) |
| MapLearn | 0.522/0.497 | 0.720/0.559 | 0.472/0.484 | 0.533/0.504 | 0.655/0.566 | 0.689/0.592 |
| HintMap  | 0.560/0.610 | 0.622/0.553 | 0.482/0.559 | 0.611/0.675 | 0.658/0.599 | 0.582/0.635 |

Reviewer comment (Experiments):
What if the MLP for corner prediction from the envelope is removed and directly using the envelope along with the hint map as the input?

Author response:
Response: Thanks for the comment. The envelope should be used in conjunction with location information (i.e., the location where the measurement is being performed). We find that explicitly telling the network about measurement locations (and shifting the local map-patches to those locations) is beneficial. It is more generalizable than directly inputting the measurement location (x,y) as two other dimensions and relying on the neural network to learn.

Reviewer comment (Reproducibility):
Will the data and code be released for reproducibility?

Author response:
Response: Yes, we will release the data and code if the paper is accepted.

Reviewer comment (Writing):
Minor: page 2: "it's introduction"

Author response:
Response: Thank you for identifying these issues. We will fix them.

Reviewer comment (Writing):
Minor: page 4: "a MLP"

Author response:
Response: Thank you for identifying these issues. We will fix them.

Reviewer comment (Writing):
Minor: page 5: "a MSE", "a L1"

Author response:
Response: Thank you for identifying these issues. We will fix them.

Reviewer comment (Writing):
Minor: Figure 5: "cGAN outpu"

Author response:
Response: Thank you for identifying these issues. We will fix them.

Reviewer comment (Evaluation):
Limitation. My major concern is the limitation of the idea. Audio signals contain much less information than visual or lidar signals for map reconstruction, which means the floorplan generated from audio reflections are not that accurate as also shown in the experiments. Moreover, audio reflections vary in environments with different painting materials, decorations, and objects in the area. These factors may further impair the accuracy and generalization.

Author response:
#### Q1: Audio has low resolution leading to impaired mapping accuracy.

Response: True, audio signals have much longer wavelengths compared to optical cameras and LIDARs, hence much lower spatial resolution. 
However, this lack of resolution hides rich details about the environment, leading to better privacy. 
We think many in-home applications (Alexa, digital twins, localization) may favor coarse-grained maps and strong privacy as opposed to the vice versa; in this tradeoff, audio lies in the sweet spot. 
Finally, even extracting this coarse-grained information is not easy; traditional signal processing methods do not work well. 
We have demonstrated that using cGAN can be effective.

Reviewer comment (Experiments):
Pose accuracy. In the paper, the location is obtained from IMU. However, IMU is very sensitive to error accumulation especially in large rooms. Currently, as GPS is not available in indoor environments, visual information is the best to obtain location information. My concern is that if we have visual information, why don’t we just recover an accurate map with SfM techniques from images? Note that audio reflections only provide very coarse floorplan.

Author response:
#### Q2: IMU location error will affect system performance.

Response: Yes, this is a valid comment. 
We acknowledge the accumulation of IMU error would impact mapping. 
However, a growing body of literature is showing improved IMU-based localization [1,2] using (re)calibration opportunities from the environment (using Wifi, BLE, magnetic, and other sensory landmarks, etc. [3,4,5]). 
In sum, IMU-based drifts can be reset periodically  allowing the error to be bounded.
We evaluated the sensitivity of map error as a function of IMU location error (Table 3). 
So long as the IMU error is unbiased, the mapping performance degrades gracefully with IMU error.

Reviewer comment (Writing):
Applications of floorplan contour. In Section 1, the authors mention that ‘a simple floorplan contour may suffice in most cases’ but don’t provide examples. Maps reconstructed by cameras and Lidars can be used for localization which is the key technique to AR/VR and robotics navigation. It is not very clear to see how to use floorplan contour in real applications.

Author response:
#### Q3: Application of the MapLearn pipeline.

Response: We mentioned the following to Reviewer 1 as well. 
Mapping with audio-reverberations is relevant when cameras or LIDARs are  not acceptable (e.g., privacy concerns in homes, poor lighting conditions, and unavailability of LIDARs).
Potential use-cases include: 

+ Digital twins of homes are useful for various applications, e.g., when away from home, a user wants to track her aging parent or her pet on a digital twin of the home floorplan.
The floorplan is valuable for visualization.
+ Conversational AI agents (e.g., Alexa and Google Home) intend to understand the spatial context of a conversation with a user.
Knowing the user's location is useful, but contextualizing that location on an even coarse-grained floorplan offers much more context (e.g., user is in kitchen, living room, etc.).
Mapping with cameras/LIDARs raise privacy concerns inside homes.
+ Localization algorithms (e.g., particle filters) routinely use floorplans to update their posterior distributions. A coarse-grained floor plan with wall location uncertainties will still significantly improve trajectory estimation.
+ Floorplans are useful to signal processing applications, e.g., WiFi radios may beamform better with an understanding of walls around it. 
On similar lines, 3D surround sound can be synthesized better with knowledge of floorplans.
+ Firefighters may rapidly need to map out a floor plan or army troops may need to map out an enemy building; cameras may be inadequate due to smoke or poor lighting conditions.
Audio reverberations (inaudible) should be more effective.

Reviewer comment (Novelty):
I'm not convinced about the motivation and practicality of this problem. While I admit there are applications wherein privacy dictates that camera or LIDARs can't be used, wouldn't the same privacy concerns persist if an audio beacon is used?

Author response:
Audio sensing will record ambient voices and sound events, which is indeed a privacy leakage.
However, we have used the ultrasonic audio bands of 20kHz-40kHz (mentioned at the end of page 5).
Human voice and music have no footprint in these frequencies.
As a result, we can preserve privacy.
Additionally, note that cameras end up ``seeing'' details of the environment (e.g., bathrooms, bedrooms, closets, text and images of documents on the table and fridge, etc.). 
Audio preserves this notion of privacy as well.

Reviewer comment (Experiments):
Results in Section 4.1 are unclear. What does the HintMap baseline indicate? Isn't Hint map part of the MapLearn method? If so, what are the contributions of the rest of the components (ablations)?

Author response:
The hint-map is the floorplan prior estimated from only the first audio reflection.
The later reflections capture additional environmental information and contribute to boosting the paper's final performance.
We have used the hint-map as a baseline (and as an ablation study to compare the value of first vs. late reflections).
In  Figure 10, row 2 shows the ground truth and row 3 shows the estimation based on the hint-map alone.
Row 4 shows the final results after including the late reflections.
With hint maps, the estimated floor plan exhibits more fake walls because the neural network tries to insert walls at the boundary of open spaces. With the full MapLearn pipeline, fake walls are fewer.

Reviewer comment (Evaluation):
Question about the metrics and visualizations — the paper does not motivate why the improvement matters. If an approximate floorplan suffices for downstream application(s), does the improvement in P(d) and R(d) matter?

Author response:
Thanks for recognizing this point.
We have struggled to design reasonable metrics -- not too harsh, not too lenient -- that can capture the quality of learnt floorplans.
Clearly, the application determines when an approximate floorplan is sufficient, but to develop some application-agnostic quantification, we decided on P(d) and R(d).
This is by no means the perfect metric and we are very open to suggestions. 
We will continue to think about better metrics for this problem but believe that P(d), R(d) are reasonably fair measures of performance.

Reviewer comment (Presentation):
It's difficult to interpret images in Fig 10. I'm not sure if Row 2 is better than Row 3 or vice versa.

Author response:
The hint-map is the floorplan prior estimated from only the first audio reflection.
The later reflections capture additional environmental information and contribute to boosting the paper's final performance.
We have used the hint-map as a baseline (and as an ablation study to compare the value of first vs. late reflections).
In  Figure 10, row 2 shows the ground truth and row 3 shows the estimation based on the hint-map alone.
Row 4 shows the final results after including the late reflections.
With hint maps, the estimated floor plan exhibits more fake walls because the neural network tries to insert walls at the boundary of open spaces. With the full MapLearn pipeline, fake walls are fewer.

Reviewer comment (Experiments):
Demonstrating an application where a more accurate floorplan from audio is needed would be useful in showing the benefit of the improvement in floorplan estimate (as the goal is anyways to obtain an approximate floorplan).

Author response:
When a user walks around a lot (covering say 80% of the home) then the hint-map generates a reasonably good floorplan (and the late audio reflections are less crucial).
For such cases, the hint-map and MapLearn can both satisfy the same applications. 
However, when the user walks less, the hint-map degrades quicker and the value of late reflections becomes pronounced.
If this paper is accepted, we will add results that visually compares floorplan accuracy between MapLearn and the hint-map, when the user has walked say 60%. 
This will highlight the additional accuracy gain from late reflections.  


Now, what applications will need more accuracy than just a good visual approximation? 
Many indoor localization applications use particle filters [1] on a floorplan. 
Very briefly, a user's possible location is modeled by many particles, each associated with a probability. 
Each particle is a hypothesis of where the user may be located.
As the user moves, the particles are propagated in the measured walking direction.
When a particle moves through a wall, these particles must be removed since they are an infeasible hypothesis. 
Said differently, the floorplan helps update the posterior distribution of the user's location.
The floorplan accuracy directly impacts the convergence (and accuracy) in such Bayesian methods for localization.


#### [1] Del Moral, Pierre. "Nonlinear filtering: Interacting particle resolution." Comptes Rendus de l'Académie des Sciences-Series I-Mathematics 325.6 (1997): 653-658.

Reviewer comment (Reproducibility):
I'm not sure how they train the GAN network, especially how they collect the ground truth. BatMapper doesn't need any ground truth. If they need the ground truth, it will harm its application and user experimence.

Author response:
Response: We respectfully disagree with this. 
We only need ground-truth for training and since we train on simulation data, the ground-truth is easily available.
We carefully performed the simulation (channel models developed on real signal propagation physics) so that it resembles real world as much as possible. This is why the proposed method works in real world as well. 
We do not need ground-truth when mapping an unknown floorplan in a real building (we validated this in 4 real buildings).

Reviewer comment (Experiments):
Their evaluations are based on simulation, and why not implement a prototype and conduct experiment in real buildings.

Author response:
Response: We indeed implemented the prototype and performed the experiments in real world across 4 different buildings.
Please find details regarding experiment setup in section 3 (Figure 8) and results in section 4 (Figure 11).

Reviewer comment (Experiments):
How to gather the ground truth, and what is the usage scenario.

Author response:
Response: Mapping with audio-reverberations is relevant when cameras or LIDARs are  not acceptable (e.g., privacy concerns in homes, poor lighting conditions, presence of fog or smoke, and unavailability of LIDARs). 
Potential use-cases include: 

+ Digital twins of homes are useful for various applications, e.g., when away from home, a user wants to track her aging parent or her pet on a digital twin of the home floorplan.
The floorplan is valuable for visualization.
+ Conversational AI agents (e.g., Alexa and Google Home) intend to understand the spatial context of a conversation with a user.
Knowing the user's location is useful, but contextualizing that location on a floorplan offers much more context (e.g., kitchen, living room, etc.).
Mapping with cameras/LIDARs raise privacy concerns inside homes.
+ Localization algorithms or state estimation problems (e.g., particle filters) routinely use floorplans to update their posterior distributions.
+ Floorplans are useful to signal processing applications, e.g., WiFi radios may beamform better with an understanding of walls around it. 
On similar lines, 3D surround sound can be synthesized better with knowledge of floorplans.
+ Firefighters may rapidly need to map out a floor plan; cameras may be inadequate due to smoke and poor lighting conditions.
Audio reverberations should be more effective.

Reviewer comment (Experiments):
Comparison with recent work, e.g., BatMapper.

Author response:
Response: Thanks for bringing this up; we should have elaborated more in the paper. 
BatMapper uses a pure signal processing-based approach under the assumptions that: (1) floorplans are rectangular shaped rooms or corridors, and (2) users need to walk quite close to the walls.
This is necessary because BatMapper only utilizes information from early reflections (similar to our hint map) to ultimately infer the parameters of the rectangle.
Our model is crucially different since it uses all the audio reverberations (i.e., the room impulse response) to learn any layout of the environment (including partitions, L or U-shaped rooms, etc.).
Results show that we need the user to walk less and our maps are far more complex than BatMapper (see our Figure 10 and 11 vs. BatMapper Figure 17).

Reviewer comment (Experiments):
Develop a prototype and test in real buildings, instead of simulation.

Author response:
Response: We indeed implemented the prototype and performed the experiments in real world across 4 different buildings.
Please find details regarding experiment setup in section 3 (Figure 8) and results in section 4 (Figure 11).