Original: V ISUAL T IMING F OR S OUND S OURCE D EPTH E STIMATION IN THE W ILD
Revised: V ISUAL T IMING F OR S OUND S OURCE D EPTH E STIMATION IN THE W ILD

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Depth estimation enables a wide variety of 3D applications, such as robotics and autonomous driving.
Revised: Depth estimation enables a wide variety of 3D applications, such as robotics and autonomous driving.

Original: Despite significant work on various depth sensors, it is challenging to develop an all-in-one method to meet multiple basic criteria.
Revised: Despite significant work on various depth sensors, it is challenging to develop an all-in-one method to meet multiple basic criteria.

Original: In this paper, we propose a novel audio-visual learning scheme by integrating semantic features with physical spatial cues to boost monocular depth with only one microphone.
Revised: In this paper, we propose a novel audio-visual learning scheme by integrating semantic features with physical spatial cues to boost monocular depth with only one microphone.

Original: Inspired by the flash-to-bang theory, we develop FBDepth, the first passive audio-visual depth estimation framework.
Revised: Inspired by the flash-to-bang theory, we develop FBDepth, the first passive audio-visual depth estimation framework.

Original: It is based on the difference between the time-of-flight (ToF) of the light and the sound.
Revised: It is based on the difference between the time-of-flight (ToF) of the light and the sound.

Original: We formulate sound source depth estimation as an audio-visual event localization task for collision events.
Revised: We formulate sound source depth estimation as an audio-visual event localization task for collision events.

Original: To approach decimeter-level depth accuracy, we design a coarse-to-fine pipeline to push the temporary localization accuracy from event-level to millisecond-level by aligning audio-visual correspondence and manipulating optical flow.
Revised: To approach decimeter-level depth accuracy, we design a coarse-to-fine pipeline to push the temporary localization accuracy from event-level to millisecond-level by aligning audio-visual correspondence and manipulating optical flow.

Original: FBDepth feeds the estimated visual timestamp together with the audio clip and object visual features to regress the source depth.
Revised: FBDepth feeds the estimated visual timestamp together with the audio clip and object visual features to regress the source depth.

Original: We use a mobile phone to collect 3.6K+ video clips with 24 different objects at up to 60 m .
Revised: We use a mobile phone to collect 3.6K+ video clips with 24 different objects at up to 60 m .

Original: FBDepth shows superior performance especially at a long range compared to monocular and stereo methods.
Revised: FBDepth shows superior performance especially at a long range compared to monocular and stereo methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Depth estimation has been a popular topic owing to many important applications.

Original: It is the fundamental functionality to enable 3D perception and manipulation.
Revised: Depth estimation is the fundamental functionality to enable 3D perception and manipulation.

Original: Although there have been significant efforts on developing depth estimation methods with various sensors. current depth estimation schemes fail to achieve a good balance on multiple basic metrics including accuracy, range, angular resolution, cost, and power consumption.
Revised: Although there have been significant efforts on developing depth estimation methods with various sensors, current depth estimation schemes fail to achieve a good balance on multiple basic metrics including accuracy, range, angular resolution, cost, and power consumption.

Original: Active depth sensing methods actively emit signals, such as LiDAR (Caesar et al., 2020), structuredlight (Zhang, 2012), mmWave (Barnes et al., 2020), ultrasound (Mao et al., 2016), WiFi (Vasisht et al., 2016).
Revised: Active depth sensing methods actively emit signals, such as LiDAR (Caesar et al., 2020), structuredlight (Zhang, 2012), mmWave (Barnes et al., 2020), ultrasound (Mao et al., 2016), WiFi (Vasisht et al., 2016).

Original: They compare the reflected signal with the reference signal to derive time-of-flight (ToF), phase change, or Doppler shift to estimate the depth.
Revised: They compare the reflected signal with the reference signal to derive time-of-flight (ToF), phase change, or Doppler shift to estimate the depth.

Original: Active methods can achieve high accuracy because  the  modulated sensing signal is well designed.
Revised: Active methods can achieve high accuracy because of the physical fundamental and well-designed modulated sensing signals.

Original: Lidar is the most attractive active senor due to its large sensing range and dense point cloud.
Revised: Lidar is the most attractive active senor due to its large sensing range and dense point cloud.

Original: However, the density is not sufficient enough to enable a small angular resolution.
Revised: However, the density is not sufficient enough to enable a small angular resolution.

Original: Therefore, the points are too sparse to be recognized at a long distance.
Revised: Therefore, the points are too sparse to be recognized at a long distance.

Original: Besides, the prohibitive cost and power consumption limit the availability of Lidar on general sensing devices.
Revised: Besides, the prohibitive cost and power consumption limit the availability of Lidar on general sensing devices.

Original: Passive depth sensing takes signals from the environment for sensing directly.
Revised: Passive depth sensing takes signals from the environment for sensing directly.

Original: It commonly uses
Revised: It commonly uses

Original: RGB monocular camera (Bhoi, 2019; Laga et al., 2020), stereo camera (ste), thermal camera (Lu & Lu, 2021), or multi-view cameras (Long et al., 2021a).
Revised: RGB monocular camera (Bhoi, 2019; Laga et al., 2020), stereo camera (Cheng et al., 2020), thermal camera (Lu & Lu, 2021), or multi-view cameras (Long et al., 2021a).

Original: These sensors can achieve pixel-wise angular resolution and consume pretty less energy due to omitting the signal emitting.
Revised: These sensors can achieve pixel-wise angular resolution and consume pretty less energy due to omitting the signal emitting.

Original: Among them, stereo matching can effectively estimate the disparity and infer a dense depth map.
Revised: Among them, stereo matching can effectively estimate the disparity and infer a dense depth map since it transforms the spatial depth to the visual disparity based on the solid physical law.

Original: The base line of the stereo camera determines the effective range and accuracy.
Revised: The baseline of the stereo camera determines the effective range and accuracy.

Original: Therefore, the dimension of the stereo camera is placed as the critical trade-off with sensing metrics.
Revised: Therefore, the dimension of the stereo camera is placed as the critical trade-off with sensing metrics.

Original: Thanks to the advance in deep learning, the cheap monocular depth estimation keeps on improving performance with new network structures and high-quality datasets.
Revised: Thanks to the advance in deep learning, the cheap monocular depth estimation keeps on improving performance with new network structures and high-quality datasets.

Original: However, the accuracy is still not satisfactory especially at a long range because it can only regress depth based on the implicitly visual cues. 
Revised: However, the accuracy is still not satisfactory especially at a long range because it can only regress depth based on the implicit visual cues. It is ill-posed without any physical formulation.

Original: Besides, it also requires domain adaption and camera calibration for various camera intrinsics (Li et al., 2022).
Revised: Besides, it heavily relies on the dataset. It requires domain adaption and camera calibration for various camera intrinsics (Li et al., 2022).

Original: In this paper, we propose to add only one microphone to enable explicit physical depth measurement and boost the performance of a single RGB camera.
Revised: In this paper, we propose to add only one microphone to enable explicit physical depth measurement and boost the performance of a single RGB camera.

Original: It does not rely on camera intrinsic  and implicit visual cues.
Revised: It does not rely on the intrinsic of cameras and implicit visual cues.

Original: We develop a novel passive depth estimation scheme, called Flash-to-Bang Depth (FBDepth).
Revised: We develop a novel passive depth estimation scheme with a solid physical formulation, called Flash-to-Bang Depth (FBDepth).

Original: It is inspired by a well-known phenomenon – ’flash-to-bang’, which is used to estimate the distance to the lightning strike according to the difference between the arrival time of a lightning flash and a thunder crack.
Revised: Flash-to-Bang is used to estimate the distance to the lightning strike according to the difference between the arrival time of a lightning flash and a thunder crack.

Original: This works because light travels a million times faster than sound.
Revised: This works because light travels a million times faster than sound.

Original: When the sound source is several miles away, the delay is large enough to be perceptible.
Revised: When the sound source is several miles away, the delay is large enough to be perceptible.

Original: Applying it to our context, FBDepth can estimate the depth of a collision that triggers audio-visual events.
Revised: Applying it to our context, FBDepth can estimate the depth of a collision that triggers audio-visual events.

Original: The collision event has been explored for navigation and physical search in (Gan et al., 2022), but our work is the first that uses the collision for depth estimation.
Revised: The collision event has been explored for navigation and physical search in (Gan et al., 2022), but our work is the first that uses the collision for depth estimation.

Original: Collisions are common and can arise when a ball bounces on the ground, a person takes a step,  a musician hits a drum.
Revised: Collisions are common and can arise when a ball bounces on the ground, a person takes a step, or a musician hits a drum.

Original: We identify and exploit several unique properties related to various collisions in the wild.
Revised: We identify and exploit several unique properties related to various collisions in the wild.

Original: First, the duration of a collision is short and collision events are sparse.
Revised: First, the duration of a collision is short and collision events are sparse.

Original: Thus, there are few overlapped collisions.
Revised: Thus, there are few overlapped collisions.

Original: Second, though the motion of objects changes dramatically after the collision, they are almost static at the collision moment.
Revised: Second, though the motion of objects changes dramatically after the collision, they are almost static at the collision moment.

Original: Third, the impact sound is loud enough to propagate to a long range.
Revised: Third, the impact sound is loud enough to propagate to a long range.

Original: While Flash-to-Bangphenonmenon is intuitive and widely known, using it for  depth estimation poses several significant challenges: (i) It is challenging toget the ground truth collision time fromboth video and audio.
Revised: Flash-to-Bang is applied to the range of miles for human perception. Using it for general depth estimation poses several significant challenges: (i) It is inaccessible to ground truth collision time from video and audio.

Original: Video only offers up to 240 frames per second, and may not capture the exact instance when the collision occurs.
Revised: Video only offers up to 240 frames per second(fps), and may not capture the exact instance when the collision occurs.

Original: Audio has a higher sampling rate but it is hard to detect the start of a collision solely based on the collision sound due to different sound pattern arising from collisions as well as ambient noise.
Revised: Audio has a high sampling rate but it is hard to detect the start of a collision solely based on the collision sound due to different sound patterns arising from collisions as well as ambient noise.

Original: (ii) We need highly accurate collision time.
Revised: (ii) We need highly accurate collision time.

Original: 1 ms error can result in a depth error of 34 cm.
Revised: 1 ms error can result in a depth error of 34 cm.

Original: (iii) Noise present in both audio and video further exacerbate the problem.
Revised: (iii) Noise present in both audio and video further exacerbate the problem.

Original: To realize our idea, we formulate the sound source depth estimation as the audio-visual localization task.
Revised: To realize our idea, we formulate the sound source depth estimation as the audio-visual localization task.

Original: Whereas existing work (Wu et al., 2019; Xia & Zhao, 2022) still focuses on 1-second-segment level localization.
Revised: Whereas existing work (Wu et al., 2019; Xia & Zhao, 2022) still focuses on 1-second-segment level localization.

Original: FBdepth performs event-level localization by aligning correspondence between the audio and the video.
Revised: FBdepth performs event-level localization by aligning correspondence between the audio and the video.

Original: Apart from audio-visual semantic features as input in existing work (Tian et al., 2018; Chen et al., 2021a), we incorporate optical flow to exclude static objects with similar visual appearances.
Revised: Apart from audio-visual semantic features as input in existing work (Tian et al., 2018; Chen et al., 2021a), we incorporate optical flow to exclude static objects with similar visual appearances.

Original: Furthermore, FBDepth applies the impulse change of optical flow to locate collision moments at the frame level.
Revised: Furthermore, FBDepth applies the impulse change of optical flow to locate collision moments at the frame level.

Original: Finally, we formulate the ms-level estimation as an optimization problem of video interpolations.
Revised: Finally, we formulate the ms-level estimation as an optimization problem of video interpolations.

Original: FBDepth succeeds to interpolate the best collision moment by maximizing the intersection between extrapolations of before-collision and after-collision flows.
Revised: FBDepth succeeds to interpolate the best collision moment by maximizing the intersection between extrapolations of before-collision and after-collision flows.

Original: With the estimated timestamp of visual collision, we regress the sound source depth with the audio clip and visual features.
Revised: With the estimated timestamp of visual collision, we regress the sound source depth with the audio clip and visual features.

Original: FBdepth avoids the requirement to know the timestamp of audio collision.
Revised: FBdepth avoids the requirement to know the timestamp of audio collision.

Original: Besides, different objects have subtle differences in audio-visual temporal alignment.
Revised: Besides, different objects have subtle differences in audio-visual temporal alignment.

Original: For example, a rigid body generates the sound peak once it touches another body.
Revised: For example, a rigid body generates the sound peak once it touches another body.

Original: But an elastic body produces little sound during the initial collision and takes several ms to produce the peak with the maximum deformation.
Revised: But an elastic body produces little sound during the initial collision and takes several ms to produce the peak with the maximum deformation.

Original: We feed semantic features to enable the network aware of the material, size, etc.
Revised: We feed semantic features to enable the network aware of the material, size, etc.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: To the best of our knowledge, FBDepth is the first passive audio-visual depth estimation.
Revised: To the best of our knowledge, FBDepth is the first passive audio-visual depth estimation.

Original: It brings the physical propagation property to audio-visual learning.
Revised: It brings the physical propagation property to audio-visual learning.

Original: We introduce the ms-level audio-visual localization task.
Revised: We introduce the ms-level audio-visual localization task.

Original: We propose a novel coarse-to-fine method to improve temporal resolution by leveraging the unique properties of collisions.
Revised: We propose a novel coarse-to-fine method to improve temporal resolution by leveraging the unique properties of collisions.

Original: We collect 3.6K+ audio-visual samples across 24 different objects in the wild.
Revised: We collect 3.6K+ audio-visual samples across 24 different objects in the wild.

Original: Our extensive evaluation shows that FBDepth achieves 0.64m absolute error(AbsErr) and 2.98% AbsRel across a wide range from 2 m to 60 m. Especially, FBDepth shows more improvement in the longer range.
Revised: Our extensive evaluation shows that FBDepth achieves 0.64m absolute error(AbsErr) and 2.98% AbsRel across a wide range from 2 m to 60 m. Especially, FBDepth shows more improvement in the longer range.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-modality Depth estimation.
Revised: Multi-modality Depth estimation.

Original: Recent work on depth estimation has shown the benefits of fusing cameras and other active sensors.
Revised: Recent work on depth estimation has shown the benefits of fusing cameras and other active sensors.

Original: (Qiu et al., 2019; Imran et al., 2021) recover dense depth maps from sparse Lidar point clouds and a single image.
Revised: (Qiu et al., 2019; Imran et al., 2021) recover dense depth maps from sparse Lidar point clouds and a single image.

Original: (Long et al., 2021b) associate pixels with pretty sparse radar points to achieve superior accuracy.
Revised: (Long et al., 2021b) associate pixels with pretty sparse radar points to achieve superior accuracy.

Original: The effective range can be increased as well by Lidar-camera (Zhang et al., 2020) or Radar-camera (Zhang et al., 2021).
Revised: The effective range can be increased as well by Lidar-camera (Zhang et al., 2020) or Radar-camera (Zhang et al., 2021).

Original: However, these methods are still expensive in cost and power consumption.
Revised: However, these methods are still expensive in cost and power consumption.

Original: (Gao et al., 2020; Parida et al., 2021) emit audio chirps and learn the depth map implicitly with audio reflections and a single image.
Revised: (Gao et al., 2020; Parida et al., 2021) emit audio chirps and learn the depth map implicitly with audio reflections and a single image.

Original: However, these methods require many nearby acoustic reflectors to produce effective echos sothat the setup is limited in rooms.
Revised: However, these methods require many nearby acoustic reflectors to produce effective echos so the setup is limited in rooms.

Original: Besides, they are evaluated in an audiovisual simulator.
Revised: Besides, they are evaluated in an audiovisual simulator.

Original: FBDepth only uses one extra microphone to perceive natural sounds directly.
Revised: FBDepth only uses one extra microphone to perceive natural sounds directly.

Original: It keeps the passive design of the audio but applies the physical measurement explicitly.
Revised: It keeps the passive design of the audio but applies the physical measurement explicitly.

Original: The one-path sound propagation has a longer effective range than echoes.
Revised: The one-path sound propagation has a longer effective range than echoes.

Original: Sound source localization.
Revised: Sound source localization.

Original: Previous systems localize sound sources with microphone arrays (Valin et al., 2003; Rascon & Meza, 2017) or one microphone with a camera (Hershey & Movellan, 1999).
Revised: Previous systems localize sound sources with microphone arrays (Valin et al., 2003; Rascon & Meza, 2017) or one microphone with a camera (Hershey & Movellan, 1999).

Original: They intend to estimate the direction of arrival(DOA) or the distance.
Revised: They intend to estimate the direction of arrival(DOA) or the distance.

Original: The DOA is inferred by the subtle difference in arrival time from the sound source to each microphone(Mao et al., 2019; Sun et al., 2022) or by semantic matching with the visual appearance if given images(Tian et al., 2018; Arandjelovic & Zisserman, 2018).
Revised: The DOA is inferred by the subtle difference in arrival time from the sound source to each microphone(Mao et al., 2019; Sun et al., 2022) or by semantic matching with the visual appearance if given images(Tian et al., 2018; Arandjelovic & Zisserman, 2018).

Original: The distance can be estimated by triangulation methods with multiple DOAs and room structures(Wang et al., 2021;
Revised: The distance can be estimated by triangulation methods with multiple DOAs and room structures(Wang et al., 2021;

Original: Shen et al., 2020).
Revised: Shen et al., 2020).

Original: It can be fully learning-based to predict depth with room acoustic features (Chen et al., 2021b).
Revised: Many work study the room acoustic and the distance cues from the reverberation(Singh et al., 2021; Chen et al., 2021b) but (Zahorik, 2002) shows that the reverberation has a coarse coding with the distance.

Original: Compared to these methods, FBDepth directly estimates the distance by the ToF and achieves superior accuracy to indirect triangulation methods and implicitly depth learning networks.
Revised: Compared to these methods, FBDepth directly estimates the distance by the ToF and achieves superior accuracy to indirect triangulation methods and implicitly depth learning networks on reverberation.

Original: Audio-visual event localization aims to detect and localize events in videos.
Revised: Audio-visual event localization aims to detect and localize events in videos.

Original: (Tian et al., 2018) first propose the taskthat detects events which are both audible and visible.
Revised: (Tian et al., 2018) first propose the task and build up the audio-visual event(AVE) dataset.

Original: Theybuild up the audio-visual event(AVE) dataset and apply an audio-guided visual attention mechanism to learn visual regions with the related sounding object or motions.
Revised: They apply an audio-guided visual attention mechanism to learn visual regions with the related sounding object or motions.

Original: Recent works developplenty of fusion strategies to leverage the global features on this task, such as dual-modality sequence-sequence framework (Lin et al., 2019) and dual attention matching mechanism (Wu et al., 2019).
Revised: Recent works develop dual-modality sequence-sequence framework (Lin et al., 2019) and dual attention matching mechanism (Wu et al., 2019) to leverage global features.

Original: However, the temporal event boundary is second-level in AVE dataset so it is split as 1s-long segments.
Revised: However, the temporal event boundary is 1s-level in AVE dataset so it is split as 1s-long segments.

Original: We study the instant collision event and come across the coarse boundary problem as well.
Revised: We study the instant collision event and solve the coarse boundary problem as well.

Original: Thanks to the unique properties of collisions and the coarse-to-fine strategy, we can achieve ms-level temporal resolution without the ground truth timestamp.

Original: (Gan et al., 2022) has a similar setup to ours.
Revised: (Gan et al., 2022) has a similar setup to ours.

Original: They use an embodied robot agent to localize a dropped object in 3D virtual rooms.
Revised: They use an embodied robot agent to navigate to a dropped object in 3D virtual rooms.

Original: They integrate asynchronous vision and audition and navigate to the object with imitation learning, reinforcement learning, and modular planning.
Revised: They integrate asynchronous vision and audition and navigate to the object.

Original: The asynchronism comes from the invisibility of the object.
Revised: The asynchronism comes from the invisibility of the object.

Original: Even though their simulator has been pretty vivid enough for semantic tasks, it has a gap to real-world collision for the ms-level formulation.
Revised: Even though their simulator has been pretty vivid enough for semantic tasks, it has a gap in the real-world collision for the mslevel formulation.

Original: Falling objects dataset(Kotera et al., 2020), TbD dataset(Kotera et al., 2019) and TbD-3D dataset(Rozumnyi et al., 2020) explores falling motions and fast movings but they do not have audio and depth information.
Revised: Falling objects dataset(Kotera et al., 2020), TbD dataset(Kotera et al., 2019) and TbD-3D dataset(Rozumnyi et al., 2020) explore falling motions and fast movements but they do not have audio and depth information.

Original: Video frame interpolation aims to synthesize intermediate frames between existing ones of a video.
Revised: Video frame interpolation aims to synthesize intermediate frames between existing ones of a video.

Original: Most state-of-the-art approaches explicitly or implicitly assume a simplistic linear motion.
Revised: Most state-of-the-art approaches explicitly or implicitly assume a simplistic linear motion.

Original: Warping-based methods (Baker et al., 2011;
Revised: Warping-based methods (Baker et al., 2011;

Original: Park et al., 2020) apply optical flow and forward warping to shift pixels to intermediate frames linearly.
Revised: Park et al., 2020) apply optical flow and forward warping to shift pixels to intermediate frames linearly.

Original: Phase-based methods (Meyer et al., 2015; 2018) combine the phase information across different scales but the phase is modeled as a linear function of time.
Revised: Phase-based methods (Meyer et al., 2015; 2018) combine the phase information across different scales but the phase is modeled as a linear function of time.

Original: Recent methods are developed to approximate non-linear motion, such as kernelbased methods (Niklaus et al., 2017a;b), quadratic interpolation (Xu et al., 2019), cubic motion modeling (Chi et al., 2020), etc.
Revised: Recent methods are developed to approximate non-linear motion, such as kernelbased methods (Niklaus et al., 2017a;b), quadratic interpolation (Xu et al., 2019), cubic motion modeling (Chi et al., 2020), etc.

Original: However, they still fail to complex non-linear motions because precise motion dynamics cannot be captured in the blind time between keyframes.
Revised: However, they still fail to complex non-linear motions because precise motion dynamics cannot be captured in the blind time between keyframes.

Original: Unfortunately, collisions are super non-linear and instant.
Revised: Unfortunately, collisions are super non-linear and instant.

Original: Given two keyframes before and after the collision, it is ambiguous to decide whether there is a collision.
Revised: Given two keyframes before and after the collision, it is ambiguous to decide whether there is a collision.

Original: Hence, these methods are not applicable.
Revised: Hence, these methods are not applicable.

Original: We analyze the motions before and after the collision and extrapolate optical flows to find the most potential collision position.
Revised: We analyze the motions before and after the collision and extrapolate optical flows to find the most potential collision position.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We formulate the depth estimation by the physical law of wave propagation.
Revised: We formulate the depth estimation by the physical law of wave propagation.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: where the depth of the sound source is d and the difference between the ToF of sound and light is T .
Revised: where the depth of the sound source is d and the difference between the ToF of sound and light is T .

Original: c and v denote the propagation speeds of light and sound, respectively.
Revised: c and v denote the propagation speeds of light and sound, respectively.

Original: We can estimate d based on d = cvT c − v ≈ vT since c ≫ v .
Revised: We can estimate d based on d = cvTc − v ≈ vT since c ≫ v .

Original: We observe T
Revised: We observe T

Original: = T audio − T video + T hardware , where T audio and T video denote the event time in the audio and video recordings, respectively, and T hardware denotes the start time difference in the audio and video recordings.
Revised: = T audio − T video + T hardware , where T audio and T video denote the event time in the audio and video recordings, respectively, and T hardware denotes the start time difference in the audio and video recordings.

Original: It can be small as well as have a small variance with a well-designed media system such as the Apple AVFoundation framework.
Revised: It can be small as well as have a small variance with a well-designed media system such as the Apple AVFoundation framework.

Original: We regard it as a constant unknown bias to learn.
Revised: We regard it as a constant unknown bias to learn.

Original: It is impossible to label the precise T video and T audio manually.
Revised: It is impossible to label the precise T video and T audio manually.

Original: T video can be tagged at most frame-level.
Revised: T video can be tagged at most frame-level.

Original: Even though many commercial cameras can support up to 240 FPS, it results in a 4-ms segment and 1.43m depth variation.
Revised: Even though many commercial cameras can support up to 240 FPS, it results in a 4-ms segment and 1.43m depth variation.

Original: Moreover, it is tough to determine the exact frame that is nearest to the collision in high FPS mode by a human being due to the constrained view of the camera.
Revised: Moreover, it is tough to determine the exact frame that is nearest to the collision in high FPS mode by a human being due to the constrained view of the camera.

Original: T audio is challenging to recognize in the wild as well.
Revised: T audio is challenging to recognize in the wild as well.

Original: Although the audio sampling rate is high enough, we can recognize the significant early peaks instead of the first sample triggered by the collision.
Revised: Although the audio sampling rate is high enough, we can recognize the significant early peaks instead of the first sample triggered by the collision.

Original: The best effort of segmentation is 10-ms level based on real data.
Revised: The best effort of segmentation is 10-ms level based on real data.

Original: We cannot learn the timestamp with supervision.
Revised: We cannot learn the timestamp with supervision.

Original: We propose a 2-stage estimation framework.
Revised: We propose a 2-stage estimation framework.

Original: The goal of the first stage is to estimate the numerical T video .
Revised: The goal of the first stage is to estimate the numerical T video .

Original: As figure 1 shows, we localize the audiovisual event in the stream and then take advantage of the unique optical flow of the collision to estimate T video at ms-level.
Revised: As figure 1 shows, we localize the audiovisual event in the stream and then take advantage of the unique optical flow of the collision to estimate T video at ms-level.

Original: In the second stage, we place the T video as an anchor into the audio clip and direct regress the depth with depth supervision.
Revised: In the second stage, we place the T video as an anchor into the audio clip and direct regress the depth with depth supervision.

Original: We make the network optimize T audio automatically with knowledge of the T video , the audio waveform and visual features.
Revised: We make the network optimize T audio automatically with knowledge of the T video , the audio waveform and visual features.

Original:  A PPROACH
Revised:  A PPROACH

Original: We demonstrate a novel coarse-to-fine pipeline to localize the collision with a super temporal resolution in the video.
Revised: We demonstrate a novel coarse-to-fine pipeline to localize the collision with a super temporal resolution in the video.

Original: This method does not require annotations on ms-level, which is at least two orders of magnitude finer than previous approaches.
Revised: This method does not require annotations on ms-level, which is at least two orders of magnitude finer than previous approaches.

Original: They rely on the supervision of segment annotations, such as AVE dataset with 1-second segments (Tian et al., 2018), Lip Reading Sentencesdataset with word-level segments (Chung & Zisserman, 2016), BOBSL with sentence-level alignments (Bull et al., 2021).
Revised: They rely on the supervision of segment annotations, such as AVE dataset with 1-second segments (Tian et al., 2018), Lip Reading Sentencesdataset with word-level segments (Chung & Zisserman, 2016), BOBSL with sentence-level alignments (Bull et al., 2021).

Original: 4. E VENT - LEVEL L OCALIZATION
Revised: 4. E VENT - LEVEL L OCALIZATION

Original: Audio-visual modeling for collisions.
Revised: Audio-visual modeling for collisions.

Original: In this step, our goal is to localize the audio-visual event for the region and the period of interest.
Revised: In this step, our goal is to localize the audio-visual event for the region and the period of interest.

Original: It is similar to (Tian et al., 2018), but the unique properties of collisions bring new opportunities to learning strategy.
Revised: It is similar to (Tian et al., 2018), but the unique properties of collisions bring new opportunities to learning strategy.

Original: Collisions have a significant motion than other sound sources.
Revised: Collisions have a significant motion than other sound sources.

Original: We can use the optical flow to inform the network of moving pixels.
Revised: We can use the optical flow to inform the network of moving pixels.

Original: Besides, the impact sound is highly correlated to the rich information of objects (Gan et al., 2022), such as shape, materials, size, mass, etc.
Revised: Besides, the impact sound is highly correlated to the rich information of objects (Gan et al., 2022), such as shape, materials, size, mass, etc.

Original: It makes audio-visual
Revised: It makes audio-visual

Original: cross-matching easier than general audiovisual events so that we do not need to apply a complex scheme to learn.
Revised: cross-matching easier than general audiovisual events so that we do not need to apply a complex scheme to learn.

Original: Another fact is that collisions are pretty sparse temporally in the wild because the duration of collisions is extremely short.
Revised: Another fact is that collisions are pretty sparse temporally in the wild because the duration of collisions is extremely short.

Original: It is rare to come across overlapped collisions based on our empirical study on the basketball court.
Revised: It is rare to come across overlapped collisions based on our empirical study on the basketball court.

Original: Only two frames have double collisions among all 1208 frames and a total of 203 collisions when 7 basketballs are played during a 40-s duration.
Revised: Only two frames have double collisions among all 1208 frames and a total of 203 collisions when 7 basketballs are played during a 40-s duration.

Original: We propose a motion-guided audio-visual correspondence network (MAVNet).
Revised: We propose a motion-guided audio-visual correspondence network (MAVNet).

Original: Similar to (Tian et al., 2018; Wu et al., 2019), MAVNet performs the cross-matching for the audio features and the
Revised: Similar to (Tian et al., 2018; Wu et al., 2019), MAVNet performs the cross-matching for the audio features and the

Original: RGB-F channels.
Revised: RGB-F channels.

Original: Besides, it predicts the audio-visual segmentation to capture the whole pixels of the target object.
Revised: Besides, it predicts the audio-visual segmentation to capture the whole pixels of the target object.

Original: It can achieve fine-grained audio-visual scene understanding (Zhou et al., 2022).
Revised: It can achieve fine-grained audio-visual scene understanding (Zhou et al., 2022).

Original: We use the segmentation mask to filter flows of interest and perform high-resolution estimation in the next steps.
Revised: We use the segmentation mask to filter flows of interest and perform high-resolution estimation in the next steps.

Original: MAVNet has two backbones to deal with RGB-F channels and audio clips respectively.
Revised: MAVNet has two backbones to deal with RGB-F channels and audio clips respectively.

Original: A UNet (Ronneberger et al., 2015) style encoder is applied to extract the frame features conditioned by optical flows.
Revised: A UNet (Ronneberger et al., 2015) style encoder is applied to extract the frame features conditioned by optical flows.

Original: It uses a series of convolution layers to extract visual features.
Revised: It uses a series of convolution layers to extract visual features.

Original: Another branch is the audio encoder which takes in the time-domain signal.
Revised: Another branch is the audio encoder which takes in the time-domain signal.

Original: It has a 1D convolution layer to learn an
Revised: It has a 1D convolution layer to learn an

Original: STFT-like representation and a stack of 2D convolution layers with batch normalization to learn the semantic audio features.
Revised: STFT-like representation and a stack of 2D convolution layers with batch normalization to learn the semantic audio features.

Original: We replicate the audio feature, tile them to match the visual feature dimension, and concatenate the audio and visual feature maps.
Revised: We replicate the audio feature, tile them to match the visual feature dimension, and concatenate the audio and visual feature maps.

Original: MAVNet has two output heads as well.
Revised: MAVNet has two output heads as well.

Original: the
Revised: the

Original: U-Net decoder applies a series of up-convolutions and skip-connections from the RGB-F encoder to fused feature maps to learn the binary segmentation mask M .
Revised: U-Net decoder applies a series of up-convolutions and skip-connections from the RGB-F encoder to fused feature maps to learn the binary segmentation mask M .

Original: Meanwhile, the fused feature map is fed into a binary classification head consisting of convolution layers and linear layers to predict the audio-visual event relevance y ∈ { 0 , 1 } .
Revised: Meanwhile, the fused feature map is fed into a binary classification head consisting of convolution layers and linear layers to predict the audio-visual event relevance y ∈ { 0 , 1 } .

Original:  We use the weighted sum Binary Cross Entropy (BCE) loss as the training objective for both segmentation and the cross matching, We train all components to jointly optimize the location predictions and energy reconstruction.
Revised: Training We use the weighted sum Binary Cross Entropy (BCE) loss as the training objective for both segmentation and the cross matching, We train all components to jointly optimize the location predictions and energy reconstruction.

Original: We minimize the total loss L total =
Revised: We minimize the total loss L total =

Original: BCE ( M, ˆ M ) + λ ∗
Revised: BCE ( M, ˆ M ) + λ ∗

Original: BCE ( y, ˆ y ) where λ is the hypermeter to set.
Revised: BCE ( y, ˆ y ) where λ is the hypermeter to set.

Original: Inference We only use low FPS to perform MAVNet to avoid dense inference at this stage.
Revised: Inference We only use low FPS to perform MAVNet to avoid dense inference at this stage.

Original: Moreover, we do not need to activate the segmentation head until the audio clip and the frame are highly matched.
Revised: Moreover, we do not need to activate the segmentation head until the audio clip and the frame are highly matched.

Original: Finally, MAVNet uses this audio clip to retrieve a sequence of frames including the full collision procedure.
Revised: Finally, MAVNet uses this audio clip to retrieve a sequence of frames including the full collision procedure.

Original: 4. F RAME - LEVEL L OCALIZATION
Revised: 4. F RAME - LEVEL L OCALIZATION

Original: Given a sequence of video frames, our goal is to split them into two sets: the frames before the collision V 0 and the frames after the collision V 1 .
Revised: Given a sequence of video frames, our goal is to split them into two sets: the frames before the collision V 0 and the frames after the collision V 1 .

Original: This essentially requires us to determine the last frame I e in V 0 before the collision and the first frame I s in V 1 after the collision.
Revised: This essentially requires us to determine the last frame I e in V 0 before the collision and the first frame I s in V 1 after the collision.

Original: Thus, we locate the collision between the frame I e
Revised: Thus, we locate the collision between the frame I e

Original: and I s .
Revised: and I s .

Original: Based on the analysis of the physical motion, we make an important observation that can help determine I e
Revised: Based on the analysis of the physical motion, we make an important observation that can help determine I e

Original: and I s .
Revised: and I s .

Original: The collision results in a significant acceleration change due to the strong impulse force.
Revised: The collision results in a significant acceleration change due to the strong impulse force.

Original: Let a t = v t − v t − 1 and δa t = a t − a t − 1 denote the acceleration and acceleration change of frame I t , respectively.
Revised: Let a t = v t − v t − 1 and δa t = a t − a t − 1 denote the acceleration and acceleration change of frame I t , respectively.

Original: δa between I e
Revised: δa between I e

Original: and I s is large, while δa between adjacent frames before or after the collision is small.
Revised: and I s is large, while δa between adjacent frames before or after the collision is small.

Original: If the object stops moving immediately after the collision, we take the static frame I e +1 as I s .
Revised: If the object stops moving immediately after the collision, we take the static frame I e +1 as I s .

Original: Finally, we select the frames before I e to generate V 0 , and select the frames after I s to generate V 1 .
Revised: Finally, we select the frames before I e to generate V 0 , and select the frames after I s to generate V 1 .

Original: We use the retrieved mask in the last stage to determine the object positions in the frames and calculate the velocity, acceleration, and acceleration change.
Revised: We use the retrieved mask in the last stage to determine the object positions in the frames and calculate the velocity, acceleration, and acceleration change.

Original: We find the I e
Revised: We find the I e

Original: and I s at the low FPS and then replicate the procedure for frames between I e
Revised: and I s at the low FPS and then replicate the procedure for frames between I e

Original: and I s at high FPS.
Revised: and I s at high FPS.

Original: Finally, we locate I e and
Revised: Finally, we locate I e and

Original: I s in the high FPS mode efficiently.
Revised: I s in the high FPS mode efficiently.

Original: 4. M S - LEVEL L OCALIZATION
Revised: 4. M S - LEVEL L OCALIZATION

Original: To further locate the exact moment of the collision, we try to interpolate frames between I e
Revised: To further locate the exact moment of the collision, we try to interpolate frames between I e

Original: and I s to recover the skipped frame.
Revised: and I s to recover the skipped frame.

Original: Unfortunately, the common assumption of frame-based interpolation is fully broken down.
Revised: Unfortunately, the common assumption of frame-based interpolation is fully broken down.

Original: Motion consistency is fundamental for spatio-temporal video processing.
Revised: Motion consistency is fundamental for spatio-temporal video processing.

Original: If the motion of the object is temporally stable across several frames ( e.g. , due to a constant force), the position and pose can be predicted in the future frames as well as be interpolated between two frames.
Revised: If the motion of the object is temporally stable across several frames ( e.g. , due to a constant force), the position and pose can be predicted in the future frames as well as be interpolated between two frames.

Original: We denote it as motion first consistency .
Revised: We denote it as motion first consistency .

Original: However, the impact sound is caused by an impulse force, which results in a rapid change of the motion status.
Revised: However, the impact sound is caused by an impulse force, which results in a rapid change of the motion status.

Original: It breaks the motion continuity and consistency.
Revised: It breaks the motion continuity and consistency.

Original: When we observe I e
Revised: When we observe I e

Original: and I s , we cannot determine whether a collision happens or the object just flies in the air.
Revised: and I s , we cannot determine whether a collision happens or the object just flies in the air.

Original: Luckily, the collision moment retains a new form of motion consistency.
Revised: Luckily, the collision moment retains a new form of motion consistency.

Original: We denote it as motion second consistency .
Revised: We denote it as motion second consistency .

Original: It reveals that the motions before and after the collision share the same intersection position.
Revised: It reveals that the motions before and after the collision share the same intersection position.

Original: Besides, they keep the motion first consistency separately.
Revised: Besides, they keep the motion first consistency separately.

Original: Therefore, we can extrapolate the motions based on the motion first consistency and search for the most similar motion extrapolations by leveraging motion second consistency .
Revised: Therefore, we can extrapolate the motions based on the motion first consistency and search for the most similar motion extrapolations by leveraging motion second consistency .

Original: Note that our final goal is to find the timestamp of the collision instead of the motion status at the shared position.
Revised: Note that our final goal is to find the timestamp of the collision instead of the motion status at the shared position.

Original: (Kotera et al., 2019; Rozumnyi et al., 2020) try to recover the sub-frame motions and trajectories as well but they require the high FPS ground truth to guide the training.
Revised: (Kotera et al., 2019; Rozumnyi et al., 2020) try to recover the sub-frame motions and trajectories as well but they require the high FPS ground truth to guide the training.

Original: In our context, we care more about when the collision happens than what it looks like.
Revised: In our context, we care more about when the collision happens than what it looks like.

Original: Optical flow extrapolation Optical flow is widely used for frame prediction () and interpolation () by warping the frame with the estimated optical flow.
Revised: Optical flow extrapolation Optical flow is widely used for frame prediction () and interpolation (Baker et al., 2011) by warping the frame with the estimated optical flow.

Original: Because it can capture all motions of pixels and get a finer understanding of the object dynamics.
Revised: Because it can capture all motions of pixels and get a finer understanding of the object dynamics.

Original: The optical flow sequence is usually generated by adjacent video frames.
Revised: The optical flow sequence is usually generated by adjacent video frames.

Original: However, it is not efficient for extrapolation.
Revised: However, it is not efficient for extrapolation.

Original: The drift of pixels in the flow requires extra iterative wrappings to align the corresponding pixels, which results in accumulation errors.
Revised: The drift of pixels in the flow requires extra iterative wrappings to align the corresponding pixels, which results in accumulation errors.

Original: Therefore, we compute the optical flows from an anchor frame I a to the frame sequence V as { I 0 , I 1 , ...I n } .
Revised: Therefore, we compute the optical flows from an anchor frame I a to the frame sequence V as { I 0 , I 1 , ...I n } .

Original: We can estimate the flow sequence F a →V as { f a → 0 , f a → 1 , ...
Revised: We can estimate the flow sequence F a →V as { f a → 0 , f a → 1 , ...

Original: f a → n } .
Revised: f a → n } .

Original: As f a → n ( x, y ) represents the movement of the pixel I a ( x, y ) to I n , F a →V ( x, y ) describes how the pixel in I a ( x, y ) moves across the frame sequence V .
Revised: As f a → n ( x, y ) represents the movement of the pixel I a ( x, y ) to I n , F a →V ( x, y ) describes how the pixel in I a ( x, y ) moves across the frame sequence V .

Original: Hence, F a →V tracks the global motion of each pixel without iterative warpings.
Revised: Hence, F a →V tracks the global motion of each pixel without iterative warpings.

Original: With the historical positions of I a ( x, y ) from frame I 0 to I n , we can regress the motion of this pixel and extrapolate the flow to f a → n + δt , which is the relative pixel position to I n + δt with an arbitrary δt .
Revised: With the historical positions of I a ( x, y ) from frame I 0 to I n , we can regress the motion of this pixel and extrapolate the flow to f a → n + δt , which is the relative pixel position to I n + δt with an arbitrary δt .

Original: In our context, We pick k consecutive frames before the collision V pre as { I e −
Revised: In our context, We pick k consecutive frames before the collision V pre as { I e −

Original: k
Revised: k

Original: +1 , I e − k +2 , ..., I e } and after the collision V post as { I s + k − 1 , I s + k − 2 , ..., I s } .
Revised: +1 , I e − k +2 , ..., I e } and after the collision V post as { I s + k − 1 , I s + k − 2 , ..., I s } .

Original: We select the frame I e as the anchor frame.
Revised: We select the frame I e as the anchor frame.

Original: It is near the collision moment, so its motion to other frames is not dramatic and easy to be estimated.
Revised: It is near the collision moment, so its motion to other frames is not dramatic and easy to be estimated.

Original: Hence, we can estimate the optical flow sequences F e →V pre and F e →V post Meanwhile, we apply the predicted segmentation mask of I e to filter the pixels of the target object.
Revised: Hence, we can estimate the optical flow sequences F e →V pre and F e →V post Meanwhile, we apply the predicted segmentation mask of I e to filter the pixels of the target object.

Original: In the last step, we build up regressors R for each pixel’s motion individually and predict future locations in any sub-frame.
Revised: In the last step, we build up regressors R for each pixel’s motion individually and predict future locations in any sub-frame.

Original: Optical flow interpolation We have construct pixel level regressors for F e →V pre and corresponding F e →V post .
Revised: Optical flow interpolation We have construct pixel level regressors for F e →V pre and corresponding F e →V post .

Original: They can extrapolate the flow f e
Revised: They can extrapolate the flow f e

Original: → e + δt 0 and f a → s + δt 1 , respectively.
Revised: → e + δt 0 and f a → s + δt 1 , respectively.

Original: δt 0 , δt 1 are extrapolation steps.
Revised: δt 0 , δt 1 are extrapolation steps.

Original: The optimization goal is to
Revised: The optimization goal is to

Original: [Equation]
Revised: [Equation]

Original: The collision duration is s + δt 1 − ( e + δt 0 ) , which is always more than 0.
Revised: The collision duration is s + δt 1 − ( e + δt 0 ) , which is always more than 0.

Original: e + δt 0 is the target mslevel localization ˆ T video .
Revised: e + δt 0 is the target mslevel localization ˆ T video .

Original: We can apply this interpolation methodology to search the intersection of the object’s center trajectory or maximize the Intersection over Union (IoU) of the object’s bounding box.
Revised: We can apply this interpolation methodology to search the intersection of the object’s center trajectory or maximize the Intersection over Union (IoU) of the object’s bounding box.

Original: However, both only use several key points so they cannot achieve a fine granularity since the optical flow takes advantage of thousands of pixels.
Revised: However, both only use several key points so they cannot achieve a fine granularity since the optical flow takes advantage of thousands of pixels.

Original: 4. D EPTH R EGRESSION
Revised: 4. D EPTH R EGRESSION

Original: Based on the estimation ˆ T video , we directly regress the depth to fit the T audio and the bias T Hardware with the supervision of ground truth depth.
Revised: Based on the estimation ˆ T video , we directly regress the depth to fit the T audio and the bias T Hardware with the supervision of ground truth depth.

Original: We observe that the sound generation procedure varies a lot across different objects, materials, shapes, and motions.
Revised: We observe that the sound generation procedure varies a lot across different objects, materials, shapes, and motions.

Original: On one hand, the diverse waveforms make it impractical to measure the exact T audio manually.
Revised: On one hand, the diverse waveforms make it impractical to measure the exact T audio manually.

Original: On the other hand, each specific waveform has significant implications on what is the best T audio corresponding to ˆ T video .
Revised: On the other hand, each specific waveform has significant implications on what is the best T audio corresponding to ˆ T video .

Original: To combat the background noise from other sources, we also feed the RGB-F crop of the target object from frame
Revised: To combat the background noise from other sources, we also feed the RGB-F crop of the target object from frame

Original: I e to the depth predictor.
Revised: I e to the depth predictor.

Original: It includes the semantic features of the object as well as the motion status just before the collision.
Revised: It includes the semantic features of the object as well as the motion status just before the collision.

Original: These cues can guide the predictor to find the waveform pattern easily.
Revised: These cues can guide the predictor to find the waveform pattern easily.

Original: We select a sequence of audio samples starting from I e and label some anchor samples as 1 at ˆ
Revised: We select a sequence of audio samples starting from I e and label some anchor samples as 1 at ˆ

Original: T video .
Revised: T video .

Original: It informed the audio sequence about the timestamp of the visual collision directly.
Revised: It informed the audio sequence about the timestamp of the visual collision directly.

Original: We feed the enriched sequence into the 1D convolution layer to extract a 2D representation.
Revised: We feed the enriched sequence into the 1D convolution layer to extract a 2D representation.

Original: It is followed by two residual blocks to learn high-dimension features.
Revised: It is followed by two residual blocks to learn high-dimension features.

Original: Meanwhile, we use ResNet-18 (He et al., 2015) to extract the RGB-F features of the target object.
Revised: Meanwhile, we use ResNet-18 (He et al., 2015) to extract the RGB-F features of the target object.

Original: We tile and concatenate the RGB-F features to the audio features along the channel dimension and append another two residual blocks to fuse the features.
Revised: We tile and concatenate the RGB-F features to the audio features along the channel dimension and append another two residual blocks to fuse the features.

Original: Finally, it is followed by a pooling layer and a fully connected layer to predict the depth.
Revised: Finally, it is followed by a pooling layer and a fully connected layer to predict the depth.

Original: The output maps to depth by the 2D projection.
Revised: The output maps to depth by the 2D projection.

Original: We use Mean Square Error (MSE)
Revised: We use Mean Square Error (MSE)

Original: L depth =
Revised: L depth =

Original: || d, ˆ d ||as the learning objective where d and ˆ d are the target depth and the predicted depth.
Revised: || d, ˆ d ||as the learning objective where d and ˆ d are the target depth and the predicted depth.

Original:  E XPERIMENTS 5. S ETUP
Revised:  E XPERIMENTS 5. S ETUP

Original: Dataset platform and collection We use an iPhone XR with a 240-fps slow-motion mode to collect the video with audio.
Revised: Dataset platform and collection We use an iPhone XR with a 240-fps slow-motion mode to collect the video with audio.

Original: The audio sampling rate is 48Khz.
Revised: The audio sampling rate is 48Khz.

Original: T hardware is small as 1 ms and has a small variance within 1 ms on the iPhone.

Original: Moreover, its frame rate is stable and does not have a frame rate drifting.

Original: Thus, we can transform the frame number to the timestamp accurately.

Original: The calibration of the audio-visual media framework is out of the scope in this work.

Original: To capture the remote scene clearly, the telephoto lens has become indispensable in recent smartphones.

Original: Samsung Ultra 20 can support 10x optical zoom and 100x hybrid zoom, and Pixel 6 pro has 20x zoom.

Original: The iPhone XR is not equipped with a telephoto lens, so we mount an ARPBEST monocular telescope to enlarge the scene at a large distance. We observed that the image quality of our setup is a bit worse than the one captured by Pixel 6 Pro’s telephoto len. Overall, our setup resembles the hardware available on commercial mobile phones. We set aZED 2 stereo camerawith a 12 cm baseline and a Livox Mid70 lidar together.
Revised: We set a stereo camera and a Lidar together to collect ground truth.

Original: It is pretty difficult to measure the depth of a small object by the lidar, especially when it is far and moving. Thus, we use the lidar to measure the depth of an anchor position and throw objects to the anchor position precisely. We use 24 objects including various masses, sizes, shapes, and six common materials: wood, metal, foam, rubber, plastic, and paper. They keep the falling motion or hand-driven motion to produce the impact sound.
Revised: We include details of data collection in the Appendix B.

Original:  We collectthe collision from 2 meters to 60 meters in different environments. After data cleaning and annotation, we get 3.6K+ raw audio-visual sequences.
Revised: AVD Dataset We collect 3.6K+ raw audio-visual sequences with a single collision event as the audio-visual depth(AVD) dataset.

Original: , including 280K+ frames. Each sequence has about 40 to 120 frames and a corresponding audio clip corresponding. We use the stereo camera to capture static images and use the lidar to capture static depth maps. AVD Dataset We augment the raw audio-visual sequences to be a large audio-visual depth(AVD) dataset.
Revised: The train/val/eval splits have 2600/500/522 sequences.

Original: We generate the sequence with multi-collisions by crop one moving object from a raw video sequence and insert it to another raw sequence with a random temporal location.
Revised: We augment the raw sequences by cropping one moving object from a raw video sequence and inserting it into another raw sequence with a random temporal location.

Original: Meanwhile, we add up the audio sequence with same temporal shift as the video.

Original: We have 10K audio-visual sequences.

Original: For the event-level localization stage, we segment an audio clip of 66.7ms including the impact sound and sample 20 frames including visible objects from each sequence and pair them as positive pairs.

Original: Negative samples pair the frame with the audio clip without impact sounds or with irrelevant impact sounds.

Original: Finally, we generate around 400K audio-visual pairs.

Original: Besides, we augment the raw depth with a maximum 3% random change to diverse the depth and shift audio samples accordingly to the video timestamp.
Revised: Besides, we augment the raw depth with a maximum 3% random change to diversify the depth and shift audio samples accordingly to the video timestamp.

Original: It can solve the problem of discrete anchor depths. The change cannot be significant because the impulse response of sound is also related to the depth. It requires more transformation than just shifting audio samples. We also augment images with low light, flip and rotation and audio with diverse background noise from WHAM!(Wichern et al., 2019).
Revised: More details are described in Appendix B.

Original: Baselines We include three types of baseline for comparison.
Revised: Baselines We include three types of baseline for comparison.

Original: We compare to a monocular depth estimation method NeWCRFs
Revised: We compare to a monocular depth estimation method NeWCRFs

Original: (Yuan et al., 2022), a state-of-the-art(SOTA) on multiple benchmarks.
Revised: (Yuan et al., 2022), a state-of-the-art(SOTA) on multiple benchmarks.

Original: We also compare to stereo matching methods including the ZED built-in ultra depth estimation SDK and a SOTA method LEAStereo (Cheng et al., 2020).
Revised: We also compare to stereo matching methods including the ZED built-in ultra depth estimation SDK and a SOTA method LEAStereo (Cheng et al., 2020).

Original: We use dense depth maps collected by the lidar to finetune these depth models if necessary.
Revised: We use dense depth maps collected by the Lidar to finetune the NeWCRFs and LEAStereo on images collected by the stereo camera.

Original: Despite optical flow based interpolation, we compare to interpolation using key points such as the trajectories of center or bounding boxes.
Revised: Despite optical flow based interpolation, we compare to interpolation using key points such as the trajectories of center or bounding boxes.

Original: Metrics We use the mean absolute depth errors as AbsErr = 1 n (cid:80) ni
Revised: Metrics We use the mean absolute depth errors as AbsErr = 1 n (cid:80) ni

Original: =1 | d − ˆ d | , root mean square absolute relative errors RMSE = (cid:113) 1 n (cid:80) ni
Revised: =1 | d − ˆ d | , root mean square absolute relative errors RMSE = (cid:113) 1 n (cid:80) ni

Original: =1 ( d − ˆ d ) 2 , AbsRel = 1 n (cid:80) ni
Revised: =1 ( d − ˆ d ) 2 , AbsRel = 1 n (cid:80) ni

Original: =1 | d − ˆ d | d as the end-to-end performance metrics.
Revised: =1 | d − ˆ d | d as the end-to-end performance metrics.

Original: FBDepth is a sparse depth estimation.
Revised: FBDepth is a sparse depth estimation.

Original: We evaluate the depth of each target object.
Revised: We evaluate the depth of each target object.

Original: However, monocular and stereo baselines have dense depth estimations for all pixels of the object.
Revised: However, monocular and stereo baselines have dense depth estimations for all pixels of the object.

Original: We evaluate the median estimation depth with the median depth of the ground truth dense map.
Revised: We evaluate the median estimation depth with the median depth of the ground truth dense map.

Original: We provide the results over different distance ranges as close( ≤ 10 m ), mid( 10 m - 30 m )  and far( ≥ 30 m ).
Revised: We provide the results over different distance ranges as close( ≤ 10 m ), mid( 10 m - 30 m ) [Table] and far( ≥ 30 m ).

Original: Intuitively, there is a upper bound for the temporal resolution so AbsRel at close depths performs worse than  further distance.
Revised: Intuitively, there is an upper bound for the temporal resolution so AbsRel at close depths performs worse than at further distances [Table]

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: [Table]
Revised: [Table]

Original: Table 1 shows the results on the depth estimation.
Revised: Table 1 shows the results on the depth estimation.

Original: In all, FBDepth can achieve better performance on all metrics than baselines across different FPS.
Revised: In all, FBDepth can achieve better performance on all metrics than baselines across different FPS.

Original: Several important trends can be observed.
Revised: Several important trends can be observed.

Original: Stereo matching methods perform extraordinarily on close objects, where more clear view difference can be captured.
Revised: Stereo matching methods perform extraordinarily on close objects, where more clear view difference can be captured.

Original: The AbsErr and RMSE increase dramatically as the targets become further because the limited baseline cannot resolve the view difference easily.
Revised: The AbsErr and RMSE increase dramatically as the targets become further because the limited baseline cannot resolve the view difference easily.

Original: In the other side, the AbsErr and
Revised: In the other side, the AbsErr and

Original: RMSE of FBDepth grows slowly with the increasing distance while its AbsRel decreases gradually.
Revised: RMSE of FBDepth grows slowly with the increasing distance while its AbsRel decreases gradually.

Original: Intuitively, there is a upper bound for the temporal resolution due to the limited FPS, the lack of the accurate timestamp and the small disturbance of audio-video software.
Revised: Intuitively, there is a upper bound for the temporal resolution due to the limited FPS, the lack of the accurate timestamp and the small disturbance of audio-video software.

Original: Thus FBDepth may not achieve the centimeter level easily.
Revised: Thus FBDepth may not achieve the centimeter level easily.

Original: A Further depth can break the assumption of stereo matching methods as well as monocular methods which has a fixed depth range of training data, but FBDepth still holds the physical propagation law in this condition.
Revised: A Further depth can break the assumption of stereo matching methods as well as monocular methods which has a fixed depth range of training data, but FBDepth still holds the physical propagation law in this condition.

Original: FBdepth also shows advantages on NeWCRFs.
Revised: FBdepth also shows advantages on NeWCRFs.

Original: The monocular methods rely on the training set, which includes various scenarios and depths.
Revised: The monocular methods rely on the training set, which includes various scenarios and depths.

Original: Although we apply camera remapping with intrinsic matrix and finetuning, NeWCRFs still cannot achieve the best performance as the one in the pretrained dataset.
Revised: Although we apply camera remapping with intrinsic matrix and finetuning, NeWCRFs still cannot achieve the best performance as the one in the pretrained dataset.

Original: The implicit depth regression has difficulty in domain adaption.
Revised: The implicit depth regression has difficulty in domain adaption.

Original: In the contrast, stereo methods can be directly applied to the new scenario and achieve awesome estimation because its fundamental is the explicit spatial view difference on stereo images.
Revised: In the contrast, stereo methods can be directly applied to the new scenario and achieve awesome estimation because its fundamental is the explicit spatial view difference on stereo images.

Original: FBDepth applies the explicit spatial measurement and does not reply on the camera and scenarios heavily.
Revised: FBDepth applies the explicit spatial measurement and does not reply on the camera and scenarios heavily.

Original: It requires several learning models but these model can be applied to common cameras and microphones.
Revised: It requires several learning models but these model can be applied to common cameras and microphones.

Original: FBDepth can be more general with a more diverse dataset.
Revised: FBDepth can be more general with a more diverse dataset.

Revised: We show some visual qualitative results in the

Revised: Appendix B.3.

Revised: Compared to other methods, the delay between audio and video can be visually recognized, which is similar to object detection.

Revised: In another word, FBDepth transforms the tough depth estimation problem to a simple interpretable problem.

Original: 5. A BLATION
Revised: 5. A BLATION

Original: In the ablation study, we show how each stage contributes to the final results.
Revised: In the ablation study, we show how each stage contributes to the final results.

Original: [Table]
Revised: [Table]

Original: Event-level localization We invest how the optical flow can help detect the collision event as well as contour the object mask.
Revised: Event-level localization We invest how the optical flow can help detect the collision event as well as contour the object mask.

Original: We define recall and precision as the percentage of correct recognized audio-visual events in all audio-visual events and all recognized events with an IoU more than 0.5, respectively.
Revised: We define recall and precision as the percentage of correct recognized audio-visual events in all audio-visual events and all recognized events with an IoU more than 0.5, respectively.

Original: With the flow, both recall and precision improves as the flow can work as a pre-mask to guide the network.
Revised: With the flow, both recall and precision improve as the flow can work as a pre-mask to guide the network.

Original: The main failures in recall come fromthe weak collision sounds or simultaneous collisions.
Revised: The main failures in recall come from weak collision sounds or simultaneous collisions.

Original: The incorrect recognition is mainly due to similar objects in the frame.
Revised: The incorrect recognition is mainly due to similar objects in the frame.

Original: [Figure]
Revised: [Figure]

Original: Frame-level localization Frame rate is most related to  frame-level stage.
Revised: Frame-level localization Frame rate is most related to the frame-level stage.

Original: We observe that increasing the frame rate reduces the numerical error of FBDepth in Table 1.
Revised: We observe that increasing the frame rate reduces the numerical error of FBDepth in Table 1.

Original: Especially, increasing 30 FPS to 60 FPS yields the largest improvement, and the benefit gradually tapers off with a further increase in the frame rate.
Revised: Especially, increasing 30 FPS to 60 FPS yields the largest improvement, and the benefit gradually tapers off with a further increase in the frame rate.

Original: We observe that 30 FPS is too slow to capture sudden movements and fast dynamics while 60 FPS is around the borderline.
Revised: We observe that 30 FPS is too slow to capture sudden movements and fast dynamics while 60 FPS is around the borderline.

Original: It is consistent with the trend to set 60 FPS as the default video recording and playing.
Revised: It is consistent with the trend to set 60 FPS as the default video recording and playing.

Original: The motion in 120FPS and 240 FPS is even slower sothat it is more difficult to distinguish the frame I e .
Revised: The motion in 120FPS and 240 FPS is even slower so it is more difficult to distinguish the frame I e .

Original: The frame error is no more than the one in the low FPS mode.
Revised: The frame error is no more than the one in the low FPS mode.

Original: Thus, 120 FPS and 240 FPS bring less improvement.
Revised: Thus, 120 FPS and 240 FPS bring less improvement.

Original: Ms-level localization We investigate our special interpolation in two perspectives.
Revised: Ms-level localization We investigate our special interpolation in two perspectives.

Original: First, we need to verify whether this method works.
Revised: First, we need to verify whether this method works.

Original: However, there is no ground truth timestamp so we cannot directly quantify the accuracy.
Revised: However, there is no ground truth timestamp so we cannot directly quantify the accuracy.

Original: We set the estimation of 240 FPS as a baseline and compare  with  estimation of lower FPS.
Revised: We set the estimation of 240 FPS as a baseline and compare it with the estimation of lower FPS.

Original: If it can get similar numerical results from independent input, which means the algorithm is reliable.
Revised: If it can get similar numerical results from independent input, which means the algorithm is reliable.

Original: In Figure ˜ refmicro2, the median temporal error for 30, 60, and120 FPS is 2.3ms, 0.65ms, and 0.5ms respectively.
Revised: In Figure ˜refmicro2, the median temporal error for 30, 60, and FPS is 2.3ms, 0.65ms, and 0.5ms respectively.

Original: Considering the frame resolution, we can compute the improvement ratio as frame duration temporal error .
Revised: Considering the frame resolution, we can compute the improvement ratio as frame duration temporal error .

Original: The 60 FPS has the largest 25x improvement over the frame duration.
Revised: The 60 FPS has the largest 25x improvement over the frame duration.

Original: This isa strong evidence that our ms-level localization is reasonable and robust.
Revised: This is strong evidence that our ms-level localization is reasonable and robust.

Original: Second, we compare the performance of depth estimation with different interpolation strategies in
Revised: Second, we compare the performance of depth estimation with different interpolation strategies in

Original: Table 2.
Revised: Table 2.

Original: We use the result from frame localization to predict the depth when there is no interpolation.
Revised: We use the result from frame localization to predict the depth when there is no interpolation.

Original: The error is large since this timestamp is ambiguous for the depth prediction.
Revised: The error is large since this timestamp is ambiguous for the depth prediction.

Original: Interpolation with the traces of centers or bounding boxes does not work well.
Revised: Interpolation with the traces of centers or bounding boxes does not work well.

Original: A few key points cannot capture the dynamics ina fine granularity.
Revised: A few key points cannot capture the dynamics in fine granularity.

Original: Depth regression Without the RGB-F channel of the target object in depth regression, the estimation will be less robust due to the ambient sound and the background noise as shown in the Table  
Revised: Depth regression Without the RGB-F channel of the target object in depth regression, the estimation will be less robust due to the ambient sound and the background noise as shown in the Table  L IMITATION AND F UTURE W ORKS

Revised: We classify audio-visual events into 3 categories by the quality and the quantity of visual cues during the sound production.

Revised: Obvious visual cues during sound production ( e.g. collision)

Revised: This is the main scenario we try to address in this paper.

Revised: It requires both visible procedure and audible sound to estimate the depth.

Original: C ONCLUSION
Revised: We can apply it to sports analytics, human stepping, etc. Moreover, it can collect the sparse depth point and accumulate depth points by time. According to existing work on depth completion (), adding some accurate depth points can boost the performance of monocular depth. Indirect visual cues during sound production ( e.g. speech, playing the piano)

Revised: This scenario is challenging but common every day.

Revised: They do not show the vibration visually.

Revised: Fortunately, they still leave lots of visual cues for us to learn.

Revised: Existing work on speech synthesis with lip motion(Ephrat &

Revised: Peleg, 2017), music generation with pose(Gan et al., 2020) indicates the strong semantic relationship between video and audio.

Revised: The spatial correlation still holds here.

Revised: We propose to apply a highresolution multi-frame alignment between the video and audio to find the accurate propagation delay.

Revised: No visual cues during sound production (e.g. car engines, mobile phone speaker)

Revised: We admit that we have no idea to estimate the depth when these sound source are static because we cannot see them at all.

Revised: Luckily, we still have chance when these sound sources move.

Revised: We propose a Doppler-like formulation to associate visual cues and audio cues.

Revised:  C ONCLUSION

Original: In this paper, we develop a novel depth estimation method based on the ”flash-to-bang” phenomenon.
Revised: In this paper, we develop a novel depth estimation method based on the ”Flash-to-Bang”.

Original: By aligning the video with the audio and detecting the events from both, we can estimate the depth in the wild without calibration or prior knowledge about the environment or target.
Revised: By aligning the video with the audio and detecting the events from both, we can estimate the depth in the wild without calibration or prior knowledge about the environment or target.

Original: Our extensive evaluation shows that our approach yields similar errors across varying distances.
Revised: Our extensive evaluation shows that our approach yields similar errors across varying distances.

Original: In comparison, the errors of several existing methods increase rapidly with distance.
Revised: In comparison, the errors of several existing methods increase rapidly with distance.

Original: Therefore, our method is particularly attractive for large distances.
Revised: Therefore, our method is particularly attractive for large distances.

Original: As part of our future work, we are interested in further enhancing the accuracy of our method, generalizing to more contexts, and using the estimated depth to the collision to estimate the depth to other objects in the scene. 
Revised: As part of our future work, we are interested in further enhancing the accuracy of our method, generalizing to more contexts, and using the estimated depth to the collision to estimate the depth to other objects in the scene. A B ACKGROUND OF D EPTH S ENSORS

Revised: We add more details on the performance of various depth sensors on multiple criteria in Table 3 and

Original: A A PPENDIX
Revised: Table 4. We especially demo the available depth sensors and corresponding APIs on iPhone Proin Table 5 as a typical example that depth estimation is well studied at the short range [Table] B D ATASET D ETAILS

Original: You may include other additional sections here.
Revised: We describe the details to build up the data collection pipeline for this novel task and discuss the trade-off during the data collection [Table] [Table] [Table]

Revised: B. P LATFORM AND C OLLISION O BJECTS

Revised: Figure 3 shows the data collection platform.

Revised: It includes three devices.

Revised: Lidar: We use a Livox Mid-70 Lidar(LIVOX, 2021) to collect the ground truth depth.

Revised: The detection range is 90 m @ 10% reflectivity.

Revised: The range precision is 2cm.

Revised: Although the point rate of Midis low, it has a special non-repetitive scan pattern so that the point cloud can be very dense by accumulation.

Revised: Thus, it is best to be used to collect the depth in the static scene.

Revised: Stereo Camera: We use a ZED 2i stereo camera(StereoLab, 2021) with a 12 cm baseline and a focal length of 4mm.

Revised: The large focal length is designed to increase the maximum effective range.

Revised: The image resolution is 1242 by 2208 pixels.

Revised: Table 3 shows detailed performance.

Revised: We use the ZED 2i camera as an important depth estimation baseline.

Revised: Video Recorder:

Revised: A pair of a camera and a microphone can play the basic functionalities of the video recorder.

Revised: However, it is very challenging to satisfy all the criteria for the audio-visual depth estimation.

Revised: In this experiment, we use an iPhone XR and record the video by the default Camera app.

Revised: It has several promising advantages.

Revised: First, we can record slow-motion 1080P videos withfps.

Revised: The frame duration is constant so that we can transform the frame number to the timestamp accurately and align it with the audio track which has a 48kHz sampling rate.

Revised: Second, the audiovisual recording delay T hardware is small as 1 ms and has a small variance within 1 ms on the iPhone.

Revised: Both specifications above are critical to the audio-visual depth but cannot be satisfied on other platforms such as Android phones.

Revised: The calibration of the audio-visual recording framework is out of the scope of this work.

Revised: It is unexpected that the calibration is pretty difficult based on our experience.

Revised: To capture the remote scene clearly, the telephoto lens has become indispensable in recent smartphones.

Revised: Samsung Ultra 22 can support 10x optical zoom and 100x hybrid zoom, and Pixel 6 pro has 20x zoom in all.

Revised: Their zoom performance is much superior to iPhone.

Revised: The iPhone XR is not equipped with a telephoto lens, so we mount an ARPBEST monocular telescope to enlarge the scene at a large distance.

Revised: As shown in Figure 4, the image quality of our setup is a bit worse than the one captured by Pixel 6 Pro’s telephoto lens.

Revised: Thus, our setup does not provide superior image quality compared to existing commercial camera modules on smartphones.

Revised: The image taken by Pixel Prois sharp but noisy while the one taken by iPhone XR with the telescope is a bit blurred.

Revised: Our setup does not take advantage of the external telescope from this perspective.

Revised: Overall, our setup resembles the hardware available on commercial mobile phones.

Revised: [Figure]

Revised: Collision Objects:

Revised: In Figure 8, We use 24 objects including various masses, sizes, shapes, and six common materials: wood, metal, foam, rubber, plastic, and paper.

Revised: These objects are ubiquitous every day.

Revised: Besides, they do not break down during the collision

Revised: B

Revised: [Figure]

Revised:  C OLLECTION M ETHODOLOGY

Revised: Sensor setup: We mount the Lidar, the stereo camera, and the iPhone on one slide.

Revised: We perform camera Lidar calibration between the left camera of the stereo camera and the Lidar according to

Revised: (Yuan et al., 2021).

Revised: We use the left camera to evaluate the monocular depth estimation and use the stereo camera to evaluate the stereo depth estimation.

Revised: The mobile phone changes the field of view to fit the object at different distances.

Revised: Hence, its intrinsic is not constant.

Revised: We use the frames recorded by iPhone only for FBDepth.

Revised: Collision setup: Since the point cloud is too sparse to measure the instant collision, we control the collision position to get the ground truth depth.

Revised: First, we select an anchor position and measure the depth from the slide to the anchor by the Lidar.

Revised: Second, we perform the collision at the anchor.

Revised: For example, we throw an object to collide with the anchor or strike a hammer into the anchor or step the shoes on the anchor.

Revised: Finally, the iPhone records the collision procedure.

Revised: Besides, the Lidar and the stereo camera record the object placed at the anchor.

Revised: They record the static object corresponding to the moving object in the video frames.

Revised: We set up various anchors from 2 meters to 60 meters in different environments.

Revised: Data Augmentation

Revised: After data cleaning and annotation, we get 3.6K+ raw audio-visual sequences, including 280K+ frames as te AVD dataset.

Revised: Each sequence has about 40 to 120 frames and a corresponding audio clip corresponding.

Revised: We use the stereo camera to capture static images and use the lidar to capture static depth maps.

Revised: We augment the raw audio-visual sequences to have more than a single collision by cropping one moving object from a raw video sequence and augmenting it to another raw sequence with a random temporal location.

Revised: Meanwhile, we add up the audio sequence with the same time shift as the video.

Revised: We have 10K audio-visual sequences.

Revised: For the event-level localization stage, we segment an audio clip of 66.7ms including the impact sound and sample 20 frames including visible objects from each sequence and pair them as positive pairs.

Revised: Negative samples pair the frame with the audio clip without impact sounds or with irrelevant impact sounds.

Revised: Finally, we generate around 400K audiovisual pairs.

Revised: Besides, we augment the raw depth with a maximum 3% random change to diversify the depth and shift audio samples accordingly to the video timestamp.

Revised: It can solve the problem of discrete anchor depths.

Revised: The change cannot be significant because the impulse response of sound is also related to depth.

Revised: It requires more transformation than just shifting audio samples.

Revised: We also augment images with low light, flip and rotation, and audio with diverse background noise from

Revised: WHAM!Wichern et al.

Revised: B. S AMPLES AND V ISUAL Q UALITATIVE R ESULTS

Revised: We provide some samples and visual qualitative results.

Revised: Considering the objects are small in the normal camera, we only show the region of interest in the RGB image and depth map.

Revised: The most intuitive observation is that our approach simplifies the difficult depth estimation problem to be easily estimated from the visual samples.

Revised: Humans can give a coarse estimation with the given timestamps, frames, and waveforms.

Revised: However, we can have no idea to know the depth from the

Revised: RGB image or stereo image visually.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]