Structure from Silence: Learning Scene Structure from Ambient Sound
[Figure]
[Figure]
Abstract: The sounds that a robot hears within a scene subtly vary as it movesthrough it.
In this paper, we ask whether these ambient sounds convey informationabout scene structure.
To study this, we collect a dataset of “in-the-wild” pairedaudio and RGB-D recordings in a variety of quiet, indoor scenes, typical of whata robot would encounter when performing navigation tasks.
Using this data, wetrain models that successfully estimate depth in a number of evaluation settings.
Finally, we show that these sounds provide a useful learning signal, and that wecan obtain a useful representation by associating a visual signal with sound.
Theseresults suggest that ambient sound conveys a surprising amount of informationabout scene structure, and that this information can be successfully exploited bylearning systems.
Introduction
Humans make extensive use of acoustic signals to help understand the physical structure of a scene.
Recent work has reproduced aspects of these capabilities in machines, typically by using distinctivesound landmarks, such as ringing phones [1].
However, since these methods have largely beenstudied in simulation and it is unclear how often these types of sound are available to robots inpractice.
Real-world scenes are often very quiet, and the sounds that robots hear within them — e.g.,ventilation noise, the hum of a machine — are ambiguous and confounded with reverberation andother artifacts.
Human hearing, by contrast, is capable of estimating scene structure through a variety of subtle cues,often from only quiet ambient sound.
For example, work in psychology has proposed that humanscan—perhaps surprisingly—use spatial variations in the ambient sound spectrum to tell whether theyare near a wall, and use “sound shadows” to detect obstructions [2].
Humans are also able to usereverberation to interpret scene geometry [3], and potentially exploit high-level regularities in scenecomposition, such as the typical locations of vents, doors, and windows.
Not only do these ambient sounds convey scene structure, but they can also provide a “free” learningsignal for other modalities.
Visual and audio signals often change in predictable ways, such as when a buzzing refrigerator simultaneously gets louder and visually enlarges as one moves toward it.
Amodel that can successfully associate vision and sound must understand the underlying 3D structurethat explains them both.
In this paper, we ask whether ambient sounds that occur in real-world scenes convey 3D structure.
To study this, we collect a large dataset of “in-the-wild” audio recordings from quiet, indoor scenestypical of what a robot would encounter when solving navigation tasks.
Each sound in our dataset ispaired with a corresponding recording from an RGB-D sensor, which provides a visual signal andpseudo ground-truth depth estimate.
The resulting dataset, which we call the Quiet Campus dataset(Fig.
1a), covers a variety of room shapes, background sounds, and materials.
Using this dataset, we ﬁrst conduct an experimental study for depth estimation from audio.
Weshow that audio can be used to estimate distance to nearby walls in a variety of scenarios, includingprediction of the relative depth between two recordings.
We demonstrate that the model can be usedas part of a very simple robotic navigation system, in which a wheeled robot moves along a wallusing ambient audio cues (Fig. 1b).
Next, we ask whether this ambient audio can be used as a learning signal, without explicit use ofdepth.
We study several multimodal self-supervised learning formulations, each of which requires amodel to associate subtle changes in sound with a visual signal (Fig. 1c).
We show that the resultingmodel learns a feature set that can be used to solve downstream distance-to-wall prediction tasks.
We make the following contributions: i) a dataset of paired audio and RGB-D data from real-worldscenes, ii) an experimental study of in-the-wild audio-based depth estimation, iii) we show that ambientsound, when paired with image data, can provide useful self-supervision for depth estimation tasks.
Our results suggest that ambient sound conveys a surprising amount of information about structure,and that it provides a useful training signal for multimodal learning.
Related Work
Human auditory perception.
Psychoacoustics researchers have explored what humans can inferfrom sounds, such as size, material, shape, and depth.
Perhaps the most familiar cue is echolocation ,which is often used by people with visual impairments to estimate the distance with objects viaself-produced sounds [4, 5].
However, humans also infer a great deal from passive observation ofexisting sounds.
Our work is inspired by Ashmead et al .
’s work in studying the navigation abilityof blind children in ambient sound ﬁeld [2, 6, 7].
They propose that blind people might use theaccumulations of low-frequency ambient sounds to distinguish obstacles, and use physical modelingto study the difference between audio near and far from walls.
Traer and McDermott [3] show thathumans can use phase change of natural reverberation to distinguish different spaces.
While thesecues have been studied extensively in psychoacoustics, they are not widely used in robotic perceptionand self-supervised learning.
Audio-based depth estimation.
Previous works have shown that audio contains rich spatial information.
Gan et al .
use stereo sound to perform cross-modal auditory localization for vehicles [8].
Yang et al .
learn spatial correspondence of sight and binaural audio [9].
Purushwalkam et al .
usean audio-visual signal to reconstruct ﬂoor plans.
To do this, they use a simulated environment inwhich distinctive sound effects (e.g. toilet ﬂush, alarm clock, dishwasher) from FreeSound [10]are synthetically inserted into the scenes via a physical simulation.
In contrast, we experimentallystudy audio-based depth estimation in real-world scenes, using ambient sounds (e.g. ventilationnoise).
Other work measures the reﬂections of self-produced echolocation sounds to provide aconsistent audio problem.
BatVision [11] predicts depth map in real scenes from echo responsesonly while VisualEchoes [12] learns visual representations by echolocation in a simulation called
SoundSpaces [13].
Instead of obtaining scene structure from active sensing, our method worksentirely via passive observation of sounds, and does not require special hardware.
Audio-visual learning.
Many recent works have proposed to use paired audio-visual data forrepresentation learning.
In seminal work, de Sa [14] proposed multimodal self-supervised learningas an alternative to single-modal models.
Later, Ngiam et al .
[15] learned a multimodal Boltzmannmachine.
Owens et al .
[16] learned visual representations for materials from impact sounds, and usedambient sound to learn visual features [17].
In the latter, the ambient sounds are taken from internetvideo and thus encompass a much wider range of events, whereas we only consider very quiet indoorscenes.
Later work learned multisensory representations [18, 19, 20, 21].
Other work has learnedcross-modal distillation [22], sound localization [23, 24, 25, 19, 26, 27, 28, 29, 30], active speaker
[Figure]
detection [31], sound source separation [32, 33, 34, 35].
We take inspiration from this work and showthat quiet, indoor sounds also provide a useful self-supervised learning signal.
We also propose newaudio-visual models that learn about relative depth in multiple modalities.
Visual depth prediction.
Recent works have tackled recovering 3D structure from RGB imagesthrough learning-based approaches and researchers have developed increasingly detailed single imagereconstructions [36, 37, 38, 39, 40, 41, 42], via voxel grids, point clouds, and meshes [43, 44, 45, 46].
At the same time, the signiﬁcant progress has been made in predicting the relative camera pose [47,48, 49, 50, 51].
We take inspiration from the techniques proposed in these models, though we predictdepth from audio instead of vision.
Robot navigation.
Robots often use visual streams to navigate to a destination in unseen environments [52, 53, 54, 55].
Recently, researchers leverage audio signals as useful cues for robot navigationtasks.
Those robotic systems are designed to localize the traceable sound sources and navigate toaudio goals in indoor environments [56, 57, 13, 58, 1, 59].
Unlike these methods, which use knownand clear sound sources, we take advantage of ambient sounds collected in real-world scenes.
The Quiet Campus Dataset
[Figure]
To study the structure-from-audio problem, we collected a dataset containing a largenumber of “in-the-wild” indoor ambient audio recordings, paired with concurrentlyrecorded RGB-D data.
We collected our data from a large number of classrooms andhallways on a university campus.
To ensure that the sounds within the recordingswere subtle ambient sounds (e.g. ventilation noise), rather than distinctive audioevents (e.g. speech), we collected the data at times when there were few peoplearound.
The recorded audio qualitatively sounds very quiet 1 .
Please refer to oursupplementary material for a random sample of sounds.
We collected two types of data: static recordings in which the microphone andcamera are stationary, and motion recordings where a human collector moves slowlytoward or away from a wall.
We show some visual examples from our collecteddataset in Fig.
In our experiments, we split the data so that there is no physicaloverlap, such that the rooms in the training and tests sets are disjoint.
Static recordings.
We collect data in 46 classrooms in 12 buildings, containingapproximately 200 minutes audio.
For each classroom, we select 16 − 30 positionsand record 10 secs.
of audio
at each one.
Since we are interested in predicting thedistance to the wall, we point the camera and microphone toward the nearest wall when recording, sothat the distance to the wall is well-deﬁned.
Static recordings with dense coverage.
In order to evaluate generalization to different orientations,we use four classrooms from Static recordings whose size ranges from 20 to 35 cells total, and collect10 hours of audio data densely with a wide range of angles (rather than always facing a wall).
We callthis these Static-dense recordings.
These two datasets were recorded at different times of the yearand with different usage patterns.
The Static recordings were made in spring 2021, when buildingswere at reduced occupancy due to the COVID-19 pandemic, and when air conditioners may havebeen in use, while Static-dense were recorded in the winter of 2020, when buildings were at normaloccupancy and when heating may have been in use.
Motion recordings.
To help understand whether small motion (and the subtle changes in audio overspace) provides useful information about structure, we collect approximately 90 minutes of videos inmotion (222 videos total).
To ensure that there are no unintended obstructions nearby, we collectthis data in hallways.
During recording, the microphone and RGB-D camera move toward or awayfrom a wall (Fig. 2).
As in the static setting, we point the microphone toward the wall.
To reducepossible sources of sound, the human operator moves slowly and does not take footsteps.
We split
VisionNet VisionNet
[Figure]
AudioNet AudioNet
[Figure]
Objective
[Figure] [Figure]
this dataset based on building , i.e ., the videos in the training and test set are hallways recorded indifferent buildings.
Estimating wall distance.
For experiments that require distance to the wall, we obtain this bycropping the center 320 × 240 region from the depth image, and averaging the depth values.
Sincethe camera is oriented toward the wall, this is generally a simple and accurate approximation.
Hardware.
We use a ZOOM H1n Handy Recorder for collecting audio since it is a simple, inexpensive, and widely-used stereo audio recorder.
For the RGB-D data, we use Intel Real-Sense Depth
Camera D415 (Fig. 3).
Predicting Structure from Ambient Sound
We design models and experiments that will experimentally evaluate the ability of a model to learn topredict depth from audio in natural scenes.
4.1 Depth estimation tasks
Inspired by work in human obstacle detection with sound [5], we study the ability of a machinehearing system to predict the distance to a wall ahead of them.
We pose this problem in multipleways, which has applications in various robotic navigation scenarios.
Obstacle detection.
First, we pose the problem of estimating whether the microphone is within asmall distance of a wall (we use 0.5 meters).
Relative depth order.
In many cases, a robotic system needs only to move toward or away from anobstacle, rather than to know its actual distance.
We also hypothesize that relative depth estimationmay be an easier task than absolute estimation, as it is in visual depth perception [60, 36].
Given twoaudio clips, we train a network to predict which one is closer to a wall.
Relative depth estimation.
Given two audio clips, we train a network to predict the differencebetween their distance to the wall.
We also consider predicting the log of their difference ( i.e ., theratio), following the work in visual depth estimation, which ﬁnds this easier to predict [38, 37], andas a multi-way classiﬁcation task, after clustering depth differences in the training set with k -means.
Absolute depth estimation.
In the end, we train a model to regress the distance to a wall, rather thana thresholded “near-or-far” prediction.
4.2 Self-supervised audio-visual learning
Having evaluated our model’s ability to estimate depth directly from audio, we now ask whether wecan learn about structure through audio-visual self-supervision.
We hypothesize that, through the taskof associating a raw (RGB) visual signal with audio, we will learn a representation with features thatwill be helpful for downstream depth estimation tasks.
One simple model, considered in prior work, is to solve a audio-visual synchronization task [31, 19,27]:
given the video, distinguish whether audio and visual stream are misaligned by randomly shiftingaudio.
We call this model the AV-Sync model.
Finally, we pose a new audio-visual learning problemthat requires a model to learn relative depth between examples.
We train a model to distinguishbetween ordered audio-visual pairs, which we call the AV-Order model.
Given a pair of audio tracksand a pair of images (corresponding to the two audio tracks), we train a network to determine whetherthe two pairs are in the same (or different) order, which we frame as a classiﬁcation problem.
4.3 Audio-based robotic navigation
To demonstrate the effectiveness of our depth estimation model, we use it to guide a wheeled robot(TurtleBot) to follow a wall.
The robot uses the obstacle detection model to predict whether there iswall near the left/right of the robot, and moves accordingly.
We use a simple policy, based on that of
Gandhi et al .
[61] (please
It senses the audio on its left side, then rotates 180 ◦ and senses audio to its right.
We run the obstacledetection model on both audio recordings, and choose an orientation such that the robot moves away
[Figure]
from the wall that has higher probability of being near a wall.
[Figure]
Network backbone.
We represent audio using the VGGish network [62].
For our experiments,we either replace the ﬁnal layer with a new regression or classiﬁcation head, or we use the (128dimensional) penultimate layer.
The network takes 0.96 sec of audio as input, in the form of a log-melspectrogram.
Due to the importance of low-frequency sound in physical models of wall-soundinteraction [7], we do not remove the very lowest frequencies of the audio, in contrast to commonpractice in speech and audio event recognition [62].
We analyze the importance of this decisionthrough experiments.
We convert the stereo audio recordings to mono by averaging the two channels.
For models that use visual data, we use ResNet-18 [63], with 224 × 224 images, and modify the lastlinear layer to output the feature vectors with the same dimension as audio features.
In addition, wealso ﬁnetune the weights from ImageNet [64] to see the upper bound performances.
Learning details.
For fully supervised tasks, we use SGD [65] optimizer with a learning rate 10 − 3 ,momentum 0 .
9 , and weight decay 5 × 10 − 4 .
For the self-supervised learning experiments, we use
Adam [66] with a learning rate of 10 − 4 for representation learning.
For linear probing experiments,we use SGD with a learning rate of 10 − 2 .
We schedule learning rates via cosine decay and pick thebest model from validation for all the training jobs.
Audio processing.
Our audio preprocessing resembles that of [62].
We resample the audio to 16 kHzand extract segments of 0.96 secs.
We then convert them to spectrograms by applying a Short-time
Fourier transform at a stride of 10 ms and 25 ms, and convert them into 96 × 64 log-mel spectrograms.
For augmentation, we randomly crop the waveform in time and multiply the amplitudes by a randomfactor uniformly sampled from [0.5, 1.5].
Depth estimation networks.
For the regression and binary classiﬁcation tasks, we add a smallmulti-layer perceptron on top of the VGGish network.
For the relative estimation tasks, our networkaims to identify objectives with two audio inputs.
We approach this by building a Siamese networkthat fuses the extracted feature vectors via concatenation prior to the multi-layer perceptron.
Forregression, we normalize depths to be in the range [ − 0 .
5] during training.
Network for self-supervision tasks.
For AV-Sync task, we concatenate image and audio featuresbefore passing through the multi-layer perceptron.
For the AV-Order model, we fuse Siamesenetworks from both modalities via concatenation and add a multi-layer perceptron (Fig. 4).
In thedownstream task, we take the output features from the last convolution layer after global averagepooling and freeze them.
We pass our 512-d learned features to a linear layer to get the objectivesand only optimize it during training.
Experiments
Having collected a dataset of in-the-wild audio and models for inferring structure from it, we conductan experimental study of what can be inferred from ambient sound in real-world scenes.
We reportour results along with 95% conﬁdence intervals.
An experimental
study of
audio-
based depth
estimation
Obstacle detection.
First, we evaluate our model’s ability to solve the obstacle detection problem(Tab. 1).
We split the data such that the test rooms were not heard during training.
We report accuracyon this binary classiﬁcation task, as well as average precision (to account for possible dataset shiftfrom using held-out rooms).
We evaluated our model in both the static and motion settings.
To helpinterpret the effectiveness of audio in the context of other modalities, we also included simple visualmodels, with both random initialization and ImageNet pretraining 2 .
Interestingly, we see that
the
[Table]
audio-based model performs better than chance in both cases, suggesting that ambient sound, indeed,conveys information about scene structure.
We also evaluated our model’s ability to generalize across orientations and recording times.
We foundthat a model trained on Static recordings could achieve 86.1% AP and 72.9% Acc on Static-denserecordings, indicating strong generalization performance.
We qualitatively demonstrate the obstacle detection results by training a version of the model on the
Static-dense data, since it contains a large number of viewpoint variations and angles (see supplementfor quantitative results).
We apply the learned obstacle detection model to every frame of a video inwhich a person walks through a room (Fig. 5).
In Fig.
6, we also apply this model densely within aroom divided according to its spatial grid.
We averaged the probability of the model for each gridcell.
We see that it successfully distinguishes between the boundaries of the room and the center.
[Table]
Relative depth order.
In Tab.
1,we evaluate our model’s ability to predict which of two audio clips is closer to a wall.
Once again, we ﬁnd that themodel obtains better results thanchance (see supplementary formore results).
We also see that,as is the case with visual prediction tasks, the relative predictiontask is easier than predicting obstacles.
[Table]
Relative depth ratio.
Since thenetwork is able to tell relativedepth order well, we explore relative depth setup further by predicting depth difference of audio pairs (log depth ratio).
Weuse regression to predict the logratio from ambient sounds with
L1 Loss function and via classiﬁcation.
We report the performance of mean absolute error,median error, and R 2 .
For classiﬁcation, we report top-1 and top-5 accuracy, and report average distance from the ground truthbin using the distance between centers of paired bins.
We report the performance of zeroing outtwo inputs as a baseline.
As shown in Tab. 2, we can see that our method outperforms chance and,interestingly, has a relatively small gap between the vision-based approach.
Absolute depth estimation.
We evaluated our model’s ability to predict depth from a single audioexample, using similar experiments as the relative depth ratio (Tab. 3).
Our model is more accuratethan chance (which we obtain for all metrics by simply ablating the input and training).
Likewise, theregression-by-classiﬁcation model improves over chance performance.
[Figure]
[Table]
We also consider another CNN model inspired by the physical model of Ashmead et al
[Table] [6] and ourrelative depth model.
This new model estimates depth for a given audio input after conditioning on areference sound recorded within the same scene and its depth (see the supplementary material fordetails).
We call the conditional absolute depth model.
This model is better than our model withonly a single audio input
[Table]
suggesting that the information from reference audio is a useful cue.
5.2 Audio-visual representation learning
[Table]
Finally, we investigate whether welearn useful representation with AVOrder and AV-Sync models.
In the
AV-Order binary classiﬁcation pretexttask, our model obtains 85.8% APand 75.8% accuracy (80.3% AP and70.
% accuracy with random initialization) on the test set where the random chance is 50%.
Given its performance on the pretext task, we testwhether the learned audio and visualrepresentations about depth are usefulfor the downstream tasks: (1) obstacledetection and (2) relative depth estimation.
To evaluate the quality of ourspatial representation, we freeze thelearned features and train a linear classiﬁer on the depth estimation tasks (Section 4.1).
In the downstream tasks, we validate image andaudio features individually and compare them with random features, pre-trained ImageNet features [64, 63], pre-trained AudioSet features [62] and AV-Sync features.
Since AudioSet pretrainedweights require a special audio pre-processing that differs from ours, we provide another randomfeature baseline based on the same processing as [62].
Relative depth.
We report results for relative depth order and relative depth ratio in Tab. 4 and Tab.
5respectively.
Our method outperforms baselines in both audio and visual modalities suggesting thatwe learn useful depth representations.
Moreover, our results are often close to supervised features.
Obstacle detection.
As the results shown in Tab. 4, AV-Order model outperforms Scratch and
AudioSet pretraining, while it loses to AV-Sync model.
It is reasonable that AV-Order model getworse performance since AV-Order model deals with two inputs during the proxy task.
We providean additional experiment by ﬁne-tuning our learned AV-Order feature and we compare with the fullysupervised method in Tab. 1.
This obtains a 4% AP boost (70.0% v.s. 65.6%), suggesting that ourlearned representation is helpful.
Furthermore, the models with both audio and visual features as inputs perform consistently betterthan visual features solely, suggesting the audio is useful in addition to the visual signal.
5.3 Audio cues
[Figure]
We ask what audio information that the network uses to solve its task.
We take inspiration from physical models of audio spectra nearwalls [6], which suggest that the distribution oflow-frequency sounds (20-200Hz) are an important cue.
We zero out different frequency rangesin the mel spectrogram, then re-train the relativedepth order model.
As can be seen in Fig.
8, theaccuracy of the model with only very low frequencies (0-250Hz) is comparable to the modelwith the full frequency range, suggesting thatthese frequencies are indeed very important.
Forcomparison, we conducted the same experimentwith instrument categories from the VGGSound dataset [67] (see supp.
for details).
Interestingly,there these low frequencies are less important, suggesting that our setting relies on different cues.
5.4 Robotic navigation with ambient sound
[Figure]
We ask whether our depth estimation model can be used toguide a robot to follow a wall, solely using ambient sound.
For this, we used a model trained on Static recordings,and evaluate on a classroom that was not included in thetraining set.
The track is 8m long, and forces the robotto turn a corner (Fig. 7).
The robot starts navigating fromone of several unknown locations (40cm, 80cm, or 120cmfrom the wall) and orientations ( 30 ◦ left or right, or facingforward).
We repeat the experiment 18 times and measurethe average distance along the track it attained beforecolliding with a boundary.
Quantitative results are shownin Fig.
We compare with a random policy, a “straight line” policy that always drives forward, anda policy based on the vision model.
For the random policy, the setting is the same as our method,except we use a randomly initialized network for obstacle prediction.
The straight line policy iseffective when the robot starts with an orientation facing front, and it has the advantage that it doesnot rotate to sense audio, and hence avoids accumulating errors 3 .
It’s not surprising that the visionmodel outperforms our audio model, since we have known that vision performs well for those kind oftasks.
We see that our method performs better than random and straight line policy, and successfullyguides the robot pass the corner in nearly half of the trials.
Conclusion
In this paper, we have shown that real-world, indoor ambient sound provides a great deal of information about structure.
We contributed the Quiet Campus dataset, a large dataset of in-the-wild pairedaudio and RGB-D recordings.
We showed that distance to a wall can be predicted from ambientaudio in a number of settings.
We also showed that these ambient sounds can be used for multimodalself-supervised learning, including with a novel formulation that captures relative depth.
We see this work as opening two directions.
The ﬁrst is in creating robotic systems that exploit realworld, passively obtained audio signals.
While related systems have previously been demonstratedin simulation with distinctive audio events (or with active echolocation), we take a step towardinterpreting in-the-wild ambient sound.
Second, we see our work as a step toward using the “free”audio signals to help train other modalities, such as vision, about scene structure.
Limitations.
While ambient sound provides information about scene structure, it is a weakersignal when compared to vision and potentially self-produced sound, and we do not yet have a fullunderstanding of what can (and cannot) be perceived from it, e.g. whether there are fundamentallimitations on resolution, material, or angle.
Additionally, while we will release our pretrained modelfor others to use, our experiments were conducted on a single college campus, and hence may not berepresentative of all indoor spaces. A Dataset Examples
We show video samples of Static recordings ( static_near.mp4 and static_far.mp4 ) andvideo samples of Motion recordings ( motion.mp4 ) along with audio.
B Additional robot navigation experiments
We run more robot navigation experiments to further verify the generalization ability to learn fromambient sound.
For these, we use a robot equipped with two microphones, oriented in oppositedirections, simply to avoid the need for the robot to rotate to sense both sides of the scene (the modelis otherwise unchanged).
Experiment setting.
We evaluate our model in 3 additional rooms, different from the one in themain paper.
These 4 rooms are located in 4 individual buildings and are not included in the trainingset.
Their ﬂoor plans and track designs are shown in Fig.
Similarly, the robot starts from oneof several unknown locations (40cm, 80cm, or 120cm from the wall) and orientations ( 30 ◦ left orright, or facing forward).
For each room, we repeat the experiment 9 times and measure the longestdistance along the track it attained before colliding with a boundary.
For the 'L'-shaped track in Room#2, we determine the distance by projecting the ﬁnal position to the center line of the track.
Algorithm.
The detailed algorithm to control the robot is shown in Algorithm 1.
When computingthe near-wall probability P ( L ) and P ( R ) , we average the predictions of 20 clips of 3s audio tomake the prediction more accurate and robust.
When running the experiments in different rooms,we manually adjust the threshold p , which essentially controls the expected distance to the wall, toaccommodate the different environments.
[Figure]
Experimental results.
We comparewith the same “straight line” policy asin the main paper.
We measure the performance by computing the percentage of the completed track, and takethe average of all trials, as well as theaverage of the trials starting with thesame orientation, as shown in Fig.
It can be seen that the “straight line”policy performs well when reachingthe goal does not require turning, asexpected.
However, it performs signiﬁcantly worse than our model whenthe robot starts at an angle that doesnot follow the wall.
In contrast, ourmodel’s performance is more consistent with all three starting orientations,indicating that the model successfullyguides the robot to navigate along thewall.
When looking at the average performance with different starting orientation on individualrooms, we also ﬁnd that the “straight line” policy only outperforms our method when starting facingfront on straight tracks, and performs worse than ours with any other starting orientation or on the'L'-shaped track in Room #2.
Vision model.
While we suspect that vision-based navigation is very effective (and is capable ofsigniﬁcantly outperforming audio-based navigation), we found that a vision-based model, whentrained solely on our paired RGB-D data, did not generalize well to out-of-distribution viewpoints.
Therefore, it performed very poorly (worse than the audio-based model) when deployed on a robot.
To make the vision baseline work, we collect extra data in the similar scenes with more diverseviewpoints and ﬁnetune the model to apply to the robotic system.
Algorithm for vision model.
Additional, we provided the detailed algorithm for the vision model inour paper, which is shown in Algorithm 2.
To keep the control algorithm as close as possible to thecontroller for audio model, we made a small modiﬁcation to Algorithm 1 that the turning angle is
[Figure]
based on the prediction of near-wall side.
The reason for this is to avoid the noise from the predictionof far-wall side, considering the threshold p is usually very high (we set it as 0.97 in the experiment).
Video demo.
We provided demo video ( robot_demo.mp4 ) in the supplementary.
Please checkthe video for qualitative results of robot navigation.
C Obstacle detection on Static-dense recordings
[Table]
We evaluate the obstacle detection task for Static-denserecordings by training a model on 3 rooms and testingon an unseen room with the same method as the mainpaper.
Furthermore, to evaluate whether the success ratefor wall perception increases when participants have selfmotion [6], we simulate the motion of the recorder byconcatenating two audio with the same angle from adjacentgrids and predict whether the agent is moving towards orfar away from the obstacles.
We report the performance ofthe model ( Ours-Static ) trained on Static recordings forcomparison as well.
As shown in Table 6, the average precision above 90% in Static-dense recordings with both settings can claim the performance is extremely good.
Comparing experimental results of with orwithout motion, we can ﬁnd the network predicts better with motion involved which denotes motion of the agent improves the performance in the obstacle detection task.
We also evaluate our
Static-dense model on Static recordings
and it could achieve 62.8% AP and 60.9% Acc, closeto the Static model.
Obstacle detection video demo of Fig.
[5] is available in the supplementary ( obstacle_detect_video_demo.mp4 ).
D Indoor localization on Static-dense recordings
We are also interested in if ambient sounds can tell us the actual location in the rooms, therefore, weconduct indoor localization experiment for Static-dense recordings.
We use the same network as the obstacle detection task and replace the last linear layer with a N -class
FC layer, where N is the number of grids in the room.
We formulate it as a multi-way classiﬁcationtask and predict which grid the given audio comes from.
We train the model with the cross entropyloss and evaluate it with top-1, top-5 and average distance ( i.e ., the Euclidean distance distancebetween our predicted grid and the ground truth grid).
Grid classiﬁcation.
We ask to what extent ambient sound can allow a model to predict which gridcell within a room the audio was recorded in.
For the train/test split, we randomly select audiosamples with three angles in each grid cell as training, then test on sound with the remaining angle.
The experimental results are shown in Table 7.
It can be seen that the performance is signiﬁcantlyhigher the random guessing baselines.
Interestingly, we see that our model does not generalize wellto other rooms.
We believe that this is because there is a great deal of variation within a room.
Whilecertain aspects of the sound are predictable, such as distance to a wall, this is not true in general
[Table]
Generalization over time.
Next, we evaluate our model’s ability to predict the audio samplesrecorded from a different time to see how well the model could generalize over time.
In thisexperiment, training and test sets are from the same room but different times.
The time intervals for
Room #4, #5 and #6 are 1 week, 1 week, and 1 hour, respectively.
We show the experimental results in Table 7.
It can be seen that the performances drops compared tothe non time-shifted experiments, conﬁrming the assumption that the network does cheat using globaltime changing of audio.
Yet the performances are still signiﬁcantly better than random guessing,even recording the audio after one week later.
Overall, this experiment shows that the ambient soundconveys information of location, but that the sounds that one hears in a room may change over time.
E Relative depth order on Motion recordings
We design experiments to investigate the accuracy of relative depth estimation as the magnitude ofthe depth difference changes.
We paired audio in the test set using 5 different distance ranges.
We teston those sampled data without re-training and compare audio-based and vision-based methods (withrandom initialization).
The results are shown in Figure 12.
We see that AP and accuracy increase asthe relative depth increases, and the models are more conﬁdent when the distances are at least 0.2meters.
Along with distance increasing, the gap between vision-based and audio-based method arereduced.
F Generalization between Motion and Static recordings
[Table]
[Table]
We give the results of the model trained on Motion recordings (recorded in hallways) and tested on Static recording(recorded in the classrooms).
The experiment results areshown in Table 8 .
Our model can still perform well aboverandom chance despite evaluating on a different domain.
G Effects of Non-ambient sounds
To investigate how non-ambient sounds from environment could affect our prediction, we augment
Static recordings with various clear sounds from FreeSound Dataset [68] to simulate other soundsource in the scenes and re-perform the obstacle detection experiment.
We test our models withoutor with re-training on the augmented data and the results are shown in Fig.
We can see that theperformance drops as the volume of distracting sound increases while it still outperforms chance andre-training the model with augmentation makes our model more robust.
[Figure]
[Figure]
Latest
[Figure]
MLP outputs
[Figure]
Latest
[Figure]
share weights
[Figure]
ResNet
ResNet
[Figure]
concatenate
MLP
[Figure]
share weights concatenate
MLP logits
[Figure]
[Figure]
H Conditional absolute depth estimation
Implementation.
We use a Siamese network from relative tasks to build conditional absolutedepth ( CAD ) model.
With three inputs: audio s 1 , reference audio s 2 and ground-truth depth x 2 , weﬁrst extract audio features with VGGish backbone and calculate the depth embedding following [69].
We add up the depth embedding and reference audio feature and fuse features via concatenation priorto the multi-layer perceptron ( i.e ., FC-ReLU-FC-ReLU-FC layers).
During the training, the referencesounds are randomly selected from the same scene while those are ﬁxed for each sample in the testset (Fig. 14).
I Self-supervised audio-visual learning
Implementation.
We approach AV-Order model by building two Siamese networks for both audioand visual modalities.
As shown in Fig.
15, for each modality, we ﬁrst extract features from inputsand concatenate them prior to the multi-layer perceptron.
Then we concatenate two 128-d featurevectors from image and audio branches, and forward it to MLP ( i.e ., FC-ReLU-FC layers, all the
MLP blocks are the same) for the ﬁnal logits prediction.
AudioSet feature.
The low performance of the AudioSet features in Table 4, 5 could be due todifferences in the types of sounds considered, as well as the preprocessing in each method.
Thepreprocessing procedure used in the AudioSet model (which is standard on the dataset) throws awayall but the 125Hz-7500Hz frequency range, which our analysis (Section 5.3) suggests is important.
To test this, we trained a random feature baseline with (and without) this preprocessing, and see alarge difference in their performances (Table 4, 5).
[Figure]
Few-shot learning.
To help understand how audiovisual self-supervision improves ﬁne-tuned models,we measure the performance using various numbersof labeled examples as training set (5%, 10%, 20%,40%, 100%).
The experiment results are shown in 16.
We can see a large improvement from self-supervisedinitialization when there is more unlabeled data thanlabeled data, especially in few-shot training regimes.
As expected, when the number of labeled examplesequals the number of unlabeled examples in our experiment, the trained-from-scratch model catches upsince the two datasets are exactly the same and the labels provide strictly more information than the audio.
J VGGSound-Instrument dataset
To perform the frequency analysis experiment, we sampled 37 video classes of musical instrumentswith 32k video clips of 10s length from VGGSound dataset, and we call this subset VGGSoundInstruments.
Video classes are listed as below:playing accordion playing acoustic guitar playing banjo playing bass drum playing bass guitar playing bongo playing cello playing clarinet playing congas playing cornet playing cymbal playing djembe playing double bass playing drum kit playing electric guitar playing electronic organ playing erhu playing ﬂute playing glockenspiel playing guiro playing hammond organ playing harp playing harpsichord playing mandolin playing marimba, xylophone playing piano playing saxophone playing sitar playing snare drum playing steel guitar, slide guitar playing tabla playing timbales playing trumpet playing ukulele playing vibraphone playing violin, ﬁddle playing zither
Require: threshold p , α ; step length v 1: while No crash do 2:
Input: left and right side sound 3:
P ( L ) =
Network(left sound) 4:
P ( R )
=
Network(right sound) 5:
Angle ∝ | 0 .
5( P ( L ) + P ( R ))
− p | 6: if | P ( L ) − P ( R ) | <
α then 7:
Go forward 8: else 9: if 0 .
( P ( L ) + P ( R )) > p then 10: # Robot might be near to the wall 11: if P(L) > P(R) then 12:
Turn right and go forward 13: else 14:
Turn left and go forward 15: end if 16: else 17: # Robot might be far from the wall 18: if P(L) > P(R) then 19:
Turn left and go forward 20: else 21:
Turn right and go forward 22: end if 23: end if 24: end if 25: end while using
RGB images