Structure from Silence: Learning Scene Structure from Ambient Sound
[Figure]
[Figure]
Abstract: The sounds that a robot hears within a scene subtly vary as it movesthrough it.
In this paper, we ask whether these ambient sounds convey informationabout scene structure.
To study this, we collect a dataset of “in-the-wild” pairedaudio and RGB-D recordings in a variety of quiet, indoor scenes, typical of whata robot would encounter when performing navigation tasks.
Using this data, wetrain models that successfully estimate depth in a number of evaluation settings.
Finally, we show that these sounds provide a useful learning signal, and that wecan obtain a useful representation by associating a visual signal with sound.
Theseresults suggest that ambient sound conveys a surprising amount of informationabout scene structure, and that this information can be successfully exploited bylearning systems.
Introduction
Humans make extensive use of acoustic signals to help understand the physical structure of a scene.
Recent work has reproduced aspects of these capabilities in machines, typically by using distinctivesound landmarks, such as ringing phones [1].
However, since these methods have largely beenstudied in simulation and it is unclear how often these types of sound are available to robots inpractice.
Real-world scenes are often very quiet, and the sounds that robots hear within them — e.g.,ventilation noise, the hum of a machine — are ambiguous and confounded with reverberation andother artifacts.
Human hearing, by contrast, is capable of estimating scene structure through a variety of subtle cues,often from only quiet ambient sound.
For example, work in psychology has proposed that humanscan—perhaps surprisingly—use spatial variations in the ambient sound spectrum to tell whether theyare near a wall, and use “sound shadows” to detect obstructions [2].
Humans are also able to usereverberation to interpret scene geometry [3], and potentially exploit high-level regularities in scenecomposition, such as the typical locations of vents, doors, and windows.
Not only do these ambient sounds convey scene structure, but they can also provide a “free” learningsignal for other modalities.
Visual and audio signals often change in predictable ways, such as when a buzzing refrigerator simultaneously gets louder and visually enlarges as one moves toward it.
Amodel that can successfully associate vision and sound must understand the underlying 3D structurethat explains them both.
In this paper, we ask whether ambient sounds that occur in real-world scenes convey 3D structure.
To study this, we collect a large dataset of “in-the-wild” audio recordings from quiet, indoor scenestypical of what a robot would encounter when solving navigation tasks.
Each sound in our dataset ispaired with a corresponding recording from an RGB-D sensor, which provides a visual signal andpseudo ground-truth depth estimate.
The resulting dataset, which we call the Quiet Campus dataset(Fig.
1a), covers a variety of room shapes, background sounds, and materials.
Using this dataset, we ﬁrst conduct an experimental study for depth estimation from audio.
Weshow that audio can be used to estimate distance to nearby walls in a variety of scenarios, includingprediction of the relative depth between two recordings.
We demonstrate that the model can be usedas part of a very simple robotic navigation system, in which a wheeled robot moves along a wallusing ambient audio cues (Fig. 1b).
Next, we ask whether this ambient audio can be used as a learning signal, without explicit use ofdepth.
We study several multimodal self-supervised learning formulations, each of which requires amodel to associate subtle changes in sound with a visual signal (Fig. 1c).
We show that the resultingmodel learns a feature set that can be used to solve downstream distance-to-wall prediction tasks.
We make the following contributions: i) a dataset of paired audio and RGB-D data from real-worldscenes, ii) an experimental study of in-the-wild audio-based depth estimation, iii) we show that ambientsound, when paired with image data, can provide useful self-supervision for depth estimation tasks.
Our results suggest that ambient sound conveys a surprising amount of information about structure,and that it provides a useful training signal for multimodal learning.
Related Work
Human auditory perception.
Psychoacoustics researchers have explored what humans can inferfrom sounds, such as size, material, shape, and depth.
Perhaps the most familiar cue is echolocation ,which is often used by people with visual impairments to estimate the distance with objects viaself-produced sounds [4, 5].
However, humans also infer a great deal from passive observation ofexisting sounds.
Our work is inspired by Ashmead et al .
’s work in studying the navigation abilityof blind children in ambient sound ﬁeld [2, 6, 7].
They propose that blind people might use theaccumulations of low-frequency ambient sounds to distinguish obstacles, and use physical modelingto study the difference between audio near and far from walls.
Traer and McDermott [3] show thathumans can use phase change of natural reverberation to distinguish different spaces.
While thesecues have been studied extensively in psychoacoustics, they are not widely used in robotic perceptionand self-supervised learning.
Audio-based depth estimation.
Previous works have shown that audio contains rich spatial information.
Gan et al .
use stereo sound to perform cross-modal auditory localization for vehicles [8].
Yang et al .
learn spatial correspondence of sight and binaural audio [9].
Purushwalkam et al .
usean audio-visual signal to reconstruct ﬂoor plans.
To do this, they use a simulated environment inwhich distinctive sound effects (e.g. toilet ﬂush, alarm clock, dishwasher) from FreeSound [10]are synthetically inserted into the scenes via a physical simulation.
In contrast, we experimentallystudy audio-based depth estimation in real-world scenes, using ambient sounds (e.g. ventilationnoise).
Other work measures the reﬂections of self-produced echolocation sounds to provide aconsistent audio problem.
BatVision [11] predicts depth map in real scenes from echo responsesonly while VisualEchoes [12] learns visual representations by echolocation in a simulation called
SoundSpaces [13].
Instead of obtaining scene structure from active sensing, our method worksentirely via passive observation of sounds, and does not require special hardware.
Audio-visual learning.
Many recent works have proposed to use paired audio-visual data forrepresentation learning.
In seminal work, de Sa [14] proposed multimodal self-supervised learningas an alternative to single-modal models.
Later, Ngiam et al .
[15] learned a multimodal Boltzmannmachine.
Owens et al .
[16] learned visual representations for materials from impact sounds, and usedambient sound to learn visual features [17].
In the latter, the ambient sounds are taken from internetvideo and thus encompass a much wider range of events, whereas we only consider very quiet indoorscenes.
Later work learned multisensory representations [18, 19, 20, 21].
Other work has learnedcross-modal distillation [22], sound localization [23, 24, 25, 19, 26, 27, 28, 29, 30], active speaker
[Figure]
detection [31], sound source separation [32, 33, 34, 35].
We take inspiration from this work and showthat quiet, indoor sounds also provide a useful self-supervised learning signal.
We also propose newaudio-visual models that learn about relative depth in multiple modalities.
Visual depth prediction.
Recent works have tackled recovering 3D structure from RGB imagesthrough learning-based approaches and researchers have developed increasingly detailed single imagereconstructions [36, 37, 38, 39, 40, 41, 42], via voxel grids, point clouds, and meshes [43, 44, 45, 46].
At the same time, the signiﬁcant progress has been made in predicting the relative camera pose [47,48, 49, 50, 51].
We take inspiration from the techniques proposed in these models, though we predictdepth from audio instead of vision.
Robot navigation.
Robots often use visual streams to navigate to a destination in unseen environments [52, 53, 54, 55].
Recently, researchers leverage audio signals as useful cues for robot navigationtasks.
Those robotic systems are designed to localize the traceable sound sources and navigate toaudio goals in indoor environments [56, 57, 13, 58, 1, 59].
Unlike these methods, which use knownand clear sound sources, we take advantage of ambient sounds collected in real-world scenes.
The Quiet Campus Dataset
[Figure]
To study the structure-from-audio problem, we collected a dataset containing a largenumber of “in-the-wild” indoor ambient audio recordings, paired with concurrentlyrecorded RGB-D data.
We collected our data from a large number of classrooms andhallways on a university campus.
To ensure that the sounds within the recordingswere subtle ambient sounds (e.g. ventilation noise), rather than distinctive audioevents (e.g. speech), we collected the data at times when there were few peoplearound.
The recorded audio qualitatively sounds very quiet 1 .
Please refer to oursupplementary material for a random sample of sounds.
We collected two types of data: static recordings in which the microphone andcamera are stationary, and motion recordings where a human collector moves slowlytoward or away from a wall.
We show some visual examples from our collecteddataset in Fig.
In our experiments, we split the data so that there is no physicaloverlap, such that the rooms in the training and tests sets are disjoint.
Static recordings.
We collect data in 46 classrooms in 12 buildings, containingapproximately 200 minutes audio.
For each classroom, we select 16 − 30 positionsand record 10 secs.
of audio
at each one.
Since we are interested in predicting thedistance to the wall, we point the camera and microphone toward the nearest wall when recording, sothat the distance to the wall is well-deﬁned.
Static recordings with dense coverage.
In order to evaluate generalization to different orientations,we use four classrooms from Static recordings whose size ranges from 20 to 35 cells total, and collect10 hours of audio data densely with a wide range of angles (rather than always facing a wall).
We callthis these Static-dense recordings.
These two datasets were recorded at different times of the yearand with different usage patterns.
The Static recordings were made in spring 2021, when buildingswere at reduced occupancy due to the COVID-19 pandemic, and when air conditioners may havebeen in use, while Static-dense were recorded in the winter of 2020, when buildings were at normaloccupancy and when heating may have been in use.
Motion recordings.
To help understand whether small motion (and the subtle changes in audio overspace) provides useful information about structure, we collect approximately 90 minutes of videos inmotion (222 videos total).
To ensure that there are no unintended obstructions nearby, we collectthis data in hallways.
During recording, the microphone and RGB-D camera move toward or awayfrom a wall (Fig. 2).
As in the static setting, we point the microphone toward the wall.
To reducepossible sources of sound, the human operator moves slowly and does not take footsteps.
We split
VisionNet VisionNet
[Figure]
AudioNet AudioNet
[Figure]
Objective
[Figure] [Figure]
this dataset based on building , i.e ., the videos in the training and test set are hallways recorded indifferent buildings.
Estimating wall distance.
For experiments that require distance to the wall, we obtain this bycropping the center 320 × 240 region from the depth image, and averaging the depth values.
Sincethe camera is oriented toward the wall, this is generally a simple and accurate approximation.
Hardware.
We use a ZOOM H1n Handy Recorder for collecting audio since it is a simple, inexpensive, and widely-used stereo audio recorder.
For the RGB-D data, we use Intel Real-Sense Depth
Camera D415 (Fig. 3).
Predicting Structure from Ambient Sound
We design models and experiments that will experimentally evaluate the ability of a model to learn topredict depth from audio in natural scenes.
4.1 Depth estimation tasks
Inspired by work in human obstacle detection with sound [5], we study the ability of a machinehearing system to predict the distance to a wall ahead of them.
We pose this problem in multipleways, which has applications in various robotic navigation scenarios.
Obstacle detection.
First, we pose the problem of estimating whether the microphone is within asmall distance of a wall (we use 0.5 meters).
Relative depth order.
In many cases, a robotic system needs only to move toward or away from anobstacle, rather than to know its actual distance.
We also hypothesize that relative depth estimationmay be an easier task than absolute estimation, as it is in visual depth perception [60, 36].
Given twoaudio clips, we train a network to predict which one is closer to a wall.
Relative depth estimation.
Given two audio clips, we train a network to predict the differencebetween their distance to the wall.
We also consider predicting the log of their difference ( i.e ., theratio), following the work in visual depth estimation, which ﬁnds this easier to predict [38, 37], andas a multi-way classiﬁcation task, after clustering depth differences in the training set with k -means.
Absolute depth estimation.
In the end, we train a model to regress the distance to a wall, rather thana thresholded “near-or-far” prediction.
4.2 Self-supervised audio-visual learning
Having evaluated our model’s ability to estimate depth directly from audio, we now ask whether wecan learn about structure through audio-visual self-supervision.
We hypothesize that, through the taskof associating a raw (RGB) visual signal with audio, we will learn a representation with features thatwill be helpful for downstream depth estimation tasks.
One simple model, considered in prior work, is to solve a audio-visual synchronization task [31, 19,27]:
given the video, distinguish whether audio and visual stream are misaligned by randomly shiftingaudio.
We call this model the AV-Sync model.
Finally, we pose a new audio-visual learning problemthat requires a model to learn relative depth between examples.
We train a model to distinguishbetween ordered audio-visual pairs, which we call the AV-Order model.
Given a pair of audio tracksand a pair of images (corresponding to the two audio tracks), we train a network to determine whetherthe two pairs are in the same (or different) order, which we frame as a classiﬁcation problem.
4.3 Audio-based robotic navigation
To demonstrate the effectiveness of our depth estimation model, we use it to guide a wheeled robot(TurtleBot) to follow a wall.
The robot uses the obstacle detection model to predict whether there iswall near the left/right of the robot, and moves accordingly.
We use a simple policy, based on that of
Gandhi et al .
[61] (please
It senses the audio on its left side, then rotates 180 ◦ and senses audio to its right.
We run the obstacledetection model on both audio recordings, and choose an orientation such that the robot moves away
[Figure]
from the wall that has higher probability of being near a wall.
[Figure]
Network backbone.
We represent audio using the VGGish network [62].
For our experiments,we either replace the ﬁnal layer with a new regression or classiﬁcation head, or we use the (128dimensional) penultimate layer.
The network takes 0.96 sec of audio as input, in the form of a log-melspectrogram.
Due to the importance of low-frequency sound in physical models of wall-soundinteraction [7], we do not remove the very lowest frequencies of the audio, in contrast to commonpractice in speech and audio event recognition [62].
We analyze the importance of this decisionthrough experiments.
We convert the stereo audio recordings to mono by averaging the two channels.
For models that use visual data, we use ResNet-18 [63], with 224 × 224 images, and modify the lastlinear layer to output the feature vectors with the same dimension as audio features.
In addition, wealso ﬁnetune the weights from ImageNet [64] to see the upper bound performances.
Learning details.
For fully supervised tasks, we use SGD [65] optimizer with a learning rate 10 − 3 ,momentum 0 .
9 , and weight decay 5 × 10 − 4 .
For the self-supervised learning experiments, we use
Adam [66] with a learning rate of 10 − 4 for representation learning.
For linear probing experiments,we use SGD with a learning rate of 10 − 2 .
We schedule learning rates via cosine decay and pick thebest model from validation for all the training jobs.
Audio processing.
Our audio preprocessing resembles that of [62].
We resample the audio to 16 kHzand extract segments of 0.96 secs.
We then convert them to spectrograms by applying a Short-time
Fourier transform at a stride of 10 ms and 25 ms, and convert them into 96 × 64 log-mel spectrograms.
For augmentation, we randomly crop the waveform in time and multiply the amplitudes by a randomfactor uniformly sampled from [0.5, 1.5].
Depth estimation networks.
For the regression and binary classiﬁcation tasks, we add a smallmulti-layer perceptron on top of the VGGish network.
For the relative estimation tasks, our networkaims to identify objectives with two audio inputs.
We approach this by building a Siamese networkthat fuses the extracted feature vectors via concatenation prior to the multi-layer perceptron.
Forregression, we normalize depths to be in the range [ − 0 .
5] during training.
Network for self-supervision tasks.
For AV-Sync task, we concatenate image and audio featuresbefore passing through the multi-layer perceptron.
For the AV-Order model, we fuse Siamesenetworks from both modalities via concatenation and add a multi-layer perceptron (Fig. 4).
In thedownstream task, we take the output features from the last convolution layer after global averagepooling and freeze them.
We pass our 512-d learned features to a linear layer to get the objectivesand only optimize it during training.
Experiments
Having collected a dataset of in-the-wild audio and models for inferring structure from it, we conductan experimental study of what can be inferred from ambient sound in real-world scenes.
We reportour results along with 95% conﬁdence intervals.
An experimental study of audio- based depth
estimation
Obstacle detection.
First, we evaluate our model’s ability to solve the obstacle detection problem(Tab. 1).
We split the data such that the test rooms were not heard during training.
We report accuracyon this binary classiﬁcation task, as well as average precision (to account for possible dataset shiftfrom using held-out rooms).
We evaluated our model in both the static and motion settings.
To helpinterpret the effectiveness of audio in the context of other modalities, we also included simple visualmodels, with both random initialization and ImageNet pretraining 2 .
Interestingly, we see that the [Table] audio-based model performs better than chance in both cases, suggesting that ambient sound, indeed,conveys information about scene structure.
We also evaluated our model’s ability to generalize across orientations and recording times.
We foundthat a model trained on Static recordings could achieve 86.1% AP and 72.9% Acc on Static-denserecordings, indicating strong generalization performance.
We qualitatively demonstrate the obstacle detection results by training a version of the model on the
Static-dense data, since it contains a large number of viewpoint variations and angles (see supplementfor quantitative results).
We apply the learned obstacle detection model to every frame of a video inwhich a person walks through a room (Fig. 5).
In Fig.
6, we also apply this model densely within aroom divided according to its spatial grid.
We averaged the probability of the model for each gridcell.
We see that it successfully distinguishes between the boundaries of the room and the center.
[Table]
Relative depth order.
In Tab. 1,we evaluate our model’s ability to predict which of two audio clips is closer to a wall.
Once again, we ﬁnd that themodel obtains better results thanchance (see supplementary formore results).
We also see that,as is the case with visual prediction tasks, the relative predictiontask is easier than predicting obstacles.
[Table]
Relative depth ratio.
Since thenetwork is able to tell relativedepth order well, we explore relative depth setup further by predicting depth difference of audio pairs (log depth ratio).
Weuse regression to predict the logratio from ambient sounds with L1 Loss function and via classiﬁcation.
We report the performance of mean absolute error,median error, and R 2 .
For classiﬁcation, we report top-1 and top-5 accuracy, and report average distance from the ground truthbin using the distance between centers of paired bins.
We report the performance of zeroing outtwo inputs as a baseline.
As shown in Tab. 2, we can see that our method outperforms chance and,interestingly, has a relatively small gap between the vision-based approach.
Absolute depth estimation.
We evaluated our model’s ability to predict depth from a single audioexample, using similar experiments as the relative depth ratio (Tab. 3).
Our model is more accuratethan chance (which we obtain for all metrics by simply ablating the input and training).
Likewise, theregression-by-classiﬁcation model improves over chance performance. [Figure] es t i n a t i on [Table]
We also consider another CNN model inspired by the physical model of Ashmead et al.
[6] and ourrelative depth model.
This new model estimates depth for a given audio input after conditioning on areference sound recorded within the same scene and its depth (see the supplementary material fordetails).
We call the conditional absolute depth model.
This model is better than our model withonly a single audio input [Table] suggesting that the information from reference audio is a useful cue.
5.2 Audio-visual representation learning
[Table]
Finally, we investigate whether welearn useful representation with AVOrder and AV-Sync models.
In the
AV-Order binary classiﬁcation pretexttask, our model obtains 85.8% APand 75.8% accuracy (80.3% AP and70.
% accuracy with random initialization) on the test set where the random chance is 50%.
Given its performance on the pretext task, we testwhether the learned audio and visualrepresentations about depth are usefulfor the downstream tasks: (1) obstacledetection and (2) relative depth estimation.
To evaluate the quality of ourspatial representation, we freeze thelearned features and train a linear classiﬁer on the depth estimation tasks (Section 4.1).
In the downstream tasks, we validate image andaudio features individually and compare them with random features, pre-trained ImageNet features [64, 63], pre-trained AudioSet features [62] and AV-Sync features.
Since AudioSet pretrainedweights require a special audio pre-processing that differs from ours, we provide another randomfeature baseline based on the same processing as [62].
Relative depth.
We report results for relative depth order and relative depth ratio in Tab. 4 and Tab.
5respectively.
Our method outperforms baselines in both audio and visual modalities suggesting thatwe learn useful depth representations.
Moreover, our results are often close to supervised features.
Obstacle detection.
As the results shown in Tab. 4, AV-Order model outperforms Scratch and
AudioSet pretraining, while it loses to AV-Sync model.
It is reasonable that AV-Order model getworse performance since AV-Order model deals with two inputs during the proxy task.
We providean additional experiment by ﬁne-tuning our learned AV-Order feature and we compare with the fullysupervised method in Tab. 1.
This obtains a 4% AP boost (70.0% v.s. 65.6%), suggesting that ourlearned representation is helpful.
Furthermore, the models with both audio and visual features as inputs perform consistently betterthan visual features solely, suggesting the audio is useful in addition to the visual signal.
5.3 Audio cues
[Figure]
We ask what audio information that the network uses to solve its task.
We take inspiration from physical models of audio spectra nearwalls [6], which suggest that the distribution oflow-frequency sounds (20-200Hz) are an important cue.
We zero out different frequency rangesin the mel spectrogram, then re-train the relativedepth order model.
As can be seen in Fig.
8, theaccuracy of the model with only very low frequencies (0-250Hz) is comparable to the modelwith the full frequency range, suggesting thatthese frequencies are indeed very important.
Forcomparison, we conducted the same experimentwith instrument categories from the VGGSound dataset [67] (see supp.
for details).
Interestingly,there these low frequencies are less important, suggesting that our setting relies on different cues.
5.4 Robotic navigation with ambient sound
[Figure]
We ask whether our depth estimation model can be used toguide a robot to follow a wall, solely using ambient sound.
For this, we used a model trained on Static recordings,and evaluate on a classroom that was not included in thetraining set.
The track is 8m long, and forces the robotto turn a corner (Fig. 7).
The robot starts navigating fromone of several unknown locations (40cm, 80cm, or 120cmfrom the wall) and orientations ( 30 ◦ left or right, or facingforward).
We repeat the experiment 18 times and measurethe average distance along the track it attained beforecolliding with a boundary.
Quantitative results are shownin Fig.
We compare with a random policy, a “straight line” policy that always drives forward, anda policy based on the vision model.
For the random policy, the setting is the same as our method,except we use a randomly initialized network for obstacle prediction.
The straight line policy iseffective when the robot starts with an orientation facing front, and it has the advantage that it doesnot rotate to sense audio, and hence avoids accumulating errors 3 .
It’s not surprising that the visionmodel outperforms our audio model, since we have known that vision performs well for those kind oftasks.
We see that our method performs better than random and straight line policy, and successfullyguides the robot pass the corner in nearly half of the trials.
Conclusion
In this paper, we have shown that real-world, indoor ambient sound provides a great deal of information about structure.
We contributed the Quiet Campus dataset, a large dataset of in-the-wild pairedaudio and RGB-D recordings.
We showed that distance to a wall can be predicted from ambientaudio in a number of settings.
We also showed that these ambient sounds can be used for multimodalself-supervised learning, including with a novel formulation that captures relative depth.
We see this work as opening two directions.
The ﬁrst is in creating robotic systems that exploit realworld, passively obtained audio signals.
While related systems have previously been demonstratedin simulation with distinctive audio events (or with active echolocation), we take a step towardinterpreting in-the-wild ambient sound.
Second, we see our work as a step toward using the “free”audio signals to help train other modalities, such as vision, about scene structure.
Limitations.
While ambient sound provides information about scene structure, it is a weakersignal when compared to vision and potentially self-produced sound, and we do not yet have a fullunderstanding of what can (and cannot) be perceived from it, e.g. whether there are fundamentallimitations on resolution, material, or angle.
Additionally, while we will release our pretrained modelfor others to use, our experiments were conducted on a single college campus, and hence may not berepresentative of all indoor spaces.