The Boombox: Visual Reconstruction from Acoustic Vibrations
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Interacting with bins and containers is a fundamental task in robotics,making state estimation of the objects inside the bin critical.
While robots oftenuse cameras for state estimation, the visual modality is not always ideal due toocclusions and poor illumination.
We introduce The Boombox, a container thatuses sound to estimate the state of the contents inside a box.
Based on the observation that the collision between objects and its containers will cause an acousticvibration, we present a convolutional network for learning to reconstruct visualscenes.
Although we use low-cost and low-power contact microphones to detectthe vibrations, our results show that learning from multimodal data enables stateestimation from affordable audio sensors.
Due to the many ways that robots usecontainers, we believe the box will have a number of applications in robotics.
Introduction
In order for robots to robustly grasp and pick up objects inside containers, they need to accurately localize and estimate the state of the objects inside the container.
Vision-based perception has enablednumerous advances for state estimation [1, 2, 3] in object manipulation and grasping, such as package bin-picking and object retrieval in household settings.
Due to the importance of this problem,pose and state estimation from vision has been long studied in robot vision [4, 5, 6, 7, 8, 9].
However, environmental conditions are not always ideal for cameras.
Although robots frequentlyneed to interact with containers, objects inside containers often become occluded from cameras,making state estimation from vision impractical.
In unconstrained settings, environments can alsolack ideal lighting conditions, with limited visual signals to indicate the location and state of objects.
Moreover, camera systems require extensive calibration for 3D state estimation, which isoften fragile during contact and collisions.
In this paper, we propose to use sound as another sensing modality for object state estimation inrobotics.
Unlike vision, sound remains robust during occlusions or poor illuminations.
For example,when an object is dropped inside a container, it may not be visible to any camera, but the collisionbetween the object and the bin will cause sound that can be easily picked up by a contact microphone.
The exact incidental vibration will depend on the location and pose of the objects inside the bin.
Wedemonstrate how to use this audio signal to reconstruct the objects’ states inside containers.
We introduce The Boombox, a smart container that uses the vibration of itself to reconstruct animage of its contents.
The box is no larger than a cubic square foot.
Unlike most containers, the boxuses contact microphones to detect its own vibration.
Exploiting the link between acoustic and visualstructure, we show that a convolutional network can use these vibrations to predict the visual sceneinside the container within centimeters, even under total occlusion and poor illumination.
Figure 1illustrates our box and one reconstruction from the sound.
The main contribution of this paper is an integrated hardware and software platform for using soundto reconstruct the visual structure inside containers.
The remainder of this paper will describe The
Boombox in detail.
In section 2, we ﬁrst review background on this topic.
In section 3, we introduceour perception hardware, and in section 4, we describe our learning model.
Finally, in section 5, we
[Figure]
We introduce a “smart” container that is able to reconstruct an image of its inside contents.
The approach works even when the camera cannot see into the container.
The box has four contact microphones on each face.
When objects interact with the box, they cause incidental acoustic vibrations.
From these vibrations, we learn to predict the visual scene inside the box.
quantitatively and qualitatively analyze the performance and capabilities of our approach.
We willopen-source all hardware designs, software, models, and data.
Related Works
Vision and Sound.
Our paper builds on work that integrates sound and visual perception.
Givenvisual observations, there has been extensive studies to enhance sounds [10, 11], ﬁll in missingsounds [12], and generate sounds entirely from video [13, 14].
Further, there have been recentworks in integrating vision and sound to improve recognition of environmental properties [15, 16]and object properties, such as geometry and materials [17, 18].
Audiovisual data have also beenstudied for representation learning [19, 20].
Lastly, there has been work for generating a face givena voice [21] and a scene from ambient sound [22].
In contrast, our work uses sound to predict the3D visual structure inside a container.
Sound in Robotics.
Most related research to our work is on audiovisual object understanding forrobot perception and control.
Gandhi et al.
[23] investigates vision and sound in a robot setting topredict which robot actions caused the given sound.
Matl et al.
[24] leverages audio signals from ballbouncing motions to calibrate the stochastic dynamical events for “sim2real” tasks.
Bu and Huang[25] predicts the trajectory of a falling cube for robot object retrieval outside visible region.
Insteadof predicting the 2D trajectory for a single cube, this paper learns to predict the entire visual sceneunder fully occluded conditions by outputting RGB and depth images on three different blocks.
Audio Analysis.
The primary features in audio that are used for sound localization [26] are timedifference of arrival and level (amplitude) difference.
Speciﬁcally calculating these exact features isnon-trivial, especially in situations where the signal is not broad-band and in motion [27, 28, 29, 30].
Furthermore, these rough approximations can only be used to localize the object, whereas our goalis to not only localize objects, but also predict the 3D structure, which includes shape and orientationof the object.
As such, we develop a model that learns the necessary features for reconstruction.
The Boombox
In this section, we present the The Boombox hardware.
We also discuss the characteristics of theacoustic signals captured by it.
3.1 Detecting Vibrations
The Boombox, shown in Figure 2A, is a plastic storage container that is 15 .
5 cm × 26 cm × 13 cm(width × length × height) with an open top.
The box is a standard object that one can buy atany local hardware store.
When an object collides with the box, a small acoustic vibration will beproduced in both the air and the solid box itself.
We have attached contact microphones on each wallof the plastic cuboid storage bin in order to detect this vibration.
Unlike air microphones, contactmicrophones are insensitive to the vibrations in the air (which human ears hear as sound).
Instead,they detect the vibration of solid objects.
The microphones are attached on the outer side of the walls, resulting in four audio channels.
Wearrange the microphones roughly at the horizontal center of each wall and close to the bottom.
As
[Figure]
S a m p l e S a m p l e
S a m p l e our approach will not require calibration, the microphone displacements can be approximate.
Weused TraderPlus piezo contact microphones, which are very affordable (no more than $5 each).
3.2 Vibration Characteristics
[Figure]
When objects collide with the box, the contact microphones will capture the resulting vibrations.
Figure 3 shows an example of the vibration captured from two of the microphones.
We aim torecover the visual structure from this signal.
As these vibrations are independent of the visualconditions, they allow perception despite occlusion and poor illumination.
There is rich structure inthe raw acoustic signal.
For example, the human auditory system uses inter-aural timing difference(ITD), which is the time difference of arrival between both ears.
Humans also locate sounds withinter-aural level difference (ILD), which is the amplitude level difference between both ears [31].
However, in our settings, extracting these characteristics is challenging.
In practice, objects willbounce around in the container before arriving at their stable position, as shown in Figure 4.
Eachbounce will produce another, potentially interfering vibration.
In our attempts to analytically usethis signal, we found that the third bounce has the best signal for the time difference of arrival, butas can be seen from Figure 3, even on the third bounce the time difference of arrival is unclear inthe actual waveform.
There are a multitude of factors that make analytical approaches not robustto our real-world signals.
Firstly, we are working with a moving signal, whereas time difference ofarrival calculations work best on stationary signals due to the fact that it compares the time taken fora signal to travel from a ﬁxed location.
This makes it very difﬁcult to analytically segment the signalinto chunks of roughly the same location.
Secondly, there are echos that make non-learning basedmethods difﬁcult to identify phase shifts as the environment is a small container.
Finally, the factthat the microphones are close together means that the time difference of arrival is encompassed infew samples, thus making it susceptible to noise.
Instead of hand crafting features, we will train a model to identify the fraction of the signal thatis most robust for ﬁnal localization.
Our model will learn to identify the useful features from thesignals to reconstruct a 3D scene, which includes the shape, orientation, and position of the contents.
3.3 Multimodal Training Dataset
Since vision and sound are naturally synchronized, we will use vision as self-supervision to learnrobust characteristic features of the acoustic signal.
We collected a multimodal training dataset bydropping objects into the box and capturing resulting images and vibrations.
We position an Intel
RealSense D435i camera that looks inside the bin to capture both RGB and depth images,
which we
[Figure]
use during training.
2 We use three wooden blocks with different shapes to create our dataset.
The blocks have the same color and materials, and we show these objects in Figure 2B.
We hold theobject above the bin, and freely drop it.
After dropping, the objects bounce around in the box a fewtimes before settling into a resting position.
We record the full process from all the microphones andthe top-down camera.
Overall, our collection process results in diverse falling trajectories across allshapes with a total of 1,575 sequences.
Figure 2C shows an overview of the dataset.
After learning, our approach will be able to reconstruct the 3D visual scene from the
Predicting the Visual Scene from Vibration
To estimate the state inside the container, we will learn to reconstruct the visual modality from theaudio modality.
We present a convolutional network that translates vibrations into images.
4.1 Model
We will ﬁt a model that reconstructs the visual contents from the vibrations.
Let A i be a spectrogramof the vibration captured by microphone i such that i ∈ { 1 , 2 , 3 , 4 } .
Our model will predict theimage ˆ X RGB =
f RGB (
A ; θ ) where f is a neural network parameterized by θ .
The network will learnto predict the image of a top-down view into the container.
We additionally have a correspondingnetwork to produce a depth image ˆ X depth =
f depth ( A ; θ ) .
Reconstructing a pixel requires the model to have access to the full spectrogram.
However, we alsowant to take advantage of the spatio-temporal structure of the signal.
We therefore use a fully convolutional encoder and decoder architecture.
The network transforms a spectrogram representation(time × frequency) into an C -channel embedding with width and height being 1 × 1, such that thereceptive ﬁeld of every dimension reaches every magnitude in the input and every pixel in the output.
Unlike image-to-image translation problems [32, 33, 34], our task requires translation acrossmodalities.
We use a multi-scale decoder network [35, 36, 37].
Speciﬁcally, each decoder layer consists of twobranches.
One branch is a transposed convolutional layer to up-sample the intermediate feature.
Theother branch passes the input feature ﬁrst to a convolutional layer and then a transposed convolutionso that the output for the second branch matches the size of the ﬁrst branch.
We then concatenate theoutput from these two branches along the feature dimension as the input feature for the next decoderlayer.
We perform the same operation for each decoder layer except the last layer where only onetransposed convolution layer is needed to predict the ﬁnal output image.
We use a spectrogram to represent audio signals.
We apply a Fourier Transform before convertingthe generated spectrogram to Mel scale.
Since we have four microphones, audio clips are concatenated together along a third dimension in addition to the original time and frequency dimension.
4.2 Learning
In most cases, we care about predicting the resting position of the object.
We therefore train thenetwork f to predict the ﬁnal stable image.
For RGB image predictions, we train the network to minimize the expected squared error between the image X RGB and the predictions from audio A :
[Equation]
In order to reconstruct shape, we also train the network to predict a depth image from the audioinput.
We train the model to minimize the expected L1 distance:
[Equation]
Since ground truth depth often has outliers and substantial noise, we use an L1 loss [38].
We usestochastic gradient descent to estimate the network parameters θ and φ .
After learning, the modelpredicts both the RGB image and the depth image from just the vibration.
The visual modality isonly supervising representations for the audio modality, allowing reconstructions when cameras arenot viable, such during occlusions or low illumination.
4.3 Implementation Details
Our network takes in the input size of 128 × 128 × 4 where the last dimension denotes the numberof microphones.
The output is a 128 × 128 × 3 RGB image or a 128 × 128 × 1 depth image.
We usethe same network architecture for both the RGB and depth output representations except the featuredimension in the last layer for different modalities.
All network details are listed in the Appendix.
Our networks are conﬁgured in PyTorch [39] and PyTorch-Lightning [40].
We optimized all thenetworks for 500 epochs with Adam [41] optimizer and batch size of 32 on a single NVIDIA RTX2080 Ti GPU.
The learning rates starts from 0.001 and decrease by 50% at epoch 20, 50, and 100.
Experiments
Our experiments analyze the hardware and software at reconstructing an image of the contents fromaudio.
In this section, we ﬁrst quantitatively evaluate the performance.
We then show qualitativeresults for the reconstructions.
Finally, we visualize the learned representations.
Since the physics behind our dataset is chaotic, everytime an object is dropped into the container,we obtain a unique example with a different resting position and orientation.
We randomly split thedataset into a training set (80%), a validation set (10%), and a testing set (10%).
All of our resultsare evaluated with three random seeds for training and evaluation with various splits of the dataset.
We report the mean and the standard error of the mean for all outcomes.
5.1 Evaluation Metrics
We use two evaluation metrics for our ﬁnal scene reconstruction that focus on the object state.
IoU measures how well the model reconstructs both shape and location.
Since the model predictsan image, we subtract the background to convert the predicted image into a segmentation mask.
Similarly, we performed the same operation on the ground-truth image.
IoU metric then computesintersection over union with the two binary masks.
Localization score evaluates whether the model produces an image with the block in the right spatialposition.
This metric is especially useful for object picking tasks with a suction gripper where thespatial location of the block matters.
With the binary masks obtained in the above process, we canﬁt a bounding box with minimum area around the object region.
We denote the distance betweenthe center of the predicted bounding box and the center of the ground-truth bounding box as d , andthe length of the diagonal line of ground-truth box as l .
We report the fraction of times the predictedlocation is less than half the diagonal: 1 N (cid:80)
Ni =
[ d i ≤ l/ 2] .
5.2 Baselines
Time Difference of Arrival (TDoA).
We compare against an analytical prediction of the location.
A standard practice is to localize sound sources by estimating the time difference of arrival acrossan array of microphones.
In our case, the microphones surround the sound source.
There are severalways to estimate the time difference of arrival, and we use the Generalized Cross Correlation with
Phase Transform (GCC-PHAT), which is a established, textbook approach [42].
Once we have our
Audio Input
Prediction
[Figure]
time difference of arrival estimate, we ﬁnd the location in the box that would yield a time differenceof arrival that is closest to our estimate.
Random Sampling.
To evaluate if the learned models simply memorize the training data, we compared our method against a random sampling procedure.
This baseline makes a prediction by randomly sampling an image from the training set and using it as the prediction.
Average Bounding Box.
The average bounding box baseline measures to what extent the modellearns the dataset bias.
We extracted object bounding boxes from all the training data throughbackground subtraction and rectangle ﬁtting to obtain the average center location, box sizes and boxorientation.
This baseline uses the average bounding box as the prediction for all the test samples.
Nearest Neighbor.
To evaluate the generalization performance from training data distribution, weconstruct a nearest neighbor baseline.
For each test input audio, we use the resulted image from thetraining data with most similar audio as the prediction.
The similarity is measured by a L2 distance. 5.3 Reconstruction with Mixed Shapes
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We ﬁrst analyze how well The Boombox reconstructs its contents when the shape is not knowna priori.
We train a single model with all the object shapes.
The training data for each shape aresimply combined together so that the training,validation and testing data are naturally wellbalanced with respect to the shapes.
This setting is challenging because the model needs to learn audio features for multiple shapes at once.
Figure 6 shows the convolutional networks are able to learn robust features to localize both theposition and orientation, even when shapes are mixed.
Our method outperforms TDoA often bysigniﬁcant margins, suggesting that our learning-based model is learning robust acoustic features forlocalization.
Due to the realistic complexity of the audio signal, the hand-crafted features are hardto reliably estimate.
Our model outperforms both the random sampling and average bounding boxbaseline, indicating that our model learns the natural correspondence between acoustic signals andvisual scene rather than memorizing the training data distribution.
We show qualitative predictionsfor both RGB and depth images in Figure 5.
5.4 Reconstruction with Known Shape
[Figure]
We next analyze how well the model performs when the object shape is known, but the positionand orientation is not.
We train separate models for each shape of the object independently.
Figure7 shows The Boombox is able to reconstruct both the position and orientation of the shapes.
Theconvolutional network obtains the best performance for most shapes on both evaluation metrics. [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table]
[Table] [Figure]
[Figure]
important with testing time a well with the relative ference an the time a consistent with
These results highlight the relative difﬁculty at reconstructing different shapes from sound.
Bycomparing the model performance across various shapes, the model trained on cubes achieves thebest performance while the model trained on blocks performs slightly worse.
The most difﬁcultshape is the stick
[Figure]
[Table] [Table] [Table] [Table]
Cube
Stick
Block 0.
Depth
[Table]
[Table]
[Table]
Stick
[Table]
Block
[Table]
Cube
Stick
Block 0.
Single-shape training
[-Stick 0.
-]
[Table]
[Table]
Depth Output (
[Table]
Score)
Cube
Stick
Block 0.
Depth Output (Localization) 0%
[Table]
[Table]
[Table]
Stick
[Table]
Block
[Table]
[Figure]
Stick
Block
[Figure]
Cube
Stick
Block
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
When the training data combines all shapes,the model should be able to share features between shapes, thus improving performance.
Tovalidate this, we compare performance on themulti-shape versus models trained with a singleknown shape.
Figure 9 shows that the performance on the block and stick shapes are improved by a large margin.
We notice that theperformance of the cube drops due to the con- fusion between shapes.
When the cube confuses with the stick or the block, because of the smallersurface area of these two shapes, the cube performance slightly degrades.
5.5 Ablations and Analysis
To better understand what features the model has learned speciﬁcally
[Figure]
we perform several ablationstudies, shown in Figure 8 and Figure 10.
[Table]
[Figure]
[Table]
Stick (IoU Score)
[Figure]
[Table]
Block (IoU Score)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
The microphones’ layoutshould matter for our learned model to localizethe objects.
When we ﬂipped the microphone location, due to the symmetric nature of the hardware setup, the predictions should also be ﬂippedaccordingly.
To study this, we ﬂipped the corresponding audio input in the testing set, shownin Figure 8.
Speciﬁcally, the audio input of the Mic1 and Mic4 are ﬂipped, and the audio inputof the Mic2 and Mic3 are ﬂipped.
Our results in Figure 8B shows that our model indeed producesa ﬂipped scene.
The performance in Figure 10nearly drops to zero, suggesting that the modelimplicitly learned the relative microphone locations to assist its ﬁnal prediction.
Remove amplitude.
The relative amplitude between microphones can also indicate the relativeposition of the sound source to different microphones.
We removed the amplitude informationby thresholding the spectrograms, shown in Figure 8.
We retrained the network due to potentialdistribution shift.
As expected, even though the time and frequency information are preserved,the model performs much worse (Figure 10), suggesting that our model additionally learns to useamplitude for the predictions.
Temporal shift.
We are interested to see if our model learns to capture features about the timedifference of arrival between microphones.
If so, when we shift the audio temporally, the predictionshould also shift spatially.
We experimented with various degrees of temporal shifts on the originalspectrograms.
For example, shifting 500 samples corresponds to shifting about 0.01s (500 / 44,000).
By shifting the Mic1’s spectrogram forward and Mic4’s spectrogram backward with zero padding tomaintain the same amount of time, and preforming similar operation on Mic2 and Mic3 respectively,we should expect that the predicted object position shifts towards the left-up direction.
In Figure 8,we can clearly observe this trend as temporal shift increases.
Shifting the signal in time decreases themodel’s performance, demonstrating that the model has picked up on the time difference of arrival.
[Figure]
Feature Visualization: We ﬁnally visualize the latent features inbetween our encoder and decoder network by projecting them intoa plane with t-SNE[43], shown in Figure 11.
We colorize the pointsaccording to their ground truth position and orientation.
The magnitude distance from the center of the image is represented by saturation, and the angle from the horizontal axis is represented by hue.
We ﬁnd that there is often clear clustering of the embeddings bytheir position and orientation, showing that the model is robustlydiscriminating the location of the impact from sound alone.
Moreover, the gradual transitions between colors suggest the features areable to smoothly interpolate spatially.
Conclusion
This paper introduces The Boombox, a low-cost container integrated with a convolutional networkthat uses sound to reconstruct an image of the contents inside it.
Containers are ubiquitous in everyday situations, and this paper shows that we can equip them with sound perception to estimatethe state inside the container.
When cameras are not available, such as during occlusions or poorillumination, our results suggest that sound is a promising sensing modality.