Differentiable Rendering and Identiﬁcation of Impact Sounds
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
Rigid objects make distinctive sounds during manipulation.
Thesesounds are a function of object features, such as shape and material, and of contactforces during manipulation.
Being able to infer from sound an object’s acousticproperties, how it is being manipulated, and what events it is participating in couldaugment and complement what robots can perceive from vision, especially in caseof occlusion, low visual resolution, poor lighting, or blurred focus.
Annotationson sound data are rare.
Therefore, existing inference systems mostly include asound renderer in the loop, and use analysis-by-synthesis to optimize for objectacoustic properties.
Optimizing parameters with respect to a non-differentiablerenderer is slow and hard to scale to complex scenes.
We present DiffImpact, afully differentiable model for sounds rigid objects make during impacts, basedon physical principles of impact forces, rigid object vibration, and other acousticeffects.
Its differentiability enables gradient-based, efﬁcient joint inference ofacoustic properties of the objects and characteristics and timings of each individualimpact.
DiffImpact can also be plugged in as the decoder of an autoencoder,and trained end-to-end on real audio data, so that the encoder can learn to solvethe inverse problem in a self-supervised way.
Experiments demonstrate thatour model’s physics-based inductive biases make it more resource efﬁcient andexpressive than state-of-the-art pure learning-based alternatives, on both forwardrendering of impact sounds and inverse tasks such as acoustic property inferenceand blind source separation of impact sounds.
Introduction
The sound we perceive during rigid object contact is a function of many factors, including noise inthe environment, position of the listener, reﬂecting surfaces in the surrounding environment, etc.
Themost deﬁning of these factors are the vibrations the object makes, a product of its intrinsic acousticproperties and the way it was contacted.
Extracting information about the acoustic properties ofan object could inform a robot about the object’s size, shape, material, and where and how it wasstruck.
Extracting information about the way the object was contacted could inform a robot of thevelocity, mass, shape, and hardness of the tool that was used to strike the object.
Therefore, the abilityto interpret impact sounds could augment and complement what robots can perceive from vision,especially in cases of occlusion, low visual resolution, poor lighting, or blurred focus.
Decomposing the sound an object makes into the contact forces that excited it and the idealizedacoustic impulse response is generally an ill-posed problem.
As annotations on sound data are rare,existing inference systems mostly include a sound renderer in the loop, and use analysis-by-synthesisto optimize for object acoustic properties.
Optimizing parameters with respect to a non-differentiablerenderer is slow and hard to scale to complex scenes.
We propose DiffImpact, a fully differentiable,generative sound model that uses physics-based priors as a bias.
Speciﬁcally, we introduce fully differentiable models of the object’s impulse response and the contact force during impact.
We combinethese with differentiable models of environment acoustics such as ambient noise and reverberationinto a fully differentiable generative sound model.
With this physics-based model, we can now inferphysically interpretable parameters of rigid body impact sounds through analysis-by-synthesis, even from real-world recordings, including from real YouTube videos or from a noisy and reverberantrobotics lab.
Impacts in the real world are not always clean impulses, nor do they occur in controlled acousticenvironments.
We show how DiffImpact’s differentiability allows us to learn meaningful soundmodels that are ﬁt to the sounds of multiple imperfect impact sounds over one or multiple recordings.
We ﬁrst validate our approach on single impact sounds.
We show that our model can ﬁt interpretablephysical parameters of an object’s sound model and of the contact forces to audio recordings.
We thenshow how DiffImpact enables a robot to passively learn object sound models from a real-world dataset(ASMR
YouTube videos [1]).
Our model not only extracts physically interpretable parameters, butalso outperforms model-based and model-free learning baselines in rendering realistic impact soundsfor held-out data.
Finally, we show how a robot can learn object sound models from data it collectedby striking these objects with a tool.
Speciﬁcally, we show how the robot can perform blind sourceseparation of the collected sound and thereby achieves 140% higher accuracy on classifying thematerial of the objects it strikes.
Related Work
Audio-frequency vibrations have been shown to be an effective modality for robots to ascertaincharacteristics of physical events in their environment, including instance level classiﬁcation ofeveryday objects [2, 3] and estimates of their motion [4], terrain classiﬁcation by legged robots [5],masses of granular materials being poured from a scoop the robot is holding [6], and amounts ofliquid poured by a robot [7].
Auditory scene analysis also has applications for dynamic inference ofstochastic events, enabling creation of audio based reactive robotic trackers [8].
Many of these works have a common characteristic with more recent visual perception approaches,that learning-based neural methods are subsuming many classical approaches by enabling automaticfeature extraction rather than requiring hand engineered features [9, 10].
However, whereasdifferentiable image renderers have empowered new approaches in visual perception [11, 12, 13, 14],differentiable audio renderers have been introduced only recently.
Engel et al.
[15] recently debutedthe Differential Digital Signal Processing (DDSP) library, showing applications in the musicaldomain on tasks such as disentangling the pitch and timbre of instrument sounds.
Rather than themusical domain, we focus on modeling modal impact sounds of rigid objects, and show applicationsof our physics-based model for robotics tasks.
Model-free neural audio representation methods are powerful in synthesizing realistic humanspeech [16, 17] but lack interpretability and ﬂexibility for transfer to different categories such astransient environment sounds.
Model-based methods often require less data and training time toproduce interpretable learned parameters, which can be reused and applied to other tasks moreintuitively.
They are also capable of synthesizing realistic object sounds [18, 19, 20] but depend onhand-engineered analytical methods for parameter estimation with structured audio samples.
Renet al.
[21] use real audio clips of striking objects to infer properties such as Young’s modulus and themodes of the object, then attempt to reproduce the sound based on a modal model with a residual.
However, they assume knowledge of the 3D geometry of the object as well as impact location.
Zhanget al.
[22] infer object properties such as Young’s modulus and Rayleigh damping from sounds ofobjects falling on a surface by using a physics-based modal sound generation engine with a blackbox forward model.
But their model is not fully differentiable, making it challenging to learn fromunconstrained in-the-wild audio or to optimize over many dimensions at once.
While we use amodel conceptually similar to that of [23, 22], our fully differentiable structure empowers learninga best estimate of an object’s modal model from the sound of multiple uncontrolled, imperfect,heterogeneous impacts, rather than requiring strictly controlled recordings for analysis.
Differentiable Rendering of Impact Sounds
Our DiffImpact models the sound that a rigid object makes when struck in a real environment as afunction of 1) the impact force’s magnitude over time; 2) the brief click of a sound pressure wavecaused by the rapid deceleration of the striking object, referred to as the “acceleration sound”; 3) thestruck and striking objects’ impulse responses or their idealized theoretical vibrational response to aperfect unit impulse; 4) any environment, background or recording noise; and ﬁnally 5) the impulseresponse of the local environment, reﬂecting and transmitting reverberations of any sound beingdirectly emitted by the object.
DiffImpact attempts to decompose real impact sounds into each of these contributions physics-based models in order to regularize this decomposition
3.1 Overview
Figure 1 provides a high-level overview of our approach.
There are some model variations dependenton which inference task we are considering.
For clarity, here we will give a brief overview of the modelwhenusedfortheend-to-endlearningtaskofpredictinganaudiowaveformfromamagnitudespectrogram of the ground truth audio sample, similar to the models in [15, 24, 16].
The magnitude spectrogram lacks phase information and is therefore not directly invertible to an audio waveform.
This spectrogram is input to an encoder network that outputs a coarsely temporally discretized time series of parameters that represent impact characteristics.
These parameters are the input to the Impact Force Proﬁle Generator that synthesizes an impact force proﬁle as described in more detail in Section 3.2.
Givenobject-speciﬁc, learnable parameters of a modal model of rigid body impact sound, the Modal Impulse
Response Generator synthesizes an impulse response as described in more detail in Section 3.3.
Thisimpulse response is convolved with the force proﬁle to synthesize object vibrations.
To model thecontribution of the sound pressure wave caused by the rapid deceleration of the impacting object, thisimpact force proﬁle is also multiplied by a learnable scalar and added to the object vibration sounds.
We theoretically now have all the sounds emitted directly from the point of impact on the object.
To model sound that is speciﬁc to the environment, we introduce a noise and reverb generator asdetailed in Section 3.4.
Given learnable noise parameters, the Filtered Noise Generator synthesizesrecording or background noise that is then added to the current audio.
Finally, given learnablereverb parameters, the Reverb Generator synthesizes an impulse response of the environment.
Thisresponse is then convolved with the current audio to model the sounds that have transmitted throughthe environment and reﬂected off nearby surfaces, rather than directly from the object.
This result isadded to the current audio to produce the ﬁnal waveform.
For training the aforementioned learnable parameters of the generators and the encoder, we use a lossthat compares spectrograms of the generated waveform with those of the ground truth (Section 3.5).
The ﬁnal waveform ˆ W ( t ) produced by our model can be expressed as
[Equation]
where t is the time in the ﬁnal waveform; F is the force proﬁle function that takes as input the impulsemagnitudes m , impulse timings t , and contact time scales ⌧ ; ~ denotes a convolution; ↵ is a scalarfor the acceleration sound; IR
o is the modal object impulse response function parameterized bymodal frequencies f , gains g o , and dampings d o , and t IR is time within each impulse response; Nis the ﬁltered noise parameterized by frequency band gains g n ; and IR e is the impulse response ofthe environment parameterized by frequency band gains g e and dampings d e .
We will now detailthe equations and physics-based models for each of these components we propose, and how we scalea low dimensional set of normalized parameters into input parameters for each of these functions,producing an audio frequency signal in a differentiable manner.
3.2 Differentiable impact force proﬁle and acceleration sound
To model the force proﬁles of multiple impact events occurring over the time interval of an audio clip,our proposed module converts the two time series outputs from the encoder into a contact force proﬁleat audio frequency.
This proﬁle is deﬁned by the time scales ⌧ and magnitudes m of the impulses.
First, we deﬁne a physical model for each impulse event.
Since objects will generally have convexcurvature at the points at which they strike, they can be locally approximated as spheres in contact.
Therefore, we adopt a Hertz half-sine contact model based on collisions between frictionless rigidspheres similar to that of [25].
To ensure smoothness, we use a Gaussian approximation C
i ( t ; t i , ⌧ i )of this contact model force impulse proﬁle [26, 27]:
[Equation]
where t i represents the timing of the onset of the unsmoothed impact, and ⌧ i is the time scale of thecontact force, which depends on the velocity of the impact and material properties of the object [28].
Sharper and faster contact points as well as harder materials will each contribute to reducing the valueof ⌧ i , while softer, slower, and duller contact points will have higher ⌧ i values.
Equation 2 provides a model for an individual contact event.
But in a natural audio clip, multiplesuch contact events, each with different characteristics, can occur over a given interval, such that anatural force proﬁle over time will be the sum of multiple such contact impulses, each with differentparameters and magnitudes, and each occurring at different instants during the interval.
We modelthe force proﬁle, the scalar value of the normal force over time, for multiple contacts as a weightedsum of such Gaussian impulses from Equation 2:
[Equation]
where t =
[ t 0 ,...,t M ] T , ⌧ = [ ⌧ 0 ,..., ⌧ M ] T and m =
[ m 0 ,...,m M ] T .
Each magnitude m i serves as theweight for the corresponding impact.
C ( t ; t , ⌧ ) stacks the output of Equation 2 into a vector.
Equation 2 is naturally differentiable, but a challenge to making Equation 3 fully differentiable is thatthe number of impacts occurring during an interval may not be be known a priori, and may vary between examples.
To address this, we propose an algorithm that ﬁnds M peaks in a time series of impactmagnitude estimates over time, and places model-based impulses at those peaks based on the parameters at each peak’s point in the time series.
If there are fewer than M peaks, the impact magnitudesm i at the extraneous peaks should be estimated as zero.
We detail this algorithm in Appendix A.1.
To model the acceleration sound, we must know the acceleration proﬁle of the striking tool’s tip.
Since the striking tool’s mass is ﬁxed, then the deceleration is proportional to the impulse force,which is already estimated in Equation 3.
Therefore, we model the contribution of the accelerationsound as the force proﬁle from Equation 3 multiplied by the learnable scalar variable ↵ .
3.3 Object impulse response
For the object impulse response, we adopt a modal model by modeling objects as spring-dampersystems, where the impulse response is a linear combination of exponentially decaying sinusoids,with each sinusoid having a different frequency f k , gain g k , and damping d k [18, 23].
We modelthe K most salient modes of the object through constructing a ﬁnite impulse response (FIR)as IR o
( t ; f , g o , d o ) =
g o T [exp(  d o t )  sin(2 ⇡ f t )] , where f = [ f 0 , ... , f K ] , g o =
[ g 0 , ... , g K ] ,d o =
[ d 0 ,...,d K ] and  is the Hadamard product.
Note that we make the simplifying assumption thatthe gains of each object are ﬁxed across the entire body of the object, whereas in reality for generalobjects, they vary signiﬁcantly depending on the location at which the object is impacted.
We madethis assumption to make the optimizations in our experiments more tractable since the inputs to ourframeworks were blind to the 3D models of the objects as well as the locations of each contact.
Wedetail how we scale normalized inputs to our module to produce the parameters in Appendix A.2.
3.4 Environment noise and reverberation
We approximate background and measurement noise as static ﬁltered noise, whereas we approximatereverberation from the measurement environment with exponentially decaying ﬁltered noise.
Forbackground and measurement noise, our ﬁltered noise function N ( t ; g n ) ﬁlters random white noisewith time-constant gains for each frequency band given by learnable noise parameters: N ( t ; g n ) =
g n T filt ( N ( t ) ,B n ) , where filt ( N ( t ) ,B n ) 2 R B n ⇥ T is a function that ﬁlters white noise N ( t ) with B n band pass ﬁlters spaced linearly over the interval of frequencies to the Nyquist frequency.
Theoutput matrix is multiplied with the vector g n weighting each frequency band.
For generating reverberations, we model the local environment’s response to sounds with an impulseresponse with exponentially decaying ﬁltered noise, where each noise band has its own decay rate,similar to [23]:
IR e ( t IR ; g e , d e ) =
g e T [exp(  d e t )  filt ( N ( t ) ,B e )] .
Here g e and d e are the gainsand damping factors per frequency band, respectively, and have dimension B e .
We describe how theseparameters are derived from scaling our learnable normalized reverb parameters in Appendix A.3.
3.5 Loss function
As a loss function for all tasks, we use the sum of the L 1 distance between the spectrograms and logspectrograms of multiple window sizes from the ground truth W and synthesized audio ˆ W [15, 24]:
L ( W, ˆ W )
[Equation]
where s w is the window size,  is the overlap ratio,  1 and  2 are weights, and S is the short-term
Fourier transform (STFT) or spectrogram.
Using multiple spectrogram scales ensures that the losscharacterizes an error signal in both high frequency and temporal resolution.
The log spectrogramsalso ensure that higher frequency components, which tend to have lower energy magnitude than lowfrequency components, are also weighted more strongly.
We simultaneously ﬁt estimates of parameters for both the sound the object makes when impacted, and the impacts that produce those sounds.
Results
We ﬁrst validate that DiffImpact is able to not only learn expressive sound models, but also to extractaccurate physically interpretable parameters from controlled recordings of impacting different householdobjects.
Thenweshowhowourcontributedmodulescanbeusedintwodifferentreal-worldtasks,one showing how a robot could use our models to passively learn sound models from data, and anothershowinghowourrobotcaninteractivelylearnsoundmodelsandusethemforusefuldownstreamtasks.
4.1 Analysis by synthesis: Fitting to single real impact sounds
To test whether our DiffImpact can extract contact forces from sound, we collected 3-second clipsof striking four different household objects (bottom right of Figure 4) in a recording studio withan impact hammer (PCB 086C01).
The impact hammer had a force transducer in its tip, providingground truth contact forces synchronized with the audio recorded by a microphone (PCB 378A06).
We randomly initialized the parameters for our model, and ﬁt them to each recording with respect toour loss function which only compares synthesized and ground truth spectrograms.
We use Adam asthe optimizer [29].
Our results are shown in Figure 2.
Though our model’s estimates of contact forcehave no sense of scale or calibration to a physical quantity (see Appendix E), we show that the forceproﬁles our model synthesized were quite similar in shape to those of the ground truth.
Note thatwe do not supervise directly on force proﬁles.
Furthermore, the spectrograms demonstrate that ourmodel is able to synthesize audio very similar to the ground truth.
Audio examples are included in the Supplementary Materials, and further details of this experiment and its setup are in Appendix B.
4.2 End to end learning: Learning rigid body impact sound models from in-the-wild audio
We demonstrate the utility of our modules in the end to end learning task of autoencoding realworld impact audio.
Similar to previous work [16, 15], the input to each model is the ground truthmagnitude spectrogram, and the output is the wave form of the audio.
Our experiments in this sectionare driven by evaluating perceptual realism and similarity of our model’s predictions, measured bothqualitatively by human user studies and quantitatively by perceptual loss metrics.
Dataset We curated a dataset of audio clips extracted from videos on the ASMR Bakery YouTube
Channel [1].
We selected clips from segments in which the creator is repeatedly tapping ﬁve objects of different materials: a wooden box, a glass bowl, a silicone pad, a steel bar, and a ceramic plate.
Screenshots from clips of each of the objects we selected are shown Appendix C.1.
With this dataset,we show that a robot could use our framework to learn meaningful, physically interpretable modelsof rigid objects from a rich real world dataset.
This dataset presents many challenges to the task ofestimating a modal sound model, challenges which prior works have not addressed.
First, whereasother works record the sound of a single controlled impact by a small pellet or specialized instrument [23, 18], many clips in our dataset include impacts randomly alternating between tapping withhard ﬁngernails and soft ﬁngertips within short timespans.
Some of these objects are being held in thecreator’s hand while they are being tapped, and other objects are lying on a table (see Appendix C.1).
Holding each object or resting it on a surface will dampen certain modes of vibration.
Therefore,at no point in the entire dataset does an object emit an ideal undamped modal response.
The ASMRdataset consists of around 3-5 minutes of continuously recorded audio of tapping for each object.
Wevalidated our model and its hyperparameters on a held out validation set of the steel bar clips, then splitremaining data for each object in approximately a 90-10 train-test split.
Thus, each model had 200seconds of training data for each object, and at least 20 seconds of unseen audio samples per object.
Baselines We implemented two baseline models based on DDSP [15, 24], which we refer to as“DDSP Serra” and “DDSP Sin.”
DDSP Serra is based on a time-varying harmonic model developedfor musical instruments [30].
To test the inﬂuence of the harmonic model bias, DDSP Sin replacesthe harmonic oscillator with an unrestricted time-varying sinusoidal oscillator.
We also tested
WaveGlow [16], a state of the art model-free approach for generating human speech.
Further detailson baselines are in Appendix C.2.
The WaveGlow baseline took more than two orders of magnitudemore GPU time to train ( ⇠ 1 hour vs. ⇠ 300 hours of training time for a single object on an Nvidia
Quadro P5000 GPU).
To demonstrate the necessity of this training time, we report results fromtraining this model for only ⇠ 1 order of magnitude more than the other models in Appendix C.4.
Forour evaluations, we trained a separate instance of each baseline and our model per object.
Qualitative evaluation through human studies To test DiffImpact’s capability of producing realistic impact sounds on held-out audio samples of each object compared to baselines, we re-dubbed theoriginal videos with each model’s output given the video’s original audio as input.
We then presentedparticipants on Amazon Mechanical Turk with a two-alternative forced choice to pick the best matching audio for 5-second videos.
The comparisons were between our model and either the baselines orground truth (1200 questions total).
Our model signiﬁcantly outperforms all the baselines on average(p < 0.05), as shown on the left of Figure 3.
Per object category, DiffImpact outperforms all baselinesfor the steel bar and ceramic plate.
It outperforms DDSP Serra and DDSP Sin models for all objectsexcept for the silicone pad (p < 0.05).
We suspect our model struggles with modeling the impact soundof the silicone pad due to the damping effect and roughness of the material as well as its high deformability that violates our model’s rigid body assumption.
The other models without these biases maybetter model its sound.
Furthermore, the user’s hand sometimes rubs against the surface during consequent tapping, resulting in unmodeled components in the audio which affect the performance of ourmethod.
On average, our participants chose our model’s audio over ground truth 42.9% of the time,suggesting that our generated audio is often realistic enough to be confused with ground truth audio.
Quantitative evaluation using distance metrics We also compared each model’s test outputs withthe ground truth audio using different computational distance metrics, including CDPAM, a learningbased perceptual distance metric designed for human speech applications [31], the multi-scale spectrogram loss used in [15, 24], and the envelope distance used in [32].
Since each of these metrics have widely different scales, we normalized each model’s average distance on each metric by the averagedistanceofwhitenoisewithrespecttothatmetric.
TheresultsareshownontherightofFigure3.
Theseresults align with our human studies, suggesting that our DiffImpact outperforms DDSP Serra and Sin,and it performs on par with WaveGlow, yet requires two orders of magnitude less training time.
Discussion We demonstrate that DiffImpact is expressive enough to characterize and generate realworld impact sounds, with performance competitive or better than state of the art alternatives.
Butperhaps most importantly, while completing this end-to-end learning task, our model is forced toextract physically interpretable and transferable parameters of impact forces and acoustic propertiesof the object and environment, whereas the parameters learned by other models are either uninterpretable (in the case of WaveGlow), or not directly meaningful to extracting impact event and intrinsicacoustic properties of objects.
These parameters can be used with a novel impact proﬁle to renderthe sound the object would make when impacted as such, creating a multimodal object model forsimulation environments [33, 34] merely from information learned from in-the-wild YouTube videos.
4.3 Robotic task: Source separation of kitchen utensil sounds
We use DiffImpact to perform a source separation task.
Unlike traditional audio source separation,where the goal is to isolate sounds in passive, pre-recorded audios and videos [35, 36], our goal is toenable robots to actively interact with objects in the environment and separate the sounds they make.
Consider the scenario where the robot strikes an object with a tool.
The perceived sound of the impactwill be a combination of the sounds the tool makes and the sounds the struck object makes.
We showthat by separating them, a robot can more accurately classify object materials, as different materialshave distinct acoustic properties.
Knowing the material of an object can inform the robot about itsdurability, surface friction, and category, all key factors to consider when planning grasps and actions.
Dataset We collected a dataset of a robot swinging four household tools (“tools”) at each of fourhousehold objects (“objects”) of four different materials, with each distinct material represented inexactly one element of each category.
We recorded each impact with a Shure SM57 microphone.
Photos of the robotic setup and the tools and objects are shown in Figure 4.
We ﬁltered out anysamples where the recording was too loud and clipped, which would obfuscate the frequency content,then used the two loudest remaining recordings of each pairwise interaction for our experiments, toensure a high signal to noise ratio.
Optimization setup We used the yellow elements of the model shown in Figure 1.
Compared tothe description in Section 3, we modiﬁed the model as follows: 1) instead of using the neural encoderto control impact force, the impact force proﬁle for each instance was a single impulse parameterizedby the Impulse trainable parameters, and 2) the Tool parameters were also used to produce a modalimpulse response which was convolved with the force proﬁle along with the Object modal impulseresponse.
The parameters for each tool and each object were optimized with respect to each of therecordings for which they were present.
All recordings shared the same set of Noise parameters, witheach having a single gain scalar for the noise.
We then performed analysis by synthesis on the entirebatch of 32 recordings at once, optimizing all of our trainable variables with the Adam optimizer withrespect to the loss that only compares ground truth with generated spectrograms of the non-separated sound.
For producing the separated object sound, we turned off the background noise, reverberation,acceleration noise, and the tool impulse response, then generated the model output for each recording.
For producing the separated tool sound, we turned off the background noise and the object impulseresponse, then generated the model output for each recording.
See Appendix D for more details.
Baselines We compared with non-negative matrix factorization (NMF) clustering [37] and themore recent “Deep Audio Prior” (DAP) [38] networks.
Similar to our framework, both baselinesrequire no pre-training.
Note that these baselines do not have a built-in way of specifying whichseparated source corresponds to the tool and which to the object, whereas ours does.
Metrics We provide qualitative examples of the separated sources in our Supplementary Materials.
Ground truth separated object sounds are not available to compare our results against, as there is norealistic way to even collect them in the real world.
Thus, for a quantitative comparison, we comparethe performance of material classiﬁcation on each framework’s outputs.
Speciﬁcally, we train anaudio-based material classiﬁer on balanced data of only the relevant materials from the Sound-20Kdataset, which consists of only synthetic sounds of objects of different materials bouncing on surfacesin a virtual environment [39].
We tested this trained classiﬁer on the original sound as well asthe source-separated output of each framework, to test whether the separated audio can be moreaccurately classiﬁed by this trained material classiﬁer.
[Table]
Results We report results in Table 1.
Because our baselines have no constraint on which separated source isthe tool and which is the object, weshow the results of two evaluationschemes.
Under the “Dataset Max”scheme, we compute the object material accuracy twice for each baseline:ﬁrst treating their ﬁrst output as theobject, then treating the second as the object.
We use the higher between the two as their result.
We do the same for tools.
Under the “Inst.
Max” scheme, we take the maximum for each instanceinstead of across the whole dataset.
While both schemes offer the baselines an advantage, our DiffImpact still outperformed them, while also explicitly differentiating between the object and the tool.
Allmodels, including ours, performed better for classifying objects than tools.
Object sounds dominatedthe recordings over tool sounds, since they were much larger, with more surface area for dispersingvibrations.
Their size also gave them stronger low frequency modes with lower dampings, lastinglonger in recordings, and thus easier to capture and characterize.
See Appendix D for confusion matrices from this task, as well as results from a similar experiment where our DiffImpact outperformsthe same baselines for estimating Young’s modulus of each object and tool from separated sounds.
Conclusion
We have proposed DiffImpact, a fully differentiable physics-based framework for modeling rigid object impact sounds.
This model enables robots to learn valuable physically interpretable parametersfrom sounds of objects in their everyday environments.
We ﬁrst validated that our method successfully simulates impact sounds of different everyday objects, and that it can be used in an analysis-bysynthesis approach to estimate impact forces that generated the analysed sound.
We also showed howour framework can be used as a neural rendering layer, learning physically interpretable parametersof object sound models during an end-to-end learning task from a real-world dataset.
We evaluatedits performance against state of the art audio learning baselines with both a human study and distancemetrics, and found our model to be more expressive and resource efﬁcient than alternatives for modeling real-world impact sounds.
Finally, we showed how a robot can use our model to learn soundmodels of everyday objects in its environment by striking them in a noisy environment with everyday tools.
The robot used our model to differentiate between the sounds of the tools it used and theobjects it struck with them and could classify the materials of the objects it strikes from sound 140%more accurately than from the original, unseparated audio or from generic audio source separationbaselines.
The knowledge robots can use our model to ascertain about objects, materials, and contactforces will be valuable to future applications which use our approach in robot decision making andpolicy learning.
As future work, we aim to expand the scope of our model to other contact events, suchas rubbing and scratching, or other object acoustic events, such as bouncing on different surfaces orfracturing.
We also intend to use our contributions as a foundation for models and applications whichuse other multimodal inputs, such as vision and tactile data.