Learning Neural Acoustic Fields
Abstract
Our environment is filled with rich and dynamic acoustic information.
When we walk into a cathedral, the reverberations as much as appearance inform us of the sanctuary’s wide open space.
Similarly, as an object moves around us, we expect the sound emitted to also exhibit this movement.
While recent advances in learned implicit functions have led to increasingly higher quality representations of the visual world, there have not been commensurate advances in learning spatial auditory representations.
To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene.
By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds.
We demonstrate NAFs on both synthetic and real data, and show that the continuous nature of NAFs enables us to render spatial acoustics for a listener at arbitrary locations.
We further show that the representation learned by NAFs can help improve visual learning with sparse views.
Finally we show that a representation informative of scene structure emerges during the learning of NAFs.
Project site: https://www.andrew.cmu.edu/user/afluo/Neural_Acoustic_Fields
Introduction
The sound of the ball leaving the bat, as much as its visible trajectory, tells us whether the hit is likely to be a home run or not.
Our experience of the world around us is rich and multimodal, depending on integrated input from multiple sensory modalities.
In particular, spatial acoustic cues provide us with a sense of the direction and distance of a sound source without needing visual confirmation, allow us to estimate the properties of a surrounding environment, and are critical to subjective realism in gaming and virtual simulations.
Recent progress in implicit neural representations has enabled the construction of continuous, differentiable representations of the visual world directly from raw image observations [Sitzmann et al., 2019, Mildenhall et al., 2020, Niemeyer et al., 2020, Yariv et al., 2020].
However, our perception of the physical world is informed not only by our visual observations, but also by the spatial acoustic cues present in the environment.
As a preliminary step in learning the acoustic properties of scenes, we explore an implicit model that represents the underlying impulse response of audio reverberations.
As shown in Figure 1, our model can model the spatial propagation of sound in a physical scene.
Past work has explored capturing the underlying acoustics of a scene
[Raghuvanshi and Snyder, 2014, 2018, Chaitanya et al., 2020].
These models, however, require handcrafted parameterizations which,
[Figure]
critically, prevent such approaches from being applied to arbitrary scenes.
In this work, we extend this approach by constructing an implicit neural representation which captures, in a generic manner , the underlying acoustics of a scene.
Learning a representation of scene acoustics poses several challenges compared to the visual setting.
First, how do we generate plausible audio impulse responses at each emitter-listener position?
While we may represent the visual appearance of a scene with an underlying three-dimensional vector, an acoustic reverberation (represented as an impulse response) can consist of over 10,000 time-domain values and, thus, is significantly harder to capture.
Second, how do we learn an acoustic neural representation that densely generalizes to novel emitter-listener locations?
In the visual setting, ray-tracing can enforce view consistency across large portions of a visual scene (modulo occlusions).
While in principle, in a similar manner, we may reflect acoustic "rays" in a scene represented as an implicit function to obtain an impulse response, an intractable amount of compute is necessary to obtain the desired representation [Srinivasan et al., 2021].
To address both challenges, we propose Neural Acoustic Fields (NAFs).
To capture the complex signal representation of impulse responses in a compact and spati ally continuous fashion, NAFs encode and represent an impulse-response in the time-frequency domain.
Motivated by the strong influence of nearby geometry on anisotropic reflections [Raghuvanshi and Snyder, 2018], we propose to condition NAFs on local geometric information present at both the listener and emitter locations when decoding the impulse response.
In our framework, local geometric information is learned directly from impulse responses.
Such a decomposition facilitates the transfer of local information captured from training emitter-listener pairs to novel combinations of emitters and listeners.
We show that NAFs are able to outperform baselines in modeling scene acoustics, and provide detailed analysis of the design choices in NAFs.
We further illustrate how the structure learned by
NAFs can improve cross-modal generation of novel visual views of a scene.
Finally, we illustrate how the learned representation of NAFs enable the downstream application of inferring scene structure.
Related Work
Audio Field
Coding There is a rich history of sound field representation, encoding, and interpolation methods for 3D spatial audio.
Some approaches seek to directly approximate the sound field [Mignot et al., 2013, Antonello et al., 2017, Ueno et al., 2018] while adding handcrafted priors.
Others adopt a parametric representation that seek to model only the perceptual cues [Raghuvanshi and Snyder, 2014, 2018, Chaitanya et al., 2020, Mehra et al., 2014, Ratnarajah et al., 2021].
Since the complete acoustic field of a scene is computationally prohibitive to simulate in real time, and expensive to store in full fidelity, these methods have typically relied on a handcrafted encoding of the acoustic field, prioritizing efficiency above reproduction fidelity.
In recent years, there has been interest in using deep learning to directly learn a sound field from data, without making strong assumptions about the scene.
However in practice, these approaches use either a stationary listener or emitter [Richard et al., 2020, 2022].
In contrast, our work enables the querying of the sound field for arbitrary emitter and listener locations.
Implicit representations Our approach towards modeling the underlying acoustics a scene relies on the use of a neural implicit representations.
Implicit representations have emerged as a promising representation of 3D geometry [Niemeyer et al., 2019, Chen and Zhang, 2019, Park et al., 2019, Saito et al., 2019, Hong et al., 2022] and appearance [Sitzmann et al., 2019, Mildenhall et al., 2020, Niemeyer et al., 2020, Yariv et al., 2020, Wang et al., 2021] of a scene.
Compared to traditional discrete representations, implicit representations are a continuous mapping capable of capturing data
[Figure] at an "infinite resolution". In [Jiang et al., 2020] proposed a grid based representation for implicit scenes, while more recently [DeVries et al., 2021] has adopted spatial conditioning for 3D image synthesis, where in both settings, the grid enables a higher-fidelity encoding of the scene.
Our work also leverages local grids to model acoustics, but as an inductive bias and way to generalize to novel inputs.
Audio-Visual Learning Our work is also closely related to joint modeling of vision and audio.
By leveraging the correspondence between vision and audio, work has been done to learn unsupervised video and audio representations [Aytar et al., 2016, Arandjelovic and Zisserman, 2017], localize objects that emit sound [Senocak et al., 2018, Zhao et al., 2018], and jointly use vision and audio for navigation [Chen et al., 2020].
Recent work aims to propose plausible reverberations or sounds from image input [Singh et al., 2021, Du et al., 2021], these approaches model the STFT using either convolution or implicit functions, which we also utilize.
Different from them, our work leverages the geometric features learned by modeling acoustic fields to improve the learning of 3D view generation.
Methods
We are interested in learning a generic acoustic representation of an arbitrary scene, which can capture the underlying sound propagation of arbitrary sound sources across both seen and unseen locations in a scene.
We first review relevant background information towards modeling environment reverberations.
We then describe Neural Acoustic Fields (NAFs), a neural field which we show can capture, in a generic manner, the acoustics of arbitrary scenes.
We further discuss how we can parameterize NAF so that it can capture acoustics property even at unseen sound sources and listener positions.
Finally, we discuss the implementation details of our model illustrated in Figure 2.
3.1 Background on the Propagation of Sound
The sound emitted by a sound source undergoes decay, occlusion, and scattering due to both the geometric and material properties of a scene.
For a fixed location pair ( q , q ′ ) , we define the impulseresponse at a listener position q , as the sound pressure p ( t ; q , q ′ ) induced by an impulse at q ′ .
Given an accurate model of the impulse-response p ( t ; q , q ′ ) , we may model audio reverberation of any sound waveform s ( t ) emitted at q ′ , by computing the response r ( t, q , q ′ ) at listener location q by querying the continuous field and using temporal convolution:
[Equation]
[Figure]
e qu e n c y
Time
3.2 Neural Acoustic Fields
We are interested in constructing a continuous representation of the underlying acoustics of a scene, which may specify the reverberation patterns of an arbitrary sound source.
The parameterization of an impulse-response introduced in Section 3.1 provides us with a method to model audio propagation when given an omnidirectional listener and emitter.
To construct a model of a directional listener, we need to further model the 3D head orientation θ ∈ R 2 , and ear k ∈ { 0 , 1 } (binary left or right) of a listener, in addition to the spatial position q ∈ R 3 of the listener and q ′ ∈ R 3 of the emitter.
We may then model the time domain impulse response v using a neural field Φ which takes as input the listener and emitter parameters:
[Equation]
Directly outputting the impulse-response waveform v ∈ R T in the time domain with a neural network is difficult due to its high dimensional (over 10,000 elements) and chaotic nature.
A naïve solution would be to further add t as an additional argument to our neural field, but we found that such a solution worked poorly, due to the highly non-smooth representation of the waveform (see supplementary).
We instead encode the impulse-response utilizing a short-time Fourier transform (STFT) denoted v STFT , which we find to be significantly more amenable to neural network prediction due to the smoother nature of the time-frequency space.
In Figure 3 we show magnitude spectrograms for ground truth impulse responses and those learned by our network.
As v STFT is a complex value, we further factorize v STFT into log-magnitude and phase angle components.
For phase angle, we use the instantaneous frequency (IF) representation proposed in GANSynth [Engel et al., 2019].
To compute the IF representation, the phase angle is unwrapped and has the finite difference taken across the time dimension for each frequency in the STFT.
This transformation results in a phase representation that conducive to learning due to more regular structure.
Thus, our parameterization of NAF is a neural field Ω that is trained to estimate the impulse response function ϕ , and outputs [ v STFT_mag , v STFT_IF ] for a given time and frequency coordinate:
[Equation]
We train our model using MSE loss between the generated and ground truth spectrograms v STFT :
[Equation]
across spectrogram coordinates t and f .
Where α is a scaling value used to balance the two losses.
3.3 Generalization through Local Geometric Conditioning
We are interested in parameterizing the underlying acoustic field, so that we may not only accurately represent impulse-response at emitter-listener pairs we see during training, but also at novel combina- [Figure] tions of emitter and listener seen at test time.
Such generalization may be problematic when directly parameterizing NAFs utilizing a MLP with inputs specified in Eqn (3), as the network may learn to directly overfit and entangle the relation between emitter and listener impulse-responses.
What generic information may we extract from a given impulse-response between an emitter and listener?
In principle, extracting the full dense geometric information in a scene would enable us to robustly generalize to new emitter and listener locations.
However, the amount of geometric information available in a particular impulse-response, especially for positions far away from either current emitter and listener is limited, since these positions have little impact on the underlying impulse-response.
In contrast, the local geometry near either emitter and listener positions will have a strong influence in the impulse-response, as much of the anisotropic reflection comes from such geometry [Paasonen et al., 2017].
Inspired by this observation, we aim to capture and utilize local geometric information, near either emitter or listener locations, as a means to predict impulseresponses across novel combinations.
To parameterize and represent these local geometric features, we learn a 2D grid of spatial latents which we illustrate in Figure 2.
The spatial latents are randomly initialized and uniformly distributed in the room.
When predicting an impulse-response at a given emitter and offset position, we query the learned grid features at both emitter and listener positions, and provide it as additional context into our NAF network Ω .
Such features provide rich information on the impulse-response, enabling
NAF to generalize better to unseen combinations of both emitter and listener locations.
In the rest of this work, we refer to the NAFs with local geometric features as Ω grid .
We learn grid latent features jointly with the underlying parameters of NAF.
Additional details can be found in the supplementary.
Such a design choice, however, still requires us to consider how to further combine local geometric information captured separately from either listeners or emitters.
A naïve implementation would be to maintain separate feature grids for both listener and emitter positions.
Such an approach fails to account for the fact that the local geometric information captured by emitter may also inform the local geometric information around a listener.
Examining Green’s function, which is the solution to the wave equation, we note that it is in fact symmetric with respect to exchanging either listener or emitter positions [Chaitanya et al., 2020], indicating that the impulse-response does not change when omnidirectional emitters and listeners are swapped (acoustic reciprocity).
Such a result means that we may in fact utilize the local geometric information captured near an emitter position interchangeably for either emitters and listeners.
Thus, we propose our local geometric information as a single latent grid, which we show to outperform the naïve dual grid implementation.
Experiments
In this section, we demonstrate that our model can faithfully represent the acoustic impulse response at seen and unseen locations.
Additional ablation studies verify the importance of utilizing local geometric features to enable test time generation fidelity.
Next, we demonstrate that learning acoustic fields could facilitate improved visual representations when training images are sparse.
Finally we show that the learned NAF can be used to infer scene structure. [Figure]
4.1 Setup
For each scene, we holdout 10% of the RIRs randomly as a test set.
Each scene is trained forepochs, which takes around 6 hours for the largest scenes on four Nvidia V100s .
In each batch, we sample 20 impulse responses, and randomly select 2 , 000 frequency & time pairs within each spectrogram.
An initial learning rate of 5 × 10 − 4 is used for the network and the grid features.
We add a small amount of noise sampled from N (0 , 0 . 1) to each coordinate during training to prevent degenerate solutions.
For evaluating the learned acoustic fields, we use two different datasets:
Soundspaces.
Soundspaces [Chen et al., 2020, Straub et al., 2019] is a synthetic dataset generated via ray-tracing, and is auralized with an ambisonic head-related transfer function (HRTF).
This dataset consists of R i probe points for each scene, with each probe capable of representing an emitter or listener location for up to R 2 i emitter and listener pairs.
The emitters are represented as omnidirectional, while the listener acts as a stereo receiver that can have one of four different orientations.
The listeners and emitters are at fixed height.
Our NAFs are trained on 6 representative scenes, where 2 consist of multi-room layouts; 2 consist of a single room with a non-rectangular walls; and 2 consist of a single room with rectangular walls as in Figure 4.
MeshRIR.
The MeshRIR dataset [Koyama et al., 2021] is recorded from a real scene, and contains monaural data collected from a cuboidal room.
The listener locations are at fixed height.
The emitters surround the listeners both above and below the listener plane.
4.2 Architecture Details
The Soundspaces dataset lacks the full parameterization of an acoustic field described in Equation 3, so we train NAF with a restricted parameterization that is available in the dataset.
This allows for two degrees of freedom along the x − y plane for the listener locations q ∈ R 2 and the emitter location q ′ ∈ R 2 .
The listener is binaural with k ∈ { 0 , 1 } , and can assume four possible orientations θ ∈ { 0 , 90 , 180 , 270 } , while the emitter is omnidirectional.
In particular, we utilize a parameterization of Ω grid which maps an input tuple [ x, y, x ′ , y ′ , f, t ] × { 0 , 90 , 180 , 270 } × { 0 , 1 } to two scalar values that represents the magnitude and phase for a given time and frequency in the STFT:
[Equation]
To encode the rotation θ , as there are only 4 possible discrete rotations in the dataset, we directly query into a learnable embedding matrix of shape R 4 × n , returning a R 1 × n vector.
Similarily, to encode the left and right ear k , we similarly query into a learnable embedding matrix of shape R 2 × n , returning a R 1 × n vector.
The f, t tuple representing the frequency and time respectively are scaled to ( − 1 , 1) and processed with sinusoidal encoding using 10 frequencies of sin and cos .
For MeshRIR, we set emitter q ′ ∈ R 3 to account for the emitters that can vary in height, and do not utilize the orientation or binaural embedding.
To obtain local geometric features for either an emitter or listener in a scene, we assume that our scene is contained within a set of pixels P
= { P 1 ...P k } which form a grid over the scene.
For a given position tuple ( x, y ) as query location, we then interpolate the local features.
Where L ( · ) is the interpolation function.
( p ∗ 1 . . .
p ∗ k ) are the set of all pixel that form the grid, and ˜ f ( · ) represents the features stored at a given pixel:
[Equation]
w i is determined by a Nadaraya-Watson estimator with a Gaussian weighting kernel applied to the distance between query and grid coordinates:
[Equation]
Because this interpolation function is differentiable, we jointly learn the grid features during training.
These queried features are combined with the coordinates processed with sinusoidal encoding using 10 frequencies of sin and cos functions.
We process both the listener and emitter position tuples this way.
We combine the grid based features with the sinusoidal embeddings and the discrete indexed embeddings as the input to our multilayer perceptron f ϕ .
Please refer to Figure 2 for a visualization of our model, and supplementary for further details.
We compare using a shared local geometric feature with the emitter and listener, as well as having the emitter and listener query their own individual grids.
4.3 Evaluation of Neural Acoustic Fields
We first validate that we can capture environmental acoustics at unseen emitter-listener positions.
[Table]
Baselines.
We compare our model against two widely used high performance audio coding methods: Advanced Audio Coding (AAC) and Xiph Opus.
We use low bitrates in order to attempt to approach the storage costs for our NAF.
For each method, we apply both linear and nearest neighbor interpolation to the coded acoustic fields.
Both linear and nearest neighbor approaches are widely used [Savioja et al., 1999, Raghuvanshi et al., 2010, Pörschmann et al., 2020] in modeling of spatial audio.
We further implement the binaural DSP baseline described by [Richard et al., 2020], which uses the image source method and a KEMAR HRTF.
We also compare the listener and emitter either sharing or using individual local geometric features in our NAFs.
Each method is provided with the same train-test split.
We visualize the acoustic fields produced by different methods in Figure 5.
Details of our baselines can be found in the supplementary section G.
Metrics.
We evaluate the results of our synthesis by measuring the spectral loss [Défossez et al., 2018] between the generated and the ground truth log-spectrograms, as well as measuring the percentage error between the T60 reverberation time in the time domain.
In this case, lower spectral loss and T60-error values indicates a better result.
Additional quantitative results can be found in the supplementary.
We also perform a human evaluation where subjects are presented with a two-alternative forced-choice task.
Each trial requires selecting if the NAFs or the AAC-nearest auralized music samples best match with ground truth auralization.
Results.
As shown in Table 2, our NAFs achieve significantly higher quality on the modeling of unseen impulse responses compared to strong interpolation baselines across all six scenes.
Observing the qualitative results in Figure 4, we observe that NAFs can predict smoothly varying acoustic fields that are affected by the physical surroundings.
Extending our model to MeshRIR which is captured from a real scene, we observe that our NAFs continue to perform better on both spectral and T metrics.
Comparing our results against baselines in Figure 5 and Table 1, our methods are able to better approximate the ground truth at a fraction of the storage cost, and does not exhibit the sound energy leakage present in linear interpolation.
The size of a spatial acoustic field is important for real life applications.
In our human evaluation with 21 subjects who were asked to judge 10 test-time
RIRs, 82.38% of responses indicates that our NAFs were higher quality compared to AAC-Nearest. [Table]
[Table] [Table] [Table] A comparison of using shared and dual local geometric features indicates that despite having fewer learnable parameters, we achieve better performance by sharing the local geometric features.
Examples of individual impulse responses generated by our model are shown in Figure 3.
Generalization through Geometric Conditioning.
We next assess the impact of utilizing local geometric conditioning as a means to generalize to novel combinations of emitterlistener positions.
On the "Large 1" room, in Figure 7 we evaluate test set spectral error when NAF is trained with a limited percentage of the training data either with or without local geometric conditioning.
We find that such geometric conditioning enables better test set reconstruction error, with the performance gap increasing with less data.
[Figure]
y
[Figure]
+ A c ou s ti c s
[Figure]
4.4 Cross-modal learning
In this experiment we explore the effect of jointly learning acoustics and visual information when we are given sparse visual information.
Recall that our NAF includes a local geometric feature grid
P that covers the entire scene.
For our cross-modal learning experiment, we jointly learn this feature grid with a NeRF network modified to accept local features sampled from this grid along with the traditional sinusoidal embedding.
In the acoustics branch, we query the grid using emitter and listener positions.
In the NeRF branch, we use point samples along the ray projected on the grid plane to query the features.
In both cases, the process is fully differentiable.
We use a standard implementation of NeRF with a coarse and fine network.
[Figure]
In the NeRF only setting, we minimize color C reconstruction loss for a ray r over a batch of rays R :
L RGB =
(cid:80) r ∈R
|| ˆ C ( r ) − C ( r )
|| 22 .
In contrast, in the NAF + NeRF experiment, we jointly minimize
L RGB + L NAF , where L NAF is defined in equation 4.
We utilize 64 coarse samples and 128 fine samples for each ray, and sample 1024 rays per batch.
Results.
We train on the two large rooms in our training set.
For each room 75 , 100 , 150 images are used for training, while the same 50 images of novel views are used for all configurations during testing.
In Table 3 we observe that training with acoustic information helps improve the PSNR and
SSIM of the visual output.
This effect is more significant when the training images are very sparse, the NAF network helps less when there is sufficient visual information.
Qualitative results are shown in Figure 6, we see there is a reduction of floaters in free space.
4.5 Inferring scene structure
Given a reverberant sound, humans are able to build a mental representation of the surrounding room and make a judgement about the distance of nearby obstacles Kolarik et al.
We investigate the intermediate representations constructed by our neural network in the process of learning an acoustic field, and examine if these representations can be used to decode the scene structure.
Setup.
The intermediate representation of the NAF depends on both listener locations q and emitter location q ′ , the rotation angle θ , the ear k , the time t and frequency f .
For consistency, at a given location ( x ∗ , y ∗ ) in the scene, we extract the NAF latent by setting the emitter location q ′ i =
( x ∗ , y ∗ ) i .
For the listener location, we iterate over five randomly selected points in the scene q ∈
[ q 1 , . . . , q 5 ] , which we keep constant for all q ′ i .
The rotation angle is fixed to θ = 0 , and we compute the representation average over all possible ( k, t, f ) , and concatenate latents for the selected q .
For our
NAFs, latents are extracted from the last layer prior to the output which includes 512 neurons.
As a
Explained variance
[Table]
comparison to our learned representation, we extract Mel-frequency cepstral coefficients (MFCCs) from the ground truth impulse response provided by a nearest neighbor interpolator.
We use a similar setup as above, for a given location we set this to be q ′ i , and iterate over the same five listener locations q 1 ... 5 .
We average the MFCCs over the left and right ear, and concatenate for the selected q .
After flattening, the MFCC features are approximately 500 dimensional for any given room.
We fit a single linear layer to NAF and MFCC features respectively.
For testing and visualization of the linear decoding results, we sample a regular grid of points with 0 .
1 m distance between each point.
For fitting the linear decoder, we randomly sample points within the scene such that the number of training points are 10% as many as the testing points.
For each location in the scene, we extract the distance to the nearest wall as the decoding target.
Results.
We visualize the results of our linear decoding in Figure 8.
The intermediate representation of our NAFs reveals an underlying structure that is both smooth and semantically meaningful.
In the multiroom scenes, the latent is well separated for each room.
We are able to successfully decode the scene structure with a linear layer when using our NAFs, but decoding fails when using MFCC features.
In Table 4, we show the amount of explained variance of our decoding results on the test set.
Our learned features are able to consistently achieve much higher scores than those using MFCC features.
Discussion
Limitations and Future Work.
Although our method achieves generalization and high quality representations of acoustic fields within a single scene, NAFs do not currently generalize to multiple scenes.
Future work may explore generalization to novel scenes.
One possible approaches may be to incorporate multi-modal inputs with the goal of synthesizing an acoustic field with few-shot visual or acoustic input.
Societal impact.
Our work focuses on learning a high quality representation of acoustic fields.
The primary use case for our work lies in virtual reality and gaming.
As our work can lead to more believable and higher quality representations of spatial audio than alternative methods, it is possible that our work could increase the dependency and time spent on gaming.
The more compact nature of our acoustic representations may allow for spatial audio to be deployed to more systems, and enable more equitable access.
Conclusion.
In summary, this paper introduces Neural Acoustic Fields (NAFs), a compact, continuous, and differentiable acoustic representation which can represent the underlying reverberation of different audio sources in a scene.
By conditioning NAFs locally on the underlying scene geometry, we demonstrate that our approach enables the prediction of plausible environmental reverberations even at unseen locations in the scene.
Furthermore, we demonstrate that the acoustic representations learned through NAFs are powerful, and may be utilized to facilitate audio-visual cross-modal learning, as well as to infer the structure of scenes.
Acknowledgments and Disclosure of Funding
We would like to thank Leila Wehbe for providing feedback on our paper.
This work was supported by the MIT-IBM Watson AI Lab, Amazon Research Award, ONR MURI, DARPA Machine Common
Sense program, ONR (N00014-18-1-2847), NIDCD R01 DC015988, Mitsubishi Electric, and the
Tianqiao and Chrissy Chen Institute.