L EARNING N EURAL A COUSTIC F IELDS
Anonymous authors
Paper under double-blind review
A BSTRACT
Our sensory perception of the world is rich and multimodal.
When we walk into a cathedral, acoustics as much as appearance inform us of the sanctuary’s wide open space.
Similarly, when we drop a wineglass, the sound immediately informs us as to whether it has shattered or not.
In this vein, while recent advances in learned implicit functions have led to increasingly higher quality representations of the visual world, there have not been commensurate advances in learning auditory representations.
To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene.
By modeling the acoustic properties of the scene as a linear time-invariant system, NAFs continuously map all emitter and listener location pairs to an impulse response function that can then be applied to new sounds.
We demonstrate that
NAFs capture environment reverberations of a scene with high ﬁdelity and can predict sound propagation for novel locations.
Leveraging the scene structure learned by NAFs, we also demonstrate improved cross-modal generation of novel views of the scene given sparse visual views.
Finally, the continuous nature of
NAFs enables potential downstream applications such as sound source localization.
Qualitative results: sites.google.com/view/nafs-iclr-2022/ .
I NTRODUCTION
The sound of the ball leaving the bat, as much as its visible trajectory, tells us whether the hit is likely to be a home run or not.
Our experience of the world around us is rich and multimodal, depending on integrated input from all of ﬁve sensory modalities – vision, touch, smell, hearing, and taste.
Understanding underlying scene structure relies on a continuous and diverse set of inputs: the visual appearance and geometry of the physical world, the way sounds bound off of or are blocked by surfaces, the way odors diffuse across the local environment, and the way objects feel as we walk on them or touch them.
Recent progress in implicit neural representations has enabled the construction of continuous, differentiable representations of the visual world directly from raw image observations (Sitzmann et al., 2019;
Mildenhall et al., 2020; Niemeyer et al., 2020; Yariv et al., 2020).
These models typically utilize a neural renderer in combination with a learned implicit representation to jointly capture and render images of a scene.
By leveraging the multiview consistent nature of visual observations, these methods can infer images of the same scene from novel viewpoints once trained on sparse views.
However, our perception of the physical world is informed not only by our visual observations, but also by the acoustics of environmental sounds.
When hearing a highly reverberant echo (e.g., as in a cathedral), we can immediately picture an open space consisting of lofty walls and high ceilings.
Moreover, given only the acoustic properties of sounds in this space, we may infer a variety of other properties of the environment, for example, the location of the emitted sound, the underlying room layout, and the material compositions of the walls, ﬂoors, and ceiling. As a preliminary step in acoustic modeling, we explore modeling the underlying impulse response of audio reverberations.
Past work has explored capturing the underlying acoustics of a scene (Raghuvanshi & Snyder, 2014; 2018; Chaitanya et al., 2020).
These models, however, require manually designing acoustic functions which, critically, prevent such approaches from being applied to arbitrary scenes.
In this work, we extend this approach by constructing an implicit neural representation which captures, in a generic manner , the underlying acoustics of a scene.
In particular, following (Raghuvanshi & Snyder, 2014), we deﬁne the acoustic modeling problem as modeling the impulse-response a listener receives given
[Figure]
a sound emitted at an emitter location (as illustrated in Figure 1) and across all possible emitterlistener pairs in a scene.
This learned representation inherently captures the pattern of all acoustic reverberations in a scene.
Learning a representation of scene acoustics poses several challenges compared to the visual setting.
First, how do we represent, in high ﬁdelity, the underlying impulse response at each emitter-listener position?
While we may represent the visual appearance of a scene with an underlying threedimensional vector, an acoustic reverberation (represented as an impulse response) can consist of over 20000 values and, thus, is signiﬁcantly harder to capture.
Second, how do we learn an acoustic neural representation that densely generalizes to novel emitter-listener locations?
In the visual setting, ray-tracing can enforce view consistency across large portions of a visual scene (modulo occlusions).
While in principal, in a similar manner, we may reﬂect acoustic "rays" in our scene to obtain an impulse response, a intractable number of rays are necessary to obtain the desired representation.
To address both challenges, we propose Neural Acoustic Fields (NAFs).
To capture the complex signal representation of impulse responses, NAFs encode and represent an impulse-response in the
Fourier frequency domain.
Motivated by the strong inﬂuence of nearby geometry on anisotropic reﬂections (Raghuvanshi & Snyder, 2018), we propose to condition NAFs on local geometric information present at both the listener and emitter locations when decoding the impulse response. In our framework, local geometric information is learned directly from impulse responses.
Such a decomposition facilitates the transfer of local information captured from training emitter-listener pairs to novel combinations of emitters and listeners
By modeling the dense acoustic ﬁelds of an environment, NAFs are a powerful method to extract structural information about the scene, which we demonstrate is useful across a variety of different tasks.
In the cross-modal setting, we demonstrate how the learned acoustic structure can be utilized to aid learned visual representations, improving novel view synthesis.
Further, by directly utilizing the acoustic information in NAFs, we demonstrate how one can localize different sound sources.
In summary, we present Neural Acoustic Fields (NAFs), a neural implicit representation which captures the underlying acoustics of a scene in a compact and spatially continuous fashion.
We show that NAFs are able to outperform baselines in modeling scene acoustics, and provide detailed analysis of the design choices in NAFs.
We further illustrate how the structure learned by NAFs, can improve cross-modal generation of novel visual views of a scene.
Finally, we illustrate how the continuous nature of NAFs enable the downstream application of sound source localization.
R ELATED W ORK
Audio Field
Coding There is a rich history of encoding methods for 3D spatial audio.
These approaches largely fall into two categories.
The ﬁrst approach encodes the sound ﬁeld at a user- centric location by capturing the sound from spatially distributed sources (Gerzon, 1973;
Breebaart et al., 2005; Pulkki, 2007).
While they may leverage perceptual cues that create the sense of spatial audio, they do not allow the listener to freely traverse the scene and experience sound from different locations.
The second approach aims to model the sound heard as a listener moves in a scene (Raghuvanshi & Snyder, 2014; Mehra et al., 2014; Raghuvanshi & Snyder, 2018; Chaitanya et al., 2020).
Since the complete acoustic ﬁeld of a scene is computationally prohibitive to simulate in real time, and expensive to store in full ﬁdelity, these methods have relied on a handcrafted encoding of the acoustic ﬁeld, prioritizing efﬁciency above reproduction ﬁdelity.
Our work allows a listener to move and experience sounds that come from anywhere in a scene, and can represent the acoustic ﬁeld continuously at high ﬁdelity by directly learning from data.
Implicit representation Our approach towards modeling the underlying acoustics a scene relies on the use of a neural implicit representations.
Implicit representations have emerged as a promising representation of 3D geometry Niemeyer et al.
; Chen & Zhang (2019); Park et al.
Saito et al.
and appearance Sitzmann et al.
; Mildenhall et al.
(2020); Niemeyer et al.
; Yariv et al.
(2020) of a scene.
Compared to traditional discrete representations, implicit representations are a continuous mapping capable of capturing data at an "inﬁnite resolution".
Jiang et al.
(2020) proposed a grid based representation for implicit scene reconstruction, while more recently DeVries et al.
(2021) has adopted spatial conditioning for 3D image synthesis, where in both settings, the grid enables a higher-ﬁdelity encoding of the scene.
Our work also leverages local grids to model acoustics, but as an inductive bias and way to generalize to novel inputs.
-Visual Learning Our work is also closely related to joint modeling of vision and audio.
By leveraging the correspondence between vision and audio, work has been done to learn unsupervised video and audio representations (Aytar et al., 2016; Arandjelovic & Zisserman, 2017), localize objects that emit sound (Senocak et al., 2018;
Zhao et al., 2018), and jointly use vision and audio for navigation (Chen et al., 2020).
Recent work aims to propose plausible reverberations or sounds from image input (Singh et al., 2021; Du et al., 2021), these approaches model the phase-free logmagnitude STFT using either convolution or implicit functions, which we also utilize.
Different from them, our work leverages the geometric features learned by modeling acoustic ﬁelds to improve the learning of 3D view generation.
M ETHODS
We are interested in learning a generic acoustic representation of an arbitrary scene, which can capture the underlying sound propagation of arbitrary sound sources across both seen and unseen locations in a scene.
We ﬁrst review relevant background information towards modeling environment reverberations.
We then describe Neural Acoustic Fields (NAFs), a neural ﬁeld which we show can capture, in a generic manner, the acoustics of arbitrary scenes.
We further discuss how we can parameterize NAF in a manner so that it can capture acoustics property even at unseen sound sources and listener positions.
Finally, we discuss the underlying implementation details of our model.
3. B ACKGROUND ON E NVIRONMENTAL R EVERBERATION
The sound emitted by a sound source undergoes decay, occlusion, and scattering due to both the geometric and material properties of a scene.
For a ﬁxed location pair ( q , q (cid:48) ) , we deﬁne the impulseresponse at a listener position q , as the sound pressure p ( t ; q , q (cid:48) ) induced by an impulse at q (cid:48) .
Such behavior can be concisely and elegantly modeled utilizing the linear wave equation (Pierce, 2019):
[Equation] [Equation]
where c is the speed of sound, p is the sound pressure, ( q, q (cid:48) ) being the listener and emitter location respectively, and δ the Dirac delta representing the forcing function, where we refer to sound pressure p ( t ; q , q (cid:48) ) as the impulse-response observed at listener position q .
Given an accurate model of the impulse-response p ( t ; q , q (cid:48) ) described in Eqn (1), we may model audio reverberation of any sound waveform s ( t ) emitted at q (cid:48) , by computing the response r ( t, q , q (cid:48) ) at listener location q by querying the continuous ﬁeld and using temporal convolution:
[Equation]
[Figure]
3. N EURAL A COUSTIC F IELDS
We are interested in constructing a continuous representation of the underlying acoustics of a scene, which may specify the reverberation patterns of an arbitrary sound source.
The parameterization of an impulse-response introduced in Section 3.1 provides us with a method to model audio propagation when given an omnidirectional listener and emitter.
To construct a model of a directional listener, we further model the 3D head orientation θ ∈ R 2 , and ear k ∈ { 0 , 1 } (binary left or right) of a listener, in addition to the spatial position q ∈ R 3 of the listener and q (cid:48) ∈ R 3 of the emitter.
We may then model the time domain impulse response v using a neural ﬁeld Φ which takes as input the listener and emitter parameters:
[Equation]
[Figure]
Directly outputting the impulse-response v ∈ R T with a neural network is difﬁcult to its high dimensional (over 20000 elements) and chaotic nature.
A naïve solution would be further add t as an additional argument our neural ﬁeld, but we found that such a solution worked poorly, due to the highly non-smooth representation of the waveform (see Table A2).
We instead encode the impulseresponse utilizing a short-time Fourier transform (STFT) to create a log-magnitude spectrogram denoted v STFT , which we ﬁnd to be signiﬁcantly more amenable to neural network prediction, due to the smooth nature of the frequency space.
In Figure 3 we show spectrograms for ground truth impulse responses and those learned by our network.
Thus, our parameterization of NAF is a neural ﬁeld Ω that is trained to estimate the impulse response function φ , and outputs v STFT for a given time and frequency coordinate:
[Equation]
We train our model using MSE loss between the generated and ground truth log-spectrograms v STFT :
[Equation]
across spectrogram coordinates t and f .
In our qualitative results we utilize Grifﬁn-Lim reconstruction to recover v from v STFT similar to prior work (Du et al., 2021).
While such an approximate reconstruction ignores the underlying phase of the impulse response, phase free models are typically used in spatial acoustic modeling (Pulkki, 2007; Raghuvanshi & Snyder, 2018; Singh et al., 2021).
Such approximations are used because the phase locking cutoff in humans occurs at around 1 ∼ 2kHz (Shinn-Cunningham et al., 2000; Joris & Verschooten, 2013), below which time delay provides important spatial cues.
We demonstrate in our qualitative results that NAFs with this approximation may still accurately model realistic spatial acoustic effects.
3. G ENERALIZATION THROUGH L OCAL G EOMETRIC C ONDITIONING
We are interested in parameterizing the underlying acoustic ﬁeld, so that we may not only accurately represent impulse-response at emitter-listener pairs we see during training, but also at novel combinations of emitter and listener seen at test time.
Such generalization may be problematic when directly parameterizing NAFs utilizing a MLP with inputs speciﬁed in Eqn (4), as the network may learn to directly overﬁt and entangle the relation between emitter and listener impulse-responses.
What generic information may we extract from a given impulse-response between an emitter and listener?
In principle, extracting the full dense geometric information in a scene would enable us to robustly generalize to new emitter and listener locations.
However, the amount of geometric information available in a particular impulse-response, especially for positions far away from either current emitter and listener is limited, since these positions has little impact on the underlying impulse-response.
In contrast, the local geometry near either emitter and listener positions will have a strong inﬂuence in the impulse-response, as much of the anisotropic reﬂection comes from such geometry (Paasonen et al., 2017).
Inspired by this observation, we aim to capture and utilize local geometric information, near either emitter or listener locations, as a means to predict impulseresponses across novel combinations.
To parameterize and represent these local geometric features, we learn a 2D grid of of spatial latents which we illustrate in Figure 2.
When predicting an impulse-response at a given emitter and offset position, we query the learned grid features at both emitter and listener positions using bilinear sampling, and provide it as additional context into our NAF network Ω .
Such features provide rich information on the impulse-response, enabling NAF to generalize better to unseen combinations of both emitter and listener locations. In the rest of this work, we refer to the NAFs with local geometric features as Ω grid .We learn grid latent features jointly with the underlying parameters of NAF.
Additional details can be found in Appendix B. Such a design choice, however, still requires us to consider how to further combine local geometric information captured separately from either listeners or emitters.
A naïve implementation would be to maintain separate feature grids for both listener and emitter positions.
Such an approach fails to account for the fact that the local geometric information captured by emitter may also inform the local geometric information around a listener.
Examining Eqn (1), we note that it is in fact symmetric with respect to exchanging either listener or emitter positions (Chaitanya et al., 2020), indicating that the impulse-response does not change when listener and emitter are swapped.
Such a result means that we may in fact directly utilize the local geometric information captured near an emitter position interchangeably for either emitters and listeners.
Thus, we propose our local geometric information as a single latent grid, which we show in our ablations outperform the naïve dual grid implementation.
E XPERIMENTS
In this section, we demonstrate that our model can faithfully represent the acoustic impulse response at seen and unseen locations.
Additional ablation studies verify the importance of utilizing local geometric features to enable test time generation ﬁdelity.
Next, we demonstrate that learning acoustic ﬁelds could facilitate improved visual representations when training images are sparse.
Finally we show that the learned NAF can be used for sound source localization.
4. S ETUP
For evaluating the learned acoustic ﬁelds, we use the Soundspaces dataset (Chen et al., 2020).
This dataset consists of R i probe points for each scene, with each probe capable of representing an emitter or listener location for up to O
( R 2 i ) emitter and listener pairs.
The emitters are represented as omnidirectional, while the listener acts as a stereo receiver that can have one of four different orientations.
The listeners and emitters are at ﬁxed height.
For each scene, we holdout 10% of the
RIRs randomly as a test set.
Our NAFs are trained on 6 representative scenes, selected such that 2 consist of multi-room layouts, 2 consist a single room with a non-rectangular walls, and 2 consist of a single room with rectangular walls as in Figure 4.
Each scene is trained for 400 epochs, which takes around 3 .
5 hours for the largest scenes on four Nvidia V100s .
In each batch, we sample 20 impulse responses, and randomly select 2 , 000 frequency & time pairs within each spectrogram.
An initial learning rate of 5 × 10 − 4 is used for the network, while an initial learning rate of 1 × 10 − 3 is used for the grid.
We add a small amount of noise sampled from N (0 , 0 . 02) to each coordinate during training to prevent degenerate solutions.
4. A RCHITECTURE D ETAILS
The Soundspaces dataset lacks the full parameterization of an acoustic ﬁeld described in Equation 4, so we train NAF with a restricted parameterization that is available in the dataset.
This allows for two degrees of freedom along the x − y plane for the listener locations q ∈ R 2 and the emitter location q (cid:48) ∈ R 2 .
The listener can assume four possible orientations θ ∈ { 0 , 90 , 180 , 270 } , while the emitter is omnidirectional.
In particular, we utilize a parameterization of Ω grid which maps an input tuple [ x, y, x (cid:48) , y (cid:48) , f, t ] ∈ R 8 × { 0 , 90 , 180 , 270 } × { 0 , 1 } to a single scalar value that represents the intensity for a given time and phase in the STFT:
[Equation]
To encode the rotation θ , as there are only 4 possible discrete rotations in the dataset, we directly query into a learnable embedding matrix of shape R 4 × k , returning a R 1 × k vector.
Similarily, to encode the left and right ear, we similarly query into a learnable embedding matrix of shape R 2 × k , returning a R 1 × k vector.
The f, t tuple representing the frequency and time respectively are scaled to ( − 1 , 1) and processed with sinusoidal encoding using 10 frequencies of sin and cos up to 200Hz.
To obtain local geometric features for either a emitter or listener in a scene, we assume that our scene is contained with a set of bounding pixels P
= { P 1 ...P k } which form a grid over the scene.
For a given position tuple ( x, y ) , we then query the local features:
[Equation]
Where L ( · ) is the bilinear interpolation function, ( p ∗ 1 ...p ∗ 4 ) are the four vertices that bound ( x, y ) , and ˜ f ( · ) represents the features stored at a given vertex.
These queried features are combined with the coordinates processed with sinusoidal encoding using 10 frequencies of sin and cos functions up to 100Hz.
We process both the listener and emitter position tuples this way.
We combine the grid based features with the sinusoidal embeddings and the discrete indexed embeddings as the input to our multilayer perceptron f φ .
Please refer to Figure 2 for a visualization of our model, and Appendix
B for further details. We compare using a shared local geometric feature with the emitter and listener, as well as using have the emitter and listener query their own individual grids.
4. E VALUATION ON NEURAL ACOUSTIC FIELDS
We ﬁrst validate that we can capture environmental acoustics with high ﬁdelity, at unseen emitterlistener positions.
Baselines.
We compare our model against several strong interpolation baselines.
We experiment with holding either the listener ﬁxed and interpolating based on the emitter location, as well as holding the emitter ﬁxed and interpolating based on the listener location.
Both linear and nearest neighbor approaches are widely used (Savioja et al., 1999; Raghuvanshi et al., 2010; Pörschmann et al., 2020) in modeling of spatial audio.
We also compare sharing and using individual local geometric features.
Results.
We evaluate the results of our synthesis by measuring the mean squared error (MSE) between the generated and the ground truth log-spectrograms, as well as measuring the percentage error between the T60 reverberation time in the time domain.
In this case, lower MSE and T60-error values indicates a better result.
As shown in Table 1, our NAFs achieve signiﬁcantly higher quality on [Table] [Figure] the modeling of unseen impulse responses compared to strong interpolation baselines.
A comparison of using shared and dual local geometric features indicates that despite having fewer learnable parameters, we achieve better performance by sharing the local geometric features.
Examples of individual impulse responses generated by our model are shown in Figure 3.
Figures 4 shows the different scenes and the loudness change predicted by our NAFs.
Our model is capable of predicting smoothly varying acoustic ﬁelds that are affected by the physical surroundings.
Generalization through Geometric Conditioning.
We next assess the impact of utilizing local geometric conditioning as a means to generalize to novel combinations of emitter-listener positions.
On the "Large 1" room, in Figure 5 we evaluate test set MSE error when NAF is trained with a limited percentage of the training data either with or without local geometric conditioning.
We ﬁnd that such geometric conditioning enable substantially better test set reconstruction error, with the underlying performance gap increasing with less data.
4. C ROSS - MODAL LEARNING
When NAF is trained, as shown in Figure 6, a structured geometric representation emerges.
This structured latent stands in sharp contrast with the noisy structure learned by NeRF models.
In this experiment, we explore the effect of jointly learning acoustics and visual information when we are given sparse visual information.
Recall that our NAF includes a local geometric feature grid P that covers the entire scene.
For our cross-modal learning experiment, we jointly learn this feature grid with a NeRF network modiﬁed to accept both local features along with the traditional sinusoidal embedding.
In the acoustics branch, we query the grid using emitter and listener positions.
In the
NeRF branch, we use point samples along the ray projected on the grid plane to query the features.
In both cases, the process is fully differentiable.
We use a standard implementation of NeRF with a coarse and ﬁne network.
In both the cross-modal and RGB only experiments we augment the
[Figure]
ﬁne network with a learnable local feature grid. In the NeRF only setting, we minimize color C reconstruction loss for a ray r over a batch of rays R : L RGB = (cid:80) r ∈R
|| ˆ C ( r ) − C ( r ) || 2 2 . In contrast, in the NAF + NeRF experiment, we jointly minimize L RGB + L NAF , where L NAF is deﬁned in equation 5. We utilize 64 coarse samples and 128 ﬁne samples for each ray
[Figure]
and sample 1024 rays per batch.
[Figure]
[Figure] [Table]
[Figure]
[Table]
Results.
We train on the two large rooms in our training set.
For each room 75 , 100 , 150 images are used for training, while the same 50 images of novel views are used for all conﬁgurations during testing.
In Table 2 we observe that training with acoustic information helps improve the PSNR and
MSE of the visual output.
This effect is more signiﬁcant when the training images are very sparse, the NAF network helps less when there is sufﬁcient visual information.
Qualitative results are shown in Figure 7, we see there is a reduction of ﬂoaters in free space.
We visualize the local features learned in Figure 6.
While the features learned using only RGB supervision lack coherent structure, the one we jointly learn with audio exhibits clear structure.
To assess the structure in grid latents, we ﬁt a linear layer to regress the room position of each latent. For room 1 and 2 respectively, we achieve a R 2 of 0 . 109 and 0 . 085 when trained on RGB only, compared to a R 2 of 0 .
385 and 0 .
201 when trained with RGB + Acoustics supervision.
This demonstrates the spatially informative structure captured by jointly learning NAF and NeRF.
[Figure] [Figure]
4. S OUND S OURCE L OCALIZATION
Given a high ﬁdelity representation of the acoustic ﬁeld, it is possible to identify the location of a sound source in a scene, regardless of where the listener is located?
In this task, we demonstrate that a trained NAF can be used for localizing an emitter based on sparse audio samples from the scene.
We explore 5 sounds in total, 4 of which are musical instruments and 1 is a sine sweep (Sweep).
We assume that exists a set of audio recordings V i ∈ V from an object, recorded from 10 different locations and orientations that are known.
We further assume that we are given the original unperturbed waveform V orig the object emits, with the underlying task being to locate the position of the emitter ( x (cid:48) , y (cid:48) ) using these sound samples and the unperturbed sound.
We hold the listener position and orientations ﬁxed and vary the emitter position to probe for the expected sound at each location. For each location, we render a log-spectrogram using our trained NAF and compare this against the actual log-spectrogram recorded response V i :
[Equation]
We perform the search for the emitter location by discretizing the scene in 0 .
3 m segments.
Baselines.
We compare against two baselines, a ResNet-18 network that takes as input both the spectrogram of the transformed and original sounds, and is trained to regress the object location relative to the listener frame.
Given 10 individuals, we take an average of the predicted emitter location.
We also compare against a modiﬁed maximum magnitude baseline, where given a set of recording locations, we simply select the position where the sound is loudest.
Results.
Our NAFs can achieve very accurate localization as seen in Table 3.
We also visualize the error map with the ground truth emtiter location labeled in Figure 8.
It can be observed that the real location of the emitter matches well with the location of the lowest prediction error.
C ONCLUSION
In summary, this paper introduces Neural Acoustic Fields (NAFs), a continuous, differentiable acoustic representation which can faithfully represent the reverberation of different audio sources in a scene, as well as the scene’s overall acoustic environment.
By conditioning NAFs locally on the underlying scene geometry, we demonstrate that our approach enables the accurate prediction of environmental reverberations even at unseen locations in the scene.
Furthermore, we demonstrate that the acoustic representations learned through NAFs are powerful, and may be utilized to facilitate audio-visual cross-modal learning, as well as to localize the underlying source of arbitrary sounds.
Appendix A A DDITIONAL V ISUALIZATION OF R OOMS
Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
Emitter Emitter
We show additional NAF predictions of loudness as we move an emitter inside different rooms in
Figure A1.
For each room, note how the sound is affected by the geometry.
In wide open spaces the sound is highly dispersed.
While in thin structures the sound test to concentrate locally.
As we move farther from the source, the loudness of the sound decreases.
B A RCHITECTURE AND T RAINING D ETAILS
We visualize the two alternative models that we experiment with, in Figure A2 is a network that uses different local feature grids for the emitter and receiver.
The network uses the emitter and listener to positions to sample from the two different grids.
In Figure A3 we show a model that does not utilize any kind of local geometry conditioning.
The listener, emitter, phase, and time input are transformed using sinusoidal embedding, while the orientation and left/right are retrieved.
All transformed inputs are directly fed to the network.
[Figure]
[Figure]
Each network consists of 8 fully connected layers in a feedforward fashion, as well as a skip connection consisting of two fully connected layers.
The skip connection takes the input and adds its output to that for the fourth intermediate layer.
We utilize an intermediate feature size of 512, and softplus as the activation function.
The grid is of size R 32 × 32 × 64 , where 32 is the size of the spatial dimensions, while 64 is the channel count. We initialize each element of the grid i.i.d. from
N (0 , 1 √ 64 ) . For the network excluding the grid, we utilize an initial learning rate of 5 e − 4 . For the grid, we utilize an initial learning rate of 1 e − 3 . The Adam optimizer is used when training our network.
We utilize a orientation embedding of shape R 7 × 4 × 512 where 7 is the number of intermediate outputs, 4 is the number of orientations, and 512 is the feature dimension.
For the left-right embedding, we use a shape of R 7 × 2 × 512 .
We perform additive conditioning by adding a R 512 vector to each intermediate output for both the orientation and the left/right.
For each scene, to generate a log-spectrogram for each impulse response, we compute the mean and standard deviation µ freq , σ freq for each frequency in the log-spectrogram, and normalize the data prior to training:
[Equation]
For the sinusoidal embedding, we utilize both cos and sin with 10 frequencies each for encoding position, phase, and time.
For encoding position we utilize a max frequency of 100 Hz, while for encoding time and frequency we utilize a max frequency of 200 Hz.
Since we do not know beforehand the time duration of an impulse response at an unseen location, we compute the maximum impulse length for each scene and use this length to zero pad the training impulse responses.
Because the padded regions do not contain useful information, we want the network to focus modeling efforts on the early regions of the impulse response.
We achieve this by stochastically padding the impulse response to maximum impulse length with 0 . 1 probability. Because the implicit function is trained on individual ( t, f ) coordinates within a given v STFT , training samples do not need to be of the same length. During test time, we perform inference up to the maximum duration of scene impulse response.
C D ATASET V ISUALIZATION
[Figure] In Figure A4, we visualize both the room and underlying set of probe positions in the training data.
Due to occlusion and the geometry, even slightly moving the emitter or listener position can result in different results.
As we demonstrated in Table 7, both nearest neighbor and linear interpolation perform poorly compared to our learned solution.
In contrast, recovered acoustic ﬁelds from NAF trained on these probe positions is substantially denser (Figure A1). D S TORAGE C OMPARISON [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We compare the averaged on disk storage cost of the different methods for inferring the spatial audio using a precomputed training set in Table A1.
Both linear and nearest interpolation methods require access to the entire training set, while our NAF based approaches compactly encode the acoustic scene into a ﬁxed size.
E V ISUALIZATION OF INTERPOLATED METHODS
In this section, we visualize the loudness when an interpolation method is used.
We compare nearest neighbor interpolation with linear interpolation applied to the training set, both in the time domain.
The nearest neighbor method presents sudden step changes in loudness, while the linear interpolation fails when the queried location falls outside of the convex hull of the training set.
Also note the sound leakage in (c)-(d) in the bottom wall when linear interpolation is used.
[Figure]
F A LTERNATIVE R EPRESENTATIONS
[Figure]
[Table]
[Figure]
Our current method follows prior work in learning in the log-magnitude STFT domain.
In this section, we investigate two possible alternatives: learning phase + log-magnitude and directly learning in the time domain.
The MSE and T60 error percentage is presented in Table A2.
We observe that jointly modeling phase + log-magnitude degrades the performance slightly compared to modeling just the log-magnitude while modeling in the time domain performs poorly.
We visualize the waveform at two test set listener/emitter locations in Figure A
G L 2 REGULARIZED GRID IN N E RF
[Table]
In Table A3 we compare NeRF that utilizes a grid and trained using image reconstruction loss, against a variant where a L 2 penalty with weight 1 e − 5 to ensure a smooth latent space is added to the image reconstruction loss.
There are 75 images used in the training set.
We observe degraded performance when we apply this penalty.
H I NTERPOLATION OF LEFT / RIGHT LATENT
[Figure]
As shown in Figure A7, are the output for two locations when we take linear steps between the latent representing the left ear, and the right ear.
We observe that the output smoothly changes as we move in the latent space.