S OUND N E R IR F:
R
ECEIVER
- TO
R ECEIVER
S OUND
N EURAL R OOM I MPULSE R ESPONSE F IELD
Anonymous authors
Paper under double-blind review
A BSTRACT
We present SoundNeRirF , a framework that learns a continuous receiver-to-receiver neural room impulse response field (r2r-RIR) to help robot efficiently predict the sound to be heard at novel locations.
It represents a room acoustic scene as a continuous 6D function, whose input is a reference receiver’s 3D position and a target receiver’s 3D position, and whose outputs are an inverse room impulse response (inverse-RIR) and a forward room impulse response (forward-RIR) that jointly project the sound from the reference position to the target position.
SoundNeRirF requires knowledge of neither sound source (e.g. location and number of sound sources) nor room acoustic properties (e.g. room size, geometry, materials).
Instead, it merely depends on a sparse set of sound receivers’ positions, as well as the recorded sound at each position.
We instantiate the continuous 6D function as multi-layer perceptrons (MLP), so it is fully differentiable and continuous at any spatial position.
SoundNeRirF is encouraged, during the training stage, to implicitly encode the interaction between sound sources, receivers and room acoustic properties by minimizing the discrepancy between the predicted sound and the truly heard sound at the target position.
During inference, the sound at a novel position is predicted by giving a reference position and the corresponding reference sound.
Extensive experiments on both synthetic and real-world datasets show SoundNeRirF is capable of predicting high-fidelity and audio-realistic sound that fully captures room reverberation characteristics, significantly outperforming existing methods in terms of accuracy and efficiency.
I NTRODUCTION
Room acoustic [3, 50, 1, 27] aims at accurately capturing the sound propagation process in a room environment, so as to determine the received sound characterizing reverberation at any spatial position.
Robots and other intelligent agents (e.g. voice assistants) can exploit such sound for a variety of tasks, including navigation [9, 44, 29], reconstruction [62, 8] and audio-dependent simultaneous localization and mapping (SLAM)
As a complementary sensor to vision, sound perception exhibits strengths in scenarios involving dim or no lighting and occlusion.
In most cases, room acoustic can be treated as linear time-invariant system (LTI).
Thus, the underlying sound propagation problem is to precisely derive a room impulse response (RIR) - a transfer function of time and other acoustic variables that measures the behaviour a sound takes from a sound source to a receiver.
The sound recorded at a particular position can thus be obtained by convolving the sound at the source position with the RIR.
We call such RIR source-to-receiver RIR (s2r-RIR).
Accurately deriving s2r-RIR, however, is a difficult task.
The challenge is three-fold: 1) high computational cost: traditional methods [3, 50, 1, 27, 41, 26, 4, 22] undergo rigorous mathematical derivation and extensive measurements to compute s2r-RIR, by treating sound as either rays [50, 1, 27] or waveforms [3, 41].
The corresponding computation complexity is proportional to the room geometric layout complexity and the number of sources.
2) non-scalable: the whole computation process needs to be re-executed once the source/receiver location changes slightly, or the room layout has altered (e.g. furniture movement).
3) too strong assumptions: the assumption that the source/receiver location and room acoustic properties are well-defined and known in advance is too strong to be held in real scenarios.
The sound source location is mostly unknown in real-scenarios, and localizing sound sources is an extremely difficult task [20, 23, 6, 19].
In this work, we propose SoundNeRirF , a receiver-to-receiver Sound Ne ural R oom i mpulse r esponse
F ield for efficiently predicting what sounds will be heard at arbitrary positions.
SoundNeRirF requires knowledge of neither sound sources (e.g. source number and position) nor room acoustic properties 1 , but instead represents a room acoustic scene through a sparse set of 3D spatial positions that robots have explored, as well as the sound recorded by the robot at each position.
A robot equipped with a receiver can easily collect massive such datasets by simply moving around the room and recording its position and received sound at each step.
SoundNeRirF represents a room acoustic scene by a continuous 6D function, which takes two receivers’ 3D positions (one reference and one target position) as input, and outputs two room impulse responses that jointly project the sound from the reference position to the target position.
SoundNeRirF thus learns receiver-to-receiver RIR (r2rRIR).
SoundNeRirF is reinforced to implicitly encode the interaction between sound sources and receivers, in addition to room acoustic properties by minimizing the error between predicted and the truly recorded sounds, because sound recorded at any position is naturally a result of such an interaction.
By instantiating the 6D function as multi-layer perceptrons (MLPs), SoundNeRirF is differentiable and continuous for any spatial position, and can be optimized with gradient descent.
Figure 1 visualizes SoundNeRirF’s motivation.
Specifically, SoundNeRirF splits r2r-RIR into two parts: an inverse-RIR for the reference position and a forward-RIR for the target position.
We further introduce receiver virtual position settings and a physical constraint strategy that guides SoundNeRirF to explicitly learn direct-path, specularreflection and late-reverberation that commonly exist in room acoustics.
SoundNeRirF has numerous applications in robotics, including immersive audio/video game experience in augmented reality (AR) [24, 57], sound pre-hearing without actually reaching to the location [5], and improving robot localization (by utilizing the predicted sound and truly heard sound).
In summary, we make four main contributions: First , we propose SoundNeRirF that learns r2r-RIR neural field from a sparse set of receiver recordings.
Second , SoundNeRirF requires knowledge of neither sound source nor room acoustic properties, but instead depends on more accessible data that can be collected by robot walking around in the room environment.
Third , SoundNeRirF directly predicts sound raw waveform.
It disentangles sound prediction and r2r-RIR learning, exhibiting strong generalization capability in predicting previously unheard sound (see Experiment Sec.).
Lastly , we release all datasets collected in either synthetic and real-world scenario to the public to foster more research.
R ELATED W ORK
Room Acoustics Modelling There are two main ways to model room acoustics: wave-based modelling [3, 41, 26, 4] and geometry-based modelling (aka geometrical acoustics) [50].
The wave-based modelling utilizes sound wave nature to model sound wave propagation, whereas geometry-based modelling treats sound propagation as optic rays.
Typical geometry-based modelling methods include ray tracing [27], image source method (ISM [1]), beam tracing [17] and acoustic radiosity [22, 36].
The main goal of room acoustics is to characterize the room reverberation effect, which consists of three main components: direct-path, specular-reflection and late-reverberation.
SoundNeRirF explicitly models the three components to learn r2r-RIR.
Neural Room Impulse Response (RIR)
Some recent work [46, 55, 47, 45, 12, 30, 49] proposed to learn room impulse response (RIR) with deep neural networks.
However, they all assume massive source-to-receiver RIRs (s2r-RIR) are available to train the model.
Unlike these methods, SoundNeRirF learns implicit r2r-RIR from a set of robot recorded sounds at different positions, which are easiler to collect with much less constraints.
Neural Radiance Field (NeRF) has received lots of attention in recent years, especially in computer vision community [33, 21, 59, 52, 61].
They model static or dynamic visual scenes by optimizing an implicit neural radiance field in order to render photo-realistic novel views.
Some recent work [60, 13, 58] extends to neural radiance field to 3D point cloud [60], image-text domain [58] and robotics [13, 53, 35, 28] for robot localization, position and scene representation.
A. Luo et.
al.
[30] propose to learn implicit neural acoustic fields from source-receiver pairs model how nearby physical environment affect reverberation.
However, their method requires the presence of massive
[Figure]
s2r-RIR as well.
Our proposed SoundNeRirF shares the same idea with NeRF that learns an implicit receiver-to-receiver acoustic room impulse response field.
Sound Synthesis Inferring sound waveform for a novel location partially relates to sound synthesis [38, 14, 64, 48, 15, 54, 11, 42].
WaveNet [38] learns to predict future sound waveform based on previously heard sound waveform.
WaveGAN [14] and GANSynth [15] adopt generative adversarial network (GAN [18]) to learn to generate sound.
Specifically, A. Richard et.
al.
[48] propose to infer binaural sound waveforms from mono-channel input waveform.
S OUND N E R IR F: R ECEIVER - TO -R ECEIVER N EURAL R OOM I MPULSE R ESPONSE F IELD 3. P ROBLEM D EFINITION
In a geometric acoustic room, at least one static sound source is constantly emitting sounds.
N sound receivers at various positions (or as a robot moves to different positions) are recording the acoustic environment, { ( W , P ) =
{ ( w i , p i ) } Ni =1 } , W ∈ R T × N , P ∈ R 3 × N .
Each sound w i is a time series of the sound raw waveform (of length T ) and position p i is the receiver’s coordinate [ x i , y i , z i ] .
Our target is to learn an implicit receiver-to-receiver neural room impulse response field (SoundNeRirF)
F from ( W , P ) , that is capable of efficiently inferring the sound ˆ w t that would be heard at a novel position (target position) p t .
Note that p t is previously unexplored, and we are given a reference sound w r from explored location p r ,
[Equation]
In this work, both sound source and receiver are omni-directional and monoaural, so we do not have to consider their directivity and inter-channel difference (as discussed in [20, 48]).
We also assume all receivers record the same sound content (e.g. telephone ring) but each has encoded unique room reverberation effect based on its position, so the sound at arbitrary position can be predicted by the sound from any reference position, if the room reverberation change between them can be modelled.
3. S OUND N E R IR F F RAMEWORK I NTRODUCTION
SoundNeRirF represents the receiver-to-receiver neural room impulse response field as a 6D function h , whose input is the combination of one reference receiver position p r =
[ x r , y r , z r ] and one target receiver position p t =
[ x t , y t , z t ] , and whose output is an inverse-RIR h inv that mainly depends on reference position p r and a forward-RIR h for that mainly depends on target position p t : h inv , h for =
h ( p r , p t ) .
h inv and h for jointly project the sound at the reference position w r to the target position ˆ w t via 1D convolution operation,
[Equation]
where ⊛ indicates 1D convolution operation in time domain.
Both h inv and h for are one-dimensional impulse response in the time domain, and we omit their time indexes for succinct representation.
The motivation behind learning two separate RIRs is to reduce the reference-target position arbitrarity
[Table]
vHaC/Ws/FSpOkCs2X9RPJMGITN8nPaE5Qzm2hDIt7K2EDammDG1IeRuCt/jyMmlUyt5l2bu7KFYrWRw5OIYTKIEHV1CFW6hBHRgoeIZXeHOM8+K8Ox/z1hUnmzmCP3A+fwBUHo/+</latexit> (
[Figure]
t ,p t ) <latexit sha1_base64="uUoxnD2K0MeMWc/8xk9uc5Q6/QY=">AAACB3icbVDLSsNAFJ3UV62vqktBBotQQUpSRN0IBTcuXFSwD2hCmEwn7dDJg5kbpYTu3Pgrblwo4tZfcOffOGmz0OqBGQ7n3Mu993ix4ApM88soLCwuLa8UV0tr6xubW+XtnbaKEklZi0Yikl2PKCZ4yFrAQbBuLBkJPME63ugy8zt3TCoehbcwjpkTkEHIfU4JaMkt79sBgSElIr2e4AtctYcE0nsXJsdY/0duuWLWzCnwX2LlpIJyNN3yp92PaBKwEKggSvUsMwYnJRI4FWxSshPFYkJHZMB6moYkYMpJp3dM8KFW+tiPpH4h4Kn6syMlgVLjwNOV2dZq3svE/7xeAv65k/IwToCFdDbITwSGCGeh4D6XjIIYa0Ko5HpXTIdEEgo6upIOwZo/+S9p12vWac26Oak06nkcRbSHDlAVWegMNdAVaqIWougBPaEX9Go8Gs/Gm/E+Ky0Yec8u+gXj4xu5PZiG</latexit
[Figure]
impact and explicitly exploit room acoustic RIR property: we first project the reference sound to a virtual implicit sound source field with inverse-RIR ( h inv , mainly depends on reference position) and then reproject it from the virtual implicit sound source field to the target sound with forwardRIR ( h for , mainly depends on target position).
By explicitly setting physical constraints to h inv and h for (Sec. 3.4), SoundNeRirF learns more robust r2r-RIR representation.
In this work, we instantiate both h inv and h for as two identical multi-layer perceptrons (MLP, specifically 6 layers withneurons in each layer), so they are fully differentiable and can be optimized by gradient descent algorithm.
They are also continuous at any spatial position so we can infer the sound for any position,
[Equation]
MLP representation has been successfully used for acoustic tasks such as speech enhancement [37] and audio equalization [40].
We extract the last MLP layer representation as the r2r-RIR (see Fig. 2).
Please note that h inv and h for learning are independent of the sound content, we can use them for predicting previously unheard sound (sound emitted from the same source but not trained with, see
Experiment).
Hitherto we have vanilla SoundNeRirF with raw receiver position as input to learn neural r2r-RIR field using an MLP.
We further modify it by adding extra virtual receivers and setting physical constraints to reinforce SoundNeRirF to learn a more representative r2r-RIR neural field.
3. S ETTING V IRTUAL R ECEIVER P OSITIONS
To explicitly model the direct-path, specular-reflection and late-reverberation effect that commonly exist in room acoustic RIR, we manipulate each original input 3D position p (which holds both the reference and target receiver position) and consecutively place them at two virtual further locations (in our case, we multiply the original position by different coefficients to ensure the receiver is virtually set at further positions).
The resulting three positions are leveraged to explicitly model the direct-path
RIR h d (original position), specular-reflection RIR h s and the late-reverberation RIR h l , respectively.
The final RIR is obtained by adding three separate RIRs together: h d ( α 1 · p )+ h s ( α 2 · p )+ h l ( α 3 · p ) , and three separate RIRs go through the same MLP.
Please note that the virtual receiver position setting is adopted for both the reference position and the target position.
As a result, h inv in Eqn.can be rewritten as,
[Equation]
h for can be obtained in similar formulation in Eqn. 4.
α 1 , α 2 , α 3 are hyperparameters setting the receiver to a further virtual position (In our case, they are empirically chosen, α 1 = 1 , α 1 = 2 , α 3 = 3 ).
The motivation of such design is based on sound propagating as optic rays [1, 27]: the longer the path a sound signal travels, the greater the time delay and energy decay of the RIR there are in general.
We implicitly approximate such room acoustic RIR nature caused by sound travelling path length difference by virtually setting the receiver at the further location.
Similar approximation has been used by a traditional Imaged Sourced Method (ISM [1]), in which it mirrors the sound source against a wall to get a virtual sound source that has a different distance to the receiver.
The mirroring process can be executed one or multiple times to obtain different order virtual sources.
Since we have no knowledge of the sound sources, we instead manipulate the receiver location to model the same
RIR nature (See Fig.
2 right).
Following the practice in [43, 63, 56, 33] [Figure] we add position encoding to encode the 3D position into high-dimensional representation before feeding it to neural network
[Figure]
3. P HYSICAL C ONSTRAINT ON R 2 R -RIR
[Figure]
To encourage SoundNeRirF to learn representative r2r-RIR, we set two constraints on MLP-learned RIR: First, time-delay order constraint for direct path h d , specular-reflection h s and latereverberation h l .
Second, to reflect the increased amplitude attenuation (energy decay) caused by longer travelling distance, we add monotonicity constraints to both inverse-RIR and forward-RIR.
For the MLP-learned inverse-RIR h inv (specifically h inv d , h inv s , h inv l ) and forward-RIR h for (specifically h for d , h for s , h for l ) in Eqn.
4, we associate each individual RIR with a learnable time shift parameter ( t inv d , t inv s , t inv l for inverse-RIR, t for d , t for s , t for l for forwardRIR).
By restricting the time shift parameters in ascending order (that is, t inv d ≤ t inv s ≤ t inv l , t for d ≤ t for s ≤ t for l ), we set constraints to the RIRs in Eqn.
4 to reflect different travelled distances of sound.
Moreover, to reflect the greater energy decay (attenuated amplitude) caused by longer travelling distance, we add a monotonicity constraint to each learned RIR separately.
For example, for the h inv d , the two physical constraints can be expressed as,
[Equation]
where t is the RIR index in the time domain.
The other five RIRs can be obtained in a similar way.
σ ( · ) operation ensures monotonicity.
In inverse-RIR h inv , we use increasing monotonicity which means σ = max( · ) , while in forward-RIR h for we use decreasing monotonicity σ ( · )
=
min( · ) .
Examples of typical learned inverse-RIR and forward-RIR on our synthetic dataset are shown in
Fig.
3. T RAINING AND I NFERENCE
The SoundNeRirF pipeline is shown in Fig.
It takes as input a pair of reference and target receiver positions.
Two identical MLPs with layer-wise communication [34, 6] are used to learn h inv and h for in parallel.
Position encoding is exploited to embed both the real and virtual positions to higher dimensions before feeding to MLP.
The h inv and h for are jointly optimized by minimizing the discrepancy between the predicted sound ˆ w t and the ground truth sound w t .
For the loss function, we combine mean squared error in both time and time-frequency domains,
[Equation]
(6) where STFT( · ) indicates the short-time Fourier transform.
We explicitly use the mean squared loss in the frequency domain because merely resorting to a mean squared loss in the time domain easily leads to audio quality issues and distorted signals [48].
During inference, theoretically we can leverage any robot traversed position and recorded sound (reference sound) to infer the sound (target sound) at an arbitrary novel location.
E XPERIMENT
Dataset We run experiments on three datasets:
Synthetic Dataset
Single large room with 3D source/receiver positions.
With the flexibility of arbitrarily setting source/receiver number or positions, we can test the necessity of each components of SoundNeRirF framework (ablation study).
To this end, we adopt Pyroomacoustics [51] to simulate a large shoebox-like room of size [50 m × 50 m × 30 m ] .
Seven seed sounds are collected from copy-right free website freesound.org : piano , cat-meowing , footstep , phone-ringing , baby-crying , dog-barking and people-talking .
Each seed sound is 4 seconds long and with the sampling rate 16k
Hz.
To reflect the impact of
[Figure]
[Figure]
sound source numbers, we simulate seven identical rooms to cover sound source numbers from one to seven.
In each room, we first put the corresponding sound sources at random positions, then randomly set 500 receivers at various locations (by guaranteeing at least 0.5 m distance to any sound source) to record the sound simultaneously.
In each room, we keep all the sound sources and receivers’ positions unchanged but replace the original seed sounds with new ones.
It helps test all methods’ generalization capability in predicting unheard sound.
The resulting 500 recorded sounds in each room are split into 300/100/100 for train/test/eval, respectively.
Real-World Dataset Lab/Corridor environment.
We use a Turtlebot robot equipped with a sound receiver (hosted by a Raspberry Pi).
Vicon system [32] is utilized for localizing both speaker and receiver with a high precision (millimetre-level precision).
The data-collection area consists of a cluttered lab room and two adjacent corridors.
Up to 2 omni-directional speakers are put in the lab area with fixed positions.
The Turtlebot moves to different positions to record sound.
We collected two sub-datasets: the first one involves one sound source (emitting baby-crying first and then piano .
baby-crying is for training model, piano is for unheard sound test), while the second one has two sound sources (one emitting cat-meowing and the other emitting dog-barking at the same time).
In total, we collected 120 receiver positions for one and two sound sources separately, with 90/30 for train/test split.
Cross-Room Dataset large area with multiple rooms.
To test the performance in room scenes that contain multiple rooms, and complex room layouts (e.g. wall, sofa and other furniture), we further run experiments on large-scale acoustic-realistic room environment platform SoundSpaces2.0 [10].
SoundSpaces-2.0 supports Matterport3D indoor environment [7], it synthesizes highfidelity sound (both binaural and monoaural) for any given source and receiver position.
The average scene size is larger than 100 m 2 .
In total, we have collected 5000 sound recordings at different positions throughout the 5 room scenes (each room with 1000 recordings).
We randomly split it into 700/300 for train/test, respectively.
We train the individual model for each room separately.
Each room has five sound sources that distribute to whole area, all sound sources emitting telephone-ring sound.
The comparison of the three datasets is in Table 8.
More discussion about the three datasets’ creation/motivation is given in Appendix A. Figure 4 shows the three datasets comparison.
All datasets will be publicly released to benefit the research community.
[Table] [Table]
Evaluation Metrics 1) T-MSE: time domain mean squared error (T-MSE).
It evaluates the difference between predicted sound and ground truth sound in the time domain; 2) TF-MSE: time-frequency mean squared error (TF-MSE).
We first convert the 1D sound waveform to 2D time-frequency representation by short-time Fourier transform (STFT), and compute the mean squared error in the time-frequency domain; 3) CDPM [31, 64]: Contrastive learning-based deep perceptual audio metric.
It incorporates human judgement in the loop, and is trained with a dataset carefully labelled by humans according to their perceptual similarity.
We use CDPM as an approximation of human subjective evaluation of two sounds similarity.
The lower of each of the three metrics, the better of the performance.
Test Configuration For the synthetic dataset, we create 50 reference sounds (selected from the training set) that cover the whole room.
The sound at a test position is predicted by each of the reference sounds separately.
So each test position has 50 predictions.
In addition to reporting the average metrics, we rank the 50 predictions according to their distance to the test position, and divide them into three distance-aware categories: close , medium and far .
Reporting results under the three categories helps to understand different methods’ capability in predicting sound from distant, farfield positions (see Fig.
5 middle).
We randomly select 30/50 reference sounds from the training set for real-world/cross-room test, respectively.
Comparing Methods There are no existing methods that directly model receiver-2-receiver sound prediction.
We compare SoundNeRirF with five most relevant methods: two interpolation based methods and three neural network based methods (More detailed discussion is in Appendix B):
LinInterp : neighboring linear-interpolation.
For each test position, we query its closest 5 positions with known sound.
Then we linearly interpolate the sound to be heard at the test position by merging the 5 neighboring sound recordings with per-position weight, and each weight of the sound at a known position is inversely proportional to its distance to the test position.
NNeigh : nearest neighbor.
Without any learning/interpolation process, we simply treat the nearest neighbor sound as the sound to be heard at any test (novel) position.
WaveNet [38].
The widely used WaveNet is adapted to take reference receiver position, sound and target position as input.
It exploits dilated convolution to predict sound waveform.
TCN [2].
TCN adopts convolution and recurrent neural networks to model sequence2sequence transformation.
We inject both the reference and target position to TCN to reinforce TCN to learn to predict target sound.
WarpNet [48].
WarpNet learns a warping field that maps the monoaural audio to binaural audio.
In our implementation
we feed receiver position [ x, y, z ] to the warping module to learn to warp the sound from the reference position to the target position.
Training Details We train all models with Pytorch [39] and Adam optimizer [25], with an initial learning rate 0.0005 which decays every 20 epochs.
We train 100 epochs and train each model three times and report the mean value for reducing model initialization impact (standard deviations are all within 0.01, we do not report it for brevity report).
All sounds have sampling rate 16 k Hz.
The sound length for both synthetic and real-world data is 4 s, while for cross-room data is 1 s. All sounds are normalized to [ − 1 , 1] before feeding to the neural network for train.
SoundNeRirF network architecture is given in Appendix E.
4. E XPERIMENTAL R ESULT ON S YNTHETIC D ATASET
The result on synthetic dataset is shown in Table 1 and Table 2 (T-MSE: 10 − 3 ).
We can see that
SoundNeRirF achieves the best performance over all the five comparing methods.
In unheard sound
[Table] [Table]
[Table] [Table]
test, the three learning comparing methods have observed remarkable performance drop (even worse than the two non-learning baselines), while SoundNeRirF maintains nearly the same performance and thus exhibits strong generalization capability in predicting previously unheard sound.
Receiver Density Discussion results reported in Table 1 and 2 are obtained by models trained onreceiver recordings.
To figure out the impact of receiver density (number of receiver recordings), we train another two models with much smaller densities (with 100, and 200 recordings respectively).
The result is shown in Fig.
5 left, from which we can see WarpNet [48] is the most sensitive to receiver density.
The reduced receiver density leads to significant performance reduction.
TCN [2] also suffers from low receiver density, and WaveNet [38] is less sensitive to receiver density than WarpNet and
TCN.
SoundNeRirF is invariant to receiver density change (TF-MSE stays as 0.03), which attests that
SoundNeRirF is able to learn representative r2r-RIR with limited receiver recordings.
Reference-Target Distance Discussion .
The TF-MSE variation w.r.t.
reference-target distance is shown in Fig.
5 middle, from which we can see that the performance of WarpNet and WaveNet gradually decrease as the reference-target distance increases.
SoundNeRirF and TCN are insensitive to reference-target distance, and SoundNeRirF stays as the best under all distance metrics.
Impact of Source Number .
The performance w.r.t.
sound source number is shown in Fig.
5 right, from which we can see the three comparison methods reduce in accuracy as the number of sources increases, with WarpNet [48] suffering the most.
SoundNeRirF remains the best-performing method, owing to its design in learning implicit neural r2r-RIR field.
4. A BLATION S TUDY
[Table]
To test the necessity and effectiveness of each component of SoundNeRirF design, we test three
SoundNeRirF variants: 1) SoundNeRirF without physical constraints (SoundNeRirF_noPhys), which helps test the necessity of explicitly setting physical constraints; 2) SoundNeRirF without explicitly modelling inverse-RIR and forwardRIR (SNeRirF_1MLP, instead using one MLP to predict r2r-RIR); 3) SoundNeRirF without receiver virtual position setting (SNeRirF_noVit).
The result is given in Table 7 (T-MSE: 10 − 3 ), from which we can see the necessity of each SoundNeRirF component.
Removing each component inevitably leads to inferior performance.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table] [Table]
[Figure]
4. E XPERIMENTAL R ESULT ON R EAL -W ORLD D ATASET
The result on real-world dataset is given in Table 3 (one source) and Table 4 (two sources)
(T-MSE: 10 − 4 ).
We can learn that 1) while all methods have sharp performance drop when transiting from one sound source to two sound sources, SoundNeRirF largely mitigates the dilemma and always performs the best.
Typical SoundNeRirF learned RIRs are shown in Fig.
6 and Fig.
VI,V,VII in Appendix, with reference and target receiver in different positions (lab or corridor).
We see that SoundNeRirF successfully characterizes the room acoustic and learns different r2r-RIRs for different positions.
[Table]
Cross-Wall Prediction Discussion We further divide the reference and target receivers in the test set into two main categories: those in the lab room and those in the corridor, then report the performance by differentiating the reference and target receiver positions.
We thus get four detailed results: Lab2Cor, Lab2Lab, Cor2Cor, and Cor2Lab (in reference-to-target format,
Lab2Cor indicates the reference receiver is in the lab room, and the target receiver is in a corridor).
The result is shown in Table 9.
We can see 1) SoundNeRirF outperforms all comparing methods in all Lab/Cor categories.
2) WaveNet shows advantage in Lab2Cor and Cor2Cor sound prediction, but performs poorly in the other two categories.
More discussion is in Appendix Sec.
C.1 and Table II.
4. E XPERIMENTAL R ESULT ON C ROSS -R OOM D ATASET
The result on cross-room data is given in Table 5 (T-MSE: 10 − 4 ), from which we can see that
SoundNeRirF stays as the best-performing method, it outperforms existing methods by a largemargin (especially the two non-learning baselines).
It thus shows SoundNeRirF generalizes well to large and complex room acoustic environment where various physical impediments, and room layout exist .
Non-learning based methods lead to the worst performance (huge performance drop), which shows received sounds largely vary w.r.t.
position displacement in complex room environment.
Inference Time/Parameter Size result is in in Table 6.
The average inference time is computed by averaging 100 independent tests (in each test, we use one reference sound to predict one target sound) on two hardwares: 1.6 GHz Intel Core i
CPU (single core) and Raspberry Pi 4.
For Raspberry Pidevice, we covert models to ONNX-compatible models (WarpNet [48] model and TCN [2] model cannot be converted to ONNX-compatible models).
We can see SoundNeRirF has small parameter size (just larger than WaveNet), and is extremely efficient .
We refer readers to Appendix C, D for more discussion, including learned r2r-RIR visualization, extra results and noise discussion.
Appendix
H to potential application discussion.
In sum, SoundNeRirF exhibits superiority in predicting sound to be heard at novel positions.
It generalizes well for unheard sound prediction and requires much less training data than existing methods.
It can also handle physical impediment.
C ONCLUSION AND L IMITATION D ISCUSSION
In this work, we have proposed a receiver-to-receiver acoustic neural field for predicting the sound to be heard at novel positions, given the sound heard at known positions.
We assume the acoustic scene is static and sound sources position is fixed.
Moreover, we just focus on mononaural sound.
One future work is to extend to binaural sound prediction.
A PPENDIX A A PPENDIX 1 : M ORE D ISCUSSION ON D ATASET A. D ISCUSSION ON S YNTHETIC D ATASET
In the synthetic dataset, we use Pyroomacoustics [51] to simulate a shoebox-like room environment of size [50 m × 50 m × 30 m ] .
The reason for creating a room of such size is two-fold:
A large room intrinsically involves large reverberation difference.
Sound recorded by receivers at different positions thus varies dramatically, in terms of sound waveform amplitude and other room acoustic related properties.
Simulating such a room better fits for testing algorithms’ capability in modelling room acoustics.
The simulated room size is complementary to the real-world dataset environment roughly of size [12 m × 12 m × 3 m ] , which is much smaller than simulated room and robot was collecting sound data within nearly a plane (the axis pointing to sky, or z - axis are nearly the same).
This is in contrast with the synthetic dataset in which the receivers used to collect the sound are randomly set along the x -, y - and z - axes, filling in the whole room.
To accurately model sound propagation process, we combine ray tracing [27] and image source method [1] to derive the sound recorded by any receiver.
Combining ray tracing and image source method strikes a good balance between recorded sound accuracy and computational cost.
For the room simulation, we set the origin point [0 , 0 , 0] to be located at one corner, and all receivers’ position coordinates are larger than 0.
[Figure]
[Figure]
In the real-world environment, we have collected 65 receiver recordings in the two adjacent corridors, and 55 receiver recordings in the lab environment.
These receiver locations cover all navigable area in the lab and two corridors.
The train/test split is randomly chosen and uniformly covers both the lab and two corridors.
In the real-world dataset, the x - axis range is [ − 2 . 0 m, 4 . 0 m ] , y - axis range is [ − 6 .
m, 6 . 0 m ] , z - range is [0 . 0 m, 0 .
3 m ] .
The plane formed by x - and y - axes is the 2D floor-plan map as shown in the main paper.
The TurtleBot sound receiver positions are shown in Fig.
I and detailed setup is in Fig.
II, in which we can observe that the positions TurtleBot has traversed uniformly cover both the lab area and two adjacent corridors.
The receivers used to train (color red) and test (color test) also cover the lab area and two corridors.
A. D ISCUSSION ON C ROSS -R OOM D ATA
[Figure]
To test the performance in room scenes that contain multiple rooms, complex room layout (e.g. room geometric features, sofa, bed and other furniture), we further run experiments on large-scale acoustic-realistic room environment platform SoundSpaces-2.0 [10].
SoundSpaces-2.0 supports
Matterport3D photo-realistic indoor environment [7], it synthesizes high-fidelity sound (both binaural and monoaural) for any given source position and receiver position.
In our setting, we leverage the embodied robot to efficiently explore 5 room scenes.
The robot step size is 1 .
m and turn-angle is 30 ◦ , it contains three actions: move-forward , turn-left and turn-right .
At each time step, it records a mono-channel sound that is 1 second long sound.
Each room contain five sound sources that across the whole area, both emitting telephone-ring sound.
Figure III shows two scene examples and embodied robot exploration trajectory, from which we can see Matterport 3D room scenes represent real-world large indoor room environment where cluttered physical impediments and complex room layout exist.
In total, we have collected 5000 sound recordings at different positions throughout the 5 room scenes (each room with 1000 recordings).
For the 1000 recordings in each room, we randomly split it into 700 for train and 300 for test, respectively.
We adopt the same training strategy to the all models, as we do for the other two datasets.
B A PPENDIX 2 : M ORE D ISCUSSION ON C OMPARING M ETHODS B. T WO N ON -L EARNING B ASELINES
In our experimental setting, we have a set of receivers’ recordings that distribute a room area.
We use these known recordings to predict the sound to be heard at novel positions.
Two simple baselines are just interpolating the sound at novel position from a subset of receiver recordings with known sound and positions.
Here we introduce two interpolation based baselines:
• neighboring linear-interpolation ( LinInterp ): for each test position (novel position), we query its closest 5 positions with known sound.
Then we linearly interpolate the sound to be heard at the test position by merging the 5 neighboring sound recordings with per-position weight, each weight of the sound at a known position is inversely proportional to its distance to the test position and all weights sums to 1 after softmax constraint.
• nearest neighbor ( NNeigh ): without any learning or interpolation process, we simply treat the nearest neighbor sound as the sound to be heard at any test (novel) position.
This baseline
[Table]
helps to test if deep neural network based methods truly learn meaningful representation to sound rendering.
From the results on the three datasets, we can learn that 1)
The two non-learning based baselines (LinInterp and NNeigh) lead to inferior performance than the other four deep neural network based methods (three comparing methods and our SoundNeRirF); 2) LinInterp performs better than NNeigh, which shows incorporating neighboring sounds can better capture the sound characteristic at a novel position.
In summary, we can conclude that: explicit learning process is essential for accurate room sound rendering, it helps to better characterize room acoustics .
B. T HREE L EARNING -B ASED M ETHODS
There are no existing methods that directly model receiver-to-receiver sound neural field.
We thus compare SoundNeRirF with three relevant methods: WarpNet [48], TCN [2] and WaveNet [38].
WarpNet [48] synthesizes binaural audio from monoaural audio.
It learns a warping field that maps the monoaural audio to binaural audio.
In our implementation, we feed receiver position [ x, y, z ] to warping module to learn the warp field.
TCN [2] adopts convolution and recurrent neural network to model sequence2sequence transform.
WaveNet [38] adopts dilated convolution to predict future sound, based on previous heard sound.
In our implementation of TCN [2] and WaveNet [38], we concatenate receiver position [ x, y, z ] to the input sound waveform and feed their concatenation to neural network.
Please note that normalization of the input receiver position [ x, y, z ] is applied as well, as we do in SoundNeRirF.
The comparison of learnable parameter number among these methods are shown in Table 6.
We can see from table that SoundNeRirF parameter number is much smaller than those of TCN [2] and
WarpNet [48].
SoundNeRirF parameter number is larger than the parameter number of WaveNet [38], mainly due to the two paralleling MLPs for modelling inverse-RIR and forward-RIR respectively and layerwise feature communication strategy.
We find that SoundNeRirF without learning reverse-RIR and forward-RIR respectively, but instead use one MLP to learn one general RIR (which means we just use 6-layer MLP to learn r2r-RIR, we call it SoundNeRirF_1MLP) achieves comparable and even better performance than WaveNet [38] on the synthetic dataset.
The result is shown in Table I, from which we can see SoundNeRirF_1MLP outperforms WaveNet [38] in terms of TF-MSE and
CDPM metrics, with a much smaller parameter number.
It thus shows the advantage of SoundNeRirF in modelling receiver-to-receiver sound neural field.
The average inference time of the comparing three methods and SoundNeRirF is given in Table 6.
The average inference time is computed by averaging 100 independent tests (in each test, we use one reference sound to predict one target sound) on two hardwares: 1.6 GHz Intel Core i
CPU (single core) and Raspberry Pi 4.
For Raspberry Pi 4 device, we covert trained models to ONNX-compatible models and run ONNX models on the device.
We find that WarpNet [48] model and TCN [2] model cannot be converted to ONNX-compatible models successfully due to various issues, such as operator incompatibility and unsupported operators.
Therefore, we only report the inference time of WaveNet [38] and SoundNeRirF. From Table 6, we can clearly see that SoundNeRirF’s inference time is much shorter than the three comparing methods’ inference time, with just 1/200 of the three comparing methods’ inference time on CPU device, 1/150 of WaveNet [38] inference time on Raspberry Pi 4 (although SoundNeRirF has more trainable parameters than WaveNet).
Therefore, it shows the high efficiency of SoundNeRirF .
C A PPENDIX 3 : M ORE E XPERIMENTAL R ESULTS C. T WO NON - LEARNING BASELINES ON CROSS LAB - CORRIDOR EXPERIMENT
We note that the neighboring sound recordings collected for the two non-learning based baselines lie very close to the test position.
As a result, the sound recordings used to interpolate and the sampled nearest neighbor sound spatially lie in the same local room environment with the sound to be predicted (e.g. both lie in the lab environment or corridor environment, see Fig.
I).
To better test the two non-learning baselines’ capability in predicting sound of further distance, we specifically compartment all sound recordings with known positions into different room environments (e
[Table]
g. lab and corridor), and just use sound recordings in one environment to predict the sound in another room environment.
The result (under TF-MSE evaluation metrics) is shown in Table II
[Table]
from which we can see that the performance of LinInterp and NNeigh deteriorates significantly when predicting sound in one room environment by sound recordings from another different room environment (the largest
TF-MSE increase by Lab2Cor and Cor2Lab).
It thus shows explicitly learning process is required to implicitly encoding room acoustic properties
[Table] [Table]
C. M ORE Q UALITATIVE R ESULTS , L EARNED R 2 R -RIR V ISUALIZATION
We first provide two learned r2r-RIR visualizations on MP3D dataset [7].
We provide one corridor-tocorridor (Cor2Cor, both reference and target position are in corridor) learned receiver-to-receiver RIR in Fig.
IV.
From this figure, we can clearly see that in large spatial area MP3D dataset [7] where the average area is larger than 100 m 2 , the direct-path RIR, specular-reflection RIR and late-reverberation
RIR learn different time shift parameters.
Such different time-shift parameters characterize room acoustics properties in large spatial area where the direct-path sound arrives to a receiver position earlier than specular-reflection sound, specular sound arrives earilier than later-reverberation sound.
We also provide corridor-to-corridor (Cor2Cor, both reference position and target position are in corridor) learned receiver-to-receiver RIR in Fig.
V, lab-to-corridor (Lab2Cor, reference position in lab, target position are in corridor) learned receiver-to-receiver RIR in Fig.
VI. lab-to-lab (Lab2Lab, both reference and target position are in lab) learned receiver-to-receiver RIR in Fig.
VII.
From the three figures, we can see that in much smaller area in our real-world dataset, the learned time-shift difference between direct-path, specular-reflection and later-reverberation is much smaller.
Such phenomena is compatible with real-scenario room acoustics property which shows that in small area the arrival time for direct-path, specular-reflection and late-reverberation is very close.
In summary, from such learned r2r-RIR visualizations, we can conclude that SoundNeRirF learned
RIR is capable of characterizing 3D room scene acoustically, from the receiver-to-receiver modelling perspective.
C. M ORE A BLATION S TUDY
In the main paper, we have tested three SoundNeRirF variants: 1) SoundNeRirF without physical constraints (SoundNeRirF_noPhys), which helps test the necessity of explicitly setting physical constraints; 2) SoundNeRirF without explicitly modelling inverse-RIR and forward-RIR (SNeRirF_1MLP, instead using one MLP to predict r2r-RIR.
It achives slightly better performance than
WaveNet [38]); 3) SoundNeRirF without receiver virtual position setting (SNeRirF_noVit).
The
[Figure]
t
[Figure] [Figure]
t1 =
t
[Figure]
t
[Figure] [Figure]
t
[Figure] [Figure]
t
[Figure] [Figure]
corresponding result on synthetic dataset shows the necessity and importance of each compoment of
SoundNeRirF design.
In addition to the three ablation studies
[Figure]
we further conduct three more studies to figure out the necessity of setting different monotonicity constraints to inverse-RIR and forward-RIR respectively.
Specifically, the necessity of increasing monotonicity to inverse-RIR and decreasing monotonicity
[Figure]
[Figure]
to forward-RIR.
We test two SoundNeRirF variants: both inverse-RIR and forward-RIR are with increasing monotonicity constraints (SoundNeRirF_incmono), with decreasing monotonicity constraints (SoundNeRirF_decmono), and swap the original constraint (SoundNeRirF_swapmono, use decreasing monotonicity for inverse-RIR and increasing monotonicity for forward-RIR).
The result is given in Table III.
We can conclude by this table that applying the same monotonicity constraints (either increasing or decreasing) to both inverse-RIR and forward-RIR leads to inferior performance.
Swapping the monotonicity generates the worst performance.
It in turn shows the necessity of setting increasing monotonicity to inverse-RIR and decreasing monotonicity to forward-RIR.
C. M ORE EXPERIMENT IN R EAL -W ORLD D ATASET
In the main paper, we have reported the quantitative result on both one sound source and two sound sources setting.
The result on unheard sound test is given in Table IV.
From this table, we can see that SoundNeRirF performs the best for unheard sound prediction in real-world setting.
C. N OISE D ISCUSSION
In real-scenario, robot position localization often suffers from noise, in which the robot cannot localize itself with high-precision.
In our real-world sound data collection platform, Vicon localization system [32] gives very high-precision position coordinates of the robot at each position.
To test the
[Table] [Table]
robustness of SoundNeRirF towards position noise interference, we specifically add noise pollution to
Vicon provided position coordinates and test all models’ performance over position noise interference.
Specifically, we model noise as Gaussian noise with mean 0 and standard deviation σ , G ( µ = 0 , σ ) .
Each coordinate x , y or z is added a noise sampled from the Gaussian noise.
By varying the standard deviation σ , we test all models’ tolerance w.r
[Table]
t. different level of noise interference.
In this experiment, we choose three noise levels, with σ = 0 .
5 , 1 . 0 , 1 . 5 , respectively.
The result is shown in Table V, from which we can see that SoundNeRirF exhibits stronger rubustness towards noise interference than three comparing methods.
D M ORE D ISCUSSION ON E XPERIMENTAL S ETTING
For each receiver position [ x, y, z ] , we first normalize each coordinate into [ − 1 , 1] before feeding to position encoding module, as WarpNet [48] does.
In position encoding, the corresponding position encoding embedding size is 384.
The learned time shift parameters for both the inverse-RIR and forward-RIR are quite small (both are less than 5).
Unheard Sound Test Explanation .
In our experimental settings, in each sound source and receiver positions, we collect two sounds.
While one sound is used for training the model, the other sound is hidden for training but instead is used for testing.
Since the two sounds theoretically share the same receiver-to-receiver RIR, such “unheard sound test” helps test each model’s generalization capability in predicting new sound.
Position Encoding since deep neural networks tend to learn lower frequency functions [43], we follow the practice in 3D protein structure modelling [63], Transformer network [56] and RGB image based neural radiance field [33] to encode the the low-dimensional position into high dimension.
For example, for the reference position p r (also for the two virtually set positions 2 · p r and 3 · p r ), the position encoding process works as,
[Equation]
[Equation]
) where p r = [ x r , y r , z r ] , and L is the position encoding length, we use 64 in our work (so the position embedding size is 384).
The target position p t goes the same process.
[Table]
[Table] [Table]
[Table] [Table]
E S OUND N E R IR F N ETWORK A RCHITECTURE
SoundNeRirF network architecture is given in Table VI.
In the layerwise communication, a trainable weight w is used to merge the intermediate features arising from reference position MLP (e.g. f 1 ) and target position MLP (e.g. f 2 ) before feeding them to the next layer, respectively,
[Equation]
F E XPLANATION ON F IG . 5 IN M AIN P APER
[Table]
In Fig.
5 of the main paper, the curves in the middle and right figures are not associated with the exact value, and we re-scaled the y - axis for better comparison.
To present more quantitative comparison, we provide the exact value for each curve.
The result of TF-MSE variation change w.r.t.
referencetarget distance is given in Table VII.
The result of TF-MSE variation change w.r.t.
sound source number is given in Table VIII.
G P REDICTED S OUND S AMPLES
We also provide predicted sound samples by each method with our real-world dataset, as well as the ground truth sound in the supplementary material.
The predicted sound samples use the reference sound in the lab area and target sound in the corridor area.
[Table]
H P OTENTIAL S OUND N E R IR F A PPLICATION IN R OBOTICS
The motivation of this work is to enable the robot to efficiently predict the sound to be heard at novel positions, without reply on enough prior information about the room geometric features/sound source installation and rigorous and complex mathematical derivation.
The robot can efficiently and effectively learn such neural acoustic field by simply exploring a new environment to record the position and received sound at each step.
We further show that learning such sound neural field through our proposed SoundNeRirF framework is easy to implement on a robot.
Our experiment shows our method just needs a relatively small number of receiver recordings to learn an efficient sound neural field (Fig. 5, left-most figure).
So the robot doesn’t have to walk a long time (exhaustive walking).
At the same time, SoundNeRirF is light-weight, it achieves real-time inference even on resource-constrained edge device.
For example, in the Table 6, SoundNeRirF takes 0.066s to predict the sound on Raspberry Pi.
This advantage enables the robot to predict/render sound in real-time, so it fits for tasks that lay much emphasis on efficiency.
We argue that the learned SoundNeRirF can be used for various down-stream robot-related tasks.
As far as we can see at current moment, it includes but is not limited to: 1.
SoundNeRirF can be used to improve robot self-localization capability, by comparing the predicted sound and truly recorded sound at a particular position; 2.
SoundNeRirF can be used for embodied robotics research in synthesized audio-vision environment.
For example, the audio-vision platform for embodied robotics SoundSpaces1.0 just provides discrete and pre-computed RIR, so the whole environment is not dynamic and the robot can only hear sound at particular positions.
SoundNeRirF can help to avoid such obstacles, enabling the robot to render sound at arbitrary positions.
VR/AR applications for immersive acoustic experience (audio auralisation).