Original: N EURAL S YNTHESIS OF B INAURAL S PEECH Anonymous authors
Revised: N EURAL S YNTHESIS OF B INAURAL S PEECH FROM M ONO A UDIO

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a neural rendering approach for binaural sound synthesis that can produce realistic and spatially accurate binaural sound in realtime.
Revised: We present a neural rendering approach for binaural sound synthesis that can produce realistic and spatially accurate binaural sound in realtime.

Original: The network takes, as input, a single-channel audio source and synthesizes, as output, two-channel binaural sound, conditioned on the relative position and orientation of the listener with respect to the source.
Revised: The network takes, as input, a single-channel audio source and synthesizes, as output, twochannel binaural sound, conditioned on the relative position and orientation of the listener with respect to the source.

Original: We investigate deﬁciencies of the (cid:96) 2 -loss on raw waveforms in a theoretical analysis and introduce an improved loss that overcomes these limitations.
Revised: We investigate deﬁciencies of the (cid:96) 2 -loss on raw wave-forms in a theoretical analysis and introduce an improved loss that overcomes these limitations.

Original: In an empirical evaluation, we establish that our approach is the ﬁrst to generate spatially accurate waveform outputs (as measured by real recordings) and outperforms existing approaches by a considerable margin, both quantitatively and in a perceptual study.
Revised: In an empirical evaluation, we establish that our approach is the ﬁrst to generate spatially accurate waveform outputs (as measured by real recordings) and outperforms existing approaches by a considerable margin, both quantitatively and in a perceptual study.

Original: We will release a ﬁrst-of-its-kind binaural audio dataset as a benchmark for future research.
Revised: Dataset and code are available online.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The rise of artiﬁcial spaces, in augmented and virtual reality, necessitates efﬁcient production of accurate spatialized audio.
Revised: The rise of artiﬁcial spaces, in augmented and virtual reality, necessitates efﬁcient production of accurate spatialized audio.

Original: Spatial hearing (the capacity to interpret spatial clues from binaural signals), not only helps us to orient ourselves in 3D environments, it also establishes immersion in the space by providing the brain with congruous acoustic and visual input (Hendrix & Barﬁeld, 1996).
Revised: Spatial hearing (the capacity to interpret spatial clues from binaural signals) not only helps us to orient ourselves in 3D environments, it also establishes immersion in the space by providing the brain with congruous acoustic and visual input (Hendrix & Barﬁeld, 1996).

Original: Binaural audio (left and right ear) even guides us in multi-person conversations: consider a scenario where multiple persons are speaking in a video call, making it difﬁcult to follow the conversation.
Revised: Binaural audio (left and right ear) even guides us in multi-person conversations: consider a scenario where multiple persons are speaking in a video call, making it difﬁcult to follow the conversation.

Original: In the same situation in a real environment we are able to effortlessly focus on the speech from an individual (Hawley et al., 2004).
Revised: In the same situation in a real environment we are able to effortlessly focus on the speech from an individual (Hawley et al., 2004).

Original: Indeed, auditory sensation has primacy over even visual sensation as an input modality for scene understanding: (1) reaction times are faster for auditory stimulus compared to visual stimulus (Jose & Praveen, 2010)
Revised: Indeed, auditory sensation has primacy over even visual sensation as an input modality for scene understanding: (1) reaction times are faster for auditory stimulus compared to visual stimulus (Jose & Praveen, 2010)

Original: (2) auditory sensing provides a surround understanding of space as opposed to the directionality of visual sensation.
Revised: (2) auditory sensing provides a surround understanding of space as opposed to the directionality of visual sensation.

Original: For these reasons, the generation of accurate binarual signal is integral to full immersion in artiﬁcial spaces.
Revised: For these reasons, the generation of accurate binarual signal is integral to full immersion in artiﬁcial spaces.

Original: Most approaches to binaural audio generation rely on traditional digital signal processing (DSP) techniques, where each component – head related transfer function, room acoustics, ambient noise – is modeled as a linear time-invariant system (LTI) (Savioja et al., 1999;
Revised: Most approaches to binaural audio generation rely on traditional digital signal processing (DSP) techniques, where each component – head related transfer function, room acoustics, ambient noise – is modeled as a linear time-invariant system (LTI) (Savioja et al., 1999;

Original: Zotkin et al., 2004; Sunder et al., 2015; Zhang et al., 2017).
Revised: Zotkin et al., 2004; Sunder et al., 2015; Zhang et al., 2017).

Original: These linear systems are well-understood, relatively easy to model mathematically, and have been shown to produce perceptually plausible results – reasons why they are still widely used.
Revised: These linear systems are well-understood, relatively easy to model mathematically, and have been shown to produce perceptually plausible results – reasons why they are still widely used.

Original: Real acoustic propagation, however, has nonlinear wave effects that are not appropriately modeled by LTI systems.
Revised: Real acoustic propagation, however, has nonlinear wave effects that are not appropriately modeled by LTI systems.

Original: As a consequence, DSP approaches do not achieve perceptual authenticity in dynamic scenarios (Brinkmann et al., 2017), and fail to produce metrically accurate results, i.e., the generated waveform does not resemble recorded binaural audio well.
Revised: As a consequence, DSP approaches do not achieve perceptual authenticity in dynamic scenarios (Brinkmann et al., 2017), and fail to produce metrically accurate results, i.e., the generated waveform does not resemble recorded binaural audio well.

Original: In this paper, we present an end-to-end neural synthesis approach that overcomes many of these limitations by efﬁciently synthesizing accurate and precise binaural audio.
Revised: In this paper, we present an end-to-end neural synthesis approach that overcomes many of these limitations by efﬁciently synthesizing accurate and precise binaural audio.

Original: The end-to-end learning scheme naturally captures the linear and nonlinear effects of sound wave propagation and, being fully convolutional, is efﬁcient to execute on commodity hardware.
Revised: The end-to-end learning scheme naturally captures the linear and nonlinear effects of sound wave propagation and, being fully convolutional, is efﬁcient to execute on commodity hardware.

Original: Our major contributions are (1) a novel binarualization model that outperforms existing state of the art, (2) an analysis of the shortcomings of the (cid:96) 2 -loss on raw waveforms and a novel loss mitigating these shortcomings, (3) a real-world binaural dataset captured in a non-anechoic room.
Revised: Our major contributions are (1) a novel binarualization model that outperforms existing state of the art, (2) an analysis of the shortcomings of the (cid:96) 2 -loss on raw waveforms and a novel loss mitigating these shortcomings, (3) a real-world binaural dataset captured in a non-anechoic room.

Original: Related Work. State of the art DSP techniques approach binaural sound spatialization as a stack of acoustic components, each of which is an LTI system. As accurate wave-based simulation of room impulse responses is computationally expensive and requires detailed geometry and material information, most real-time systems rely on simpliﬁed geometrical models (V¨alim¨aki et al., 2012; Savioja x 1: T c 1: T
Revised: x 1: T c 1: T

Original: Neural Time Warping neuralwarp geometricwarp ⊕ warp activation ρ 1: T warp x ( l ) 1: T x ( r ) 1: T
Revised: Neural Time Warping neuralwarp geometricwarp ⊕ warp activation ρ 1: T warp x ( l ) 1: T x ( r ) 1: T

Original: Temporal ConvNet
Revised: Temporal ConvNet

Original: H y p
Revised: H y p

Original: e r
Revised: e r

Original: C o n v
Revised: C o n v

Original: L a y e r ...
Revised: L a y e r ...

Original: H y p
Revised: H y p

Original: e r
Revised: e r

Original: C o n v
Revised: C o n v

Original: L a y e r
Revised: L a y e r

Original: N y ( l ) 1: T y ( r ) 1: T
Revised: N y ( l ) 1: T y ( r ) 1: T

Revised: Related Work.

Revised: State of the art DSP techniques approach binaural sound spatialization as a stack of acoustic components, each of which is an LTI system.

Revised: As accurate wave-based simulation of room impulse responses is computationally expensive and requires detailed geometry and material information, most real-time systems rely on simpliﬁed geometrical models (V ¨ alim ¨ aki et al., 2012; Savioja & Svensson, 2015).

Revised: Head-related transfer functions are measured in an anechoic chamber (Li &

Revised: Peissig, 2020) and high-quality spatialization requires binaural recordings at almost 10k different spatial positions (Armstrong et al., 2018).

Original:  Given their success in speech synthesis (Wang et al., 2017) neural networks gained increased attention for audio generation recently.
Revised: To generate binaural audio the DSP-based binaural renderers typically perform a series of convolutions with these component impulse responses (Savioja et al., 1999; Zotkin et al., 2004; Sunder et al., 2015; Zhang et al., 2017). For a more detailed discussion, see Appendix A.4. Given their success in speech synthesis (Wang et al., 2017), neural networks gained increased attention for audio generation recently.

Original: While most approaches focus on models in frequency domain (Choi et al., 2018; Vasquez & Lewis, 2019), raw waveform models were long neglected due to the difﬁculty to model long-range dependencies on a high-frequency audio signal.
Revised: While most approaches focus on models in frequency domain (Choi et al., 2018; Vasquez & Lewis, 2019), raw waveform models were long neglected due to the difﬁculty to model long-range dependencies on a high-frequency audio signal.

Original: With the success of WaveNet (Van Den Oord et al., 2016) however, direct wave-to-wave modeling is of increasing interest (Fu et al., 2017; Luo & Mesgarani, 2018; Donahue et al., 2019) and shows major improvements in speech enhancement (Defossez et al., 2020) and denoising (Rethage et al., 2018), speech synthesis (Kalchbrenner et al., 2018), and music style translation (Mor et al., 2019).
Revised: With the success of WaveNet (Van Den Oord et al., 2016) however, direct wave-to-wave modeling is of increasing interest (Fu et al., 2017; Luo & Mesgarani, 2018; Donahue et al., 2019) and shows major improvements in speech enhancement (Defossez et al., 2020) and denoising (Rethage et al., 2018), speech synthesis (Kalchbrenner et al., 2018), and music style translation (Mor et al., 2019).

Original: More recently, ﬁrst steps towards neural sound spatialization have been undertaken with a focus on predicting spatial sound conditioned on visual information.
Revised: More recently, ﬁrst steps towards neural sound spatialization have been undertaken.

Original: One of the ﬁrst works by Morgado et al.
Revised: Gebru et al. (2021) showed that HRTFs can be implicitly learned by neural networks trained on raw waveforms. Focusing on predicting spatial sound conditioned on visual information, a work by Morgado et al.

Original: (2018) aims to spatialize sound conditioned on 360 ◦ video.
Revised: (2018) aims to spatialize sound conditioned on 360 ◦ video.

Original: Yet, their work is limited to ﬁrst order ambisonics and can not model detailed binaural effects.
Revised: Yet, their work is limited to ﬁrst order ambisonics and can not model detailed binaural effects.

Original: More closely related is a line of papers originating from the 2.5D visual sound system by Gao & Grauman (2019b).
Revised: More closely related is a line of papers originating from the 2.5D visual sound system by Gao & Grauman (2019b).

Original: In this work, binaural audio is generated conditioned on a video frame embedding such that object locations can contribute to where sound comes from.
Revised: In this work, binaural audio is generated conditioned on a video frame embedding such that object locations can contribute to where sound comes from.

Original: Yang et al.
Revised: Yang et al.

Original: ; Lu et al.
Revised: ; Lu et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2020) build upon the same idea.
Revised: (2020) build upon the same idea.

Original: Unfortunately, all these works have in common that they pose the spatialization task as an upmixing problem, i.e., their models are trained with a mixture of left and right ear binaural recording as pseudo mono input.
Revised: Unfortunately, all these works have in common that they pose the spatialization task as an upmixing problem, i.e., their models are trained with a mixture of left and right ear binaural recording as pseudo mono input.

Original: By design, these methods fail to model time delays and reverberation effects caused by the difference between source and listener position.
Revised: By design, these methods fail to model time delays and reverberation effects caused by the difference between source and listener position.

Original:  A N EURAL N ETWORK FOR B INAURAL S YNTHESIS
Revised:  A N EURAL N ETWORK FOR B INAURAL S YNTHESIS

Original: We consider the problem where a monoaural (single-channel) signal x 1: T = ( x 1 , . . .
Revised: We consider the problem where a monaural (single-channel) signal x 1: T = ( x 1 , . . .

Original: , x T ) of length
Revised: , x T ) of length

Original: T is to be transformed into a binaural (stereophonic) signal y ( l ) 1: T , y ( r ) 1: T representing the listener’s left ear and right ear, given a conditioning temporal signal c 1: T .
Revised: T is to be transformed into a binaural (stereophonic) signal y ( l ) 1: T , y ( r ) 1: T representing the listener’s left ear and right ear, given a conditioning temporal signal c 1: T .

Original: This conditioning signal is the position and orientation of source and listener, respectively.
Revised: This conditioning signal is the position and orientation of source and listener, respectively.

Original: Here x t , and correspondingly y ( l ) t and y ( r ) t , are scalars representing an audio sample at time t .
Revised: Here x t , and correspondingly y ( l ) t and y ( r ) t , are scalars representing an audio sample at time t .

Original: In other words, we aim to produce a function,
Revised: In other words, we aim to produce a function,

Original: [Equation]
Revised: [Equation]

Original: where ∆ is a temporal receptive ﬁeld.
Revised: where ∆ is a temporal receptive ﬁeld.

Original: Each c t ∈ R 14 contains the 3D position of source and listener (three values each) and their orientations as quaternions (four values each).
Revised: Each c t ∈ R 14 contains the 3D position of source and listener (three values each) and their orientations as quaternions (four values each).

Original: Note that in practice, c often is of lower frequency than the input and output signals x 1: T and y ( l/r ) 1: T – source and listener positions would likely not be updated at 48kHz but rather at typical camera frame rates such as 30120Hz.
Revised: Note that in practice, c often is of lower frequency than the input and output signals x 1: T and y ( l/r ) 1: T – source and listener positions would likely not be updated at 48kHz but rather at typical camera frame rates such as 30120Hz.

Original: To simplify notation, we assume that c has already been upsampled to the same temporal resolution as the audio signals.
Revised: To simplify notation, we assume that c has already been upsampled to the same temporal resolution as the audio signals.

Original: Our overall framework is shown in Figure 1.
Revised: Our overall framework is shown in Figure 1.

Original: A neural time warping module ﬁrst warps the singlechannel input signal x 1: T into a two-channel signal x ( l/r ) 1: T , where the channels represent left and right ear.
Revised: A neural time warping module ﬁrst warps the singlechannel input signal x 1: T into a two-channel signal x ( l/r ) 1: T , where the channels represent left and right ear.

Original: The time warping compensates for coarse temporal effects and differences in time of sound arrival at left and right ear caused by the distance between source and listener.
Revised: The time warping compensates for coarse temporal effects and differences in time of sound arrival at left and right ear caused by the distance between source and listener.

Original: The second block in Figure 1 is a stack of N layers, each of which is a conditioned hyper-convolution (see
Revised: The second block in Figure 1 is a stack of N layers, each of which is a conditioned hyper-convolution (see

Original: Section 2.2) followed by a sine activation, which has been shown to be beneﬁcial for modeling higher frequencies (Sitzmann et al., 2020).
Revised: Section 2.2) followed by a sine activation, which has been shown to be beneﬁcial for modeling higher frequencies (Sitzmann et al., 2020).

Original: Following the design of WaveNet, we use kernel size 2 and double the dilation factor in each layer to increase the receptive ﬁeld.
Revised: Following the design of WaveNet, we use kernel size 2 and double the dilation factor in each layer to increase the receptive ﬁeld.

Original: This temporal ConvNet models nuanced effects caused by room reverberations, head and ear shape, or changing head orientations.
Revised: This temporal ConvNet models nuanced effects caused by room reverberations, head and ear shape, or changing head orientations.

Original: 2. N EURAL T IME W ARPING
Revised: 2. N EURAL T IME W ARPING

Original: Time warping is the task of mapping a source temporal sequence onto a target sequence and has a long tradition in temporal signal processing.
Revised: Time warping is the task of mapping a source temporal sequence onto a target sequence and has a long tradition in temporal signal processing.

Original: Most prominently, dynamic time warping (DTW) ﬁnds application in tasks like speech recognition (Juang, 1984) or audio retrieval (Deng & Leung, 2015).
Revised: Most prominently, dynamic time warping (DTW) ﬁnds application in tasks like speech recognition (Juang, 1984) or audio retrieval (Deng & Leung, 2015).

Original: DTW can be characterized as ﬁnding a warpﬁeld ρ 1: T that warps a source signal x 1: T to a target signal ˆ x 1: T such that the distance between the signals is minimized,
Revised: DTW can be characterized as ﬁnding a warpﬁeld ρ 1: T that warps a source signal x 1: T to a target signal ˆ x 1: T such that the distance between the signals is minimized,

Original: [Equation]
Revised: [Equation]

Original: where ρ t ∈ { 1 , . . .
Revised: where ρ [Equation]

Original: , T } , (1) where the warpﬁeld is typically constrained to respect physical properties such as monotonicity ( ρ t ≥ ρ t − 1 ) and causality ( ρ t ≤ t ).
Revised: [Equation] 1) where the warpﬁeld is typically constrained to respect physical properties such as monotonicity ( ρ t ≥ ρ t − 1 ) and causality ( ρ t ≤ t ).

Original: For binaural audio, there is a clear monotonous and causal relationship between source and target signal but the target signal is unknown at inference time.
Revised: For binaural audio, there is a clear monotonous and causal relationship between source and target signal but the target signal is unknown at inference time.

Original: Additionally, the warping from mono to binaural signals goes far beyond simple linear time-shifts.
Revised: Additionally, the warping from mono to binaural signals goes far beyond simple linear time-shifts.

Original: For example, consider the source moving from the front to the left of the listener.
Revised: For example, consider the source moving from the front to the left of the listener.

Original: This causes the delay between source and left ear to decrease but the delay between source and right ear to increase.
Revised: This causes the delay between source and left ear to decrease but the delay between source and right ear to increase.

Original: If source and listener areboth moving, other non-linear physical effects such as the Doppler effect inﬂuence how the signal needs to be warped from the source to the listener’s left and right ear.
Revised: If source and/or listener are moving, other wave effects such as the Doppler effect inﬂuence how the signal needs to be warped from the source to the listener’s left and right ear.

Original: We are therefore interested in estimating a(nonlinear) warpﬁeld from the conditioning input c 1: T , i.e., from the spatial position and orientation of source and listener.
Revised: We are therefore interested in estimating a warpﬁeld from the conditioning input c 1: T , i.e., from the spatial position and orientation of source and listener.

Original: A simple, parameter-free approach is geometric warping based on the speed of sound ν sound and the distance between source and listener.
Revised: A simple, parameter-free approach is geometric warping based on the speed of sound ν sound and the distance between source and listener.

Original: Let p (src) t and p (lstn) t be the source and listener positions at time t (which are part of c t ).
Revised: Let p (src) t and p (lstn) t be the source and listener positions at time t (which are part of c t ).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: audio sample rate
Revised: audio sample rate

Original: [Equation]
Revised: [Equation]

Original: This approach, however, fails to model important nuances such as the displacement between the left and right ear or diffraction delays as sound travels around the listener’s head rather than straight through.
Revised: This approach, however, fails to model important nuances such as the displacement between the left and right ear or diffraction delays as sound travels around the listener’s head rather than straight through.

Original: In order to correct for those effects that geometric warping can not model properly, we estimate a neural warpﬁeld ρ (neural) 1: T = WarpNet( c 1: T ) and add it to the geometric warpﬁeld (cf.
Revised: In order to correct for those effects that geometric warping can not model properly, we estimate a neural warpﬁeld ρ (neural) 1: T = WarpNet( c 1: T ) and add it to the geometric warpﬁeld (cf.

Original: Figure 1),ρ
Revised: Figure 1),

Original: [Equation]
Revised: [Equation]

Original: where σ (warp) ( ρ t − 1 , ˆ ρ t )
Revised: where σ (warp) ( ρ t − 1 , ˆ ρ t )

Original: =
Revised: =

Original: max( ρ t − 1 , min( t, ˆ ρ t )) is a recursive activation function that ensures monotonicity and causality.
Revised: max( ρ t − 1 , min( t, ˆ ρ t )) is a recursive activation function that ensures monotonicity and causality.

Original: The WarpNet is a shallow temporal convolutional network with four layers and 64 channels each.
Revised: The WarpNet is a shallow temporal convolutional network with four layers and 64 channels each.

Original: The warped signal can now be computed using the predicted warpﬁeld.
Revised: The warped signal can now be computed using the predicted warpﬁeld.

Original: Since the warpﬁeld elements ρ t are typically not integers, we deﬁne the warped signal ˆ x 1: T to be the linear interpolation of the original signal x 1: T at positions (cid:98) ρ t (cid:99) and (cid:100) ρ t (cid:101) ,
Revised: Since the warpﬁeld elements ρ t are typically not integers, we deﬁne the warped signal ˆ x 1: T to be the linear interpolation of the original signal x 1: T at positions (cid:98) ρ t (cid:99) and (cid:100) ρ t (cid:101) ,

Original: [Equation]
Revised: [Equation]

Original: In practice two warpﬁelds are generated, one for each ear.
Revised: In practice two warpﬁelds are generated, one for each ear.

Original: Note how we explicitly enforce physical constraints in the warping by σ (warp) : min( t, ˆ ρ t ) ensures causality by enforcing that the t -th element of the warpﬁeld can not be larger than t itself.
Revised: Note how we explicitly enforce physical constraints in the warping by σ (warp) : min( t, ˆ ρ t ) ensures causality by enforcing that the t -th element of the warpﬁeld can not be larger than t itself.

Original: Monotonicity is enforced by max( ρ t − 1 , · ) : if an element has been warped from ρ t − 1 to position t − 1 , the next element at position t must be warped from ρ t − 1 or a succeeding position.
Revised: Monotonicity is enforced by max( ρ t − 1 , · ) : if an element has been warped from ρ t − 1 to position t − 1 , the next element at position t must be warped from ρ t − 1 or a succeeding position.

Original: In contrast to related approaches such as deformable convolutions (Dai et al., 2017) and spatial transformer networks (Jaderberg et al., 2015), our neural time warping therefore allows for constrained warping of input signals with arbitrary lengths and directly models a physical phenomenon of sound.
Revised: In contrast to related approaches such as deformable convolutions (Dai et al., 2017) and spatial transformer networks (Jaderberg et al., 2015), our neural time warping therefore allows for constrained warping of input signals with arbitrary lengths and directly models a physical phenomenon of sound.

Original: 2. C ONDITIONED HYPER - CONVOLUTIONS
Revised: 2. C ONDITIONED HYPER - CONVOLUTIONS

Original: Raw waveform models where the output depends on an input signal and an additional conditioning temporal signal have primarily been studied in speech synthesis (Van Den Oord et al., 2016).
Revised: Raw waveform models where the output depends on an input signal and an additional conditioning temporal signal have primarily been studied in speech synthesis (Van Den Oord et al., 2016).

Original: The predominant approach towards such conditional temporal convolutions is an additive combination of the input signal x 1: T and the conditioning signal c 1: T , i.e., z 1: T
Revised: The predominant approach towards such conditional temporal convolutions is an additive combination of the input signal x 1: T and the conditioning signal c 1: T , i.e., z 1: T

Original: = W ∗ x 1: T
Revised: = W ∗ x 1: T

Original: + V ∗ c 1: T + b , such that the result of the convolution at time t is
Revised: + V ∗ c 1: T + b , such that the result of the convolution at time t is

Original: [Equation]
Revised: [Equation]

Original:  Here, W ∈ R C out × C in × K and V ∈ R C out × C cond × K are tensors containing the weights for temporal convolutions of the C in -dimensional input signal x 1: T and the C cond -dimensional
Revised: [Equation] Here, W ∈ R C out × C in × K and V ∈ R C out × C cond × K are tensors containing the weights for temporal convolutions of the C in -dimensional input signal x 1: T and the C cond -dimensional

Original: conditional signal c 1: T with a kernel size of K .
Revised: conditional signal c 1: T with a kernel size of K .

Original: Note that the convolutional weights W and V in this formulation are constant over time.
Revised: Note that the convolutional weights W and V in this formulation are constant over time.

Original: Binaural ﬁlters in traditional digital signal processing, on the contrary, depend on the position of the sound source.
Revised: Binaural ﬁlters in traditional digital signal processing, on the contrary, depend on the position of the sound source.

Original: Inspired by the DSP formulation, we predict the convolutional weights for the input x 1: T of a layer and the bias as functions of the conditioning input c 1: T ,
Revised: Inspired by the DSP formulation, we predict the convolutional weights for the input x 1: T of a layer and the bias as functions of the conditioning input c 1: T ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: This formulation is similar to the use of hyper-networks in Ha et al.
Revised: This formulation is similar to the use of hyper-networks in Ha et al.

Original: (2017) but rather than generating them from intermediate feature maps, weights are generated from the conditioning input c 1: T that contains physical information about the relation between source and listener.
Revised: (2017) but rather than generating them from intermediate feature maps, weights are generated from the conditioning input c 1: T that contains physical information about the relation between source and listener.

Original: H ( W ) and H ( b ) are small convolutional hyper-networks that receive c 1: t as input and predict the convolutional weights and the bias as their output, respectively.
Revised: H ( W ) and H ( b ) are small convolutional hyper-networks that receive c 1: t as input and predict the convolutional weights and the bias as their output, respectively.

Original: Therefore, not only is the input to the convolutional layer a temporal sequence but the weights and biases change over time as well.
Revised: Therefore, not only is the input to the convolutional layer a temporal sequence but the weights and biases change over time as well.

Original: We show in Appendix A. that if H ( W ) and H ( b ) are linear networks, hyper-convolutions equal equation 5 plus a biliear term.
Revised: We show in Appendix A. that if H ( W ) and H ( b ) are linear networks, hyper-convolutions equal equation 5 plus a biliear term.

Original: 2. D EFICIENCIES OF THE (cid:96) 2 - LOSS ON R AW W AVEFORMS
Revised: 2. D EFICIENCIES OF THE (cid:96) 2 - LOSS ON R AW W AVEFORMS

Original: Training a generative audio model with an (cid:96) 2 -loss on the raw waveform is generally considered to result in poor audio quality and distorted signals particularly for speech.
Revised: Training a generative audio model with an (cid:96) 2 -loss on the raw waveform is generally considered to result in poor audio quality and distorted signals particularly for speech.

Original: Therefore, a number of mostly spectrogram oriented alternative loss functions have been introduced over recent years (Kolbæk et al., 2020).
Revised: Therefore, a number of mostly spectrogram oriented alternative loss functions have been introduced over recent years (Kolbæk et al., 2020).

Original: Here, we provide an analytical explanation for a fundamental problem of phase estimation with the (cid:96) 2 -loss on the waveform and show that a simple additional loss term mitigates the problem.
Revised: Here, we provide an analytical explanation for a fundamental problem of phase estimation with the (cid:96) 2 -loss on the waveform and show that a simple additional loss term mitigates the problem.

Original: While correct phase estimation is not critical for single-channel audio, it is crucial for binaural audio as our ears are sensitive to interaural time differences as small as 10 µ s (Brown &
Revised: While correct phase estimation is not critical for single-channel audio, it is crucial for binaural audio as our ears are sensitive to interaural time differences as small as 10 µ s (Brown &

Original: Duda, 1998).
Revised: Duda, 1998).

Original: To start the analysis, let
Revised: To start the analysis, let

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  be the time-domain (cid:96) 2 -loss between the predicted audio signal y 1: T and the target ˆ y 1: T and let
Revised: ) be the time-domain (cid:96) 2 -loss between the predicted audio signal y 1: T and the target ˆ y 1: T and let

Original: Y k , ˆ Y k ∈ C denote the k -th frequency component of y 1: t and ˆ y 1: T in the Fourier domain.
Revised: Y k , ˆ Y k ∈ C denote the k -th frequency component of y 1: t and ˆ y 1: T in the Fourier domain.

Original: We denote the amplitude error and angular phase error of the k -th frequency component as
Revised: We denote the amplitude error and angular phase error of the k -th frequency component as

Original: [Equation]
Revised: [Equation]

Original: where | · | is the modulus (or magnitude) of the complex number.
Revised: where | · | is the modulus (or magnitude) of the complex number.

Revised: ε ( ‘ 2 -loss) | ˆ Y | (t a r g e t e n e r g y )

Revised: (a) E Y (cid:0) L (amp)

Revised: ( Y, ˆ Y ) (cid:1) ε ( ‘ 2 -loss) | ˆ Y | (t a r g e t e n e r g y )

Revised: (b) E Y (cid:0) L (phase)

Revised: ( Y, ˆ Y ) (cid:1)

Original: Lemma 1.
Revised: Lemma 1.

Original: Let ˆ Y ∈ C be a ﬁxed complex number and Y ∈ B ε, ˆ Y =
Revised: Let ˆ Y ∈ C be a ﬁxed complex number and Y ∈ B ε, ˆ Y =

Original: { Y ∈ C :
Revised: { Y ∈ C :

Original: | Y − ˆ Y | = ε } be any complex number that has distance ε from ˆ Y .
Revised: | Y − ˆ Y | = ε } be any complex number that has distance ε from ˆ Y .

Original: Then, the expected amplitude error and the expected angular phase error with respect to ˆ Y are
Revised: Then, the expected amplitude error and the expected angular phase error with respect to ˆ Y are

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: dϕ and

Original: [Equation]
Revised: [Equation]

Original: ε ( ‘ 2 -loss)

Original: | ˆ Y | (t a r g e t e n e r g y ) (a) E Y (cid:0) L (amp) ( Y, ˆ Y ) (cid:1) ε ( ‘ 2 -loss) | ˆ Y | (t a r g e t e n e r g y ) (b) E Y (cid:0) L (phase)
Revised: [Equation]

Original: ( Y, ˆ Y ) (cid:1)

Original: Proof.
Revised: Proof.

Original: See Appendix A.1.
Revised: See Appendix A.1.

Original: Using Parseval’s theorem, we write the time-domain (cid:96) 2 -loss as the (cid:96) 2 -loss on the complex spectrum,
Revised: Using Parseval’s theorem, we write the time-domain (cid:96) 2 -loss as the (cid:96) 2 -loss on the complex spectrum,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Now, consider a single summand from equation 11 and denote the distance | Y k − ˆ Y k | as ε .
Revised: Now, consider a single summand from equation 11 and denote the distance | Y k − ˆ Y k | as ε .

Original: Lemma 1 allows us to analyze the expected amplitude and phase errors along this k -th frequency component.
Revised: Lemma 1 allows us to analyze the expected amplitude and phase errors along this k -th frequency component.

Original: In Figure 2 we plot equation 9 and equation 10 as a function of the (cid:96) 2 -value ε and the target energy
Revised: In Figure 2 we plot equation 9 and equation 10 as a function of the (cid:96) 2 -value ε and the target energy

Original: | ˆ Y | .
Revised: | ˆ Y | .

Original: There are two key insights.
Revised: There are two key insights.

Original: First , the expected amplitude error is low even for large (cid:96) 2 -values – that is, in the early stage of training – as long as the target signal has high energy (top right part of
Revised: First , the expected amplitude error is low even for large (cid:96) 2 -values – that is, in the early stage of training – as long as the target signal has high energy (top right part of

Original: Figure 2a).
Revised: Figure 2a).

Original: The phase, on the contrary, is barely optimized at all early in training when the (cid:96) 2 -loss is large, even for high energy components, see Figure 2b.
Revised: The phase, on the contrary, is barely optimized at all early in training when the (cid:96) 2 -loss is large, even for high energy components, see Figure 2b.

Original: Second , over the course of training, i.e., when the (cid:96) 2 -loss decreases over time, the expected amplitude error among all target energies decreases.
Revised: Second , over the course of training, i.e., when the (cid:96) 2 -loss decreases over time, the expected amplitude error among all target energies decreases.

Original: The expected phase error, on the other hand, improves primarily for high energy components and mid- and low energy components tend to have poor phase accuracy even for small (cid:96) 2 -values.
Revised: The expected phase error, on the other hand, improves primarily for high energy components and mid- and low energy components tend to have poor phase accuracy even for small (cid:96) 2 -values.

Original: The above analysis shows that optimizing raw waveforms with a time-domain (cid:96) 2 -loss leads to a strong focus on ﬁtting the amplitudes but accurate phase reconstruction falls short.
Revised: The above analysis shows that optimizing raw waveforms with a time-domain (cid:96) 2 -loss leads to a strong focus on ﬁtting the amplitudes but accurate phase reconstruction falls short.

Original: Since the models have limited capacity, the training data usually can only be ﬁt up to an (cid:96) 2 -loss ε min .
Revised: Since the models have limited capacity, the training data usually can only be ﬁt up to an (cid:96) 2 -loss ε min .

Original: If this ε min is not sufﬁciently small, the signal’s amplitude can be modeled well but phase errors will always be signiﬁcant.
Revised: If this ε min is not sufﬁciently small, the signal’s amplitude can be modeled well but phase errors will always be signiﬁcant.

Original: This can be critical since small amplitude errors lead to a slight change in speech coloration but phase errors introduce perceivable distortions.
Revised: This can be critical since small amplitude errors lead to a slight change in speech coloration but phase errors introduce perceivable distortions.

Original: To overcome the deﬁciencies of the time-domain (cid:96) 2 -loss in phase optimization, we add an explicit phase term to the loss function,
Revised: To overcome the deﬁciencies of the time-domain (cid:96) 2 -loss in phase optimization, we add an explicit phase term to the loss function,

Original: [Equation]
Revised: [Equation]

Original: where STFT( y 1: T ) is the short-term Fourier transform of the audio signal y 1: T . 
Revised: where STFT( y 1: T ) is the short-term Fourier transform of the audio signal y 1: T . [Figure] [Table]

Revised: power spectrum phase spectrum

Original: E VALUATION
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: encoding 0 .0 .1 .

Revised: [Table]

Revised: 886 (cid:96)

Revised: [Table]

Revised:  E VALUATION

Original: Dataset.
Revised: Dataset.

Original: We recorded a total of 2 hours of paired mono and binaural data at 48kHz from eight different speakers, four male and four female.
Revised: We recorded a total of 2 hours of paired mono and binaural data at 48kHz from eight different speakers, four male and four female.

Original: The listener is a mannequin equipped with binaural microphones in its ears.
Revised: The listener is a mannequin equipped with binaural microphones in its ears.

Original: Participants were asked to walk around the mannequin an a circle with 1.5m radius and have an unscripted conversation with it.
Revised: Participants were asked to walk around the mannequin an a circle with 1.5m radius and have an unscripted conversation with it.

Original: We used an OptiTrack system to track position and orientation of source and listener throughout the captures.
Revised: We used an OptiTrack system to track position and orientation of source and listener throughout the captures.

Original: To the best of our knowledge, this is the only in-the-wild (i.e., not recorded in an anechoic chamber) binaural dataset of such size.
Revised: To the best of our knowledge, this is the only in-the-wild (i.e., not recorded in an anechoic chamber) binaural dataset of such size.

Original: The dataset will be made available to the public on acceptance of the paper.

Original: We use a validation sequence and the last two minutes from each participant as test data and train the models on the remaining data.
Revised: We use a validation sequence and the last two minutes from each participant as test data and train the models on the remaining data.

Original: See Appendix A.4 for a more detailed description.
Revised: See Appendix A.5 for a more detailed description.

Original: Network Architecture.
Revised: Network Architecture.

Original: The WarpNet architecture is as described in Section 2.1.
Revised: The WarpNet architecture is as described in Section 2.1.

Original: The temporal convolutional network consists of three sequential blocks.
Revised: The temporal convolutional network consists of three sequential blocks.

Original: Each block is a stack of ten hyperconvolution layers with 64 channels, kernel size 2, and the dilation size is doubled after each layer.
Revised: Each block is a stack of ten hyperconvolution layers with 64 channels, kernel size 2, and the dilation size is doubled after each layer.

Original: We train our models for 100 epochs using an Adam optimizer.
Revised: We train our models for 100 epochs using an Adam optimizer.

Original: Learning rates are decreased if between two epochs the loss on the training set did not improve.
Revised: Learning rates are decreased if between two epochs the loss on the training set did not improve.

Original: At inference, our model can produce binaural audio in real-time.[Figure] [Table] spectrum
Revised: At inference, our model can produce binaural audio in real-time.

Original: [Table]

Original: spectrum

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 3. L OSS E VALUATION
Revised: 3. L OSS E VALUATION

Original: In order to empirically validate our ﬁndings from Section 2.3, we train our proposed network with time-domain (cid:96) 2 -loss only and with the loss proposed in equation 12.
Revised: In order to empirically validate our ﬁndings from Section 2.3, we train our proposed network with time-domain (cid:96) 2 -loss only and with the loss proposed in equation 12.

Original: Figure 3 shows how the phase error and amplitude error develop during training as the time-domain (cid:96) 2 -loss decreases.
Revised: Figure 3 shows how the phase error and amplitude error develop during training as the time-domain (cid:96) 2 -loss decreases.

Original: The model trained with (cid:96) 2 -loss only (Figure 3a) shows the behaviour that the analysis in Section 2.3 suggests: the amplitude is optimized aggressively, particularly in the beginning in training when the (cid:96) 2 -loss is still high.
Revised: The model trained with (cid:96) 2 -loss only (Figure 3a) shows the behaviour that the analysis in Section 2.3 suggests: the amplitude is optimized aggressively, particularly in the beginning in training when the (cid:96) 2 -loss is still high.

Original: The phase, on the contrary, does hardly improve at all in the beginning and shows only moderate improvements as the (cid:96) 2 -loss becomes smaller.
Revised: The phase, on the contrary, does hardly improve at all in the beginning and shows only moderate improvements as the (cid:96) 2 -loss becomes smaller.

Original: When training with time-domain (cid:96) 2 -loss and phase loss (Figure 3b), this effect is being compensated for.
Revised: When training with time-domain (cid:96) 2 -loss and phase loss (Figure 3b), this effect is being compensated for.

Original: The amplitude is optimized less aggressively and phase improves from the beginning of training on.
Revised: The amplitude is optimized less aggressively and phase improves from the beginning of training on.

Original: Various audio losses have been proposed over time, ranging from optimizing the power spectrum only and copying the input’s phase (Zhao et al., 2018; Gao & Grauman, 2019a) over a multiscale
Revised: Various audio losses have been proposed over time, ranging from optimizing the power spectrum only and copying the input’s phase (Zhao et al., 2018; Gao & Grauman, 2019a) over a multiscale

Original: STFT loss for high frequency and high time resolution (Yamamoto et al., 2020) to optimization of the scale-invariant signal to distortion ratio (si-SDR, Le Roux et al.
Revised: STFT loss for high frequency and high time resolution (Yamamoto et al., 2020) to optimization of the scale-invariant signal to distortion ratio (si-SDR, Le Roux et al.

Original: ; Heitkaemper et al.
Revised: ; Heitkaemper et al.

Original: Luo & Mesgarani (2019)).
Revised: Luo & Mesgarani (2019)).

Original: With the introduction of WaveNet for speech synthesis (Van Den Oord et al., 2016), interpreting audio optimization as categorical optimization on a µ -law encoded signal has become a prominent technique.
Revised: With the introduction of WaveNet for speech synthesis (Van Den Oord [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: et al., 2016), interpreting audio optimization as categorical optimization on a µ -law encoded signal has become a prominent technique.

Original: As Table 1 shows, all these approaches fail to predict accurate phase and mostly result in meager power spectral and waveform optimization.
Revised: As Table 1 shows, all these approaches fail to predict accurate phase and mostly result in meager power spectral and waveform optimization.

Original: Overall, our proposed loss retains accurate (cid:96) 2 and power spectral estimations while outperforming other criteria by a huge margin in phase error.
Revised: Overall, our proposed loss retains accurate (cid:96) 2 and power spectral estimations while outperforming other criteria by a huge margin in phase error.

Original: Perceptually, we observe a strong correlation between the phase error and noise and distortions in the generated binaural signal.
Revised: Perceptually, we observe a strong correlation between the phase error and noise and distortions in the generated binaural signal.

Original: In particular, our proposed loss was the only one that produced clean speech without perceivable distortions.
Revised: In particular, our proposed loss was the only one that produced clean speech without perceivable distortions.

Original: This is consistent with our perceptual study in Table 4, where other approaches with different losses and architectures have been ranked less favorable.
Revised: This is consistent with our perceptual study in Table 4, where other approaches with different losses and architectures have been ranked less favorable.

Original: 3. M ODEL E VALUATION
Revised: 3. M ODEL E VALUATION

Original: Ablation Study.
Revised: Ablation Study.

Original: In Table 2, we show the impact of our model’s individual components compared to a vanilla temporal convolutional network baseline with a WaveNet-like architecture and ReLU activations.
Revised: In Table 2, we show the impact of our model’s individual components compared to a vanilla temporal convolutional network baseline with a WaveNet-like architecture and ReLU activations.

Original: Number of layers, channels, and kernel sizes are the same as in our ﬁnal system.
Revised: Number of layers, channels, and kernel sizes are the same as in our ﬁnal system.

Original: Keeping amplitudes unchanged but compensating for interaural time differences, it is not surprising that neural time warping leads to a huge improvement in phase.Replacing regular convolutions with [Table] [Figure]
Revised: Keeping amplitudes unchanged but compensating for interaural time differences, it is not surprising that neural time warping leads to a huge improvement in phase.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: r ec

Original: [Table]

Original: d

Original: [Table]

Original: d g

Original: [Table]

Original: m

Original: [Table] [Table] [Figure] hyper-convolutions, on the contrary, is particularly beneﬁcial to improve the power spectrum.
Revised: Replacing regular convolutions with hyper-convolutions, on the contrary, is particularly beneﬁcial to improve the power spectrum.

Original: Finally, replacing the ReLU activations by sine functions, which have been proven to retain high frequency details more reliably (Sitzmann et al., 2020), leads to an additional moderate improvement along waveform, phase, and amplitude error.
Revised: Finally, replacing the ReLU activations by sine functions, which have been proven to retain high frequency details more reliably (Sitzmann et al., 2020), leads to an additional moderate improvement along waveform, phase, and amplitude error.

Original: Neural Time Warping.
Revised: Neural Time Warping.

Original: The purpose of neural time warping is a strong initial alignment of the mono source signal to the left and right ear listener signal, respectively.
Revised: The purpose of neural time warping is a strong initial alignment of the mono source signal to the left and right ear listener signal, respectively.

Original: Note the signiﬁcant temporal shift between the mono signal and recorded left ear signal in Figure 4a.
Revised: Note the signiﬁcant temporal shift between the mono signal and recorded left ear signal in Figure 4a.

Original: In the same ﬁgure, observe how geometric warping provides an approximate alignment to the reference signal, while the learned neural warping successfully corrects the inaccurate geometric warping and aligns the peaks and valleys more accurately.
Revised: In the same ﬁgure, observe how geometric warping provides an approximate alignment to the reference signal, while the learned neural warping successfully corrects the inaccurate geometric warping and aligns the peaks and valleys more accurately.

Original: Although those adjustments seem small, the impact of neural warping on the phase error is signiﬁcant, as shown in Figure 4b (red bars).
Revised: Although those adjustments seem small, the impact of neural warping on the phase error is signiﬁcant, as shown in Figure 4b (red bars).

Original: Naturally, neural warping can not improve the amplitude (blue bars).
Revised: Naturally, neural warping can not improve the amplitude (blue bars).

Original: Temporal HyperConv Network.
Revised: Temporal HyperConv Network.

Original: Neural warping provides an accurate alignment between input and target signal.
Revised: Neural warping provides an accurate alignment between input and target signal.

Original: This raises the question if a deep network is required on top of the warping module or if a linear amplitude adjustment can already yield convincing results.
Revised: This raises the question if a deep network is required on top of the warping [Figure] standard convolution hyper convolution [Table]

Revised: [Table]

Revised: module or if a linear amplitude adjustment can already yield convincing results.

Original: We therefore apply a learned bilinear term to the warped result,
Revised: We therefore apply a learned bilinear term to the warped result,

Original: [Equation]
Revised: [Equation]

Original:  given the conditioning c t and the warped signal x (warped) for the left or right ear, respectively.
Revised: (13) given the conditioning c t and the warped signal x (warped) for the left or right ear, respectively.

Original: Figure 4b shows that this leads to a slight improvement of the amplitude error but falls way behind the performance of the full system with a deep temporal network of hyper-convolutions after the warping module.
Revised: Figure 4b shows that this leads to a slight improvement of the amplitude error but falls way behind the performance of the full system with a deep temporal network of hyper-convolutions after the warping module.

Original: Inspection of the mono and recorded signal in Figure 4a in fact reveals that the binaural recording undergoes nonlinear transformations beyond warping.
Revised: Inspection of the mono and recorded signal in Figure 4a in fact reveals that the binaural recording undergoes additional transformations beyond warping.

Original: Room reverberations, source speech directivity, and modiﬁcations caused by the shape of the listener’s ear, for instance, are highly nonlinear effects and require complex transformations of the warped signal.
Revised: Room reverberations, source speech directivity, and modiﬁcations caused by the shape of the listener’s ear, for instance, are physical effects that require complex transformations of the warped signal.

Original: Many of these subtle effects depend on the position and orientation of source and listener in the room.
Revised: Many of these subtle effects depend on the position and orientation of source and listener in the room.

Original: It is therefore plausible that conditioned hyper-convolutions, which can model more complex dependencies between inputs and conditioning variables in a single layer, show better performance than standard convolutions, cf.
Revised: It is therefore plausible that conditioned hyper-convolutions, which can model more complex dependencies between inputs and conditioning variables in a single layer, show better performance than standard convolutions, cf.

Original: Table 2 (b) versus (c).
Revised: Table 2 (b) versus (c).

Original: As Figure 5 reveals, hyper-convolutions also converge signiﬁcantly faster than standard convolutions in the early stages of training.[Figure] standard convolution hyper convolution [Table]
Revised: As Figure 5 reveals, hyper-convolutions also converge signiﬁcantly faster than standard convolutions in the early stages of training.

Original: [Table]

Original:  S TATE OF THE A RT C OMPARISON
Revised: 3. S TATE OF THE A RT C OMPARISON

Original: In Table [Table] we compare our approach to other neural binauralization approaches and to a DSP baseline, which is the de-facto state of the art for binauralization.
Revised: In Table 3, we compare our approach to other neural binauralization approaches and to a DSP baseline, which is the de-facto state of the art for binauralization.

Original: The recently proposed 2.5D visual sound (Gao & Grauman, 2019b) network operates in frequency domain and predicts a complex mask which the input is multiplied with to obtain left and right ear outputs.
Revised: The recently proposed 2.5D visual sound (Gao & Grauman, 2019b) network operates in frequency domain and predicts a complex mask which the input is multiplied with to obtain left and right ear outputs.

Original: We compare to their approach and replace the visual features with our conditioning features c 1: T .
Revised: We compare to their approach and replace the visual features with our conditioning features c 1: T .

Original: For the STFT, we use a window size of 1,600 samples and a hop length of 480 samples (10ms).
Revised: For the STFT, we use a window size of 1,600 samples and a hop length of 480 samples (10ms).

Original: Therefore, modeling delays of less than 10ms requires non-trivial manipulation of the phase information in the complex  spectrogram, which is a more difﬁcult operation than modeling delays in time-domain.
Revised: Therefore, modeling delays of less than 10ms requires non-trivial manipulation of the phase information in the complex [Table] spectrogram, which is a more difﬁcult operation than modeling delays in time-domain.

Original: We also provide a comparison to a WaveNet that proved to be generally strong in various generative audio problems (Rethage et al., 2018; Engel et al., 2017).
Revised: We also provide a comparison to a WaveNet that proved to be generally strong in various generative audio problems (Rethage et al., 2018; Engel et al., 2017).

Original: For the conditioning on source and listener positions, we follow the approach of Van Den Oord et al.
Revised: For the conditioning on source and listener positions, we follow the approach of Van Den Oord et al.

Original: (2016), i.e., the source and listener positions are appended to the input of each temporal convolutional layer.
Revised: (2016), i.e., the source and listener positions are appended to the input of each temporal convolutional layer.

Original: Note that we use the WaveNet in a non-autoregressive setup since the input audio, i.e., the mono signal, is fully available at inference time.
Revised: Note that we use the WaveNet in a non-autoregressive setup since the input audio, i.e., the mono signal, is fully available at inference time.

Original: Overall, our approach performs signiﬁcantly better than other methods.
Revised: Overall, our approach performs signiﬁcantly better than other methods.

Original: For a perceptual evaluation, we asked 100 participants to rank a total of 2,000 audio snippets fromto 5 on a Likert scale according to three criteria: cleanliness of the signal, spatialization quality, and overall realism, see Table 4.
Revised: For a perceptual evaluation, we asked 100 participants to rank a total of 2,000 audio snippets fromto 5 on a Likert scale according to three criteria: cleanliness of the signal, spatialization quality, and overall realism, see Table 4.

Original: All scores are below a 5 (indistinguishable from reality) because participants listened to results for a generic head-related transfer function rather to one that takes their explicit head and ear shape into account.
Revised: All scores are below a 5 (indistinguishable from reality) because participants listened to results for a generic head-related transfer function rather to one that takes their explicit head and ear shape into account.

Original: Additionally, user’s headphones are of different quality and not equalized.
Revised: Additionally, user’s headphones are of different quality and not equalized.

Original: Note that the binaural (ground truth) recordings score lower on cleanliness because they contain ambient noise that is uncorrelated to the source input and therefore not modeled by our approach.
Revised: Note that the binaural (ground truth) recordings score lower on cleanliness because they contain ambient noise that is uncorrelated to the source input and therefore not modeled by our approach.

Original: WaveNet leads to a particularly noisy audio signal, which is caused by two major factors.
Revised: WaveNet leads to a particularly noisy audio signal, which is caused by two major factors.

Original: First, audio is generated at 48kHz, which is more difﬁcult to model than 16-24kHz audio.
Revised: First, audio is generated at 48kHz, which is more difﬁcult to model than 16-24kHz audio.

Original: Arik et al.
Revised: Arik et al.

Original: (2017) show that the quality of WaveNet degrades with higher sampling rates.
Revised: (2017) show that the quality of WaveNet degrades with higher sampling rates.

Revised: Second,

Original: Second, WaveNet has to spend a considerable amount of capacity on modeling the non-linear source-to-listener time [Table] [Table] shift between the mono and binaural signal and, in consequence, struggles more to generate clean audio.
Revised: WaveNet has to spend a considerable amount of capacity on modeling large source-to-listener time shifts between the mono and binaural signals and, in consequence, struggles more to generate clean audio.

Original: Our approach ranks favorably against other neural binauralization systems and is also preferred in terms of cleanliness and realism over the DSP baseline.
Revised: Our approach ranks favorably against other neural binauralization systems and is also preferred in terms of cleanliness and realism over the DSP baseline.

Original: A t-test showed that all results in
Revised: A t-test showed that all results in

Original: Table 4 are statistically signiﬁcant with the exception of spatialization between ours and DSP, which is ranked at almost equal quality.
Revised: Table 4 are statistically signiﬁcant with the exception of spatialization between ours and DSP, which is ranked at almost equal quality.

Original: Since DSP is the perceptually closest competitor to our approach, we performed an additional perceptual side-by-side study between the two systems that comﬁrms the results presented in Table 4, see Appendix A.5.
Revised: Since DSP is the perceptually closest competitor to our approach, we performed an additional perceptual side-by-side study between the two systems that comﬁrms the results presented in Table 4, see Appendix A.6.

Original: Runtime.
Revised: Runtime.

Original: When analyzing the runtime of a system, two cases are important.
Revised: When analyzing the runtime of a system, two cases are important.

Original: The ﬁrst is ofﬂine processing , where a user provides the complete mono audio to be binauralized in advance.
Revised: The ﬁrst is ofﬂine processing , where a user provides the complete mono audio to be binauralized in advance.

Original: The real-time-factor is the computation time divided by the duration of the input.
Revised: The real-time-factor is the computation time divided by the duration of the input.

Original: Table 5 shows that our system allows for a rapid binauralization.
Revised: Table 5 shows that our system allows for a rapid binauralization.

Original: On a single NVidia Tesla V100, our approach can binauralize 100 seconds of mono audio in just 6.9 seconds.
Revised: On a single NVidia Tesla V100, our approach can binauralize 100 seconds of mono audio in just 6.9 seconds.

Original: Note that the DSP baseline does not run on a GPU but is purely CPU-based.
Revised: Note that the DSP baseline does not run on a GPU but is purely CPU-based.

Original: The second case is a streaming scenario where binaural audio has to be computed on-the-ﬂy, e.g., when a user navigates through a 3D environment in a game or in virtual reality.
Revised: The second case is a streaming scenario where binaural audio has to be computed on-the-ﬂy, e.g., when a user navigates through a 3D environment in a game or in virtual reality.

Original: In this case, systems are required to have low latency.
Revised: In this case, systems are required to have low latency.

Original: On an NVidia Tesla V100, our approach runs with roughly 33ms latency, which is low enough to have non-observable delays for videos or games that render frames at 30Hz.
Revised: On an NVidia Tesla V100, our approach runs with roughly 33ms latency, which is low enough to have non-observable delays for videos or games that render frames at 30Hz.

Original: Note that this is measured with a naive pytorch implementation and allows for several improvements to further lower the latency.
Revised: Note that this is measured with a naive pytorch implementation and allows for several improvements to further lower the latency.

Original: The 2.5D sound network can not efﬁciently be applied in a streaming mode because (a) it is acausal, i.e., requires access to future audio, and (b) it summarizes as much as 0 .
Revised: The 2.5D sound network can not efﬁciently be applied in a streaming mode because (a) it is acausal, i.e., requires access to future audio, and (b) it summarizes as much as 0 .

Original: 32 seconds of audio in a single temporal step in its bottleneck layer due to the UNet structure.
Revised: 32 seconds of audio in a single temporal step in its bottleneck layer due to the UNet structure.

Original: Unseen Subjects.

Original: While previous evaluations were based on unseen audio data from speakers that are part of the training set, we evaluate the performance on unseen speakers here.

Original: We train our model in a leave-one-speaker-out setup, i.e., we train eight models, each with another speaker being held out.

Original: Table 6 shows that our approach still outperforms the DSP baseline by a large margin on unseen subjects.

Original: This is remarkable, considering that only seven different subjects are seen during training.

Original: With a more diverse dataset, we expect the generalization quality of our approach to increase signiﬁcantly.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our neural sound binauralization approach is the ﬁrst purely data-driven end-to-end model that shows convincing performance compared to traditional state of the art binauralization methods.
Revised: Our neural sound binauralization approach is the ﬁrst purely data-driven end-to-end model that shows convincing performance compared to traditional state of the art binauralization methods.

Original: We were able to show effectiveness of our model both quantitatively and in a perceptual user study.
Revised: We were able to show effectiveness of our model both quantitatively and in a perceptual user study.

Original: Moreover, we unveiled and mitigated a fundamental issue with (cid:96) 2 -optimization on the raw waveform that affects not only this task but is relevant to other generative audio problems as well.
Revised: Moreover, we unveiled and mitigated a fundamental issue with (cid:96) 2 -optimization on the raw waveform that affects not only this task but is relevant to other generative audio problems as well.

Original: The code and dataset will be released on acceptance of the paper.

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Before we start with the formal proof, let us get a more intuitive idea what Lemma 1 means and how phase and amplitude error relate to the (cid:96) 2 -loss in time-domain and complex spectral domain, respectively.
Revised: Before we start with the formal proof, let us get a more intuitive idea what Lemma 1 means and how phase and amplitude error relate to the (cid:96) 2 -loss in time-domain and complex spectral domain, respectively.

Original: Reconsider the (cid:96) 2 -loss in time-domain and use Parseval’s theorem to relate it to the complex frequency domain,
Revised: Reconsider the (cid:96) 2 -loss in time-domain and use Parseval’s theorem to relate it to the complex frequency domain,

Original: [Equation]
Revised: [Equation]

Original: where k runs over all frequency components of the signal.
Revised: where k runs over all frequency components of the signal.

Original: The time-domain (cid:96) 2 -loss is therefore a sum of the (cid:96) 2 -loss of each individual frequency component in spectral domain.
Revised: The time-domain (cid:96) 2 -loss is therefore a sum of the (cid:96) 2 -loss of each individual frequency component in spectral domain.

Original: For the analysis, let us consider one ﬁxed frequency component k .
Revised: For the analysis, let us consider one ﬁxed frequency component k .

Original: Our ﬁndings hold for all frequency components equally.
Revised: Our ﬁndings hold for all frequency components equally.

Original: Figure 6 illustrates this case in the complex plane.
Revised: Figure 6 illustrates this case in the complex plane.

Original: 2 Given a target ˆ Y and a prediction Y that has distance ε to ˆ Y , optimizing the (cid:96) 2 -loss is equal to optimization of the amplitude (the length difference of Y and ˆ Y ) if θ Y
Revised: 3 Given a target ˆ Y and a prediction Y that has distance ε to ˆ Y , optimizing the (cid:96) 2 -loss is equal to optimization of the amplitude (the length difference of Y and ˆ Y ) if θ Y

Original: = θ ˆ Y .
Revised: = θ ˆ Y .

Original: If the phases θ Y and θ ˆ Y are different, however, the relation is not that obvious.
Revised: If the phases θ Y and θ ˆ Y are different, however, the relation is not that obvious.

Original: Lemma 1 makes a statement about the expected amplitude and phase error for a random prediction Y that has (cid:96) 2 -distance ε to ˆ Y , i.e., that lies on the circle deﬁned by B ε, ˆ Y .
Revised: Lemma 1 makes a statement about the expected amplitude and phase error for a random prediction Y that has (cid:96) 2 -distance ε to ˆ Y , i.e., that lies on the circle deﬁned by B ε, ˆ Y .

Original: [Figure]
Revised: [Figure]

Original: [Equation] 
Revised: [Equation] [Figure] [Equation]

Original: [Figure]
Revised: [Figure]

Original: (cid:12)(cid:12)
Revised: (cid:12)(cid:12)

Original: | Y | − | ˆ Y | (cid:12)(cid:12)
Revised: | Y | − | ˆ Y | (cid:12)(cid:12)

Original: With this in mind, we prove
Revised: With this in mind, we prove

Original: Lemma 1.
Revised: Lemma 1.

Original: Let ˆ Y ∈ C be a ﬁxed complex number and Y ∈ B ε, ˆ Y =
Revised: Let ˆ Y ∈ C be a ﬁxed complex number and Y ∈ B ε, ˆ Y =

Original: { Y ∈ C :
Revised: { Y ∈ C :

Original: | Y − ˆ Y | = ε } be any complex number that has distance ε from ˆ Y .
Revised: | Y − ˆ Y | = ε } be any complex number that has distance ε from ˆ Y .

Original: Then, the expected amplitude error and the expected angular phase error with respect to ˆ Y are
Revised: Then, the expected amplitude error and the expected angular phase error with respect to ˆ Y are

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let ˆ Y ∈ C and Y ∈ B ε, ˆ Y .
Revised: Let ˆ Y ∈ C and Y ∈ B ε, ˆ Y .

Original: Then, amplitude loss and phase loss as deﬁned in equation 8 are given by
Revised: Then, amplitude loss and phase loss as deﬁned in equation 8 are given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where | · | denotes the modulus (or magnitude) of a complex number.
Revised: where | · | denotes the modulus (or magnitude) of a complex number.

Original: Without loss of generality, let Im( ˆ Y )
Revised: Without loss of generality, let Im( ˆ Y )

Original: = 

Revised: =

Original: and Re( ˆ Y ) ≥ 0 .
Revised: and Re( ˆ Y ) ≥ 0 .

Original: This can always be achieved by a unitary rotation of angle θ ˆ Y around the origin of the complex plane which preserves the lengths and angles between Y and ˆ Y .
Revised: This can always be achieved by a unitary rotation of angle θ ˆ Y around the origin of the complex plane which preserves the lengths and angles between Y and ˆ Y .

Original: In that case, the phase loss simpliﬁes to
Revised: In that case, the phase loss simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: Since Y ∈ B ε, ˆ Y is a point on the circle around ˆ Y with radius ε (see Figure 6 for an illustration),
Revised: Since Y ∈ B ε, ˆ Y is a point on the circle around ˆ Y with radius ε (see Figure 6 for an illustration),

Original: B ε, ˆ Y can equivalently be written as
Revised: B ε, ˆ Y can equivalently be written as

Original: [Equation]
Revised: [Equation]

Original: Then, each Y ∈ B ε, ˆ Y is uniquely deﬁned by some ϕ ∈ [ − π, π ] and we can rewrite L (amp) as a function of ϕ by substituting Y =
Revised: Then, each Y ∈ B ε, ˆ Y is uniquely deﬁned by some ϕ ∈ [ − π, π ] and we can rewrite L (amp) as a function of ϕ by substituting Y =

Original: εe iϕ + ˆ Y ,
Revised: εe iϕ + ˆ Y ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Observe that due to Im( ˆ Y ) = 0 and Re( ˆ Y ) ≥ 0 , it follows that Re( ˆ Y ) =
Revised: Observe that due to Im( ˆ Y ) = 0 and Re( ˆ Y ) ≥ 0 , it follows that Re( ˆ Y ) =

Original: | ˆ Y | and
Revised: | ˆ Y | and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore,
Revised: Therefore,

Revised: L

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12) .

Original: For the phase error, we apply the same substitution and obtain
Revised: For the phase error, we apply the same substitution and obtain

Original: [Equation]
Revised: [Equation]

Original: The expected amplitude and phase error can now be written as
Revised: The expected amplitude and phase error can now be written as

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Assuming that Y ∈ B ε, ˆ Y is uniformly distributed on the circle around ˆ Y , i [Equation] e., p ( ϕ ) follows a circular uniform distribution, we have p ( ϕ ) =
Revised: Assuming that Y ∈ B ε, ˆ Y is uniformly distributed on the circle around ˆ Y , i.e., p ( ϕ ) follows a circular uniform distribution, we have p ( ϕ ) =

Original: 12 π and plug in equation 23 and equation 24 to obtain the claim from Lemma 1.
Revised: 12 π and plug in equation 23 and equation 24 to obtain the claim from Lemma 1.

Original: A. I NTERPRETATION OF HYPER - CONVOLUTIONS
Revised: A. F REQUENCY -D OMAIN L OSS F ORMULATION

Revised: In our original formulation of the phase-enhanced loss in equation 12, phase is technically penalized twice: once implicitly in the (cid:96) 2 -loss on the raw waveform and once explicitly in the phase loss term.

Revised: It is possible to formulate the phase-enhanced loss in frequency domain such that there is a clear separation of magnitude and phase loss in two different additive terms.

Revised: Due to Parseval’s theorem, optimizing the (cid:96) 2 -loss in time domain is equivalent to optimizing the (cid:96) 2 -loss in frequency domain.

Revised: We transform the time domain audio signal into frequency domain using a short-term Fourier transform and denote the result of the transformation as

Revised: [Equation]

Revised: where K is the number of frequency bins of the discrete Fourier transform and S is the number of STFT steps, i.e., S = T/ hop length.

Revised: Since the frequency spectrum can be expressed by it’s magnitude and phase spectrum, we can reformulate equation 12 as

Revised: [Equation]

Revised: Note that this form is not equivalent to the original loss since separating the frequency spectrum into magnitude and phase is a non-unitary operation.

Revised: However, we do not ﬁnd

Revised: the differences to be signiﬁcant, as Table 6 shows.

Revised: In fact, the clear separation allows for a better adjustment of the phase weight λ and can even lead to slightly improved results.

Revised: A. I NTERPRETATION OF HYPER - CONVOLUTIONS

Original: Hyper-convolutions are a generalization of stardard convolutions as long as H ( W ) and H ( b ) are able to learn a linear transformation.
Revised: Hyper-convolutions are a generalization of stardard convolutions as long as H ( W ) and H ( b ) are able to learn a linear transformation.

Original: Speciﬁcally, if H ( W ) and H ( b ) are linear convolutional networks, hyper-convolutions extend equation 5 by a bilinear term.
Revised: Speciﬁcally, if H ( W ) and H ( b ) are linear convolutional networks, hyper-convolutions extend equation 5 by a bilinear term.

Original: We show this property in the following and provide an example for a simple fully connected layer.
Revised: We show this property in the following and provide an example for a simple fully connected layer.

Original: Recall the deﬁnition of hyper-convolutions from equation 6,
Revised: Recall the deﬁnition of hyper-convolutions from equation 6,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Table]

Original: Lemma 2.
Revised: Lemma 2.

Original: Let H ( W ) and H ( b ) be linear convolutional networks with kernel size K .
Revised: Let H ( W ) and H ( b ) be linear convolutional networks with kernel size K .

Original: Let further x 1: T and c 1: T be a sequence of input and conditional vectors in R C in and R C cond , and the output z 1: T be a sequence of vectors in R C out .
Revised: Let further x 1: T and c 1: T be a sequence of input and conditional vectors in R C in and R C cond , and the output z 1: T be a sequence of vectors in R C out .

Original: Then, the hyper-convolution from equation 27 reduces to
Revised: Then, the hyper-convolution from equation 29 reduces to

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: with U ∈ R ( C out × C in × K ) ×
Revised: with U ∈ R ( C out × C in × K ) ×

Original: ( C cond × K ) , W ∈ R C out × C in × K , V ∈ R C out × C cond × K and b ∈ R C out .
Revised: ( C cond × K ) , W ∈ R C out × C in × K , V ∈ R C out × C cond × K and b ∈ R C out .

Original: Proof.
Revised: Proof.

Original: We start from equation 27 and use that both H ( W ) and H ( b ) are linear (and therefore, w.l.o.g. single-layer) convolutional networks with kernel size K , such that
Revised: We start from equation 29 and use that both H ( W ) and H ( b ) are linear (and therefore, w.l.o.g. single-layer) convolutional networks with kernel size K , such that

Original: [Equation]
Revised: [Equation]

Original: First, consider H ( b ) : R C cond × K (cid:55)→ R C out .
Revised: First, consider H ( b ) : R C cond × K (cid:55)→ R C out .

Original: Since H ( b ) is linear, it can be written as
Revised: Since H ( b ) is linear, it can be written as

Original: [Equation]
Revised: [Equation]

Original:  with V ∈ R C out × C cond × K and b ∈ R C out .
Revised: ) with V ∈ R C out × C cond × K and b ∈ R C out .

Original: Note that this already yields the last two terms in equation 28.
Revised: Note that this already yields the last two terms in equation 30.

Original: While H ( b ) only generates a bias, H ( W ) needs to generate a R C out × C in × K sized tensor to realize the convolution with x 1: T .
Revised: While H ( b ) only generates a bias, H ( W ) needs to generate a R C out × C in × K sized tensor to realize the convolution with x 1: T .

Original: Thus, H ( W ) :
Revised: Thus, H ( W ) :

Original: R C cond × K (cid:55)→ R C out × C in × K and due to H ( W ) being a linear function, the component at ( i, j, k ) of the output tensor is given as
Revised: R C cond × K (cid:55)→ R C out × C in × K and due to H ( W ) being a linear function, the component at ( i, j, k ) of the output tensor is given as

Original: [Equation]
Revised: [Equation]

Original: where U ∈ R ( C out × C in × K )
Revised: where U ∈ R ( C out × C in × K )

Original: ×
Revised: ×

Original: ( C cond × K ) is the weight tensor of the hypernetwork H ( W ) and W ∈
Revised: ( C cond × K ) is the weight tensor of the hypernetwork H ( W ) and W ∈

Original: R C out × C in × K is its bias.
Revised: R C out × C in × K is its bias.

Original: For simplicity of notation, let ˆ z t be z t from equation 29 without
Revised: For simplicity of notation, let ˆ z t be z t from equation 31 without

Original: H ( b ) ( c t − K +1: t ) .
Revised: H ( b ) ( c t − K +1: t ) .

Original: Using equation 31, ˆ z t is then given by
Revised: Using equation 33, ˆ z t is then given by

Original: [Equation]
Revised: [Equation]

Original: Together with equation 30, this yields the claim from Lemma 2,
Revised: Together with equation 32, this yields the claim from Lemma 2,

Original: [Equation]
Revised: [Equation]

Original: The last row of equation 33 is exactly the deﬁnition of a standard conditioned temporal convolution from equation 5.
Revised: The last row of equation 35 is exactly the deﬁnition of a standard conditioned temporal convolution from equation 5.

Original: The row before is a bilinear combination of c 1: T and x 1: T .
Revised: The row before is a bilinear combination of c 1: T and x 1: T .

Original: As a consequence of
Revised: As a consequence of

Original: Lemma 2, conditioned (linear) hyper-convolutions are therefore a strict generalization of the standard conditioned temporal convolutions from equation 5.
Revised: Lemma 2, conditioned (linear) hyper-convolutions are therefore a strict generalization of the standard conditioned temporal convolutions from equation 5.

Original: Note that any non-linear H ( W ) and H ( b ) that is capable of learning a linear transformation is therefore also a strict generalization of equation 5.
Revised: Note that any non-linear H ( W ) and H ( b ) that is capable of learning a linear transformation is therefore also a strict generalization of equation 5.

Original: Example: K = 1 , C out = 1 .
Revised: Example: K = 1 , C out = 1 .

Original: We illustrate the case for K = 1 , i.e., the convolutions break down to a simple fully connected layer.
Revised: We illustrate the case for K = 1 , i.e., the convolutions break down to a simple fully connected layer.

Original: To simplify notation, we also restrict this example to a single output channel C out = 1 .
Revised: To simplify notation, we also restrict this example to a single output channel C out = 1 .

Original: With linear hyper-convoltions and C out = 1 , we have
Revised: With linear hyper-convoltions and C out = 1 , we have

Original: [Equation]
Revised: [Equation]

Original: 34) for a weight matrix U ∈ R C in × C cond and a bias vector w ∈ R C in as parameters of H ( W ) as well as weights v ∈ R C cond and bias b ∈ R as parameters for H ( b ) .
Revised: 36) for a weight matrix U ∈ R C in × C cond and a bias vector w ∈ R C in as parameters of H ( W ) as well as weights v ∈ R C cond and bias b ∈ R as parameters for H ( b ) .

Original: Inserting this into equation 27 leads to
Revised: Inserting this into equation 29 leads to

Original: [Equation]
Revised: [Equation]

Original: Compared to the standard formulation from equation 5,
Revised: Compared to the standard formulation from equation 5,

Original: [Equation]
Revised: [Equation]

Original: equation 35 adds the bilinear term x
Revised: equation 37 adds the bilinear term x

Original: Tt Uc t while all other terms are still remaining.
Revised: Tt Uc t while all other terms are still remaining.

Original: A. S IGNAL P ROCESSING B ASELINE
Revised: A. S IGNAL P ROCESSING B ASELINE

Original: Sound produced in a scene arrives at the left and right ear at offset times due to the marginal difference in their distance from the source of sound (Wightman & Kistler, 1992).
Revised: Sound produced in a scene arrives at the left and right ear at offset times due to the marginal difference in their distance from the source of sound (Wightman & Kistler, 1992).

Original: The coronal (back/front) asymmetry of the outer ear (pinnae) further transforms the incoming sound wave differently depending on the direction of the source (Asano et al., 1990; Cheng & Wakeﬁeld, 2001).
Revised: The coronal (back/front) asymmetry of the outer ear (pinnae) further transforms the incoming sound wave differently depending on the direction of the source (Asano et al., 1990; Cheng & Wakeﬁeld, 2001).

Original: Room effects such reverberation inﬂuence auditory localization as well (Shinn-Cunningham et al., 2005).
Revised: Room effects such reverberation inﬂuence auditory localization as well (Shinn-Cunningham et al., 2005).

Original: These binaural disparities allow listeners to localize sources of sound in three dimensions (Begault et al., 2000) and gain a more complete sense of the state of the space around them.
Revised: These binaural disparities allow listeners to localize sources of sound in three dimensions (Begault et al., 2000) and gain a more complete sense of the state of the space around them.

Original: Traditionally different effects that inﬂuence propagation between the source and the listener, i.e., different components of the mapping function between the input mono signal and output binaural signals, are addressed separately.
Revised: Traditionally different effects that inﬂuence propagation between the source and the listener, i.e., different components of the mapping function between the input mono signal and output binaural signals, are addressed separately.

Original: The components are assumed to be linear time-invariant (LTI) systems and therefore completely characterized by their impulse responses.
Revised: The components are assumed to be linear time-invariant (LTI) systems and therefore completely characterized by their impulse responses.

Original: They are then combined to produce the output signals with a series of convolution operations (Savioja et al., 1999; Zotkin et al., 2004; Sunder et al., 2015; Zhang et al., 2017).
Revised: They are then combined to produce the output signals with a series of convolution operations (Savioja et al., 1999; Zotkin et al., 2004; Sunder et al., 2015; Zhang et al., 2017).

Original: A short overview follows.
Revised: A short overview follows.

Original: Source.
Revised: Source.

Original: People are not omnidirectional sound sources and, unlike a loudspeaker whose spatial directivity depends only on the frequency, the human directivity pattern may depend on speech content and pose as well.
Revised: People are not omnidirectional sound sources and, unlike a loudspeaker whose spatial directivity depends only on the frequency, the human directivity pattern may depend on speech content and pose as well.

Original: Though some studies of speech directivity in controlled settings exist (Kocon &
Revised: Though some studies of speech directivity in controlled settings exist (Kocon &

Original: Monson, 2018; Bellows & Leishman, 2019), in practical applications it is either ignored or approximated with a simple pattern such as a cardioid.
Revised: Monson, 2018; Bellows & Leishman, 2019), in practical applications it is either ignored or approximated with a simple pattern such as a cardioid.

Original: Environment: room acoustics.
Revised: Environment: room acoustics.

Original: The reverberation is caused by the interaction of the sound ﬁeld with the surrounding environment.
Revised: The reverberation is caused by the interaction of the sound ﬁeld with the surrounding environment.

Original: Different approaches to room acoustic modeling exist
Revised: Different approaches to room acoustic modeling exist

Original: but they are mainly divided in two groups (V ¨ alim ¨ aki et al., 2012): (1) physically accurate but computationally expensive wave-based methods that, given detailed geometry and material information, numerically solve the wave equation; and (2) methods based on geometrical acoustics (Savioja & Svensson, 2015) that ignore the wave nature of sound and assume a ray-like behaviour, and are therefore more suitable for real-time operation (wave phenomena such as diffraction is usually modeled separately (Rungta et al., 2018)).
Revised: but they are mainly divided in two groups (V¨alim¨aki et al., 2012): (1) physically accurate but computationally expensive wave-based methods that, given detailed geometry and material information, numerically solve the wave equation; and (2) methods based on geometrical acoustics (Savioja & Svensson, 2015) that ignore the wave nature of sound and assume a ray-like behaviour, and are therefore more suitable for real-time operation (wave phenomena such as diffraction is usually modeled separately (Rungta et al., 2018)).

Original: For the real-world rooms the room impulse responses (RIRs) are either measured or computed using simpliﬁed geometric models informed by some estimated room parameters.
Revised: For the real-world rooms the room impulse responses (RIRs) are either measured or computed using simpliﬁed geometric models informed by some estimated room parameters.

Original: It is important to note that measurement procedure is infeasible for fully dynamic scenarios since RIRs depend on both source and listener spatial positions.
Revised: It is important to note that measurement procedure is infeasible for fully dynamic scenarios since RIRs depend on both source and listener spatial positions.

Original: The length of the RIR ﬁlters depends on the reverberation time of the environment, i.e., the time it takes for sound to decay by 60dB, which can go from less than half a second for typical ofﬁce spaces to a couple of seconds for auditoriums and concert halls, and around ten seconds for large cathedrals.
Revised: The length of the RIR ﬁlters depends on the reverberation time of the environment, i.e., the time it takes for sound to decay by 60dB, which can go from less than half a second for typical ofﬁce spaces to a couple of seconds for auditoriums and concert halls, and around ten seconds for large cathedrals.

Original: Environment: background noise.
Revised: Environment: background noise.

Original: Even in absence of other interfering sound sources, there is always some degree of ambient noise present in the environment.
Revised: Even in absence of other interfering sound sources, there is always some degree of ambient noise present in the environment.

Original: Usually this noise is assumed to be diffuse and independent from the listener position.
Revised: Usually this noise is assumed to be diffuse and independent from the listener position.

Original: It is also often assumed to be stationary and it can be estimated from short silence intervals.
Revised: It is also often assumed to be stationary and it can be estimated from short silence intervals.

Original: Listener.
Revised: Listener.

Original: The human body, most notably pinna, head and torso, modify the incoming acoustic waves in a way that is crucial for the spatial perception of sound.
Revised: The human body, most notably pinna, head and torso, modify the incoming acoustic waves in a way that is crucial for the spatial perception of sound.

Original: Traditionally the head-related transfer function (HRTF) is used to model these effects (Cheng & Wakeﬁeld, 2001).
Revised: Traditionally the head-related transfer function (HRTF) is used to model these effects (Cheng & Wakeﬁeld, 2001).

Original: In theory, the HRTF is personalized to the individual (listener) and depends on the source position relative to the listener.
Revised: In theory, the HRTF is personalized to the individual (listener) and depends on the source position relative to the listener.

Original: However, most practical implementations use a generic (not-personalized) HRTF though HRTF penalization is an active research area (Bilinski et al., 2014; Yamamoto & Igarashi, 2017; Guezenoc & Seguier, 2018).
Revised: However, most practical implementations use a generic (not-personalized) HRTF though HRTF penalization is an active research area (Bilinski et al., 2014; Yamamoto & Igarashi, 2017; Guezenoc & Seguier, 2018).

Original: Moreover, measuring HRTF in a volume is impractical and it is usually measured on a ﬁxed radius (Li & Peissig, 2020), making the HRTF a function of the source direction only.
Revised: Moreover, measuring HRTF in a volume is impractical and it is usually measured on a ﬁxed radius (Li & Peissig, 2020), making the HRTF a function of the source direction only.

Original: In alternative, the boundary element method (BEM) (Katz, 2001) or ﬁnite-difference time-domain (FDTD) method (Prepelit , ˘a et al., 2016) can be used for numerical HRTF simulation using head and torso scans.
Revised: In alternative, the boundary element method (BEM) (Katz, 2001) or ﬁnite-difference time-domain (FDTD) method (Prepelit , ˘a et al., 2016) can be used for numerical HRTF simulation using head and torso scans.

Original: Depending on the dataset, the HRTF ﬁlters are usually 2 .
Revised: Depending on the dataset, the HRTF ﬁlters are usually 2 .

Original: 5 − 20ms long after removing the initial onset delays.
Revised: 5 − 20ms long after removing the initial onset delays.

Original: Equipment.
Revised: Equipment.

Original: While above components are enough to model the physics involved in the mapping between the input and the output signals, in a practical setting the capture/reproduction equipment plays a role as well.
Revised: While above components are enough to model the physics involved in the mapping between the input and the output signals, in a practical setting the capture/reproduction equipment plays a role as well.

Original: To compensate for the frequency response of the equipment and the signalprocessing chain, the equalization ﬁlter should be applied.
Revised: To compensate for the frequency response of the equipment and the signalprocessing chain, the equalization ﬁlter should be applied.

Original: It is often assumed that this ﬁlter does not change in time and it can be estimated from a test capture.
Revised: It is often assumed that this ﬁlter does not change in time and it can be estimated from a test capture.

Original: Note that each of above steps introduces some degree of estimation, measurement or modeling errors that accumulate down the pipeline and, not being formulated in an end-to-end fashion, the solution is sub-optimal from the perspective of the particular application.
Revised: Note that each of above steps introduces some degree of estimation, measurement or modeling errors that accumulate down the pipeline and, not being formulated in an end-to-end fashion, the solution is sub-optimal from the perspective of the particular application.

Original: Furthermore, a study showed that even using binaural room impulse responses measured for the test subject inside the test environment, the perceptual authenticity between virtual and real sound sources was not fully achieved for a dynamic scenario that allowed natural head movements of the listeners (Brinkmann et al., 2017).
Revised: Furthermore, a study showed that even using binaural room impulse responses measured for the test subject inside the test environment, the perceptual authenticity between virtual and real sound sources was not fully achieved for a dynamic scenario that allowed natural head movements of the listeners (Brinkmann et al., 2017).

Original: Our implementation, used as the DSP baseline, computes the output binaural signals y
Revised: Our implementation, used as the DSP baseline, computes the output binaural signals y

Original: ( l/r ) ( t ) , for left and right ear respectively, from the input mono signal x ( t ) , t being the sample index, as follows:
Revised: ( l/r ) ( t ) , for left and right ear respectively, from the input mono signal x ( t ) , t being the sample index, as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: • x ( t ) is assumed to be a clean input signal; ∗ indicates the convolution operation; • h rir ( t, θ (src) t , p (src) t , p (lstn) t ) is the RIR computed using the image source method (Allen & Berkley, 1979) assuming a simple rectangular room and reverberation time 0 .
Revised: • x ( t ) is assumed to be a clean input signal; ∗ indicates the convolution operation; • h rir ( t, θ (src) t , p (src) t , p (lstn) t ) is the RIR computed using the image source method (Allen & Berkley, 1979) assuming a simple rectangular room and reverberation time 0 .

Original: 2s ; p (src) t
Revised: 2s ; p (src) t

Original: and
Revised: and

Original: p (lstn) t are the source and listener positions, and θ (src) t indicates the source orientation used to simulate cardioid directivity; • h ( l/r ) hrtf ( t, θ (src,lstn) t ) is the head related impulse response (HRTF in the time-domain) for the left / right ear; the HRTF of a KEMAR mannequin, measured in an anechoic chamber at 9600 unique discrete positions on a sphere of radius 2m is used; θ (src,lstn) t indicates the direction on which the source is found with respect to the listener’s front; • h ( l/r ) eq ( t ) is the equalization ﬁlter for the given channel, estimated from a test capture in presence of the source signal; and • w ( l/r ) ( t ) is the random noise added to the given channel, generated with a power spectral density estimated from a test capture during a silence period.
Revised: p (lstn) t are the source and listener positions, and θ (src) t indicates the source orientation used to simulate cardioid directivity; • h ( l/r ) hrtf ( t, θ (src,lstn) t ) is the head related impulse response (HRTF in the time-domain) for the left / right ear; the HRTF of a KEMAR mannequin, measured in an anechoic chamber at 9600 unique discrete positions on a sphere of radius 2m is used; θ (src,lstn) t indicates the direction on which the source is found with respect to the listener’s front; • h ( l/r ) eq ( t ) is the equalization ﬁlter for the given channel, estimated from a test capture in presence of the source signal; and • w ( l/r ) ( t ) is the random noise added to the given channel, generated with a power spectral density estimated from a test capture during a silence period.

Original: Since the ﬁlters depend on source and listener positions, which in a dynamic scenario are continuously changing, the computation is done on a frame-by-frame basis using the overlap-add method, with frame length of 1024 samples and 75% overlap (at sampling rate of 48kHz).
Revised: Since the ﬁlters depend on source and listener positions, which in a dynamic scenario are continuously changing, the computation is done on a frame-by-frame basis using the overlap-add method, with frame length of 1024 samples and 75% overlap (at sampling rate of 48kHz).

Original: A. D ATASET D ESCRIPTION
Revised: A. D ATASET D ESCRIPTION

Original: Dataset Overview.
Revised: Dataset Overview.

Original: We recorded eight different subjects, four male and four female, in an acoustically treated room.
Revised: We recorded eight different subjects, four male and four female, in an acoustically treated room.

Original: The capture contains unidirectional conversational speech, i.e., we asked participants to talk to a mannequin for 15 minutes each while walking around.
Revised: The capture contains unidirectional conversational speech, i.e., we asked participants to talk to a mannequin for 15 minutes each while walking around.

Original: We collected approximately 2 hours of mono-to-binaural audio data in total.
Revised: We collected approximately 2 hours of mono-to-binaural audio data in total.

Original: Source and listener head positions are tracked and synchronized with the recorded audio.
Revised: Source and listener head positions are tracked and synchronized with the recorded audio.

Original: We use the last two minutes from each subject and a separately recorded validation sequence as test data and kept the remaining data as training data.
Revised: We use the last two minutes from each subject and a separately recorded validation sequence as test data and kept the remaining data as training data.

Original: To the best of our knowledge, this is the ﬁrst binaural data capture of its kind, i.e., modeling moving trajectories between receiver and transmitter position and recorded in a regular room rather than an anechoic chamber.
Revised: To the best of our knowledge, this is the ﬁrst binaural data capture of its kind, i.e., modeling moving trajectories between receiver and transmitter position and recorded in a regular room rather than an anechoic chamber.

Original: Data Capture Details.
Revised: Data Capture Details.

Original: The acoustic head and torso simulator is the GRAS KEMAR mannequin with the size large anthropometric pinnae inserts.
Revised: The acoustic head and torso simulator is the GRAS KEMAR mannequin with the size large anthropometric pinnae inserts.

Original: Participants were free to walk around a 1 .
Revised: Participants were free to walk around a 1 .

Original: 5 m radius circle around the KEMAR mannequin, and prompted to cover as much area as a normal social conversation would.
Revised: 5 m radius circle around the KEMAR mannequin, and prompted to cover as much area as a normal social conversation would.

Original: The KEMAR mannequin was wearing a B&K 4101B binaural microphone headset.
Revised: The KEMAR mannequin was wearing a B&K 4101B binaural microphone headset.

Original: The subjects wore a DPA 4060-OC microphone taped next to their mouth to capture their speech.
Revised: The subjects wore a DPA 4060-OC microphone taped next to their mouth to capture their speech.

Original: The participants wore a modiﬁed bicycle helmet with reﬂective markers for head-pose tracking using an OptiTrack system.
Revised: The participants wore a modiﬁed bicycle helmet with reﬂective markers for head-pose tracking using an OptiTrack system.

Original: Although the KEMAR mannequin did not move, KEMAR wore a headband with reﬂective markers for head-pose tracking for complete source/listener headpose tracking.
Revised: Although the KEMAR mannequin did not move, KEMAR wore a headband with reﬂective markers for head-pose tracking for complete source/listener headpose tracking.

Original: All tracking information was captured with a ﬁeld array of 24 OptiTrack Prime 17W cameras.
Revised: All tracking information was captured with a ﬁeld array of 24 OptiTrack Prime 17W cameras.

Original: The audio data is recorded at 48kHz sampling rate and rigid body tracking data is collected at 120fps via motion capture software, Motive.
Revised: The audio data is recorded at 48kHz sampling rate and rigid body tracking data is collected at 120fps via motion capture software, Motive.

Original: LTC signal is used to synchronize the audio recordings with OptiTrack data.
Revised: LTC signal is used to synchronize the audio recordings with OptiTrack data.

Original: The capture layout is schematically illustrated in Figure 7.[Figure] [Table]
Revised: The capture layout is schematically illustrated in Figure 7.

Original:  E XTENDEDP ERCEPTUAL E VALUATION
Revised: A. E XTENDED E VALUATION

Original:  In order to back the results of the perceptual study in Table [Table] we performed a side-by-side evaluation of our system and the DSP baseline, which was ranked to be the strongest competitor to our approach.
Revised: Additional Perceptual Evaluation. In order to back the results of the perceptual study in Table 4, we performed a side-by-side evaluation of our system and the DSP baseline, which was ranked to be the strongest competitor to our approach.

Original: In this study [Table] participants were presented an audio snipped rendered with DSP and the same snippet rendered with our approach.
Revised: In this study, participants were presented an audio snipped rendered with DSP and the same snippet rendered with our approach.

Original: The snippets are presented side-by-side in random order to ensure an unbiased evaluation.
Revised: The snippets are presented side-by-side in random order to ensure an unbiased evaluation.

Original: Participants were then asked to decide which of the two methods is preferable in terms of cleanliness, spatialization [Table] and realism.
Revised: Participants were then asked to decide which of the two methods is preferable in terms of cleanliness, spatialization, and realism.

Original: We additionally gave participants the option to select can not tell the difference as an answer.
Revised: We additionally gave participants the option to select can not tell the difference as an answer.

Original: Overall [Table] 30 participants evaluatedbinaural snippets generated with DSP and our method, respectively.
Revised: Overall,participants evaluated 360 binaural snippets generated with DSP and our method, respectively.

Original: The results in Table 7 support our ﬁndings from Table 4: our approach is preferred in terms of cleanliness and realism.
Revised: The results in Table 7 support our ﬁndings from Table 4: our approach is preferred in terms of cleanliness and realism.

Original: For spatialization, most participants could not ﬁnd a clear favorite, which is consistent with the mean opinion score of 3 . 75 vs. 3 .
Revised: For spatialization, most participants could not ﬁnd a clear favorite, which is consistent with the mean opinion score of 3 . 75 vs. 3 .

Original: 76 that is reported in Table 4.
Revised: 76 that is reported in Table 4.

Original: A. A CTIVATION F UNCTIONS
Revised: Unseen Subjects. While previous evaluations were based on unseen audio data from speakers that are part of the training set, we evaluate the performance on unseen speakers here. We train our model in a leave-one-speaker-out setup, i.e., we train eight models, each with another speaker being held out. Table 8 shows that our approach still outperforms the DSP baseline by a large margin

Revised: [Figure]

Revised: [Table]

Revised: on unseen subjects.

Revised: This is remarkable, considering that only seven different subjects are seen during training.

Revised: With a more diverse dataset, we expect the generalization quality of our approach to increase signiﬁcantly.

Original:  In recent works (Sitzmann et al., 2020; Tancik et al., 2020), sine activations have been found to preserve high frequency information better than other commonly used activation functions if weights are initialized appropriately.
Revised: Activation Functions. In recent works (Sitzmann et al., 2020; Tancik et al., 2020), sine activations have been found to preserve high frequency information better than other commonly used activation functions if weights are initialized appropriately.

Original: As reconstructing high frequencies is particularly important for audio modeling, we adopt this strategy in our network.
Revised: As reconstructing high frequencies is particularly important for audio modeling, we adopt this strategy in our network.

Original: Table 8 shows a comparison of our network with ReLUs, gated convolutions as used in Van Den Oord et al.
Revised: Table 9 shows a comparison of our network with ReLUs, gated convolutions as used in Van Den Oord et al.

Original: (2016), and sine activations as used in Sitzmann et al.
Revised: (2016), and sine activations as used in Sitzmann et al.

Original: ReLUs do not perform well on audio data: their sparse outputs are not well suited to model the smooth and sinusoid nature of waveforms.
Revised: ReLUs do not perform well on audio data: their sparse outputs are not well suited to model the smooth and sinusoid nature of waveforms.

Original: For this reason,WaveNet originally raw waveform power [Table] ours with [Table]
Revised: For this reason,

Original: ours

Original: [Table] ours with sine activation [Table] used gated convolutions, which we also ﬁnd to work better than ReLUs in our task.
Revised: WaveNet originally used gated convolutions, which we also ﬁnd to work better than ReLUs in our task.

Original: Overall, however, we still ﬁnd sine activations to produce the best results.
Revised: Overall, however, we still ﬁnd sine activations to produce the best results.

Original: A. Q UALITATIVE R ESULTS
Revised: Qualitative Results. We show qualitative results on the raw waveform in Figure 8. Note the consid-

Original: [Figure]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: mono) l e f t e a r r i g h t e a r binaural recording l e f t e a r r
Revised: l e f t e a r r i g h t e a r binaural recording l e f t e a r r

Original: i g h t e a r
Revised: i g h t e a r

Original: DSP l e f t e a r r
Revised: DSP l e f t e a r r

Original: i g h t e a r binaural recording l e f t e a r r
Revised: i g h t e a r binaural recording l e f t e a r r

Original: i g h t e a r
Revised: i g h t e a r

Original: WaveNet l e f t e a r r i g h t e a r binaural recording l e f t e a r r
Revised: WaveNet l e f t e a r r i g h t e a r binaural recording l e f t e a r r

Original: i g h t e a r 2.5D
Revised: i g h t e a r 2.5D

Original: Sound l e f t e a r r i g h t e a r binaural recording l e f t e a r r
Revised: Sound l e f t e a r r i g h t e a r binaural recording l e f t e a r r

Original: i g h t e a r
Revised: i g h t e a r

Original: Ours l e f t e a r r
Revised: Ours l e f t e a r r

Original: i g h t e a r

Original: We show qualitative results on the raw waveform in Figure 8.

Original: Note the considerable temporal shift between the mono signal captured at the source’s microphone and the binaural recording.
Revised: i g h t e a r erable temporal shift between the mono signal captured at the source’s microphone and the binaural recording.

Original: A strong binauralization system is required to accurately model not only this temporal shift but also produce a metrically correct waveform, i.e., match the shape of the binaural recording’s waveform.
Revised: A strong binauralization system is required to accurately model not only this temporal shift but also produce a metrically correct waveform, i.e., match the shape of the binaural recording’s waveform.

Original: When comparing the results for WaveNet and the 2.5D Sound architecture, it is apparent that both approaches lack in their ability to accurately match the recording’s waveform.
Revised: When comparing the results for WaveNet and the 2.5D Sound architecture, it is apparent that both approaches lack in their ability to accurately match the recording’s waveform.

Original: Additionally, the 2.5D sound approach fails to align its output to the recordings.
Revised: Additionally, the 2.5D sound approach fails to align its output to the recordings.

Original: This comes of no surprise as the model is inherently designed to solve an upmixing problem, i.e., a problem where temporal shifts do not exist.
Revised: This comes of no surprise as the model is inherently designed to solve an upmixing problem, i.e., a problem where temporal shifts do not exist.

Original: Also note that due to µ -law quantization that is typically applied in WaveNet, its results are non-smooth and introduce high-frequency noise due to quantization bins being misclassiﬁed.
Revised: Also note that due to µ -law quantization that is typically applied in WaveNet, its results are non-smooth and introduce high-frequency noise due to quantization bins being misclassiﬁed.

Original: The DSP approach, which is to day the de-facto state of the art, performs more favorably in terms of temporal alignment and overall matching of the recording’s waveform.
Revised: The DSP approach, which is to day the de-facto state of the art, performs more favorably in terms of temporal alignment and overall matching of the recording’s waveform.

Original: Compared to our approach, however, there are signiﬁcant inaccuracies – an observation that is consistent with our evaluation in Section 3.3.
Revised: Compared to our approach, however, there are signiﬁcant inaccuracies – an observation that is consistent with our evaluation in Section 3.3.