D oppler BAS : Binaural Audio Synthesis Addressing Doppler Effect (2023)

Abstract: Recently, binaural audio synthesis (BAS) has emerged as a promising research field for its applications in augmented and virtual realities. Binaural audio helps ususers orient themselves and establish immersion by providing the brain with interaural time differences reflecting spatial information. However, existing BAS methods are limited in terms of phase estimation, which is crucial for spatial hearing. In this paper, we propose the DopplerBAS method to explicitly address the Doppler effect of the moving sound source. Specifically, we calculate the radial relative velocity of the moving speaker in spherical coordinates, which further guides the synthesis of binaural audio. This simple method introduces no additional hyper-parameters and does not modify the loss functions, and is plug-and-play: it scales well to different types of backbones. DopperBAS distinctly improves the representative WarpNet and BinauralGrad backbones in the phase error metric and reaches a new state of the art (SOTA): 0.780 (versus the current SOTA 0.807). Experiments and ablation studies demonstrate the effectiveness of our method.

## introduction binaural audio synthesis (bas), which aims to render binaural audio from the monaural counterpart, has become a prominent technology in artificial spaces (e.g. augmented and virtual reality) @xcite @xcite @xcite @xcite . binaural rendering provides users with an immersive spatial and social presence @xcite @xcite , by producing stereophonic sounds with accurate spatial information. unlike traditional single channel audio synthesis @xcite , bas places more emphasis on * equal contribution. currently, there are three types of neural networks (nn) to synthesize binaural audio. firstly, @xcite collects a paired monauralbinaural speech dataset and provides an end-to-end baseline with geometric and neural warping technologies. secondly, to simplify the task, @xcite decompose the synthesis into a two-stage paradigm: the common information of the binaural audio is generated in the first stage, based on which the binaural audio is generated in the second stage. they also propose to use the generative model ddpm @xcite to improve the audio naturalness. thirdly, to increase the generalization capability for the out-of-distribution audio, @xcite renders the speech in the fourier space. these non-linear nn-based methods outperform the traditional digital signal processing systems based on a linear time-invariant system @xcite @xcite . however, these nn methods still have room for improvement in accuracy, especially phase accuracy. @xcite claims that the correct phase estimation is crucial for binaural rendering foot_0 . actually, the previous works tend to view the scene "statically", and only take into account the series of positions and head orientations. this motivates us to propose dopplerbas, which facilitates phase estimation by explicitly introducing the doppler effect @xcite into neural networks. specifically, 1) we calculate the 3d velocity vector of the moving sound source in the cartesian coordinates and then decompose this 3d velocity vector into a velocity vector in the spherical coor-dinates relative to the listener; 2) according to the doppler effect, we use the radial relative velocity as an additional condition of the neural network, to incentivize the model to sense the moving objects. we also analyze the efficacy of different types of velocity conditions through extensive experiments. naturally, dopplerbas can be applied to different neural binaural renderers without tuning hyperparameters. we pick two typical recent backbones to demonstrate the effectiveness of our method: 1) warpnet @xcite , a traditional neural network optimized by reconstruction losses; 2) binauralgrad ( leng et al., 2022 ), a novel diffusion model optimized by maximizing the evidence bound of the data likelihood. experiments on warp-net and binauralgrad are representative and could show the generalizability of our proposed doppler-bas on other conditions based on gains on these two models. the contributions of this work can be summarized as follows: • we propose dopplerbas, which distinctly improves warpnet and binauralgrad in the phase error metric and produces a new state of the art performance: 0.780 (vs. the current state of the art 0.807). ## method in this work, we focus on the most basic bas scenario where only the monaural audio, the series of positions and head orientations are provided @xcite , rather than other scenarios where extra modalities @xcite are present. note that scenarios with extra modalities present are different tasks. also, as demonstrated in this paper, our proposed dopplerbas is plug-and-play and can be easily integrated into other more complex scenarios. in this section, we will introduce the doppler effect as the preliminary knowledge, and then introduce the proposed method dopplerbas. we will describe how to calculate and decompose the velocity vec-tor, and how to apply this vector to two different backbones. ## experiments in this section, we first introduce the commonly used binaural dataset, and then introduce the training details for warpnet-based and binauralgradbased models. after that, we describe the evaluation metrics that we use to evaluate baselines and our methods. finally, we provide the main results with analytical experiments on bas. ## conclusion in this work, we proposed dopplerbas to address the doppler effect of the moving sound source in binaural audio synthesis, which is not explicitly considered in previous neural bas methods. we calculate the radial relative velocity of the moving source in the spherical coordinate system as the additional conditions for bas. experimental results show that dopplerbas scales well to different types of backbones and reaches a new sota.