BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis
Anonymous Author(s)
AffiliationAddress email
Abstract
Binaural audio plays a significant role in constructing immersive augmented andvirtual realities.
As it is expensive to record binaural audio from the real world,synthesizing them from mono audio has attracted increasing attention.
This synthesis process involves not only the basic physical warping of the mono audio,but also room reverberations and head/ear related filtrations, which, however, aredifficult to accurately simulate in traditional digital signal processing.
In this paper,we formulate the synthesis process from a different perspective by decomposingthe binaural audio into a common part that shared by the left and right channelsas well as a specific part that differs in each channel.
Accordingly, we propose
BinauralGrad, a novel two-stage framework equipped with diffusion models tosynthesize them respectively.
Specifically, in the first stage, the common information of the binaural audio is generated with a single-channel diffusion modelconditioned on the mono audio, based on which the binaural audio is generated by atwo-channel diffusion model in the second stage.
Combining this novel perspectiveof two-stage synthesis with advanced generative models (i.e., the diffusion models),the proposed BinauralGrad is able to generate accurate and high-fidelity binauralaudio samples.
Experiment results show that on a benchmark dataset, BinauralGradoutperforms the existing baselines by a large margin in terms of both object andsubject evaluation metrics (Wave L2: 0 .
128 vs. 0 .
157 , MOS: 3 . 80 vs. 3 . 61 ).
Thegenerated audio samples are available online 1 .
Introduction
Human brains have the ability to decode spatial properties from real-world stereophonic sounds, whichhelps us to locate and interact with the environment.
While in artificial spaces such as augmented andvirtual reality, generating accurate binaural audio from the mono one is therefore essential to provideimmersive environments for listeners.
Traditional Digital Signal Processing (DSP) systems generatebinaural audio by warping the mono audio according to the time difference between two ears andmodeling room reverberations (e.g., room impulse response (RIR)) and head/ear related filtration (e.g.,head-related transfer functions (HRTF))
through a linear time-invariant system [32, 45, 35].
However,they fail to generate accurate and immersive binaural audio due to the non-linear nature of soundpropagation and the simplification in DSP systems, such as simplifying the physical modeling of RIRand utilizing general HRTFs instead of personalized.
In addition, the exact physical information ofthe recording environment is not available most of the time.
Instead of following the complicated synthesis process of binaural audio in traditional digital signalprocessing which is difficult to model, in this paper, we formulate the process from a different and
[Figure]
novel perspective.
We first decompose the synthesis of binaural audio into two steps: 1) the originalmono audio is emitted by the object and then diffuses to the listener.
The audio that arrives at theleft and right ears are similar, as the distance between the two ears is marginal compared with thatbetween the object and the listener; 2) the binaural audio is then encoded by human head dependingon the marginal difference of audio received at two ears [2, 41].
This process is personalized as themorphology of each listener is unique, which will change the internal encoding results.
Preciselyprocessing the two stages is difficult due to the complexity of physical system.
Therefore, as analternative, given that the two channels of the binaural audio is similar but with marginal differences,we propose to divide the the representation of the binaural audio y
=
( y l , y r ) as two parts.
One isthe general part that contains the common information of two channels, which are represented asthe average ¯ y ; the other one is the specific part that is distinct between the two channels encoded byhuman head in the end, which are denoted as δ l and δ r .
Then we can represent the binaural audio asy l = ¯ y + δ l and y r
= ¯ y
+ δ r respectively.
The common part represents the fundamental informationof the source mono audio, while the specific parts contain the difference between the left and rightchannels caused by room reverberance and the acoustical influence of human head.
Based on the representation, we propose a two-stage framework to model the two parts respectively.
The common information of the two channels of binaural audio is generated in the first stage, andtheir difference are generated in the second stage.
We therefore term them as first/common andsecond/specific stages.
The common stage is encouraged to generate ¯ y while the specific stage triesto model the difference to the left and right audio.
Specifically, we build our framework on denoisingdiffusion probabilistic models (diffusion models for short) [12], which have been shown effective andefficient to generate high-fidelity audio on the task of speech synthesis [15].
In the common stage, themono audio is taken as the condition of a single-channel diffusion model, which is supervised by theaverage over the two channels of the golden binaural speech ¯ y , encouraging the model to generate thecommon information of two channels.
And in the specific stage, conditioned on the single-channeloutput of the first stage, we utilize a two-channel diffusion model to generate the audio for the leftand right ears respectively.
We provide an illustration of the proposed framework in Figure 1.
The main contributions of this work are summarized as follows:
• We formulate the representation of binaural audio as two parts, and propose a novel two-stage framework to respectively synthesize their common and specific information conditioned on
the mono audio.• Equipped with denoising probabilistic diffusion models, the proposed two-stage framework is able to generate accurate and high-fidelity binaural audio.
• We conduct experiments on the benchmark dataset released in [29] and the proposed
framework outperforms existing baselines by a large margin, both on automatic (Wave L2:
128 vs. 0 .
and human evaluation (MOS: 3 . 80 vs. 3 . 61 ) metrics.
Related Works2.1 Binaural Audio Synthesis
Based on the physical process of sound rendering, human listening can be generally considered asa source-medium-receiver model [3].
The sound waves emitted by the object will travel throughthe medium, i.e., free air and then be received by listener.
In medium, the sound will be diffused,reverberated and effected by the physical objects such as walls during the propagation.
Room impulseresponse (RIR) [19, 36, 1, 30] describes the distortion effect caused by surrounding environmentusing filters.
Besides the interaction with medium, the morphology of listener (torso, head, and pinna)will also change the sound received by the eardrum.
The head-related transfer functions (HRTF) [3, 6]are utilized to describe the transformation of the sound from a direction in free air to that it arrivesat the eardrum [18].
Digital Signal Processing (DSP) methods synthesize the binaural speech bycombining RIR and HRTF [45, 35], both of which are expensive and tedious to measure, as well asmissed from most open-sourced datasets in addition.
Therefore, DSP methods usually utilize genericfunctions instead, resulting in sub-optimal generation results as the recording environments are highlyspecialized among different datasets.
Recently, neural rendering approaches are proposed for binaural speech synthesis.
Gebru et al.
[10]utilizing neural networks to learn HRTFs implicitly.
Richard et al.
[29] propose a convolution neuralnetwork based model with an additional neural time warping module to learn the time shifts from themono to binaural audio.
Some works incorporate extra modalities such as the visual information tohelp the synthesis of binaural speech [44, 43, 26].
In this work, we consider the most basic scenariowhere only the mono audio is available.
2.2 Denoising Diffusion Probabilistic Models
Denoising diffusion probabilistic models (diffusion models for short) have achieved the state-of-theart (SOTA) generation results in various tasks, including image [34, 22, 8, 7, 33, 37, 42] and superresolution image generation [13, 31, 39, 25], text-to-image generation [23, 11, 14, 28], text-to-speechsynthesis [4, 15, 27, 17, 16, 5] and speech enhancement [20, 21, 40].
Especially, in audio synthesis,diffusion models have shown strong ability in modelling both spectrogram features [27, 17] and rawwaveforms [4, 15, 5].
Diffusion models require a large number of inference steps to achieve high sample quality.
For highresolution data generation, some works [8, 13, 28] even cascade multiple diffusion models togetherand achieve the SOTA quality in text-to-image generation [28].
In this cascaded structure, multiplediffusion models are trained at different resolution scales, and while inference, low-resolution datasamples are firstly generated with the base diffusion model, and then stacked conditional diffusionmodels are employed for super resolution, where high-fidelity results can be achieved with moderatetime cost.
In this paper, for generating binaural audio waveform with high resolution ( 48 kHz sampling rate), weintegrate diffusion models into the proposed two-stage framework, and generate the common andspecific information of binaural audio with single- and two-channel diffusion models respectively.
Methods
We introduce the proposed framework BinauralGrad in this section.
We start with the preliminaryknowledge about binaural audio and geometric warping, and then introduce the proposed two-stageframework as well as the structure of the diffusion models.
3.1 Preliminary
Problem Definition Given the source mono audio x ∈ R N with length N and the relative positionbetween the source and listener p , we aim to generate the binaural audio y
=
( y l , y r ) that containstwo channels of audio for left and right ears through( y l ( n ) , y r ( n ))
=
f ( x ( n ) , p ) , (1) where y l and y r ∈ R N , and f indicates the transformation function parameterized by the proposedframework.
We denote the average of the binaural audio as ¯ y
= mean ( y l , y r ) .
Diffusion Model 𝜃 𝑐 𝑁(0,1) ത𝑦 𝑐 = (
[Equation]
[Equation]
𝛼
[Equation]
ҧ𝑥 𝑥
DiffusionModel 𝜃 𝑠 𝑁(0,1)
𝑦 𝑙 𝑐 = (𝑝 𝑠 ,𝑝 𝛼 ) 𝑁(0,1) 𝑦 𝑟 𝑥
ො𝑥 𝑙 ො𝑥
𝑟 ത𝑦 Specifically, the relative position can be described with a tuple p =
( p s , p α ) , where p s =
( p x , p y , p z )is the spatial position that indicated by the coordinates, and p α
= ( qx, qy, qz, qw ) is quaternions thatindicates the head orientation from the listener to the source.
The listener is stationary and set inthe origin of the coordinate system, and therefore ( p x , p y , p z ) axes indicate the front, right and updirections respectively.
Besides, we also denote the spatial position of the left and right ears of thelistener as p llstn and p rlstn respectively.
A simple way to align the temporal differences between the left and right ears is geometric warping,a non-parametric method conditioned on the distance between the source and the listener:
[Equation]
where n indicates the current time-stamp, C is a constant that calculated by the ratio between theaudio sampling rate and the speed of the sound.
As the predicted warpfield ρ ( n ) are usually floats,the warped signals can be computed with the linear interpolation: [Equation]
[Equation]
) where ⌈·⌉ and ⌊·⌋ indicate the ceil and floor function respectively.
The warping for the left and rightears can be obtained by changing the position p lstn ( n ) , and the warped binaural audio is denoted as(ˆ x l , ˆ x r ) , which are with low quality without considering the diffraction of audio.
3.2 Overview of the Framework
The proposed framework contains two stages, and we train two distinct diffusion models for eachstage, parameterized by θ c and θ s respectively.
In the first stage, the model θ c aims to synthesize thecommon information of the binaural audio supervised by ¯ y , i.e., we treat the mono average of twochannels as the golden supervision of the common information.
And in the second stage, conditionedon the synthesized audio y c of the first stage, a two-channel diffusion model θ s tries to synthesize thegolden binaural audio y
=
( y l , y r ) .
We provide an illustration of the proposed framework in Figure 2.
We will first introduce the backbone diffusion model and then the details of two stages as below.
3.2.1 Conditional Diffusion Models
Diffusion models are score-based generative models [12, 34, 8].
They are composed of a forwardprocess and a reverse process .
In the training stage, the forward process gradually destroys the datasamples into Gaussian noise with a large number of time steps.
At each time step t , the model learnsa score function that denotes the gradient information.
Then, in the reverse process, with learnedscore functions at predefined inference noise schedules, the model can generate clean data samplesfrom Gaussian noise in an iterative denoising process.
The forward process converts the data samples z 0 into the isotropic Gaussian noise ϵ ∼ N (0 , I ) witha predefined variance schedule 0 < β 1 < · · · < β t < · · · < β T < 1 .
The latent representations attime step t can be directly calculated with:
[Equation]
where the α t :=
1 − β t , and ¯ α t := (cid:81) ts =
1 α s can denote a corresponding noise level at time step t .
In sampling, the reverse process starts from the isotropic Gaussian noise p ( z T ) ∼ N (0 , I ) , anditeratively denoises the generated samples toward clean data z 0 with the conditioning information c :
[Equation]
Providing strong conditioning information c is usually helpful to reduce the number of inferencesteps and improve the generation quality.
The Gaussian transition probability of each inference stepis parameterized as:
[Equation]
where the variance σ 2 θ is usually predefined as 1 − ¯ α t − 1 1 − ¯ α t β t or β t .
Following previous works [12, 15],we use the former one in this work.
The model is trained by maximizing the variation lower bound ofthe likelihood p θ ( z 0 ) .
When we parameterize the mean function as:
[Equation]
7) a reweighted training objective is usually adopted in practice as [12, 4, 15]:
[Equation]
3.2.2 Details of Two Stages
When considering diffusion models, we find that in forward process, the high-frequency informationis firstly destroyed, then the low-frequency information is also destroyed with the time step t increases.
Correspondingly, in reverse process, the low-frequency information will be gradually generated atfirst, then details will be iteratively refined with following inference steps.
Inspired by this mechanism,we integrate the diffusion models into the proposed two-stage framework, in which a single-channeldiffusion model generates the common information of binaural audio in the first stage, while atwo-channel diffusion model is then used for modelling the specific information of both left ear andright ear.
In the first stage, the average ¯ y of the golden binaural audio is considered as the clean data in theforward process.
In this way, the diffusion model is encouraged to generate the common informationof the two channels.
The condition information is consistent for the forward and reverse processes.
In concrete, the model takes the average of the warped binaural audio ¯ x = mean (ˆ x l , ˆ x r ) as thecondition information.
In addition to the mono audio and position, the condition of the first stage canbe denoted as c 1 =
( p s , p α , ¯ x, x ) .
The position and audio information are separately processed bythe model, which will be detailed introduced in Section 3.3.
The second stage aims at synthesizing the binaural audio through a two-channel diffusion model,therefore the golden audio y
=
( y l , y r ) is taken as the clean data for the two channels respectively.
Different from that in the first stage, the condition information is distinct in the forward and reverseprocess as it includes the generation results of the first stage.
The average ¯ y of the golden binauralaudio is known in the forward process, therefore, along with the warped binaural audio, the conditioninformation can be written as c
= ( p s , p α , ˆ x l , ˆ x r , x, ¯ y ) .
For the reverse process, the generationresults y c of the first stage is utilized to replace the golden audio which is unknown in inference, i.e.,c 2 =
( p s , p α , ˆ x l , ˆ x r , x, y c ) .
While training, the two stages are trained separately by minimizing the regression loss defined in
Equation 8 utilizing different clean data and conditions.
And in inference, firstly the common stagegenerates the mono audio y c , conditioned on which the specific stage synthesizes the binaural audio,which are taken as the final output of the framework.
There exists an inconsistency between thetraining and inference of the second stage due to the different conditions, and fine-tune the secondstage conditioned on the output of the first stage may alleviate this problem.
We leave it for futurework.
3.3 Model Architecture
The model architecture of our diffusion model is shown in Figure 3.
The diffusion models utilized inthe two stages have similar architectures, except the channels of the input, output and condition are [Figure] Residual [Figure]
one and two for the common and specific stages respectively.
We introduce a Conditioner componentto deal with the condition signals introduced in previous sections.
Specifically, we utilize two separateconvolution blocks to process the position as well as the conditional audio respectively, and theoutput features are concatenated and then processed by a final convolution layer.
The conditionalaudio represents the concatenation of [¯ x, x ] and [ˆ x l , ˆ x r , x, y c ] for the two stages.
The output of the
Conditioner is taken as the representation of condition signals.
The rest of our model generally follows the design in [15].
The main network is based on thebidirectional variant of the dilated convolution layer [24] given the non-autoregressive nature of thetask.
The architecture is constructed by M residual blocks, with m dilated convolution layers ineach block.
The diffusion step t is represented by positional encoding [38] and encoded by twofully connected layers before fed to the network, and then in each block, the diffusion step willbe further transformed by another fully connected layer that is unique among blocks, where theoutput representation will be added to the input of the dilated convolution layer.
The conditionalrepresentation modeled by the Conditioner will be added to the output of the dilated layer to bring inthe conditional information.
In the end, through skip connections, the outputs of residual blocks aregathered to produce the final output of the model.
Experiments
Dataset We utilize the dataset released in [29] 2 as it is the largest binaural dataset captured in thewild up to now.
It contains 2 hours of mono-binaural parallel audio data at 48kHz from eight differentobjects, recorded in a regular room.
The position and orientation between the objects and the listenerare tracked at 120Hz and aligned with the audio.
We follow the original split of training/valid/testsets to make our results comparable.
Baselines The proposed framework is denoted as BinauralGrad , and we consider followingbaselines.
DSP is a binaural rendering approach that simulates the spatial acoustic effects of sound sources with dynamic virtual positions.
DSP utilizes RIR to model the room reverberance and
HRTF to model the acoustical influence of the human head.
We perform RIR simulation with anopen-sourced tool 3 according to the room information provided in the original paper [29].
The
HRTF data comes from [9], which was measured at a distance of 1.4 meters using the KEMAR, amanikin for acoustic testing.
And we follow the procedure in 4 to simulate the binaural sound.
Asthe exact HRTF and RIR are not provided in the dataset, therefore the DSP results are worse thanthat reported in the original paper [29].
WaveNet [24] where the positions of source and listenerare used as condition signals and fed into a ConvNet to generate binaural audio from the mono one.
WarpNet [29] which originally proposes the dataset, and stacking a neural time warping module witha temporal ConvNet to generate binaural audio.
For all baselines, we adjust their hyper-parameters tomake the number of parameters comparable with our model to ensure fair comparisons.
Evaluation Metrics We use following metrics to evaluate the quality of synthesized binaural audio.
Wave L2 , which is the mean squared error between the synthesized binaural audio and the goldenbinaural recording.
Amplitude L2 and Phase L2 , which are the mean squared errors between thesynthesized binaural speech and the binaural recording on the amplitude and phase respectivelyafter performing Short-Time Fourier Transform (STFT) on wave.
PESQ , which is the perceptualevaluation of speech quality 5 and widely used in speech enhancement.
MRSTFT , which modelsthe multi-resolution spectral loss 6 by taking the spectral convergence, log magnitude loss and linearmagnitude loss into consideration.
Except PESQ which is the higher the better, the rest of metrics arethe lower the better.
Besides object metrics, we also conduct subject evaluation (Mean Option Score,
MOS ) to verify the audio quality intuitively.
Training Configurations As described in Section 3.3, BinauralGrad consists of M = 3 residualblocks, each of which has m = 10 bidirectional dilated convolution layers.
The hidden size and thedimension of the diffusion step encoding are both set to 128 .
We train BinauralGrad on 8 Nvidia
V100 GPUs for 1M steps, and the diffusion steps are set to 200 and 6 during training and inferencerespectively, mainly following previous diffusion works [15].
We will introduce the experimental results in the rest of this section.
We first compare the proposed
BinauralGrad with baseline systems in terms of objective metrics, then verify the effectiveness of theproposed two-stage framework by comparing with single stage diffusion model.
In addition, a subjectmetric MOS test is also utilized to show the intuitive performance of the model.
Finally, to betterunderstand the two-stage model, we also conduct a thorough analysis on the output of each stage.
4.1 Main Results
We report the object metrics of BinauralGrad and other binaural speech synthesis baselines in Table 1.
The proposed framework performs consistently better than compared baselines over different metrics.
Specifically, we have several observations:
• Our method outperforms the main baseline WarpNet on Wave L2 by a large margin, showing that BinauralGrad can synthesize better binaural waveforms that are closer to the golden
recordings.• Since there are some randomness in the waveform, the metrics on the STFT results of waveform is important to evaluate the overall quality of binaural speech synthesis.
Ourmethod surpasses baseline methods by a large margin in terms of Amplitude L2 andMRSTFT, and being slight better than WarpNet on Phase L2.• PESQ is a widely used metric on the perceptual evaluation of speech quality.
The proposedBinauralGrad achieves 2 .
759 PESQ score, which is significantly better than other baselinesystems.
The naturalness of our method is further verified by the MOS results listed inTable 3.
In conclusion, the object metrics show that the proposed two-stage framework is able to generatemore accurate binaural audio than existing baselines.
[Table] [Table]
4.2 Comparison with Single Stage Diffusion Model
We compare BinauralGrad with a single stage diffusion model to verify the advantage of the twostage framework.
For the single stage diffusion model, its model architecture is similar to thetwo-channel diffusion model in the second stage of BinauralGrad
[Table]
except for two differences: 1) thesingle stage model does not use any information from the common information, i
[Table]
e., with conditionc =
( p s , p α , ˆ x l , ˆ x r , x ) ; 2) the single stage model is two times deeper so as to make the number ofparameters comparable.
The comparison of BinauralGrad with the single stage diffusion model are shown in Table 2.
Theresults show that the two-stage model performs better than the single stage one on most metricsincluding PESQ and L2 losses of waveform, amplitude and phase, while achieving comparable
MRSTFT scores, demonstrating the effectiveness of two stage framework.
It is worth noting that thesingle stage model already outperforms the major baseline WarpNet over most metrics except Phase
L2 as shown in Table 1, illustrating the stronger ability of diffusion models in synthesizing raw audiowaveform than purely convolution based models.
4.3 MOS Test
We conduct three types of MOS to verify the quality of synthesized audio with human evaluation,including: 1) MOS, where judges are asked to rate for the overall naturalness and fluency ofsynthesized audio; 2) Similarity MOS, where judges are asked to rate for the similarity between thesynthesized results and the golden binaural recordings; 3) Spatial MOS, where the judges are askedto rate for the sense of direction contained in the synthesized audio.
For all MOS tests, the judgesrate discrete score from 1 to 5, and the higher score the better quality.
The MOS results together withthe 95% confidence interval are shown in Table 3.
Firstly, the proposed BinauralGrad achieves a quite high MOS score 3 . 80 which not only surpassesall baselines, but also slightly outperforms the recording, illustrating the strong ability of the proposedframework in synthesizing natural audio waveforms.
Secondly, the similarity MOS shows that
BinauralGrad can synthesize closest binaural audio to the golden binaural recording among allcompared models, which is consistent with the conclusions from objective metrics.
Finally, theresults on the spatial MOS show that DSP, WarpNet and BinauralGrad achieve similar performanceon describing the sense of spatial, which might be attributed to the fact that the above three methodsall utilize the physical warping process that brings in the interaural time difference (ITD) and resultsin the sense of direction [41].
4.4 Analysis
In this section, we conduct a thorough analysis to verify the effectiveness of each stage in ourframework, and try to find out the potential bottleneck that brings sparks for future work.
[Table]
[Table]
In each stage
[Table]
the diffusion model synthesize audio conditioned on different information. Therefore,to verify the effectiveness of the model in each stage, we evaluate the performance of the conditionand synthesized audio separately to illustrate the improvements brought by the model.
Specifically, inaddition to the synthesized audio from the two stages (ID 3 and 5 ), we also calculate the performanceof the physical warping results (i.e., (ˆ x l , ˆ x r ) , ID 2 ) and the average ¯ y of the golden binaural audio.
Note that as the output of the first stage and ¯ y are both mono audio, we therefore calculate the scoresby duplicating them to binaural such as (¯ y, ¯ y ) for ID 4 and ( y c , y c ) for ID 3 .
The results are listed in Table 4.
Firstly, we can find that the performance of the physical warping (ID2 ) is worse as it only brings gains on Wave L2 and remain consistent on other metrics compared tothe mono audio (ID 1 ).
Secondly, the improvements from ID 2 to 3 and 4 to 5 are brought by the firstand second stage of the proposed model respectively, verifying the effectiveness and necessities ofboth stages.
In addition, we try to explore the boundary of the framework by directly feeding the perfect conditionto the second stage while inference, i.e., instead of conditioning on y c which is synthesized by the firststage, we utilize the golden average ¯ y as the condition which is unavailable in inference to test thebest performance of the model.
As a result, we can find that the second stage achieves an extremelygood result and brings a large improvement over all metrics compared with the given condition,showing that the model in the second stage has been well trained.
On the contrary, there still existsa large margin between the synthesized result from the first stage and the label (ID 3 vs. ID 4 ).
Inconclusion, the results in Table 4 show that the first stage is the bottleneck of the framework, whichpoints out potential future work directions such as the end-to-end optimization of the two stages.
Conclusion
In this paper, we propose BinauralGrad, a two-stage framework for binaural audio synthesis conditioned on mono audio.
Specifically, we formulate the synthesis process from a novel perspective anddivide the binaural audio into a common part that is shared by two channels as well as distinct parts.
Accordingly, a single-channel diffusion model is utilized to generate the common information in thefirst stage, conditioned on which a two-channel diffusion model synthesizes the binaural audio.
Theproposed framework is able to synthesize accurate and high-fidelity audio samples.
On a benchmarkdataset, BinauralGrad achieves state-of-the-art results both in object and subject evaluation metrics.
In the future, we plan to improve the training of the two stages in an end-to-end way for betterperformance or speed up the inference of BinauralGrad.
The negative impact of our method might bethe abuse of binaural speech synthesis.
Checklist
For all authors.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See future work in Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
negative impact in Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
There is no
theoretical result.
(b) Did you include complete proofs of all theoretical results?
[N/A]
There is no theoretical
result.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section 4.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Table 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A] Since the license can be found in
citation about assets creators.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the source code in supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] Since there is no obvious this kind of issue.
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[Yes] See Appendix A.1 in supplemental material.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A] Since there is no obvious this kind of risk.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[Yes] See Appendix A.1 in supplemental material.
[Table]