S OUND S YNP :
S OUND
S OURCE D ETECTION FROM R AW W AVEFORMS WITH M ULTI -S CALE S YNPERIODIC F ILTERBANKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Accurately estimating sound sources’ temporal location, spatial location and semantic identity label from multi-channel sound raw waveforms is crucial for an agent to understand the 3D environment acoustically.
Multiple sounds form a complex waveform mixture in time, frequency and space, so accurately detecting them requires a representation that can achieve high resolutions across all these dimensions.
Existing methods often cannot do it well because they either extract hand-engineered features ( i.e. STFT, LogMel) that require a great deal of parameter tuning work ( i.e. ﬁlter length, window size), or propose to learn a single ﬁlter bank to process sound waveforms in a single-scale that often leads to limited timefrequency resolution capability.
In this paper, we tackle this issue by proposing to learn a group of parameterized synperiodic ﬁlter banks.
Each synperiodic ﬁlter’s length and frequency resolution are inversely related, hence is capable of maintaining a better time-frequency resolution trade-off.
By alternating the periodicity term, we can easily obtain a group of synperiodic ﬁlter banks, where each bank differs in its temporal length.
Convolution of the proposed ﬁlterbanks with the raw waveform helps to achieve multi-scale perception in the time domain.
Moreover, applying synperiodic ﬁlter bank to recursively process a downsampled waveform enables us to also achieve multi-scale perception in the frequency domain.
Beneﬁting from the advantage of the multi-scale perception in both time and frequency domain, our proposed synperiodic ﬁlter bank groups learn a data-dependent time-frequency resolution map.
Following the learnable synperiodic ﬁlter bank group front-end, we add a Transformer-like backbone with two parallel soft-stitched branches to learn semantic identity label and spatial location representation semi-independently.
Experiments on both direction of arrival estimation task and the physical location estimation task shows our framework outperforms existing methods by
a large margin.
Replacing existing methods’ front-end with synperiodic ﬁlter bank also helps to improve the performance
I NTRODUCTION
The fundamental task for an agent to perceive and interact with the 3D environment is to know the location and semantic identity of its nearby objects.
The location includes spatial location that is either stationary or moving, temporal location like start time and end time.
Vision-based such environment perception has received large attention in the past decade and we have witnessed huge progress in tasks such as object detection(Liu et al., 2016; Lin et al., 2014; Yang et al., 2019), classiﬁcation(He et al., 2016a) and tracking(Wang et al., 2019).
Nevertheless, the sound-based counterpart research has far lagged behind, despite all the fascinating properties sound signal exhibits.
For example, sound is ubiquitous and insensitive to ambient illumination change, it has no ﬁeld-of-view (FoV) constraints and is capable of circumventing physical barriers to perceive scene beyond line-of-sight.
As a complementary sensing approach to vision, sound-based environment perception is of vital importance for acoustic scene understanding.
A typical example is the sound source detection (answer where is it, when does it happen and what is it), given the recorded multi-channel sound waveforms.
To detect sound sources, we often deploy a spatially-conﬁgured microphone array to record an acoustic environment.
Unlike camera or LiDAR scanner that directly captures RGB image or measures the range distance, sound waveform itself is a highly compressed one-dimensional points with high sampling rate, all sound source signals are compressed and mixed into the one-dimensional data format.
Since different sound sources have different frequency property, it is essential convert waveform into time-frequency representation so that frequencies hidden in the waveform are explicitly split out.
This is often achieved by projecting the raw waveform into various orthogonal frequency basis.
At the same time, a sound source’s spatial location clue lies in inter-channel difference of recorded waveforms ( i.e. phase difference in this work).
It is essential to design a neural network that jointly encodes mono-channel time-frequency representation and inter-channel phase difference from the raw waveforms in a uniﬁed, parameter-frugal and computation-efﬁcient manner.
The learned representation should have elegant resolution in both time, frequency and space domain so that sound sources can be precisely detected.
However, learning such representation is a tough task.
Challenges derive from both theoretical side and practical side.
According to Uncertainty Principle, we cannot achieve the optimal resolution in time and frequency domain at the same time, but instead keep a trade-off between them.
For example, shorter ﬁlters are often preferred to achieve high time resolution, but longer ﬁlters are required to achieve high frequency resolution, especially for low-frequency sound signal.
Traditional hand-engineered sound feature(Davis & Mermelstein, 1980; Cao et al., 2021; Tho Nguyen et al., 2020; Brandstein & Silverman, 1997) and some recently proposed learnable ﬁlter bank(Ravanelli & Bengio, 2018; Zeghidour et al., 2018) empirically set the same length for all ﬁlters, resulting in human-biased, unadjustable time-frequency resolution map.
Some other work(Zeghidour et al., 2021) correlates ﬁlter frequency response and ﬁlter length by initializing in mel-scale, but it is not scalable nor stable because the ﬁnal initialization depends on the ﬁlter number.
Moreover, all existing methods process raw waveform with one-scale ﬁlter bank, we argue that the one-scale sound perception easily leads to incomprehensive sound sources sensing, especially when sound sources have different frequency property or undergo various spatial motion.
In this paper, we ﬁrst give comprehensive theoretical analysis on the ﬁlter bank impact on its extracted feature’s time and frequency resolution.
Based on the analysis, we propose a simple yet effective synperiodic ﬁlter bank construction strategy in which each ﬁlter’s frequency response and length are correlated by rotating periodicity such that each ﬁlter’s length is inversely proportional to its frequency resolution.
The synperiodic ﬁlter bank thus internally maintains a better time-frequency resolution trade-off than traditional ﬁxed-length ﬁlter bank.
Coupling the ﬁlter length with its frequency response helps us to reduce human intervention in ﬁlter bank design.
By simply alternating the periodicity term, we further construct a group of synperiodic ﬁlter banks, with which we achieve multi-scale perception in time domain.
At the same time, by applying a synperiodic ﬁlter bank to process one raw waveform as well as its consecutively-downsampled versions, we achieve multi-scale perception in frequency domain.
The multi-scale perception in both time and frequency domain of synperiodic ﬁlter bank enables the neural network to dynamically learn better representation for sound source detection in a data-driven way.
It is worth noting that synperiodic ﬁlter bank parameter number is just linear to ﬁlter number (adds up to less than 1% of the whole parameter number) and it can be efﬁciently implemented as a 1D convolution operator.
Following the aforementioned learnable front-end, we add a Transformer-like backbone network with two paralleling branches with intermediate soft-parameter sharing to learn sound source’s semantic and spatial location related representation both jointly and separately.
Experiment on both direction-of-arrival (DoA) task and physical location estimation task shows that our proposed framework outperform comparing methods by a large margin.
Replacing existing method’s head with our proposed synperiodic ﬁlter bank also improvdes the performance.
R ELATED W ORK
Sound signal processing has been thoroughly studied in traditional digital signal processing area.
The preliminary step of sound signal processing is usually to convert raw waveform into 2D timefrequency representation.
There are two main realms: Fourier transform based and Wavelet based transform(Sturm, 2007).
Traditional sound feature design are motivated inﬂuenced by human-auditory system.
For example, they often convert frequency bins into mel-scale to imitate human hearing system, like MFCC(Davis & Mermelstein, 1980), LogMel(Cao et al., 2021;
Grondin et al., 2019), the ﬁlter length is empirically chosen and often a windowing is added to avoid spectrum leakage.
For inter-channel phase difference encoding, it is often recommended to encode in frequency domain due to the less-computation advantage.
Typical phase difference features include GCC-Phat(Brandstein & Silverman, 1997) and intensity vector(Cao et al., 2021).
Sound source detection has been previously treated as sound structure(Thrun, 2006) estimation, sound object detection(He et al., 2021) and sound event detection and localization (SELD)
problem(Adavanne et al., 2018; Thi Ngoc et al., 2021).
It involves jointly identifying a sound source’s semantic label and predicting its spatial location, the two sub-tasks have been thoroughly studied separately in acoustics(Nandwana & Hasan, 2016; Mohan et al., 2008; Sundar et al., 2020; Vera-Diaz et al., 2018) and computer vision community(He et al., 2016a). T. Kim et al.
(Kim et al., 2019) provided a review and discussion for raw-audio based event classiﬁcation.
Beneﬁting from the success of the traditional hand-engineered sound feature and the large availability of mature image-based deep neural networks(He et al., 2016a), most work(Grondin et al., 2019; Cao et al., 2021;
Grondin et al., 2019; Adavanne et al., 2018; Thi Ngoc et al., 2021) tackle the task by ﬁrst extracting hand-engineered sound feature and then feeding them to mature image-based neural networks.
This workﬂow is straightforward and often guarantees reasonably good results but it is not end-to-end trainable and heavily depend on image side which may not be optimal for sound processing.
At the same time, some work(He et al., 2021; Adavanne et al., 2018; Tho Nguyen et al., 2020) have simpliﬁed the problem by assuming no two sound sources of the same semantic label but different spatial location happen at the same time.
This assumption avoids semantic label and spatial location association issue but may not reﬂect real scenarios.
In recent years, a bunch of work tried to design neural work to directly learn from raw sound waveform, ranging from the earlier methods that directly apply stacked layers to process raw waveform(Schneider et al., 2019;
Palaz et al., 2013; Jaitly & Hinton, 2011; Sainath et al., 2013) to the recent frequencysensitive ﬁlter bank learning methods(Zeghidour et al., 2021; Ravanelli & Bengio, 2018; Zeghidour et al., 2018;
He et al., 2021; Hoshen et al., 2015; Sainath et al., 2015; Luo & Mesgarani, 2019).
The ﬁlter bank parameter is initialized in either mel-scale(Ravanelli & Bengio, 2018;
Zeghidour et al., 2021;
He et al., 2021; Zeghidour et al., 2018) or as Gammatone ﬁlter(Hoshen et al., 2015;
Sainath et al., 2015).
SoundDet(He et al., 2021) is the ﬁrst work to directly learn from multi-channel raw waveforms to detect sound sources.
It designs MaxCorr ﬁlter bank to directly convolve with multi-channel raw waveforms to learn phase difference aware features.
Multi-scale representation has a rich history in computer vision community ((Liu et al., 2016; 2021;
Lin et al., 2016;
He et al., 2016b)), in which multi-scale representation strategy has been proposed to accommodate large object scale variation.
For example, SSD(Liu et al., 2016) combines feature maps of various scale to cover potential objects of various sizes.
In addition to spatial multi-scale representation, feature space multi-scale representation has been explored as well.
For example, He et.
al.
(He et al., 2016b) incorporate features arising from intermediate layers to boost attribute recognition.
In sound signal processing, St ´ ephane Mallat(Bruna & Mallat, 2013; Mallat, 2012) proposed wavelet scattering to obtain multi-scale sound representation by iteratively treating the proceeding processed sound waveform as new virtual waveform for further process.
S OUND S OURCE D ETECTION FROM M ULTI -C HANNEL R AW W AVEFORMS 3. S OUND S OURCE D ETECTION P ROBLEM D EFINITION
We use a spatially-deployed microphone array, like four closely-bounded microphones in a circular coplanar conﬁguration, to record an acoustic environment where various sound sources undergo independent and unconstrained spatial motion.
Recorded multi-channel raw waveforms are represented as
W “ t x i p t qu 4 i “ 1 , each waveform is one-dimensional vector of size T sampled at a ﬁxed sampling rate.
We denote K sound sources involved in the waveforms by S “ t s k “ p t ks , t ke , sp k , sl k qu Kk “ 1 , one single sound source corresponds to a start time t s , end time t e , stationary or moving spatial motion sp and a semantic identity label sl .
Our goal is to learn a representation P that is representative enough to detect sound sources both temporally, spatially and semantically.
P is a temporal framewise representation of shape T 1 ˆ N .
We propose to learn P directly from the raw waveforms via a deep neural network F parameterized by θ ,
[Equation]
A learned elegant representation should have large resolution in both time, frequency and space domain and exhibit strong capability at addressing three intrinsic challenges:
Polyphonicity, sound sources are mutually independent, they can freely overlap temporally; Time-scale variance, the temporal duration of different sound sources varies enormously, from seconds to minutes long;
Datasize, sound waveform is sampled with large sampling rate, resulting in large data size.
Sound source presence mainly lies in mono-channel sound waveform time-frequency representation.
Its spatial motion lies in inter-channel waveforms interaction, such as sound signal arrival time delay and phase difference.
In this paper, our proposed synperiodic ﬁlter bank groups jointly encode time-frequency representation and inter-channel phase difference from raw waveforms in a uniﬁed way.
More details about phase difference encoding is given in the Appendix A.1.
3. M ULTI -S CALE S YNPERIODIC F ILTER B ANK
We denote the general ﬁlter bank of M ﬁlters by F “ t F i p w i , σ i , l i q “ φ p w i q ¨ ω p σ i qu Mi “ 1 .
Each single ﬁlter F i is created by multiplying the frequency-selective ﬁlter φ p w i q of frequency response w i ( i.e. sinusoidal basis or band-pass basis(Ravanelli & Bengio, 2018)) with a windowing function ω p σ i q .
The windowing function has locality property which means it is just active within a local region controlled by σ i ( i.e. Gaussian kernel window).
Then l i points are cropped around the active region and treated as a frequency-selective ﬁlter (usually l i ą σ i ).
Existing ﬁlter bank differs in their way of parameter setting, including ﬁlters’ frequency response distribution, ﬁlter length length and windowing function.
In classic hand-engineered features such as short-time Fourier transform (STFT) and LogMel feature, all these parameters are empirically chosen.
Contrary to STFT, wavelet transform(Sturm, 2007) inversely correlates window length with frequency response so that the active region of the ﬁlter is internally decided by its frequency response.
Time-Frequency Dynamic Resolution
The frequency resolution indicates the ability of discerning two adjacent frequency bins in frequency domain, while similarly time resolution corresponds to the capability of precisely localizing a sound object in time domain.
The resolution in time and frequency domain jointly inﬂuence spatial localization precision because we encode spatial location related feature on pre-extracted time-frequency feature.
According to Uncertainty Principle, however, the frequency resolution ∆ f and time resolution ∆ t satisﬁes ∆ f ¨ ∆ t ě C
( C is a constant).
It means we cannot get optimal resolution in both time and frequency domain at the same time, but rather keep a trade-off between them: increasing the resolution in one domain inevitably sacriﬁces the resolution in the other domain.
Traditional hand-engineered sound feature’s time-frequency resolution map is ﬁxed because all ﬁlter bank construction relevant parameters are empirically chosen.
Therefore, its time-frequency resolution map is evenly divided across both the time and frequency domain (see Fig.1 second last ﬁgure).
Relaxing the frequency response and windowing length as trainable(Zeghidour et al., 2021) helps to achieve a better time-frequency resolution trade-off.
The improvement is, however, limited because it uses one ﬁlter bank of the same ﬁlter length to process all sound signals (or perceive at one-scale).
We categorize these traditional ﬁlter bank as syndistance ﬁlter bank to emphasize their equal temporal length property across all frequency responses.
In this paper, we rethink ﬁlter bank design and propose a synperiodic ﬁlter bank construction strategy, and further achieve multi-scale processing in both time and frequency domain.
Our motivation is based on the fact that high-frequency sound signal can be adequately detected with a short ﬁlter because its temporal length of one period is small.
Conversely, lower frequency sound signals require wider ﬁlters.
It thus shows using the same window size for all ﬁlters across the frequency range is not a good choice, a more desirable way is to use narrower ﬁlters for ﬁlters with higher frequency responses.
With this guide in mind, we propose a synperiodic ﬁlter bank construction strategy.
To better understand the main difference between syndistance ﬁlter bank and our proposed synperiodic ﬁlter bank, we can visualize them in complex-valued plane (see Fig.1, left-most one), in which a ﬁlter is a complex exponential rotating in the complex plane counter-clockwisely, the rotating speed corresponds to the frequency response it carries.
In the complex-valued plane, all syndistance ﬁlters rotate to the same distance.
Synperiodic ﬁlters, however, rotate to a predeﬁned periodicity ρ , naturally resulting in narrow window for high-frequency ﬁlters and wide window for low-frequency ﬁlters.
Synperiodic ﬁlter bank lends us three advantages: it ﬁrst avoids us setting window length for each ﬁlter separately, which is quite empirical and random; second, the constructed ﬁlter bank naturally maintains a good time-frequency resolution trade-off by design; third, by simply varying the periodicity term ρ , we can easily obtain a group of synperiodic ﬁlter bank to process the raw waveform in multi-scale manner (as we will present in below).
Our synperiodic ﬁlter bank construction strategy shares similar idea with Wavelet transform(Sturm (2007)) where it uses a Gaussian kernel to squeeze or expand a sinusoidal basis wave to generate narrow-window-high-frequency and wide-window-lowfrequency ﬁlter bank.
The difference is that we don’t have to explicitly set the “squeezing rate” for each ﬁlter separately and synperiodic ﬁlter bank is learnable.
Speciﬁcally, our proposed synperiodic ﬁlter bank can be represented as,
[Equation]
where we can see that the windowing function just depends on the center frequency w i and the periodicity term ρ .
By varying the periodicity term ρ , we can obtain a group of synperiodic ﬁlter banks F synp “ t F ρ 1 , F ρ 2 , ¨ ¨ ¨ , F ρ n u .
The comparison between syndistance ﬁlter bank and synperiodic ﬁlter bank is shown in Fig.
There are many ways to instantiate ω p w i , ρ q , as long as we guarantee the window length gradually reduces as the the frequency response increases.
The simplest choice is to treat ω p w i , ρ q as a constant, but we ﬁnd it either results in too wide window for low-frequency ﬁlters or too narrow window for high-frequency ﬁlters.
To mitigate this dilemma, we use logarithmic windowing function
[Equation]
We set ρ as r´ 6 , ´ 11 , ´ 16 s respectively to alternate the periodicity term to construct the three synperiodic ﬁlter bank groups.
The design of this window function is motivated by mel-scale frequency initialization strategy.
By roughly setting a ﬁlter’s band width to be equal to the distance between its preceding and next neighboring frequency location in frequency domain, converting to time domain we can roughly get a logarithmic scale frequency-periodicity relationship (see Fig.4 in Appendix).
In our implementation, synperiodic ﬁlter is created by multiplying a sinusoidal ﬁlter with learnable frequency response initialized in mel-scale by a Gaussian window with learnable width initialized through the windowing function.
Please note that each synperiodic ﬁlter bank group is initialized with independent learnable frequencies and window length, they are independently updated during training stage.
3. M ULTI -S CALE L EARNING IN T IME AND F REQUENCY D OMAIN
We use the previously constructed synperiodic ﬁlter bank groups to convolve with mono-channel sound waveform with the same step size and padding strategy, resulting in the same size output for each single group.
Since different ﬁlter group has different window periodicity ( ρ ), we naturally achieve multi-scale learning in time domain.
It maximally avoids us empirically selecting one window scale ρ which might not be optimal, but instead uses a group of ﬁlter bank to enforce the neural network to strike a better time-frequency resolution trade-off in a data-driven way.
We further propose a strategy to enable multi-scale learning in frequency domain.
The strategy is hierarchical: given a raw sound waveform with sampling frequency F S , synperiodic ﬁlter bank’s frequency is initialized in the range r 0 , F S 2 s under Nyquist sampling theorem.
If we downsample the sound waveform by a factor of 2, the resulting waveform can be processed by the part of the of the original ﬁlter bank whose frequency response lies in r 0 , F S 4 s .
We can thus reuse the lowerhalf ﬁlter bank to process the downsampled waveform.
This process that 2x-downsampling the waveform to get a new waveform and further process the new waveform with the lower-half frequency ﬁlters in the current ﬁlter bank can potentially iterate a couple of times (in our case three times), resulting in a hierarchical and multi-scale convolution structure in frequency domain.
The multi-scale frequency learning strategy is illustrated in Fig.
Multi-scale learning in frequency domain brings us two extra beneﬁts: from data augmentation perspective, hierarchical 2x downsampling the raw waveform naturally generated more waveform data, which helps a lot especially when labelling audio data is technically hard and time-consuming.
From the causality perspective, the adjacent interval subsampling strategy leads to dilated convolution for lower-frequency ﬁlters because applying a ﬁlter to convolve with a downsampled waveform equals to convolve with the original waveform with dilated convolution (skip-2 connection).
The skipped wider ﬁeld of view (FoV) for lower frequency ﬁlters enables to learn better sound causality along the time axis (see Appendix A.4).
In sum, by using learnable synperiodic ﬁlter bank groups to process the raw waveform in multi-scale manner, we can achieve a dynamic time-frequency resolution map that naturally maintains a better time-frequency resolution trade-off and is adjustable w.r.t.
the the sound scene in a data-driven way.
Computational Analysis Synperiodic ﬁlter bank groups introduces very few parameters (less than 1% ) because they are parameterized ﬁlters.
The trainable parameter number increases linearly w.r.t. synperiodic ﬁlter bank number.
Their convolution with raw waveforms can also be efﬁciently implemented by matrix parallelization.
3. T ANSFORMER - LIKE B ACKBONE WITH T WO S OFT -S TITCHING B RANCHES
The aforementioned synperiodic ﬁlter bank generates initial feature representation of shape r T 0 , N 0 , C s , where T 0 is much smaller than the T but is still larger than T 1 .
N 0 equals to the ﬁlter number (in this paper, we use 256), C is channel number.
We stack the feature processed by each synperiodic ﬁlter bank to form the channel dimension.
Jointly learning framewise sound source semantic label and spatial location representation is a multi-task problem(Kendall et al., 2018;
Misra et al., 2016), and learning them in one neural network often leads to inferior performance.
To mitigate this dilemma, we propose a Transformer-like backbone with two paralleling branches to learn each sub-task separately.
To encourage information communication, we add a layerwise information exchange module.
For any two intermediate semantic label feature f is and spatial location aware feature f ig learned by the paralleling i -th network block, a learnable weight W i is introduced to linearly combine them together to get an updated f is and f ig before feeding them to the next layer, r f is , f ig s “ W i ¨ r f is , f ig s .
In practice, the Transformer-like backbone gradually reduces the [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
learned feature’s temporal length but increase the feature length accordingly.
Finally we can get the ﬁnal sound source representation of shape r T 1 , N s .
On top of the representation, we add trackwise permutation-invariant training (PIT) strategy(Cao et al., 2021) to train the whole neural network in an end-to-end manner.
The permutation invariant training strategy helps to address the polyphonicity challenge because it encapsulates a potential sound source’s semantic label and spatial location within a track, so it won’t be inﬂuenced by other sound sources.
E XPERIMENTS
We conduct experiments on two tasks: direction of arrival (DoA) and physical location estimation.
DoA estimation requires to decide arrival angle the sound source w.r.t.
the microphone array.
Physical location tends to directly localize a sound source’s physical location ( i.e. p x, y, z q coordinates).
For
DoA estimation task, we use DCASE2020 sound event detection and localization
dataset(Politis et al., 2020).
It contains 14 sound sources with azimuth range r´ 180 ˝ , 180 ˝ s and elevation range r´ 45 ˝ , 45 ˝ s .
Two recording formats are available: FOA and MIC-array (refer to Appendix for detailed discussion).
For more details about this dataset, please refer to (Politis et al., 2020).
For physical location estimation, we use pyroomacoustics(Scheibler et al., 2018) to simulate the dataset.
We adopt two metrics: segment-based and event-based metric.
Segment-based metric is widely adopted evaluation metric(Adavanne et al., 2018; Cao et al., 2021) and it evaluates in segmentwise manner by coupling semantic label and spatial location prediction result: a semantic-correctly detected sound source needs to be spatially close enough to its ground truth location in order to be regarded as true positive.
Event-based based metric is newly proposed by(He et al., 2021) to comprehensively evaluate under different conﬁdence scores(Lin et al., 2014).
We call our framework SoundSynp and compare it with four main methods: SELDNet(Adavanne et al., 2018), EIN(Cao et al., 2021), SoundDet(He et al., 2021) and Utsc-Iﬂytek(Wang et al., 2020).
SELDNet is the baseline model and it jointly trains sound source’s semantic label and spatial location with a convolutional recurrent neural network (CRNN)(Chung et al., 2014).
EIN(Cao et al., 2021) is a very recent work.
It adopts multi-heads self-attention (MHSA)(Vaswani et al., 2017) to model temporal dependency and trackwise permutation-invariant training to the train the network.
SoundDet(He et al., 2021) directly learns from raw waveform with MaxCorr kernels, followed by an encoder-decoder neural network and an object proposal generation head.
Utsc-Iﬂytek(Wang et al., 2020) is ranked ﬁrst in DECASE2020 challenge leaderboard 1 , it combines MIC and FOA recording format and ensemble different models like ResNet(He et al., 2016a) and Xception(Chollet, 2017) together to infer the ﬁnal prediction.
Implementation Detail The network architecture is shown in table 4 in Appendix material.
To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets.
The raw waveform is ﬁrst normalized to r´ 1 , 1 s before feeding the neural network.
We adopt Adam optimizer(Kingma & Ba, 2015) with an initial learning rate 0.0002 in the ﬁrst 100 epochs and 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classiﬁcation head and regression head is 1 : 2 . During training, data augmentation method SpecAugment(Park et al., 2019) is applied.
For DoA task, we regress direction of arrival angle in Cartesian coordinates r x, y, z s .
In synperiodic ﬁlter bank groups, the ﬁlter length is 1025, each group’s ﬁlter number is 256 and the step size is 600.
Particularly, we have observed the initialized learnable synperiodic ﬁlter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable.
The ﬁnal learned parameters are close to their initialization, one group’s learned parameters (like frequency response and ﬁlter window length) is different from those learned by other groups, although they are initialized as the same in the start.
The result is given in Table1, from which we see that our proposed SoundSynp achieves the best performance over all other comparing methods by a large margin, under both segment-based and event-based evaluation metric.
SELDNet, EIN and UTSC-Iﬂytek both use pre-extracted handengineered sound features, such as Logmel, GCC-Phat and Intensity Vector.
SoundDet(He et al., 2021) and SoundSynp are the only two methods that directly learn from raw waveforms.
At the same time, SoundSynp obtains better performance on FOA than MIC recording format, the same phenomena has been observed by all other methods.
It thus shows FOA better ﬁts for sound source detection than MIC.
It is worth noting that Utsc-Iﬂytek(Wang et al., 2020) ensembles different powerful image-based 2D neural networks infer sound sources.
However, our proposed SoundSynp still outperforms Utsc-Iﬂytek by a large margin.
We don’t report of mAP/mAR value of Utsc-Iﬂytek because it is a complex system and no detail about their system is available.
Ablation Study To disentangle the individual contribution of each part of our SoundSynp framework to the whole performance improvement, we conduct three ablation studies. First , the individual contribution of Synperiodic ﬁlter bank. We replace hand-engineered sound feature head of SELDNet, EIN and learnable MaxCorr ﬁlter bank with our proposed Synperiodic ﬁlter bank to test their corresponding performance.
It helps to remove the inﬂuence of the backbone neural network of different models and thus helps us to get direct comparison of synperiodic ﬁlter bank with other front-end.
Second , we replace SoundSynp’s synperiodic ﬁlter bank front with widely-used MFCC(Davis & Mermelstein, 1980) and Log-Mel that is used by SELDNet(Adavanne et al., 2018) and EIN(Cao et al., 2021). It disentangles synperiodic ﬁlter bank groups with Transformer-like backbone and helps to ﬁgure out if the performance gain is brought by backbone network.
Third , internally, we test ﬁve synperiodic variants: (1) synperiodic ﬁlter bank with just multi-scale perception in frequency domain (SoundSynp MSFreq), (2) just multi-scale perception in time domain (SoundSynp MSTime), (3) Synperiodic ﬁlter bank with frequency responses linearly initialized in Nyquist frequency range (SoundSynp Linear, compare with our mel-scale initialization), (4) just one synperiodic ﬁlter bank without multi-scale perception neither in time nor frequency domain (SoundSynp SingleScale).
(5) synperiodic ﬁlter bank with rectangular band-pass frequency response initialization (SoundSynp Sinc), like SincNet(Ravanelli & Bengio, 2018) does.
The internal comparison helps us to ﬁgure out the necessity of each part of synperiodic ﬁlter bank design.
The ablation study result is shown in Table 2. We can observe that: First , using synperiodic ﬁlter bank as a replacement of existing ﬁlter bank can help to improve the corresponding performance.
Second , replacing SoundSynp’s synperiodic ﬁlter bank with classic hand-engineered features inevitably reduces the performance under all evaluation metrics, its thus shows learning from pre-extracted ﬁxed and single-scale sound representation leads to inferior performance than our proposed multiscale synperiodic ﬁlter bank.
Thrid , the absence of multi-scale perception in either frequency domain or time domain inevitably reduces the performance.
We ﬁnd sound source semantic label detection suffers more in single-scale perception in time domain than in frequency domain (see better performance on ER 20 ˝ , and F 20 ˝ score), which shows frequency domain multi-scale perception is vital for semantic label estimation.
Similarly, we can observe that multi-scale perception in time [Table] [Table] domain is vital for sound source spatial location estimation (see better performance on LE and LR score).
Linearly initialized ﬁlter bank frequency response reduces the performance, which attests assign more ﬁlters to the lower frequency range is important.
But this conclusion might be datadependent because we ﬁnd DCASE dataset contains many low-frequency sounds like burning ﬁre and footsteps.
Moreover, reducing the synperiodic ﬁlter bank group to one groups with just singlescale perception leads to comparably the worst performance, it thus shows multi-scale perception in both time and frequency domain is essential for DoA-based sound source detection.
Lastly, SoundSynp Sinc leads to slightly inferior performance than our used mel-scale initialization strategy, it shows our proposed synperiodic ﬁlter bank is a general ﬁlter bank that can be adopted to other frequency sensitive ﬁlter bank.
One qualitative comparison is shown in Fig.
We can clearly see that SELDNet generates mixed prediction at different time steps and DoA locations.
SoundDet and EIN give non-existing sound sources (orange color).
When multiple sound sources happen at the same time (polyphonicity),
SoundDet and EIN are easily failed to predict the right spatial location (discretized blue and red color).
Our method predicts more spatially and temporally consistent sound sources by maximally keeping sound source’s continuity and consistency.
Physical Location Estimation We use Pyroomacoustics simulator(Scheibler et al., 2018) to simulate a shoebox like room with r 7 m, 5 m, 3 m s size.
Four microphones (24kHz sampling rate) are conﬁgured in a plane with 5 cm mutual distance and put in the room center ( r 3 .
5 m, 2 . 5 m, 1 . 5 m s ).
We choose three commonly heard sounds from BBC sound effect site 2 : cat meowing, baby talking and dog barking.
All these seed sounds last less than 10s.
During simulation, we put all seed sound in the plane z “ 1 .
m
(we constrain to a plane because we ﬁnd relaxing z easily leads EIN model to get stuck in a local minima like r´ 1 , ´ 1 , ´ 1 s ), but with x randomly in r 2 m, 5 m s and y randomly in r 1 m, 4 m s , up to two sounds can happen at the same time.
In sum, we have simulated 500 one-minute recordings (400 for train, 100 for test), with an average of four sounds recorded per recording.
For physical location prediction, we regress the angle and range.
The result is in Table 3.
We do not report SELDNet result as we ﬁnd it doesn’t converge during train.
We can see from the table that
SoundSynp produces the best performance than over EIN and SoundDet network.
Replacing their front-end with synperiodic ﬁlter bank also improves the performance.
It thus shows our proposed
SoundSynp is capable of learning representation for physical-location based sound source detection.
A A PPENDIX A. S PATIAL L OCATION E NCODING IN F REQUENCY D OMAIN AND R ECORDING F ORMAT D ISCUSSION
We discuss the detailed spatial location encoding for FOA and MIC sound waveform recording format.
The spatial location encoding is based on 2D feature learned by the Gabor ﬁlter bank for each sound waveform channel, which can be represented as t F i “ p R i , I i qu 4 i “ 1 .
R i
and I
i are real part and imaginary part feature of the i -th channel sound waveform, respectively.
FOA format is well-known as ﬁrst-order Ambisonics (B-format).
It contains four channels: omnidirectional, x -directional, y -directional and z -directional components, respectively.
The instantaneous sound intensity vector is often used as spatial location (or phase difference) feature, which can be computed through the cross-spectrum between the omni-directional channel to the remaining x, y, zdirectional.
As a result, we have obtained 3 channel spatial location encoding feature.
[Equation]
[Equation] where F ˚ 0 indicates the conjugate of the omni-directional feature.
The three cross-spectrum feature
IV x , IV y and IV z are stacked together and further normalized before serving as the spatial encoding feature.
MIC format is well-known as tetrahedral microphone array.
The four microphones are mounted in spherical coordinates with four distinct orientations.
We treat the four microphones equally and compute the phase difference between any two microphones.
Thus a total of six channels spatial location feature can be constructed.
Speciﬁcally, we choose to compute GCC-PHAT(Brandstein &
Silverman, 1997) like cross-spectrum feature.
For any two channel m and n , we compute the angle between the real part and imaginary part of the cross-spectrum.
[Equation]
SL indicates the spatial location feature computed by the sound waveform channel m and n .
The angle p¨q equals to a frequency amplitude normalization operation, like the GCC-PHAT(Brandstein &
Silverman, 1997) does.
Please note that all the spatial location feature computation operations are differentiable so the whole neural network becomes end-to-end trainable.
A. N ETWORK A RCHITECTURE AND T RAINING D ETAILS
SoundSynp neural network architecture is given in Table 4.
To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets.
The raw waveform is ﬁrst normalized to r´ 1 , 1 s before feeding the neural network.
We adopt Adam optimizer(Kingma & Ba, 2015) with an initial learning rate 0.0002 in the ﬁrst 100 epochs and 0.
in the following 50 epochs.
Batchsize is 16.
The loss combination weight between classiﬁcation head and regression head is 1 : 2 .
During training, data augmentation method SpecAugment(Park et al., 2019) is applied.
For DoA task, we regress direction of arrival angle in Cartesian coordinates r x, y, z s .
In synperiodic ﬁlter bank groups, the ﬁlter length is 1025, each group’s ﬁlter number is 256 and the step size is 600.
Particularly, we have observed the initialized learnable synperiodic ﬁlter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable.
The ﬁnal learned parameters are close to their initialization, one group’s learned parameters (like frequency response and ﬁlter window length) is different from those learned by other groups, although they are initialized as the same in the start.
A. S YNPERIODIC F ILTER B ANK F REQUENCY -P ERIODICITY R ELATIONSHIP D ETERMINATION
Mel-scale time-frequency representation has been widely used in both traditional sound feature like
MFCC(Davis & Mermelstein, 1980), LogMel and learnable ﬁlter
bank(Zeghidour et al., 2021).
It initializes the ﬁlter bank in frequency domain, in which high-frequency ﬁlter has wider window length.
We transform the ﬁlter bank into time domain and can naturally get a roughly logarithmic-scale frequency-periodicity relationship, in which narrower window width is associated with high-frequency
[Table]
ﬁlters.
We thus set ω p w i , ρ q “ 27 ¨ log 10 p w i q ´ ρ .
We plot our synperiodic ﬁlter bank window function and the mel-scale initialized windowing function in Fig.
4, it shows our proposed windowing function naturally approximates the mel-scale windowing function.
A. C AUSALITY IN M ULTI - SCALE FREQUENCY DOMAIN PERCEPTION
Fig.5 shows how causality is achieved in multi-scale frequency domain perception.
[Figure]
A. C OMPUTATION T IME R EPORT
The computation time of all comparing methods is given in Table 5, from which we can see that our proposed SoundSynp framework requires between 2x-3x more computation time than SELDNet, EIN and SoundDet due to its large Transformer-like backbone.
However, SoundSynp computation time is still within a controllable range.
[Table]