Interpretable Convolutional Filters with SincNet
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep learning is currently playing a crucial role toward higher levels of artiﬁcialintelligence.
This paradigm allows neural networks to learn complex and abstractrepresentations, that are progressively obtained by combining simpler ones.
Nevertheless, the internal "black-box" representations automatically discovered bycurrent neural architectures often suffer from a lack of interpretability, making ofprimary interest the study of explainable machine learning techniques.
This paper summarizes our recent efforts to develop a more interpretable neuralmodel for directly processing speech from the raw waveform.
In particular, wepropose SincNet , a novel Convolutional Neural Network (CNN) that encouragesthe ﬁrst layer to discover more meaningful ﬁlters by exploiting parametrized sincfunctions.
In contrast to standard CNNs, which learn all the elements of eachﬁlter, only low and high cutoff frequencies of band-pass ﬁlters are directly learnedfrom data.
This inductive bias offers a very compact way to derive a customizedﬁlter-bank front-end, that only depends on some parameters with a clear physicalmeaning.
Our experiments, conducted on both speaker and speech recognition,show that the proposed architecture converges faster, performs better, and is moreinterpretable than standard CNNs.
Introduction
Deep learning has recently contributed to achieving unprecedented performance levels in numeroustasks, mainly thanks to the progressive maturation of supervised learning techniques [1].
The increased discrimination power of modern neural networks, however, is often obtained at the cost of areduced interpretability of the model.
Modern end-to-end systems, whose popularity is increasing inmany ﬁelds such as speech recognition [2, 3, 4], often discover "black-box" internal representationsthat make sense for the machine but are arguably difﬁcult to interpret by humans.
The remarkablesensitivity of current neural networks toward adversarial examples [5], for instance not only highlightshow superﬁcial the discovered representations could be but also raised crucial concerns about ourcapabilities to really interpret neural models.
Such a lack of interpretability can be a major bottleneckfor the development of future deep learning techniques.
Having more meaningful insights on thelogic behind network predictions and errors, in fact, can help us to better trust, understand, anddiagnose our model, eventually guiding our efforts toward more robust deep learning.
In recent years,a growing interest has been thus devoted to the development of interpretable machine learning [6, 7],as witnessed by the numerous works in the ﬁeld, ranging from visualization [8, 9], diagnosis of DNNs[10], explanatory graphs [11], and explainable models [12], just to name a few.
Interpretability is a major concern for audio and speech applications as well [13].
CNNs and Recurrent
Neural Networks (RNNs) are the most popular architectures nowadays used in speech and speakerrecognition [2].
RNN can be employed to capture the temporal evolution of the speech signal[14, 15, 16, 17], while CNNs, thanks to their weight sharing, local ﬁlters, and pooling networksare normally employed to extract robust and invariant representations [18].
Even though standard hand-crafted features such as FBANK and Mel-Frequency Cepstral Coefﬁcients (MFCC) are stillemployed in many state-of-the-art systems [19, 20, 21], directly feeding a CNN with spectrogrambins [22, 23, 24] or even with raw audio samples [25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37]is an approach of increasing popularity.
The engineered features, in fact, are originally designedfrom perceptual evidence and there are no guarantees that such representations are optimal for allspeech-related tasks.
Standard features, for instance, smooth the speech spectrum, possibly hinderingthe extraction of crucial narrow-band speaker characteristics such as pitch and formants.
Conversely,directly processing the raw waveform allows the network to learn low-level representations that arepossibly more customized on each speciﬁc task.
The downside of raw speech processing lies in the possible lack of interpretability of the ﬁlter banklearned in the ﬁrst convolutional layer.
According to us, the latter layer is arguably the most criticalpart of current waveform-based CNNs.
This layer deals with high-dimensional inputs and is alsomore affected by vanishing gradient problems, especially when employing very deep architectures.
As will be discussed in this paper, the ﬁlters learned by CNNs often take noisy and incongruousmulti-band shapes, especially when few training samples are available.
These ﬁlters certainly makesome sense for the neural network, but they do not appeal to human intuition, nor appear to lead to anefﬁcient representation of the speech signal.
To help the CNNs discover more meaningful ﬁlters, this work proposes to add some constraints ontheir shape.
Compared to standard CNNs, where the ﬁlter-bank characteristics depend on severalparameters (each element of the ﬁlter vector is directly learned), SincNet convolves the waveformwith a set of parametrized sinc functions that implement band-pass ﬁlters [38].
The low and highcutoff frequencies are the only parameters of the ﬁlter learned from data.
This solution still offersconsiderable ﬂexibility but forces the network to focus on high-level tunable parameters that havea clear physical meaning.
Our experimental validation has considered both speaker and speechrecognition tasks.
Speaker recognition is carried out on TIMIT [39] and Librispeech [40] datasetsunder challenging but realistic conditions, characterized by minimal training data (i.e., 12-15 secondsfor each speaker) and short test sentences (lasting from 2 to 6 seconds).
With the purpose of validating
SincNet in both clean and noisy conditions, speech recognition experiments are conducted on boththe TIMIT and DIRHA dataset [41, 42].
Results show that the proposed SincNet converges faster,achieves better performance, and is more interpretable than a more standard CNN.
The remainder of the paper is organized as follows.
The SincNet architecture is described in Sec. 2.
Sec. 3 discusses the relation to prior work.
The experimental activity on both speaker and speechrecognition is outlined in Sec. 4.
Finally, Sec. 5 discusses our conclusions.
The SincNet Architecture
The ﬁrst layer of a standard CNN performs a set of time-domain convolutions between the inputwaveform and some Finite Impulse Response (FIR) ﬁlters [43].
Each convolution is deﬁned asfollows 1 :
[Equation]
where x [ n ] is a chunk of the speech signal, h [ n ] is the ﬁlter of length L , and y [ n ] is the ﬁltered output.
In standard CNNs, all the L elements (taps) of each ﬁlter are learned from data.
Conversely, theproposed SincNet (depicted in Fig. 1) performs the convolution with a predeﬁned function g thatdepends on few learnable parameters θ only, as highlighted in the following equation:
[Equation]
A reasonable choice, inspired by standard ﬁltering in digital signal processing, is to deﬁne g suchthat a ﬁlter-bank composed of rectangular bandpass ﬁlters is employed.
In the frequency domain, themagnitude of a generic bandpass ﬁlter can be written as the difference between two low-pass ﬁlters:
[Equation]
where f 1 and f 2 are the learned low and high cutoff frequencies, and rect ( · ) is the rectangularfunction in the magnitude frequency domain 2 .
After returning to the time domain (using the inverse
Fourier transform [43]), the reference function g becomes:
[Equation]
where the sinc function is deﬁned as sinc ( x ) = sin ( x ) /x .
The cut-off frequencies can be initialized randomly in the range [0 , f s / 2] , where f s represents thesampling frequency of the input signal.
As an alternative, ﬁlters can be initialized with the cutofffrequencies of the mel-scale ﬁlter-bank, which has the advantage of directly allocating more ﬁlters inthe lower part of the spectrum, where crucial speech information is located.
To ensure f 1 ≥ 0 andf 2 ≥ f 1 , the previous equation is actually fed by the following parameters:
[Equation]
Note that no bounds have been imposed to force f 2 to be smaller than the Nyquist frequency, sincewe observed that this constraint is naturally fulﬁlled during training.
Moreover, the gain of each ﬁlteris not learned at this level.
This parameter is managed by the subsequent layers, which can easilyattribute more or less importance to each ﬁlter output.
An ideal bandpass ﬁlter (i.e., a ﬁlter where the passband is perfectly ﬂat and the attenuation in thestopband is inﬁnite) requires an inﬁnite number of elements L .
Any truncation of g thus inevitablyleads to an approximation of the ideal ﬁlter, characterized by ripples in the passband and limitedattenuation in the stopband.
A popular solution to mitigate this issue is windowing [43].
Windowingis performed by multiplying the truncated function g with a window function w , which aims tosmooth out the abrupt discontinuities at the ends of g :
[Equation]
This paper uses the popular Hamming window [44], deﬁned as follows:
[Equation]
The Hamming window is particularly suitable to achieve high frequency selectivity [44].
However,results not reported here reveals no signiﬁcant performance difference when adopting other functions,such as Hann, Blackman, and Kaiser windows.
All operations involved in SincNet are fully differentiable and the cutoff frequencies of the ﬁlterscan be jointly optimized with other CNN parameters using Stochastic Gradient Descent (SGD) orother gradient-based optimization routines.
As shown in Fig.
1, a standard CNN pipeline (pooling,normalization, activations, dropout) can be employed after the ﬁrst sinc-based convolution.
Multiplestandard convolutional, fully-connected or recurrent layers [15, 16, 17, 45] can then be stackedtogether to ﬁnally perform a classiﬁcation with a softmax classiﬁer.
Fig.
2 shows some examples of ﬁlters learned by a standard CNN and by the proposed SincNet for aspeaker identiﬁcation task trained on Librispeech (the frequency response is plotted between 0 and 4kHz).
As observed in the ﬁgures, the standard CNN does not always learn ﬁlters with a well-deﬁnedfrequency response.
In some cases, the frequency response looks [Figure]
[Figure]
2.1 Model properties
The proposed SincNet has some remarkable properties:
# Epoch 0.
F E R ( % )
SincNet
CNN
Frequency [Hz] 0.
N o r m a li z ed
F il t e r
S u m
SincNet
CNN 2nd
Formant 1st
Formant
Pitch
• Fast Convergence: SincNet forces the network to focus only on the ﬁlter parameters with a major impact on performance. The proposed approach actually implements a natural inductive bias, utilizing knowledge about the ﬁlter shape (similar to feature extraction
methods
generally deployed on this task
while retaining ﬂexibility to adapt to data.
This
prior
knowledge
makes
learning
the ﬁlter characteristics much
easier
, helping
SincNet to
converge
signiﬁcantly faster
to a better
solution
Fig.
3 shows the learning curves of SincNetand CNN obtained in a speaker-id task.
These results are achieved on the TIMIT datasetand highlight a faster decrease of the Frame Error Rate ( FER % ) when SincNet is used.
Moreover, SincNet converges to better performance leading to a FER of 33.0% against a
FER of 37.7% achieved with the CNN baseline.
• Few Parameters: SincNet drastically reduces the number of parameters in the ﬁrst con- volutional layer.
For instance, if we consider a layer composed of F ﬁlters of length L , a standard CNN employs F · L parameters, against the 2 F considered by SincNet.
If F = and L = 100 , we employ 8k parameters for the CNN and only 160 for SincNet.
Moreover, if we double the ﬁlter length L , a standard CNN doubles its parameter count (e.g., we go
from 8k to 16k), while SincNet has an unchanged parameter count (only two parametersare employed for each ﬁlter, regardless its length L ).
This offers the possibility to derivevery selective ﬁlters with many taps, without actually adding parameters to the optimization problem.
Moreover, the compactness of the SincNet architecture makes it suitable in the few sample regime.
• Computational Efﬁciency : The proposed function g is symmetric.
This means we canperform convolution in a very efﬁcient way by only considering one side of the ﬁlter and inheriting the results for the other half.
This saves 50% of the ﬁrst-layer computation over a standard CNN.
Interpretability :
The SincNet feature maps obtained in the ﬁrst convolutional layer are
deﬁnitely more interpretable and human-readable than other approaches.
The ﬁlter bank, in fact, only depends on parameters with a clear physical meaning.
Fig.
4, for instance, shows the cumulative frequency response of the ﬁlters learned by SincNet and CNN on a speaker-id task.
The cumulative frequency response is obtained by summing up all the discovered ﬁlters and is useful to highlight which frequency bands are covered by the learned ﬁlters.
Interestingly, there are three main peaks which clearly stand out from the SincNet plot (seethe red line in the ﬁgure).
The ﬁrst one corresponds to the pitch region (the average pitch is133 Hz for a male and 234 for a female).
The second peak (approximately located at {+
+}
Hz) mainly captures ﬁrst formants, whose average value over the various English vowelsis indeed 500 Hz.
Finally, the third peak (ranging from 900 to 1400 Hz) captures some important second formants, such as the second formant of the vowel /a/ , which is locatedon average at 1100 Hz.
This ﬁlter-bank conﬁguration indicates that SincNet has successfullyadapted its characteristics to address speaker identiﬁcation.
Conversely, the standard CNNdoes not exhibit such a meaningful pattern: the CNN ﬁlters tend to correctly focus on the [Figure] k [Figure]
lower part of the spectrum, but peaks tuned on ﬁrst and second formants do not clearlyappear.
As one can observe from Fig.
4, the CNN curve stands above the SincNet one.
SincNet, in fact, learns ﬁlters that are, on average, more selective than CNN ones, possiblybetter capturing narrow-band speaker clues.
Fig.
5 shows the cumulative frequency response of a CNN and SincNet obtained on a noisyspeech recognition task.
In this experiment, we have artiﬁcially corrupted TIMIT with asigniﬁcant quantity of noise in the band between 2.0 and 2.5 kHz (see the spectrogram) andwe have analyzed how fast the
Related Work
Several works have recently explored the use of low-level speech representations to process audioand speech with CNNs.
Most prior attempts exploit magnitude spectrogram features [22, 23, 24, 46,47, 48].
Although spectrograms retain more information than standard hand-crafted features, theirdesign still requires careful tuning of some crucial hyper-parameters, such as the duration, overlap,and typology of the frame window, as well as the number of frequency bins.
For this reason, a morerecent trend is to directly learn from raw waveforms, thus completely avoiding any feature extractionstep.
This approach has shown promise in speech [25, 26, 27, 28, 29], including emotion tasks [30],speaker recognition [35], spooﬁng detection [34], and speech synthesis [31, 32].
Similar to SincNet, some previous works have proposed to add constraints on the CNN ﬁlters, forinstance forcing them to work on speciﬁc bands [46, 47].
Differently from the proposed approach, thelatter works operate on spectrogram features and still learn all the L elements of the CNN ﬁlters.
Anidea related to the proposed method has been recently explored in [48], where a set of parameterized
Gaussian ﬁlters are employed.
This approach operates on the spectrogram domain, while SincNetdirectly considers the raw waveform in the time domain.
This work extends our previous studies on the SincNet [38].
To the best of our knowledge, this paperis the ﬁrst that shows the effectiveness of the proposed SincNet in a speech recognition application.
Moreover, this work not only considers standard close-talking speech recognition, but it also extendsthe validation of SincNet to distant-talking speech recognition [49, 50, 51].
Results
The proposed SincNet has been evaluated on both speech and speaker recognition using differentcorpora.
This work considers a challenging but realistic speaker recognition scenario: for all theadopted corpora, we only employed 12-15 seconds of training material for each speaker, and we testedthe system performance on short sentences lasting from 2 to 6 seconds.
In the spirit of reproducibleresearch, we release the code of SincNet on GitHub 3 .
More details on the adopted datasets as well ason the SincNet and baseline setups can found in the appendix.
4.1 Speaker Recognition
Table 1 reports the Classiﬁcation Error Rates (CER%) achieved on a speaker-id task.
The tableshows that SincNet outperforms other systems on both TIMIT (462 speakers) and Librispeech (2484speakers) datasets.
The gap with a standard CNN fed by raw waveform is larger on TIMIT, conﬁrmingthe effectiveness of SincNet when few training data are available.
Although this gap is reduced when
LibriSpeech is used, we still observe a 4% relative improvement that is also obtained with fasterconvergence (1200 vs 1800 epochs).
Standard FBANKs provide results comparable to SincNet onlyon TIMIT, but are signiﬁcantly worse than our architecture when using Librispech.
With few trainingdata, the network cannot discover ﬁlters that are much better than that of FBANKs, but with moredata a customized ﬁlter-bank is learned and exploited to improve the performance.
Table 2 extends our validation to speaker veriﬁcation, reporting the Equal Error Rate (EER%)achieved with Librispeech.
All DNN models show promising performance, leading to an EERlower than 1% in all cases.
The table also highlights that SincNet outperforms the other models,showing a relative performance improvement of about 11% over the standard CNN model.
Notethat the speaker veriﬁcation system is derived from the speaker-id neural network using the d-vectortechnique.
The d-vector [19, 24] is extracted from the last hidden layer of the speaker-id network.
Aspeaker-dependent d-vector is computed and stored for each enrollment speaker by performing an L2normalization and averaging all the d-vectors of the different speech chunks.
The cosine distancebetween enrolment and test d-vectors is then calculated, and a threshold is then applied on it to rejector accept the speaker.
Ten utterances from impostors were randomly selected for each sentencecoming from a genuine speaker.
To assess our approach on a standard open-set speaker veriﬁcationtask, all the enrolment and test utterances were taken from a speaker pool different from that used fortraining the speaker-id DNN [Table] [Table] outperforms
[Table]
For the sake of completeness, experiments have also been conducted with standard i-vectors.
Althougha detailed comparison with this technology is out of the scope of this paper, it is worth noting that ourbest i-vector system achieves an EER=1.1%, rather far from what achieved with DNN systems.
Itis well-known in the literature that i-vectors provide competitive performance when more trainingmaterial is used for each speaker and when longer test sentences are employed [52, 53, 54].
Underthe challenging conditions faced in this work, neural networks achieve better generalization.
4.2 Speech Recognition
Tab. 3 reports the speech recognition performance obtained by CNN and SincNet using the TIMITand the DIRHA dataset [41].
To ensure a more accurate comparison between the architectures, ﬁveexperiments varying the initialization seeds were conducted for each model and corpus.
Table 3 thusreports the average speech recognition performance.
Standard deviations, not reported here, rangebetween 0 . 15 and 0 .
2 for all the experiments.
[Table]
For all the datasets, SincNet outperforms CNNs trained on both standard FBANK and raw waveforms.
The latter result conﬁrms the effectiveness of SincNet not only in close-talking scenarios but alsoin challenging noisy conditions characterized by the presence of both noise and reverberation.
Asemerged in Sec.2, SincNet is able to effectively tune its ﬁlter-bank front-end to better address thecharacteristics of the noise.
Conclusions and Future Work
This paper proposed SincNet, a neural architecture for directly processing waveform audio.
Ourmodel, inspired by the way ﬁltering is conducted in digital signal processing, imposes constraintson the ﬁlter shapes through efﬁcient parameterization.
SincNet has been extensively evaluated onchallenging speaker and speech recognition tasks, consistently showing some performance beneﬁts.
Beyond performance improvements, SincNet also signiﬁcantly improves convergence speed over astandard CNN, is more computationally efﬁcient due to the exploitation of ﬁlter symmetry, and itmore interpretable than standard black-box models.
Analysis of the SincNet ﬁlters, in fact, revealedthat the learned ﬁlter-bank is tuned to the speciﬁc task addressed by the neural network.
In futurework, we would like to evaluate SincNet on other popular speaker recognition tasks, such as VoxCeleb.
Inspired by the promising results obtained in this paper, in the future we will explore the use of
SincNet for supervised and unsupervised speaker/environmental adaptation.
Moreover, although thisstudy targeted speaker and speech recognition only, we believe that the proposed approach deﬁnes ageneral paradigm to process time-series and can be applied in numerous other ﬁelds.
Acknowledgement
This research was enabled in part by support provided by Calcul Québec and Compute Canada.
AppendixCorpora
To provide experimental evidence on datasets characterized by different numbers of speakers, this paper considers 394 the TIMIT (462 spks, train chunk) [39] and Librispeech (2484 spks) [40] corpora.
For speaker veriﬁcationexperiments, non-speech intervals at the beginning and end of each sentence were removed.
Moreover, the 396 Librispeech sentences with internal silences lasting more than 125 ms were split into multiple chunks.
To addresstext-independent speaker recognition, the calibration sentences of TIMIT (i.e., the utterances with the same text 398 for all speakers) have been removed.
For the latter dataset, ﬁve sentences for each speaker were used for training,while the remaining three were used for test.
For the Librispeech corpus, the training and test material have been 400 randomly selected to exploit 12-15 seconds of training material for each speaker and test sentences lasting 2-6seconds.
To evaluate the performance in a challenging distant-talking scenario, speech recognition experiments 402 have also considered the DIRHA dataset [41].
This corpus, similarly to the other DIRHA corpora [55, 56], hasbeen developed in the context of the DIRHA project [57] and is based on WSJ sentences [58] recorded in a 404 domestic environment.
Training is based on contaminating WSJ-5k utterances with realistic impulse responses[59, 60], while the test phase test phase consists of 409 WSJ sentences recorded by native speakers in a domestic 406 environment (the average SNR is 10 dB).
SincNet Setup
The waveform of each speech sentence was split into chunks of 200 ms (with 10 ms overlap), which were 409 fed into the SincNet architecture.
The ﬁrst layer performs sinc-based convolutions as described in Sec. 2,using 80 ﬁlters of length L = 251 samples.
The architecture then employs two standard convolutional layers, 411 both using 60 ﬁlters of length 5.
Layer normalization [61] was used for both the input samples and for allconvolutional layers (including the SincNet input layer).
Next, three fully-connected layers composed of 2048 413 neurons and normalized with batch normalization [62, 63] were applied.
All hidden layers use leaky-ReLU[64] non-linearities.
The parameters of the sinc-layer were initialized using mel-scale cutoff frequencies, while 415 the rest of the network was initialized with the well-known “Glorot" initialization scheme [65].
Frame-levelspeaker and phoneme classiﬁcations were obtained by applying a softmax classiﬁer, providing a set of posterior 417 probabilities over the targets.
For speaker-id, a sentence-level classiﬁcation was simply derived by averagingthe frame predictions and voting for the speaker which maximizes the average posterior.
Training used the 419 RMSprop optimizer, with a learning rate lr = 0 .
001 , α = 0 .
95 , (cid:15) =
10 − 7 , and minibatches of size 128.
All thehyper-parameters of the architecture were tuned on TIMIT, then inherited for Librispeech as well.
The speaker 421 veriﬁcation system was derived from the speaker-id neural network using the d-vector approach [19, 24], whichrelies on the output of the last hidden layer and computes the cosine distance between test and the claimed 423 speaker d-vectors.
Ten utterances from impostors were randomly selected for each sentence coming from agenuine speaker.
Note that to assess our approach on a standard open-set speaker-id task, all the impostors were 425 taken from a speaker pool different from that used for training the speaker-id DNN.
Baseline Setups
We compared SincNet with several alternative systems.
First, we considered a standard CNN fed by the raw 428 waveform.
This network is based on the same architecture as SincNet, but replacing the sinc-based convolutionwith a standard one.
A comparison with popular hand-crafted features was also performed.
To this end, we computed 39 MFCCs (13 431 static+ ∆ + ∆∆ ) and 40 FBANKs using the Kaldi toolkit [66].
These features, computed every 25 ms with 10ms overlap, were gathered to form a context window of approximately 200 ms (i.e., a context similar to that of 433 the considered waveform-based neural network).
A CNN was used for FBANK features, while a Multi-Layer
Perceptron (MLP) was used for MFCCs.
Note that CNNs exploit local correlation across features and cannot 435 be effectively used with uncorrelated MFCC features.
Layer normalization was used for the FBANK network,while batch normalization was employed for the MFCC one.
The hyper-parameters of these networks were also 437 tuned using the aforementioned approach.
For speaker veriﬁcation experiments, we also considered an i-vector baseline.
The i-vector system was imple- 439 mented with the SIDEKIT toolkit [67].
The GMM-UBM model, the Total Variability (TV) matrix, and
the
Probabilistic Linear Discriminant Analysis (PLDA) were trained on the Librispeech data (avoiding test and 441 enrollment sentences).
GMM-UBM was composed of 2048 Gaussians, and the rank of the TV and PLDAeigenvoice matrix was 400.
The enrollment and test phase is conducted on Librispeech using the same set of 443 speech segments used for DNN experiments.