Interpretable Convolutional Filters with SincNet
Abstract
Deep learning is currently playing a crucial role toward higher levels of artiﬁcial intelligence.
This paradigm allows neural networks to learn complex and abstract representations, that are progressively obtained by combining simpler ones.
Nevertheless, the internal "black-box" representations automatically discovered by current neural architectures often suffer from a lack of interpretability, making of primary interest the study of explainable machine learning techniques.
This paper summarizes our recent efforts to develop a more interpretable neural model for directly processing speech from the raw waveform.
In particular, we propose SincNet , a novel Convolutional Neural Network (CNN) that encourages the ﬁrst layer to discover more meaningful ﬁlters by exploiting parametrized sinc functions.
In contrast to standard CNNs, which learn all the elements of each ﬁlter, only low and high cutoff frequencies of band-pass ﬁlters are directly learned from data.
This inductive bias offers a very compact way to derive a customized ﬁlter-bank front-end, that only depends on some parameters with a clear physical meaning.
Our experiments, conducted on both speaker and speech recognition, show that the proposed architecture converges faster, performs better, and is more interpretable than standard CNNs.
Introduction
Deep learning has recently contributed to achieving unprecedented performance levels in numerous tasks, mainly thanks to the progressive maturation of supervised learning techniques [1].
The increased discrimination power of modern neural networks, however, is often obtained at the cost of a reduced interpretability of the model.
Modern end-to-end systems, whose popularity is increasing in many ﬁelds such as speech recognition [2, 3, 4], often discover "black-box" internal representations that make sense for the machine but are arguably difﬁcult to interpret by humans.
The remarkable sensitivity of current neural networks toward adversarial examples [5], for instance, not only highlights how superﬁcial the discovered representations could be but also raised crucial concerns about our capabilities to really interpret neural models.
Such a lack of interpretability can be a major bottleneck for the development of future deep learning techniques.
Having more meaningful insights on the logic behind network predictions and errors, in fact, can help us to better trust, understand, and diagnose our model, eventually guiding our efforts toward more robust deep learning.
In recent years, a growing interest has been thus devoted to the development of interpretable machine learning [6, 7], as witnessed by the numerous works in the ﬁeld, ranging from visualization [8, 9], diagnosis of DNNs [10], explanatory graphs [11], and explainable models [12], just to name a few.
Interpretability is a major concern for audio and speech applications as well [13].
CNNs and Recurrent
Neural Networks (RNNs) are the most popular architectures nowadays used in speech and speaker recognition [2].
RNN can be employed to capture the temporal evolution of the speech signal [14, 15, 16, 17], while CNNs, thanks to their weight sharing, local ﬁlters, and pooling networks are normally employed to extract robust and invariant representations [18].
Even though standard hand-crafted features such as FBANK and Mel-Frequency Cepstral Coefﬁcients (MFCC) are still employed in many state-of-the-art systems [19, 20, 21], directly feeding a CNN with spectrogram bins [22, 23, 24] or even with raw audio samples [25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] is an approach of increasing popularity.
The engineered features, in fact, are originally designed from perceptual evidence and there are no guarantees that such representations are optimal for all speech-related tasks.
Standard features, for instance, smooth the speech spectrum, possibly hindering the extraction of crucial narrow-band speaker characteristics such as pitch and formants.
Conversely, directly processing the raw waveform allows the network to learn low-level representations that are possibly more customized on each speciﬁc task.
The downside of raw speech processing lies in the possible lack of interpretability of the ﬁlter bank learned in the ﬁrst convolutional layer.
According to us, the latter layer is arguably the most critical part of current waveform-based CNNs.
This layer deals with high-dimensional inputs and is also more affected by vanishing gradient problems, especially when employing very deep architectures.
As will be discussed in this paper, the ﬁlters learned by CNNs often take noisy and incongruous multi-band shapes, especially when few training samples are available.
These ﬁlters certainly make some sense for the neural network, but they do not appeal to human intuition, nor appear to lead to an efﬁcient representation of the speech signal.
To help the CNNs discover more meaningful ﬁlters, this work proposes to add some constraints on their shape.
Compared to standard CNNs, where the ﬁlter-bank characteristics depend on several parameters (each element of the ﬁlter vector is directly learned), SincNet convolves the waveform with a set of parametrized sinc functions that implement band-pass ﬁlters [38].
The low and high cutoff frequencies are the only parameters of the ﬁlter learned from data.
This solution still offers considerable ﬂexibility but forces the network to focus on high-level tunable parameters that have a clear physical meaning.
Our experimental validation has considered both speaker and speech recognition tasks.
Speaker recognition is carried out on TIMIT [39] and Librispeech [40] datasets under challenging but realistic conditions, characterized by minimal training data (i.e., 12-15 seconds for each speaker) and short test sentences (lasting from 2 to 6 seconds).
With the purpose of validating
SincNet in both clean and noisy conditions, speech recognition experiments are conducted on both the TIMIT and DIRHA dataset [41, 42].
Results show that the proposed SincNet converges faster, achieves better performance, and is more interpretable than a more standard CNN.
The remainder of the paper is organized as follows.
The SincNet architecture is described in Sec. 2.
Sec. 3 discusses the relation to prior work.
The experimental activity on both speaker and speech recognition is outlined in Sec. 4.
Finally, Sec. 5 discusses our conclusions.
The SincNet Architecture
The ﬁrst layer of a standard CNN performs a set of time-domain convolutions between the input waveform and some Finite Impulse Response (FIR) ﬁlters [43].
Each convolution is deﬁned as follows 1 :
[Equation]
where x [ n ] is a chunk of the speech signal, h [ n ] is the ﬁlter of length L , and y [ n ] is the ﬁltered output.
In standard CNNs, all the L elements (taps) of each ﬁlter are learned from data.
Conversely, the proposed SincNet (depicted in Fig. 1) performs the convolution with a predeﬁned function g that depends on few learnable parameters θ only, as highlighted in the following equation:
[Equation]
A reasonable choice, inspired by standard ﬁltering in digital signal processing, is to deﬁne g such that a ﬁlter-bank composed of rectangular bandpass ﬁlters is employed.
In the frequency domain, the magnitude of a generic bandpass ﬁlter can be written as the difference between two low-pass ﬁlters:
[Equation]
[Figure]
where f 1 and f 2 are the learned low and high cutoff frequencies, and rect ( · ) is the rectangular function in the magnitude frequency domain 2 .
After returning to the time domain (using the inverse
Fourier transform [43]), the reference function g becomes:
[Equation]
where the sinc function is deﬁned as sinc ( x ) = sin ( x ) /x .
The cut-off frequencies can be initialized randomly in the range [0 , f s / 2] , where f s represents the sampling frequency of the input signal.
As an alternative, ﬁlters can be initialized with the cutoff frequencies of the mel-scale ﬁlter-bank, which has the advantage of directly allocating more ﬁlters in the lower part of the spectrum, where crucial speech information is located.
To ensure f 1 ≥ 0 and f 2 ≥ f 1 , the previous equation is actually fed by the following parameters:
[Equation]
Note that no bounds have been imposed to force f 2 to be smaller than the Nyquist frequency, since we observed that this constraint is naturally fulﬁlled during training.
Moreover, the gain of each ﬁlter is not learned at this level.
This parameter is managed by the subsequent layers, which can easily attribute more or less importance to each ﬁlter output.
An ideal bandpass ﬁlter (i.e., a ﬁlter where the passband is perfectly ﬂat and the attenuation in the stopband is inﬁnite) requires an inﬁnite number of elements L .
Any truncation of g thus inevitably leads to an approximation of the ideal ﬁlter, characterized by ripples in the passband and limited attenuation in the stopband.
A popular solution to mitigate this issue is windowing [43].
Windowing
n  n  n  f
[Hz] f [Hz] f
[Hz](
[Figure]
) CNN Filters n  n  n  f
[Hz] f [Hz] f
[Hz](
[Figure]
Filters is performed by multiplying the truncated function g with a window function w , which aims to smooth out the abrupt discontinuities at the ends of g :
[Equation]
This paper uses the popular Hamming window [44], deﬁned as follows:
[Equation]
The Hamming window is particularly suitable to achieve high frequency selectivity [44].
However, results not reported here reveals no signiﬁcant performance difference when adopting other functions, such as Hann, Blackman, and Kaiser windows.
All operations involved in SincNet are fully differentiable and the cutoff frequencies of the ﬁlters can be jointly optimized with other CNN parameters using Stochastic Gradient Descent (SGD) or other gradient-based optimization routines.
As shown in Fig.
1, a standard CNN pipeline (pooling, normalization, activations, dropout) can be employed after the ﬁrst sinc-based convolution.
Multiple standard convolutional, fully-connected or recurrent layers [15, 16, 17, 45] can then be stacked together to ﬁnally perform a classiﬁcation with a softmax classiﬁer.
Fig.
2 shows some examples of ﬁlters learned by a standard CNN and by the proposed SincNet for a speaker identiﬁcation task trained on Librispeech (the frequency response is plotted between 0 andkHz).
As observed in the ﬁgures, the standard CNN does not always learn ﬁlters with a well-deﬁned frequency response.
In some cases, the frequency response looks noisy (see the ﬁrst CNN ﬁlter), while in others assuming multi-band shapes (see the third CNN ﬁlter).
SincNet, instead, is speciﬁcally designed to implement rectangular bandpass ﬁlters, leading to more a meaningful ﬁlter-bank.
2.1 Model properties
The proposed SincNet has some remarkable properties:
• Fast Convergence: SincNet forces the network to focus only on the ﬁlter parameters with a major impact on performance.
The proposed approach actually implements a natural inductive bias, utilizing knowledge about the ﬁlter shape (similar to feature extraction methods generally deployed on this task) while retaining ﬂexibility to adapt to data.
This prior knowledge makes learning the ﬁlter characteristics much easier, helping SincNet to converge signiﬁcantly faster to a better solution.
Fig.
3 shows the learning curves of SincNet and CNN obtained in a speaker-id task.
These results are achieved on the TIMIT dataset and highlight a faster decrease of the Frame Error Rate ( FER % ) when SincNet is used.
# Epoch 0.
F E R ( % )
SincNet
CNN
Frequency [Hz] 0.
N o r m a li z ed
F il t e r
S u m
SincNet
CNN 2nd
Formant 1st
Formant
Pitch
Moreover, SincNet converges to better performance leading to a FER of 33.0% against a FER of 37.7% achieved with the CNN baseline.
• Few Parameters: SincNet drastically reduces the number of parameters in the ﬁrst convolutional layer.
For instance, if we consider a layer composed of F ﬁlters of length L , a standard CNN employs F · L parameters, against the 2 F considered by SincNet.
If F =and L = 100 , we employ 8k parameters for the CNN and only 160 for SincNet.
Moreover, if we double the ﬁlter length L , a standard CNN doubles its parameter count (e
g., we go from 8k to 16k), while SincNet has an unchanged parameter count (only two parameters are employed for each ﬁlter, regardless its length L ).
This offers the possibility to derive very selective ﬁlters with many taps, without actually adding parameters to the optimization problem.
Moreover, the compactness of the SincNet architecture makes it suitable in the few sample regime.
• Computational Efﬁciency : The proposed function g is symmetric.
This means we can perform convolution in a very efﬁcient way by only considering one side of the ﬁlter and inheriting the results for the other half.
This saves 50% of the ﬁrst-layer computation over a standard CNN.
Interpretability :
The SincNet feature maps obtained in the ﬁrst convolutional layer are deﬁnitely more interpretable
and human-readable than other approaches.
The ﬁlter bank, in fact, only depends on parameters with a clear physical meaning.
Fig.
4, for instance, shows the cumulative frequency response of the ﬁlters learned by SincNet and CNN on a speaker-id task.
The cumulative frequency response is obtained by summing up all the discovered ﬁlters and is useful to highlight which frequency bands are covered by the learned ﬁlters.
Interestingly, there are three main peaks which clearly stand out from the SincNet plot (see the red line in the ﬁgure).
The ﬁrst one corresponds to the pitch region (the average pitch is 133 Hz for a male and 234 for a female).
The second peak (approximately located at
Hz) mainly captures ﬁrst formants, whose average value over the various English vowels is indeed 500 Hz.
Finally, the third peak (ranging from 900 to 1400 Hz) captures some important second formants, such as the second formant of the vowel /a/ , which is located on average at 1100 Hz.
This ﬁlter-bank conﬁguration indicates that SincNet has successfully adapted its characteristics to address speaker identiﬁcation.
Conversely, the standard CNN does not exhibit such a meaningful pattern: the CNN ﬁlters tend to correctly focus on the lower part of the spectrum, but peaks tuned on ﬁrst and second formants do not clearly appear.
As one can observe from Fig.
4, the CNN curve stands above the SincNet one.
SincNet, in fact, learns ﬁlters that are, on average, more selective than CNN ones, possibly better capturing narrow-band speaker clues.
Fig.
5 shows the cumulative frequency response of a CNN and SincNet obtained on a noisy speech recognition task.
In this experiment, we have artiﬁcially corrupted TIMIT with a signiﬁcant quantity of noise in the band between 2.0 and 2.5 kHz (see the spectrogram) and we have analyzed how fast the two architectures learn to avoid such a useless band.
The
[Figure]
second row of sub-ﬁgures compares the CNN and the SincNet at an early training stage (i.e., after processing only one hour of speech), while the last row shows the cumulative frequency responses after completing the training.
From the ﬁgures emerges that both
CNN and SincNet have correctly learned to avoid the corrupted band at end of training, as highlighted by the holes between 2.0 and 2.5 kHz in the cumulative frequency responses.
SincNet, however, learns to avoid such a noisy band much earlier.
In the second row of sub-ﬁgures, in fact, SincNet shows a visible valley in the cumulative spectrum even after processing only one hour of speech, while CNN has only learned to give more importance to the lower part of the spectrum.
Related Work
Several works have recently explored the use of low-level speech representations to process audio and speech with CNNs.
Most prior attempts exploit magnitude spectrogram features [22, 23, 24, 46, 47, 48].
Although spectrograms retain more information than standard hand-crafted features, their design still requires careful tuning of some crucial hyper-parameters, such as the duration, overlap, and typology of the frame window, as well as the number of frequency bins.
For this reason, a more recent trend is to directly learn from raw waveforms, thus completely avoiding any feature extraction step.
This approach has shown promise in speech [25, 26, 27, 28, 29], including emotion tasks [30], speaker recognition [35], spooﬁng detection [34], and speech synthesis [31, 32].
Similar to SincNet, some previous works have proposed to add constraints on the CNN ﬁlters, for instance forcing them to work on speciﬁc bands [46, 47].
Differently from the proposed approach, the latter works operate on spectrogram features and still learn all the L elements of the CNN ﬁlters.
An idea related to the proposed method has been recently explored in [48], where a set of parameterized
Gaussian ﬁlters are employed.
This approach operates on the spectrogram domain, while SincNet directly considers the raw waveform in the time domain.
This work extends our previous studies on the SincNet [38].
To the best of our knowledge, this paper is the ﬁrst that shows the effectiveness of the proposed SincNet in a speech recognition application.
Moreover, this work not only considers standard close-talking speech recognition, but it also extends the validation of SincNet to distant-talking speech recognition [49, 50, 51].
Results
The proposed SincNet has been evaluated on both speech and speaker recognition using different corpora.
This work considers a challenging but realistic speaker recognition scenario: for all the adopted corpora, we only employed 12-15 seconds of training material for each speaker, and we tested the system performance on short sentences lasting from 2 to 6 seconds.
In the spirit of reproducible research, we release the code of SincNet on GitHub 3 .
More details on the adopted datasets as well as on the SincNet and baseline setups can found in the appendix.
4.1 Speaker Recognition
Table 1 reports the Classiﬁcation Error Rates (CER%) achieved on a speaker-id task.
The table shows that SincNet outperforms other systems on both TIMIT (462 speakers) and Librispeech ( speakers) datasets.
The gap with a standard CNN fed by raw waveform is larger on TIMIT, conﬁrming the effectiveness of SincNet when few training data are available.
Although this gap is reduced when
LibriSpeech is used, we still observe a 4% relative improvement that is also obtained with faster convergence (1200 vs 1800 epochs).
Standard FBANKs provide results comparable to SincNet only on TIMIT, but are signiﬁcantly worse than our architecture when using Librispech.
With few training data, the network cannot discover ﬁlters that are much better than that of FBANKs, but with more data a customized ﬁlter-bank is learned and exploited to improve the performance.
Table 2 extends our validation to speaker veriﬁcation, reporting the Equal Error Rate (EER%) achieved with Librispeech.
All DNN models show promising performance, leading to an EER lower than 1% in all cases.
The table also highlights that SincNet outperforms the other models, showing a relative performance improvement of about 11% over the standard CNN model.
Note that the speaker veriﬁcation system is derived from the speaker-id neural network using the d-vector technique.
The d-vector [19, 24] is extracted from the last hidden layer of the speaker-id network.
A speaker-dependent d-vector is computed and stored for each enrollment speaker by performing an L normalization and averaging all the d-vectors of the different speech chunks.
The cosine distance between enrolment and test d-vectors is then calculated, and a threshold is then applied on it to reject or accept the speaker.
Ten utterances from impostors were randomly selected for each sentence coming from a genuine speaker.
To assess our approach on a standard open-set speaker veriﬁcation task, all the enrolment and test utterances were taken from a speaker pool different from that used for training the speaker-id DNN.
For the sake of completeness, experiments have also been conducted with standard i-vectors.
Although a detailed comparison with this technology is out of the scope of this paper, it is worth noting that our best i-vector system achieves an EER=1.1%, rather far from what achieved with DNN systems.
It is well-known in the literature that i-vectors provide competitive performance when more training
[Table] [Table]
[Table] [Table]
material is used for each speaker and when longer test sentences are employed [52, 53, 54].
Under the challenging conditions faced in this work, neural networks achieve better generalization.
4.2 Speech Recognition
Tab. 3 reports the speech recognition performance obtained by CNN and SincNet using the TIMIT and the DIRHA dataset [41].
To ensure a more accurate comparison between the architectures, ﬁve experiments varying the initialization seeds were conducted for each model and corpus.
Table 3 thus reports the average speech recognition performance.
Standard deviations, not reported here, range between 0 . 15 and 0 .
2 for all the experiments.
[Table]
For all the datasets, SincNet outperforms CNNs trained on both standard FBANK and raw waveforms.
The latter result conﬁrms the effectiveness of SincNet not only in close-talking scenarios but also in challenging noisy conditions characterized by the presence of both noise and reverberation.
As emerged in Sec.2, SincNet is able to effectively tune its ﬁlter-bank front-end to better address the characteristics of the noise.
Conclusions and Future Work
This paper proposed SincNet, a neural architecture for directly processing waveform audio.
Our model, inspired by the way ﬁltering is conducted in digital signal processing, imposes constraints on the ﬁlter shapes through efﬁcient parameterization.
SincNet has been extensively evaluated on challenging speaker and speech recognition tasks, consistently showing some performance beneﬁts.
Beyond performance improvements, SincNet also signiﬁcantly improves convergence speed over a standard CNN, is more computationally efﬁcient due to the exploitation of ﬁlter symmetry, and it more interpretable than standard black-box models.
Analysis of the SincNet ﬁlters, in fact, revealed that the learned ﬁlter-bank is tuned to the speciﬁc task addressed by the neural network.
In future work, we would like to evaluate SincNet on other popular speaker recognition tasks, such as VoxCeleb.
Inspired by the promising results obtained in this paper, in the future we will explore the use of
SincNet for supervised and unsupervised speaker/environmental adaptation.
Moreover, although this study targeted speaker and speech recognition only, we believe that the proposed approach deﬁnes a general paradigm to process time-series and can be applied in numerous other ﬁelds.
Our future effort will be thus devoted to extending to other tasks, such as emotion recognition, speech separation, and music processing.
Acknowledgement
This research was enabled in part by support provided by Calcul Québec and Compute Canada.
Appendix Corpora
To provide experimental evidence on datasets characterized by different numbers of speakers, this paper considers the TIMIT (462 spks, train chunk) [39] and Librispeech (2484 spks) [40] corpora.
For speaker veriﬁcation experiments, non-speech intervals at the beginning and end of each sentence were removed.
Moreover, the Librispeech sentences with internal silences lasting more than 125 ms were split into multiple chunks.
To address text-independent speaker recognition, the calibration sentences of TIMIT (i.e., the utterances with the same text for all speakers) have been removed.
For the latter dataset, ﬁve sentences for each speaker were used for training, while the remaining three were used for test.
For the Librispeech corpus, the training and test material have been randomly selected to exploit 12-15 seconds of training material for each speaker and test sentences lasting 2seconds.
To evaluate the performance in a challenging distant-talking scenario, speech recognition experiments have also considered the DIRHA dataset [41].
This corpus, similarly to the other DIRHA corpora [55, 56], has been developed in the context of the DIRHA project [57] and is based on WSJ sentences [58] recorded in a domestic environment.
Training is based on contaminating WSJ-5k utterances with realistic impulse responses [59, 60], while the test phase test phase consists of 409 WSJ sentences recorded by native speakers in a domestic environment (the average SNR is 10 dB).
SincNet Setup
The waveform of each speech sentence was split into chunks of 200 ms (with 10 ms overlap), which were fed into the SincNet architecture.
The ﬁrst layer performs sinc-based convolutions as described in Sec. 2, using 80 ﬁlters of length L = 251 samples.
The architecture then employs two standard convolutional layers, both using 60 ﬁlters of length 5.
Layer normalization [61] was used for both the input samples and for all convolutional layers (including the SincNet input layer).
Next, three fully-connected layers composed of 2048 neurons and normalized with batch normalization [62, 63] were applied.
All hidden layers use leaky-ReLU [64] non-linearities.
The parameters of the sinc-layer were initialized using mel-scale cutoff frequencies, while the rest of the network was initialized with the well-known “Glorot" initialization scheme [65].
Frame-level speaker and phoneme classiﬁcations were obtained by applying a softmax classiﬁer, providing a set of posterior probabilities over the targets.
For speaker-id, a sentence-level classiﬁcation was simply derived by averaging the frame predictions and voting for the speaker which maximizes the average posterior.
Training used the RMSprop optimizer, with a learning rate lr = 0 .
001 , α = 0 .
95 , (cid:15) =
10 − 7 , and minibatches of size 128.
All the hyper-parameters of the architecture were tuned on TIMIT, then inherited for Librispeech as well.
The speaker veriﬁcation system was derived from the speaker-id neural network using the d-vector approach [19, 24], which relies on the output of the last hidden layer and computes the cosine distance between test and the claimed speaker d-vectors.
Ten utterances from impostors were randomly selected for each sentence coming from a genuine speaker.
Note that to assess our approach on a standard open-set speaker-id task, all the impostors were taken from a speaker pool different from that used for training the speaker-id DNN.
Baseline Setups
We compared SincNet with several alternative systems.
First, we considered a standard CNN fed by the raw waveform.
This network is based on the same architecture as SincNet, but replacing the sinc-based convolution with a standard one.
A comparison with popular hand-crafted features was also performed.
To this end, we computed 39 MFCCs (13 static+ ∆ + ∆∆ ) and 40 FBANKs using the Kaldi toolkit [66].
These features, computed every 25 ms withms overlap, were gathered to form a context window of approximately 200 ms (i.e., a context similar to that of the considered waveform-based neural network).
A CNN was used for FBANK features, while a Multi-Layer
Perceptron (MLP) was used for MFCCs.
Note that CNNs exploit local correlation across features and cannot be effectively used with uncorrelated MFCC features.
Layer normalization was used for the FBANK network, while batch normalization was employed for the MFCC one.
The hyper-parameters of these networks were also tuned using the aforementioned approach.
For speaker veriﬁcation experiments, we also considered an i-vector baseline.
The i-vector system was imple- mented with the SIDEKIT toolkit [67].
The GMM-UBM model, the Total Variability (TV) matrix, and
the
Probabilistic Linear Discriminant Analysis (PLDA) were trained on the Librispeech data (avoiding test and enrollment sentences).
GMM-UBM was composed of 2048 Gaussians, and the rank of the TV and PLDA eigenvoice matrix was 400.
The enrollment and test phase is conducted on Librispeech using the same set of speech segments used for DNN experiments.