Original: LEAF: A L EARNABLE F RONTEND FOR A UDIO C LASSIFICATION
Revised: LEAF: A L EARNABLE F RONTEND FOR A UDIO C LASSIFICATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Mel-ﬁlterbanks are ﬁxed, engineered audio features which emulate human perception and have been used through the history of audio understanding up to today.
Revised: Mel-ﬁlterbanks are ﬁxed, engineered audio features which emulate human perception and have been used through the history of audio understanding up to today.

Original: However, their undeniable qualities are counterbalanced by the fundamental limitations of handmade representations.
Revised: However, their undeniable qualities are counterbalanced by the fundamental limitations of handmade representations.

Original: In this work we show that we can train a single learnable frontend that outperforms mel-ﬁlterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classiﬁcation.
Revised: In this work we show that we can train a single learnable frontend that outperforms mel-ﬁlterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classiﬁcation.

Original: To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-ﬁlterbanks.
Revised: To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-ﬁlterbanks.

Original: Our system learns all operations of audio features extraction, from ﬁltering to pooling, compression and normalization, and can be integrated into any neural network at a negligible parameter cost.
Revised: Our system learns all operations of audio features extraction, from ﬁltering to pooling, compression and normalization, and can be integrated into any neural network at a negligible parameter cost.

Original: We perform multi-task training on eight diverse audio classiﬁcation tasks, and show consistent improvements of our model over mel-ﬁlterbanks and previous learnable alternatives.
Revised: We perform multi-task training on eight diverse audio classiﬁcation tasks, and show consistent improvements of our model over mel-ﬁlterbanks and previous learnable alternatives.

Original: Moreover, our system outperforms the current state-of-the-art learnable frontend on Audioset, with orders of magnitude fewer parameters.
Revised: Moreover, our system outperforms the current state-of-the-art learnable frontend on Audioset, with orders of magnitude fewer parameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning representations by backpropagation in deep neural networks has become the standard in audio understanding, ranging from automatic speech recognition (ASR) (Hinton et al., 2012;
Revised: Learning representations by backpropagation in deep neural networks has become the standard in audio understanding, ranging from automatic speech recognition (ASR) (Hinton et al., 2012;

Original: Senior et al., 2015) to music information retrieval (Arcas et al., 2017), as well as animal vocalizations (Lostanlen et al., 2018) and audio events (Hershey et al., 2017; Kong et al., 2019).
Revised: Senior et al., 2015) to music information retrieval (Arcas et al., 2017), as well as animal vocalizations (Lostanlen et al., 2018) and audio events (Hershey et al., 2017; Kong et al., 2019).

Original: Still, a striking constant along the history of audio classiﬁcation is the mel-ﬁlterbanks, a ﬁxed, hand-engineered representation of sound.
Revised: Still, a striking constant along the history of audio classiﬁcation is the mel-ﬁlterbanks, a ﬁxed, hand-engineered representation of sound.

Original: Mel-ﬁlterbanks ﬁrst compute a spectrogram, using the squared modulus of the short-term Fourier transform (STFT).
Revised: Mel-ﬁlterbanks ﬁrst compute a spectrogram, using the squared modulus of the short-term Fourier transform (STFT).

Original: Then, the spectrogram is passed through a bank of triangular bandpass ﬁlters, spaced on a logarithmic scale (the mel-scale) to replicate the non-linear human perception of pitch (Stevens & Volkmann, 1940).
Revised: Then, the spectrogram is passed through a bank of triangular bandpass ﬁlters, spaced on a logarithmic scale (the mel-scale) to replicate the non-linear human perception of pitch (Stevens & Volkmann, 1940).

Original: Eventually, the resulting coefﬁcients are passed through a logarithm compression, to replicate our non-linear sensitivity to loudness (Fechner et al., 1966).
Revised: Eventually, the resulting coefﬁcients are passed through a logarithm compression, to replicate our non-linear sensitivity to loudness (Fechner et al., 1966).

Original: This approach of drawing inspiration from the human auditory system to design features for machine learning has been historically successful (Davis & Mermelstein, 1980; Mogran et al., 2004).
Revised: This approach of drawing inspiration from the human auditory system to design features for machine learning has been historically successful (Davis & Mermelstein, 1980; Mogran et al., 2004).

Original: Moreover, decades after the design of mel-ﬁlterbanks, And´en & Mallat (2014) showed that they coincidentally exhibit desirable mathematical properties for representation learning, in particular shift-invariance and stability to small deformations.
Revised: Moreover, decades after the design of mel-ﬁlterbanks, And´en & Mallat (2014) showed that they coincidentally exhibit desirable mathematical properties for representation learning, in particular shift-invariance and stability to small deformations.

Original: Hence, both from an auditory and a machine learning perspective, mel-ﬁlterbanks represent strong audio features.
Revised: Hence, both from an auditory and a machine learning perspective, mel-ﬁlterbanks represent strong audio features.

Original: However, the design of mel-ﬁlterbanks is also ﬂawed by biases.
Revised: However, the design of mel-ﬁlterbanks is also ﬂawed by biases.

Original: First, not only has the mel-scale been revised multiple times (O’Shaughnessy, 1987; Umesh et al., 1999), but also the auditory experiments that led their original design could not be replicated afterwards (Greenwood, 1997).
Revised: First, not only has the mel-scale been revised multiple times (O’Shaughnessy, 1987; Umesh et al., 1999), but also the auditory experiments that led their original design could not be replicated afterwards (Greenwood, 1997).

Original: Similarly, better alternatives to log-compression have been proposed, like cubic root for speech enhancement (Lyons & Paliwal, 2008) or 10th root for ASR (Schluter et al., 2007).
Revised: Similarly, better alternatives to log-compression have been proposed, like cubic root for speech enhancement (Lyons & Paliwal, 2008) or 10th root for ASR (Schluter et al., 2007).

Original: Moreover, even though matching human perception provides good inductive biases for some application domains, e.g., ASR or music understanding, these biases may also be detrimental, e.g. for tasks that require ﬁne-grained resolution at high frequencies.
Revised: Moreover, even though matching human perception provides good inductive biases for some application domains, e.g., ASR or music understanding, these biases may also be detrimental, e.g. for tasks that require ﬁne-grained resolution at high frequencies.

Original: Finally, the recent history of other ﬁelds like computer vision, in which the rise of deep learning methods has allowed learning representations from raw pixels rather than from engineered features (Krizhevsky et al., 2012), inspired us to take the same path.
Revised: Finally, the recent history of other ﬁelds like computer vision, in which the rise of deep learning methods has allowed learning representations from raw pixels rather than from engineered features (Krizhevsky et al., 2012), inspired us to take the same path.

Original: [Figure]
Revised: [Figure]

Original: These observations motivated replacing mel-ﬁlterbanks with learnable neural layers, ranging from standard convolutional layers (Palaz et al., 2015) to dilated convolutions (Schneider et al., 2019), as well as structured ﬁlters exploiting the characteristics of known ﬁlter families, such as Gammatone (Sainath et al., 2015),
Revised: These observations motivated replacing mel-ﬁlterbanks with learnable neural layers, ranging from standard convolutional layers (Palaz et al., 2015) to dilated convolutions (Schneider et al., 2019), as well as structured ﬁlters exploiting the characteristics of known ﬁlter families, such as Gammatone (Sainath et al., 2015),

Original: Gabor (Zeghidour et al., 2018a;
Revised: Gabor (Zeghidour et al., 2018a;

Original: No ´ e et al., 2020), Sinc (Ravanelli &
Revised: No ´ e et al., 2020), Sinc (Ravanelli &

Original: Bengio, 2018; Pariente et al., 2020) or Spline (Balestriero et al., 2018) ﬁlters.
Revised: Bengio, 2018; Pariente et al., 2020) or Spline (Balestriero et al., 2018) ﬁlters.

Original: While tasks such as speech separation have already successfully adopted learnable frontends (Luo & Mesgarani, 2019;
Revised: While tasks such as speech separation have already successfully adopted learnable frontends (Luo & Mesgarani, 2019;

Original: Luo et al., 2019), we observe that most state-of-the art approaches for audio classiﬁcation (Kong et al., 2019), ASR (Synnaeve et al., 2019) and speaker recognition (Villalba et al., 2020) still employ mel-ﬁlterbanks as input features, regardless of the backbone architecture.
Revised: Luo et al., 2019), we observe that most state-of-the art approaches for audio classiﬁcation (Kong et al., 2019), ASR (Synnaeve et al., 2019) and speaker recognition (Villalba et al., 2020) still employ mel-ﬁlterbanks as input features, regardless of the backbone architecture.

Original: In this work, we argue that a credible alternative to mel-ﬁlterbanks for classiﬁcation should be evaluated across many tasks, and propose the ﬁrst extensive study of learnable frontends for audio over a wide and diverse range of audio signals, including speech, music, audio events, and animal sounds.
Revised: In this work, we argue that a credible alternative to mel-ﬁlterbanks for classiﬁcation should be evaluated across many tasks, and propose the ﬁrst extensive study of learnable frontends for audio over a wide and diverse range of audio signals, including speech, music, audio events, and animal sounds.

Original: By breaking down mel-ﬁlterbanks into three components (ﬁltering, pooling, compression/normalization), we propose LEAF, a novel frontend that is fully learnable in all its operations, while being controlled by just a few hundred parameters.
Revised: By breaking down mel-ﬁlterbanks into three components (ﬁltering, pooling, compression/normalization), we propose LEAF, a novel frontend that is fully learnable in all its operations, while being controlled by just a few hundred parameters.

Original: In a multi-task setting over 8 datasets, we show that we can learn a single set of parameters that outperforms mel-ﬁlterbanks, as well as previously proposed learnable alternatives.
Revised: In a multi-task setting over 8 datasets, we show that we can learn a single set of parameters that outperforms mel-ﬁlterbanks, as well as previously proposed learnable alternatives.

Original: Moreover, these ﬁndings are replicated when training a different model for each individual task.
Revised: Moreover, these ﬁndings are replicated when training a different model for each individual task.

Original: We also conﬁrm these results on a challenging, large-scale benchmark: classiﬁcation on Audioset (Gemmeke et al., 2017).
Revised: We also conﬁrm these results on a challenging, large-scale benchmark: classiﬁcation on Audioset (Gemmeke et al., 2017).

Original: In addition, we show that the general inductive bias of our frontend (i.e., learning bandpass ﬁlters, lowpass ﬁltering before downsampling, learning a per-channel compression) is general enough to beneﬁt other systems, and propose a new, improved version of SincNet (Ravanelli & Bengio, 2018).
Revised: In addition, we show that the general inductive bias of our frontend (i.e., learning bandpass ﬁlters, lowpass ﬁltering before downsampling, learning a per-channel compression) is general enough to beneﬁt other systems, and propose a new, improved version of SincNet (Ravanelli & Bengio, 2018).

Original: To foster application to new tasks, we will release the source code of all our models and baselines, as well as pre-trained frontends.
Revised: Our code is publicly available 1 .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In the last decade, several works addressed the problem of learning the audio frontend, as an alternative to mel-ﬁlterbanks.
Revised: In the last decade, several works addressed the problem of learning the audio frontend, as an alternative to mel-ﬁlterbanks.

Original: The ﬁrst notable contributions in this ﬁeld emerged for ASR, with Jaitly & Hinton (2011) pretraining Restricted Boltzmann Machines from the waveform, and Palaz et al.
Revised: The ﬁrst notable contributions in this ﬁeld emerged for ASR, with Jaitly & Hinton (2011) pretraining Restricted Boltzmann Machines from the waveform, and Palaz et al.

Original: (2013) training a hybrid DNN-HMM model, replacing mel-ﬁlterbanks by several layers of convolution.
Revised: (2013) training a hybrid DNN-HMM model, replacing mel-ﬁlterbanks by several layers of convolution.

Original: However, these alternatives, as well as others proposed more recently (Tjandra et al., 2017;
Revised: However, these alternatives, as well as others proposed more recently (Tjandra et al., 2017;

Original: Schneider et al., 2019), are composed of many layers, which makes a fair comparison with melﬁlterbanks difﬁcult.
Revised: Schneider et al., 2019), are composed of many layers, which makes a fair comparison with melﬁlterbanks difﬁcult.

Original: In the following section, we focus on frontends that provide a lightweight, drop-in replacement to mel-ﬁlterbanks, with comparable capacity.
Revised: In the following section, we focus on frontends that provide a lightweight, drop-in replacement to mel-ﬁlterbanks, with comparable capacity.

Original: 2. L EARNING FILTERS FROM WAVEFORMS
Revised: 2. L EARNING FILTERS FROM WAVEFORMS

Original: A ﬁrst attempt at learning the ﬁlters of mel-ﬁlterbanks was proposed by Sainath et al.
Revised: A ﬁrst attempt at learning the ﬁlters of mel-ﬁlterbanks was proposed by Sainath et al.

Original: (2013), where a ﬁlterbank is initialized using the mel-scale and then learned together with the rest of the network, taking a spectrogram as input.
Revised: (2013), where a ﬁlterbank is initialized using the mel-scale and then learned together with the rest of the network, taking a spectrogram as input.

Original: Instead, Sainath et al.
Revised: Instead, Sainath et al.

Original: and
Revised: and

Original: Hoshen et al.
Revised: Hoshen et al.

Original: (2015) later proposed to learn convolutional ﬁlters directly from raw waveforms, initialized with Gammatone ﬁl- ters (Schluter et al., 2007).
Revised: (2015) later proposed to learn convolutional ﬁlters directly from raw waveforms, initialized with Gammatone ﬁlters (Schluter et al., 2007).

Original: In the same spirit, Zeghidour et al.
Revised: In the same spirit, Zeghidour et al.

Original: (2018a) used the scattering transform approximation of mel-ﬁlterbanks (And´en & Mallat, 2014) to propose the time-domain ﬁlterbanks, a learnable frontend that approximates mel-ﬁlterbanks at initialization and can then be learned without constraints (see Figure 1).
Revised: (2018a) used the scattering transform approximation of mel-ﬁlterbanks (And´en & Mallat, 2014) to propose the time-domain ﬁlterbanks, a learnable frontend that approximates mel-ﬁlterbanks at initialization and can then be learned without constraints (see Figure 1).

Original: More recently, the SincNet (Ravanelli & Bengio, 2018) model was proposed, which computes a convolution with sine cardinal ﬁlters, a non-linearity and a max-pooling operator (see Figure 1), as well as a variant using Gabor ﬁlters (No´e et al., 2020).
Revised: More recently, the SincNet (Ravanelli & Bengio, 2018) model was proposed, which computes a convolution with sine cardinal ﬁlters, a non-linearity and a max-pooling operator (see Figure 1), as well as a variant using Gabor ﬁlters (No´e et al., 2020).

Original: We take inspiration from these works to design a new learnable ﬁltering layer.
Revised: We take inspiration from these works to design a new learnable ﬁltering layer.

Original: As detailed in Section 3.1.2, we parametrize a complex-valued ﬁltering layer with Gabor ﬁlters.
Revised: As detailed in Section 3.1.2, we parametrize a complex-valued ﬁltering layer with Gabor ﬁlters.

Original: Gabor ﬁlters are optimally localized in time and frequency (Gabor, 1946), unlike Sinc ﬁlters that require using a window function (Ravanelli & Bengio, 2018).
Revised: Gabor ﬁlters are optimally localized in time and frequency (Gabor, 1946), unlike Sinc ﬁlters that require using a window function (Ravanelli & Bengio, 2018).

Original: Moreover, unlike No´e et al.
Revised: Moreover, unlike No´e et al.

Original: (2020), who use complex-valued layers in the rest of the network, we describe in Section 3.1.2 how using a squared modulus not only brings back the signal to the real-valued domain (leading to compatibility with standard architectures), but also performs shift-invariant Hilbert envelope extraction.
Revised: (2020), who use complex-valued layers in the rest of the network, we describe in Section 3.1.2 how using a squared modulus not only brings back the signal to the real-valued domain (leading to compatibility with standard architectures), but also performs shift-invariant Hilbert envelope extraction.

Original: Zeghidour et al.
Revised: Zeghidour et al.

Original: (2018a) also apply a squared modulus non-linearity, however as described in Section 3.1.1, training unconstrained ﬁlters can lead to overﬁtting and stability issues, which we solve with our approach.
Revised: (2018a) also apply a squared modulus non-linearity, however as described in Section 3.1.1, training unconstrained ﬁlters can lead to overﬁtting and stability issues, which we solve with our approach.

Original: 2. L EARNING THE COMPRESSION AND THE NORMALIZATION
Revised: 2. L EARNING THE COMPRESSION AND THE NORMALIZATION

Original: The problem of learning a compression and/or normalization function has received less attention in the past literature.
Revised: The problem of learning a compression and/or normalization function has received less attention in the past literature.

Original: A notable contribution is the Per-Channel Energy Normalization (PCEN) (Wang et al., 2017; Lostanlen et al., 2019), which was originally proposed for keyword spotting, outperforming log-compression.
Revised: A notable contribution is the Per-Channel Energy Normalization (PCEN) (Wang et al., 2017; Lostanlen et al., 2019), which was originally proposed for keyword spotting, outperforming log-compression.

Original: Later, Battenberg et al.
Revised: Later, Battenberg et al.

Original: (2017) and Lostanlen et al.
Revised: (2017) and Lostanlen et al.

Original: (2018) conﬁrmed the advantages of PCEN, respectively for large scale ASR and animal bioacoustics.
Revised: (2018) conﬁrmed the advantages of PCEN, respectively for large scale ASR and animal bioacoustics.

Original: However, these previous works learn a compression on top of ﬁxed mel-ﬁlterbanks.
Revised: However, these previous works learn a compression on top of ﬁxed mel-ﬁlterbanks.

Original: Instead, in this work we propose a new version of PCEN and show for the ﬁrst time that combining learnable ﬁlters, learnable pooling, and learnable compression and normalization outperforms all other approaches.
Revised: Instead, in this work we propose a new version of PCEN and show for the ﬁrst time that combining learnable ﬁlters, learnable pooling, and learnable compression and normalization outperforms all other approaches.

Original:  M ODEL
Revised:  M ODEL

Original: Let x ∈ R T denote a one-dimensional waveform of T samples, available at the sampling frequency
Revised: Let x ∈ R T denote a one-dimensional waveform of T samples, available at the sampling frequency

Original: F s [ Hz ].
Revised: F s [ Hz ].

Original: We decompose the frontend into a sequence of three components: i) ﬁltering, which passes x through a bank of bandpass ﬁlters followed by a non-linearity, operating at the original sampling rate F s ; ii) pooling, which decimates the signal to reduce its temporal resolution; iii) compression/normalization, which applies a non-linearity to reduce the dynamic range.
Revised: We decompose the frontend into a sequence of three components: i) ﬁltering, which passes x through a bank of bandpass ﬁlters followed by a non-linearity, operating at the original sampling rate F s ; ii) pooling, which decimates the signal to reduce its temporal resolution; iii) compression/normalization, which applies a non-linearity to reduce the dynamic range.

Original: Overall, the frontend can be represented as a function F ψ : R T → R M × N , which maps the input waveform to a 2-dimensional feature space, where M denotes the number of temporal frames (typically M < T ),
Revised: Overall, the frontend can be represented as a function F ψ : R T → R M × N , which maps the input waveform to a 2-dimensional feature space, where M denotes the number of temporal frames (typically M < T ),

Original: N the number of feature channels (which might correspond to frequency bins) and ψ the frontend parameters.
Revised: N the number of feature channels (which might correspond to frequency bins) and ψ the frontend parameters.

Original: The features computed by the frontend are then fed to a model g θ ( · ) parametrized by θ .
Revised: The features computed by the frontend are then fed to a model g θ ( · ) parametrized by θ .

Original: The frontend and the model parameters are estimated by solving a supervised classiﬁcation problem:
Revised: The frontend and the model parameters are estimated by solving a supervised classiﬁcation problem:

Original: [Equation]
Revised: [Equation]

Original:  where ( x, y ) are samples in a labelled dataset D and L is a loss function.
Revised: (1) where ( x, y ) are samples in a labelled dataset D and L is a loss function.

Original: Our goal is to learn the frontend parameters ψ end-to-end with the model parameters θ .
Revised: Our goal is to learn the frontend parameters ψ end-to-end with the model parameters θ .

Original: To achieve this, it is necessary to make all the frontend components learnable, so that we can solve the optimization problem in equation 1 with gradient descent.
Revised: To achieve this, it is necessary to make all the frontend components learnable, so that we can solve the optimization problem in equation 1 with gradient descent.

Original: In the following we detail the design choices of each component.
Revised: In the following we detail the design choices of each component.

Original: 3. F ILTERING
Revised: 3. F ILTERING

Original: The ﬁrst block of the learnable frontend takes x as input, and computes a convolution with a bank of complex-valued ﬁlters ( ϕ n ) n =1 ..N , followed by a squared modulus operator, which brings back its output to the real-valued domain.
Revised: The ﬁrst block of the learnable frontend takes x as input, and computes a convolution with a bank of complex-valued ﬁlters ( ϕ n ) n =1 ..N , followed by a squared modulus operator, which brings back its output to the real-valued domain.

Original: This convolution step has a stride of 1, therefore keeping the input temporal resolution, and outputs the following time-frequency representation:
Revised: This convolution step has a stride of 1, therefore keeping the input temporal resolution, and outputs the following time-frequency representation:

Original: [Equation]
Revised: [Equation]

Original: where ϕ n ∈ C W is a complex-valued 1-D ﬁlter of length W .
Revised: where ϕ n ∈ C W is a complex-valued 1-D ﬁlter of length W .

Original: It is possible to compute equationwithout explicitly manipulating complex numbers.
Revised: It is possible to compute equationwithout explicitly manipulating complex numbers.

Original: As proposed by Zeghidour et al.
Revised: As proposed by Zeghidour et al.

Original: (2018a),
Revised: (2018a),

Original: to
Revised: to

Original: [Figure]
Revised: [Figure]

Original: produce the squared-modulus of a complex-valued convolution with N ﬁlters, we compute instead the convolution with 2 N real-valued ﬁlters ˜ ϕ n , n = 1 , . .
Revised: produce the squared-modulus of a complex-valued convolution with N ﬁlters, we compute instead the convolution with 2 N real-valued ﬁlters ˜ ϕ n , n = 1 , . .

Original: , 2 N , and perform squared (cid:96) 2 -pooling with size 2 and stride 2 along the channel axis to obtain the squared modulus, using adjacent ﬁlters as real and imaginary part of ϕ n .
Revised: , 2 N , and perform squared (cid:96) 2 -pooling with size 2 and stride 2 along the channel axis to obtain the squared modulus, using adjacent ﬁlters as real and imaginary part of ϕ n .

Original: Formally:
Revised: Formally:

Original: [Equation]
Revised: [Equation]

Original: We explore two different parametrizations for ϕ n .
Revised: We explore two different parametrizations for ϕ n .

Original: One relies on standard fully parametrized convolutional ﬁlters while the other makes use of learnable Gabor ﬁlters.
Revised: One relies on standard fully parametrized convolutional ﬁlters while the other makes use of learnable Gabor ﬁlters.

Original: 3.1. N ORMALIZED 1D-C ONVOLUTION
Revised: 3.1. N ORMALIZED 1D-C ONVOLUTION

Original: Inspired by Zeghidour et al.
Revised: Inspired by Zeghidour et al.

Original: (2018a)
Revised: (2018a)

Original: , the ﬁrst version of our ﬁltering component is a standard 1D convolution, initialized with a bank of Gabor ﬁlters, which approximates the computation of a melﬁlterbank.
Revised: , the ﬁrst version of our ﬁltering component is a standard 1D convolution, initialized with a bank of Gabor ﬁlters, which approximates the computation of a melﬁlterbank.

Original: Thus, at initialization, the output of the frontend is identical to that of a mel-ﬁlterbank, but during training the ﬁlters can be learned by backpropagation.
Revised: Thus, at initialization, the output of the frontend is identical to that of a mel-ﬁlterbank, but during training the ﬁlters can be learned by backpropagation.

Original: In our early experiments, we observed several limitations of this approach.
Revised: In our early experiments, we observed several limitations of this approach.

Original: First, due to the unconstrained optimization, these ﬁlters not only learn to select frequency bands of interest, but also learn a scaling factor.
Revised: First, due to the unconstrained optimization, these ﬁlters not only learn to select frequency bands of interest, but also learn a scaling factor.

Original: This might lead to instability during training, because the same form of scaling can be also computed by the ﬁlter-wise compression component, which is applied at a later stage in the frontend.
Revised: This might lead to instability during training, because the same form of scaling can be also computed by the ﬁlter-wise compression component, which is applied at a later stage in the frontend.

Original: We address this issue by applying (cid:96) 2 -normalization to the ﬁlter coefﬁcients before computing the convolution.
Revised: We address this issue by applying (cid:96) 2 -normalization to the ﬁlter coefﬁcients before computing the convolution.

Original: Second, the unconstrained parametrization increases the number of degrees of freedom, making training prone to overﬁtting.
Revised: Second, the unconstrained parametrization increases the number of degrees of freedom, making training prone to overﬁtting.

Original: The ﬁrst panel in Figure 2 shows the frequency response of a bank of N = 40 ﬁlters, each parametrized with W = 401 coefﬁcients.
Revised: The ﬁrst panel in Figure 2 shows the frequency response of a bank of N = 40 ﬁlters, each parametrized with W = 401 coefﬁcients.

Original: We observe that, at convergence, they are widely spread across the frequency axis, and include negative frequencies.
Revised: We observe that, at convergence, they are widely spread across the frequency axis, and include negative frequencies.

Original: Moreover, the ﬁlters are very spiky rather than smooth.
Revised: Moreover, the ﬁlters are very spiky rather than smooth.

Original: To alleviate these issues, in the next section we introduce a parametrization based on a bank of Gabor ﬁlters, which reduces the number of parameters to learn, while at the same time enforcing during training a stable and interpretable representation.
Revised: To alleviate these issues, in the next section we introduce a parametrization based on a bank of Gabor ﬁlters, which reduces the number of parameters to learn, while at the same time enforcing during training a stable and interpretable representation.

Original: 3.1. G ABOR 1D-C ONVOLUTION
Revised: 3.1. G ABOR 1D-C ONVOLUTION

Original: Gabor ﬁlters are produced by modulating a Gaussian kernel with a sinusoidal signal.
Revised: Gabor ﬁlters are produced by modulating a Gaussian kernel with a sinusoidal signal.

Original: These ﬁlters provide several desirable properties.
Revised: These ﬁlters provide several desirable properties.

Original: First, they have an optimal trade-off between localization in time and frequency (Gabor, 1946), which makes them a suitable choice for a convolutional network with ﬁnite-sized ﬁlters.
Revised: First, they have an optimal trade-off between localization in time and frequency (Gabor, 1946), which makes them a suitable choice for a convolutional network with ﬁnite-sized ﬁlters.

Original: This is in contrast to Sinc ﬁlters (Ravanelli & Bengio, 2018), which require using a window function to smooth abrupt variations on each side of the ﬁlter.
Revised: This is in contrast to Sinc ﬁlters (Ravanelli & Bengio, 2018), which require using a window function to smooth abrupt variations on each side of the ﬁlter.

Original: Second, the time and frequency response of Gabor ﬁlters have the same functional form, thus leading to interpretable bandpass ﬁlters, unlike the unconstrained ﬁlters described in the previous section.
Revised: Second, the time and frequency response of Gabor ﬁlters have the same functional form, thus leading to interpretable bandpass ﬁlters, unlike the unconstrained ﬁlters described in the previous section.

Original: Finally, Gabor ﬁlters are analytic (i.e., their frequency response is  zero for negative frequencies) and when combined with the squared modulus the resulting ﬁlterbank can be interpreted as a set of subband Hilbert envelopes, which are invariant to small shifts.
Revised: Finally, Gabor ﬁlters are quasi-analytic (i.e., their frequency response is almost zero for negative frequencies) and when combined with the squared modulus the resulting ﬁlterbank can be interpreted as a set of subband Hilbert envelopes, which are invariant to small shifts.

Original: Due to this desirable property, they have been previously used as (ﬁxed) features for speech and speaker recognition (Falk & Chan, 2009; Thomas et al., 2008).
Revised: Due to this desirable property, they have been previously used as (ﬁxed) features for speech and speaker recognition (Falk & Chan, 2009; Thomas et al., 2008).

Original: Formally, Gabor ﬁlters are parametrized by their center frequencies ( η n ) n =1 ..N and inverse bandwidths ( σ n ) n =1 ..N as follows:
Revised: Formally, Gabor ﬁlters are parametrized by their center frequencies .N and inverse bandwidths ( σ n ) n =1 ..N as follows:

Original: [Equation]
Revised: [Equation]

Original: The frequency response of ϕ n is a Gaussian centered at frequency η n and of bandwidth 1 /σ n , both expressed in normalized frequency units in [ − 1 / 2 , +1 / 2] .
Revised: The frequency response of ϕ n is a Gaussian centered at frequency η n and of bandwidth 1 /σ n , both expressed in normalized frequency units in [ − 1 / 2 , +1 / 2] .

Original: Therefore, learning these parameters allows learning a bank of smooth, analytic bandpass ﬁlters, with controllable center frequency and bandwidth.
Revised: Therefore, learning these parameters allows learning a bank of smooth, quasi-analytic bandpass ﬁlters, with controllable center frequency and bandwidth.

Original: In practice, to compute the output of the ﬁltering component, we obtain the impulse response of the Gabor ﬁlters ϕ n ( t ) over the range t = − W/ 2 , . . .
Revised: In practice, to compute the output of the ﬁltering component, we obtain the impulse response of the Gabor ﬁlters ϕ n ( t ) over the range t = − W/ 2 , . . .

Original: , W/ 2 , and convolve these impulse responses with the input waveform.
Revised: , W/ 2 , and convolve these impulse responses with the input waveform.

Original: To ensure stability during training, we clip the center frequencies ( η n ) n =1 ..N to be in [0 , 1 / 2] , so that they lie in the positive part of the frequency range.
Revised: To ensure stability during training, we clip the center frequencies ( η n ) n =1 ..N to be in [0 , 1 / 2] , so that they lie in the positive part of the frequency range.

Original: We also constrain the bandwidths ( σ n ) n =1 ..N in the range [4 √ 2 log 2 , 2 W √ 2 log 2] , such that the full-width at half-maximum of the frequency response is within 1 /W and 1 / 2 .
Revised: We also constrain the bandwidths ( σ n ) n =1 ..N in the range [4 √ 2 log 2 , 2 W √ 2 log 2] , such that the full-width at half-maximum of the frequency response is within 1 /W and 1 / 2 .

Original: Gabor ﬁlters have signiﬁcantly fewer parameters than the normalized 1D-convolutions described in
Revised: Gabor ﬁlters have signiﬁcantly fewer parameters than the normalized 1D-convolutions described in

Original: Section 3.1.1.
Revised: Section 3.1.1.

Original: N ﬁlters of length W are described by 2 N parameters, N for the center frequencies and N for the bandwidths, against W · N for a standard 1D-convolution.
Revised: N ﬁlters of length W are described by 2 N parameters, N for the center frequencies and N for the bandwidths, against W · N for a standard 1D-convolution.

Original: In particular, when using a window length of 25 ms , operating at a sampling rate of 16 kHz , then W = 401 samples, and
Revised: In particular, when using a window length of 25 ms , operating at a sampling rate of 16 kHz , then W = 401 samples, and

Original: Gabor-based ﬁltering accounts for 200 times fewer parameters than their unconstrained alternatives.
Revised: Gabor-based ﬁltering accounts for 200 times fewer parameters than their unconstrained alternatives.

Original: 3.1. T IME -F REQUENCY ANALYSIS AND LEARNABLE FILTERS
Revised: 3.1. T IME -F REQUENCY ANALYSIS AND LEARNABLE FILTERS

Original: A spectrogram, in linear or mel scale, provides an ordered time-frequency representation: adjacent frames represent consecutive time-steps, while frequencies monotonically increase along the feature axis.
Revised: A spectrogram, in linear or mel scale, provides an ordered time-frequency representation: adjacent frames represent consecutive time-steps, while frequencies monotonically increase along the feature axis.

Original: A learnable frontend that performs ﬁltering by means of convolution with a set of bandpass ﬁlters also preserves ordering along the temporal axis.
Revised: A learnable frontend that performs ﬁltering by means of convolution with a set of bandpass ﬁlters also preserves ordering along the temporal axis.

Original: However, the ordering along the feature axis is unconstrained.
Revised: However, the ordering along the feature axis is unconstrained.

Original: This can be problematic when applying subsequent operations that rely on frequency ordering.
Revised: This can be problematic when applying subsequent operations that rely on frequency ordering.

Original: These include, for example: i) operations that leverage local frequency information, e.g., two-dimensional convolutions, which compute a feature representation based on local time-frequency patches; ii) operations that leverage long-range dependencies along the frequency axis, e.g., to capture the harmonic structure of the underlying signal; and iii) augmentation methods that mask adjacent frequency bands, like SpecAugment (Park et al., 2019).
Revised: These include, for example: i) operations that leverage local frequency information, e.g., two-dimensional convolutions, which compute a feature representation based on local time-frequency patches; ii) operations that leverage long-range dependencies along the frequency axis, e.g., to capture the harmonic structure of the underlying signal; and iii) augmentation methods that mask adjacent frequency bands, like SpecAugment (Park et al., 2019).

Original: To evaluate the impact of enforcing an explicit ordering of the center frequencies of the learned ﬁlters, we compared the result of training a frontend using Gabor ﬁlters with or without explicitly enforcing ordering of the center frequencies.
Revised: To evaluate the impact of enforcing an explicit ordering of the center frequencies of the learned ﬁlters, we compared the result of training a frontend using Gabor ﬁlters with or without explicitly enforcing ordering of the center frequencies.

Original: Interestingly, we observe that even without an explicit constraint, ﬁlters that are ordered at the initialization tend to keep the same ordering throughout training, and that enforcing sorted ﬁlters has no effect on the performance.
Revised: Interestingly, we observe that even without an explicit constraint, ﬁlters that are ordered at the initialization tend to keep the same ordering throughout training, and that enforcing sorted ﬁlters has no effect on the performance.

Original: 3. L EARNABLE L OWPASS P OOLING
Revised: 3. L EARNABLE L OWPASS P OOLING

Original: The output of the ﬁltering component has the same temporal resolution as the input waveform.
Revised: The output of the ﬁltering component has the same temporal resolution as the input waveform.

Original: The second step of a learnable frontend is to downsample the output of the ﬁlterbank to a lower sampling rate, similarly to what happens in the STFT when computing mel-ﬁlterbanks.
Revised: The second step of a learnable frontend is to downsample the output of the ﬁlterbank to a lower sampling rate, similarly to what happens in the STFT when computing mel-ﬁlterbanks.

Original: Previous work relied on max-pooling (Sainath et al., 2015; Ravanelli & Bengio, 2018; No´e et al., 2020), lowpassﬁltering (Zeghidour et al., 2018a) or average pooling (Balestriero et al., 2018).
Revised: Previous work relied on max-pooling (Sainath et al., 2015; Ravanelli & Bengio, 2018; No´e et al., 2020), lowpassﬁltering (Zeghidour et al., 2018a) or average pooling (Balestriero et al., 2018).

Original: Zeghidour et al.
Revised: Zeghidour et al.

Original: (2018b) compare these methods on a speech recognition task and show a systematic improvement when using lowpass ﬁltering instead of max-pooling.
Revised: (2018b) compare these methods on a speech recognition task and show a systematic improvement when using lowpass ﬁltering instead of max-pooling.

Original: More recently, Zhang (2019) showed that in standard 2D convolutional architectures, including ResNet (He et al., 2016) and DenseNet (Huang et al., 2017), a drop-in replacement of max-pooling and average pooling layers with a (ﬁxed) lowpass ﬁlter improves the performance for image classiﬁcation.
Revised: More recently, Zhang (2019) showed that in standard 2D convolutional architectures, including ResNet (He et al., 2016) and DenseNet (Huang et al., 2017), a drop-in replacement of max-pooling and average pooling layers with a (ﬁxed) lowpass ﬁlter improves the performance for image classiﬁcation.

Original: In the proposed frontend, we extend the pooling layer of Zeghidour et al.
Revised: In the proposed frontend, we extend the pooling layer of Zeghidour et al.

Original: (2018a) in two ways.
Revised: (2018a) in two ways.

Original: First, while Zeghidour et al.
Revised: First, while Zeghidour et al.

Original: (2018a) adopt a single shared lowpass ﬁlter for all input channels, we implement lowpass ﬁltering by means of depthwise convolution, such that each input channel is associated with one lowpass ﬁlter.
Revised: (2018a) adopt a single shared lowpass ﬁlter for all input channels, we implement lowpass ﬁltering by means of depthwise convolution, such that each input channel is associated with one lowpass ﬁlter.

Original: This is useful because each channel in the learnable frontend is characterized by a different bandwidth, and a speciﬁc lowpass ﬁlter can be learned for each of them.
Revised: This is useful because each channel in the learnable frontend is characterized by a different bandwidth, and a speciﬁc lowpass ﬁlter can be learned for each of them.

Original: Second, we parametrize these lowpass ﬁlters to have a Gaussian impulse response:
Revised: Second, we parametrize these lowpass ﬁlters to have a Gaussian impulse response:

Original: [Equation]
Revised: [Equation]

Original: Note that this is a particular case of Gabor ﬁlters with center frequency equal to 0 and learnable bandwidth.
Revised: Note that this is a particular case of Gabor ﬁlters with center frequency equal to 0 and learnable bandwidth.

Original: With this choice, we can learn per-channel lowpass pooling functions while adding only
Revised: With this choice, we can learn per-channel lowpass pooling functions while adding only

Original: N parameters to the frontend model.
Revised: N parameters to the frontend model.

Original: We initialize all channels with a bandwidth of 0 . 4 , which gives a frequency response close to the Hann window used by mel-ﬁlterbanks.
Revised: We initialize all channels with a bandwidth of 0 . 4 , which gives a frequency response close to the Hann window used by mel-ﬁlterbanks.

Original: 3. L EARNING PER - CHANNEL COMPRESSION AND NORMALIZATION
Revised: 3. L EARNING PER - CHANNEL COMPRESSION AND NORMALIZATION

Original: When using a traditional frontend based on a mel-ﬁlterbank or STFT, the time-frequency features are typically passed through a logarithmic compression, to replicate the non-linear human perception of loudness (Fechner et al., 1966).
Revised: When using a traditional frontend based on a mel-ﬁlterbank or STFT, the time-frequency features are typically passed through a logarithmic compression, to replicate the non-linear human perception of loudness (Fechner et al., 1966).

Original: The ﬁrst limitation of this approach is that the same compression function is applied to all frequency bins, regardless of their content.
Revised: The ﬁrst limitation of this approach is that the same compression function is applied to all frequency bins, regardless of their content.

Original: The second limitation is the ﬁxed choice of the non-linearity used by the compression function.
Revised: The second limitation is the ﬁxed choice of the non-linearity used by the compression function.

Original: While, in most cases, a logarithmic function is used, other compression functions have been proposed and evaluated in the past, including the cubic root (Lyons & Paliwal, 2008) and the 10th root (Schluter et al., 2007).
Revised: While, in most cases, a logarithmic function is used, other compression functions have been proposed and evaluated in the past, including the cubic root (Lyons & Paliwal, 2008) and the 10th root (Schluter et al., 2007).

Original: This motivates learning the compression function as part of the model, rather than relying on a handcrafted choice.
Revised: This motivates learning the compression function as part of the model, rather than relying on a handcrafted choice.

Original: In particular, Per-Channel Energy Normalization (Wang et al., 2017) was proposed as a learnable alternative to log-compression and mean-variance normalization:
Revised: In particular, Per-Channel Energy Normalization (Wang et al., 2017) was proposed as a learnable alternative to log-compression and mean-variance normalization:

Original: [Equation]
Revised: [Equation]

Original: where t = 1 , . .
Revised: where t = 1 , . .

Original: , M denotes the time-step and n = 1 , . . .
Revised: , M denotes the time-step and n = 1 , . . .

Original: , N the channel index.
Revised: , N the channel index.

Original: In this parametrization, the time-frequency representation F is ﬁrst normalized by an exponential moving average of its past values M ( t, n ) =
Revised: In this parametrization, the time-frequency representation F is ﬁrst normalized by an exponential moving average of its past values M ( t, n ) =

Original: (1 − s ) M ( t − 1 , n )+ s F ( t, n ) , controlled by a smoothing coefﬁcient s and an exponent α n , with ε being a small constant used to avoid dividing by zero.
Revised: (1 − s ) M ( t − 1 , n )+ s F ( t, n ) , controlled by a smoothing coefﬁcient s and an exponent α n , with ε being a small constant used to avoid dividing by zero.

Original: An offset δ n is then added before applying compression with the exponent r n (typically in [0 , 1] ).
Revised: An offset δ n is then added before applying compression with the exponent r n (typically in [0 , 1] ).

Original: Wang et al.
Revised: Wang et al.

Original: (2017) train α n , δ n , and r n , while treating s as a hyperparameter, or learning a convex combination of exponential moving averages for different, ﬁxed values of s .
Revised: (2017) train α n , δ n , and r n , while treating s as a hyperparameter, or learning a convex combination of exponential moving averages for different, ﬁxed values of s .

Original: Instead, in this work we learn channel-dependent smoothing coefﬁcients s n , jointly with the rest of the parameters.
Revised: Instead, in this work we learn channel-dependent smoothing coefﬁcients s n , jointly with the rest of the parameters.

Original: We call this version sPCEN.
Revised: We call this version sPCEN.

Original: This approach was previously used by Schl¨uter & Lehner (2018) for singing voice detection, except that they did not learn the exponents α n , while we learn all these parameters jointly.
Revised: This approach was previously used by Schl ¨ uter & Lehner (2018) for singing voice detection, except that they did not learn the exponents α n , while we learn all these parameters jointly.

Original: Our ﬁnal frontend cascades a Gabor 1D-convolution, a Gaussian lowpass pooling, and sPCEN.
Revised: Our ﬁnal frontend cascades a Gabor 1D-convolution, a Gaussian lowpass pooling, and sPCEN.

Original: In the rest of the paper, we refer to this model as LEAF, for “LEarnable Audio Frontend”.
Revised: In the rest of the paper, we refer to this model as LEAF, for “LEarnable Audio Frontend”.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our frontend on three supervised learning problems: i) single-task classiﬁcation; ii) multi-task classiﬁcation and iii) multi-label classiﬁcation on Audioset.
Revised: We evaluate our frontend on three supervised learning problems: i) single-task classiﬁcation; ii) multi-task classiﬁcation and iii) multi-label classiﬁcation on Audioset.

Original: As baselines, we compare our system to log-compressed mel-ﬁlterbanks, learnable Time-Domain ﬁlterbanks (Zeghidour et al., 2018a) 1 and SincNet (Ravanelli & Bengio, 2018) 2 .
Revised: As baselines, we compare our system to log-compressed mel-ﬁlterbanks, learnable Time-Domain ﬁlterbanks (Zeghidour et al., 2018a) 2 and SincNet (Ravanelli & Bengio, 2018) 3 .

Original: We reimplemented these baselines to match their ofﬁcial implementation.

Original: In all our experiments, we keep the same common backbone network architecture, which consists of a frontend, a convolutional encoder, and one (or more) head(s).
Revised: In all our experiments, we keep the same common backbone network architecture, which consists of a frontend, a convolutional encoder, and one (or more) head(s).

Original: We adopt the lightweight version of EfﬁcientNet (Tan & Le, 2019)
Revised: We adopt the lightweight version of EfﬁcientNet (Tan & Le, 2019)

Original: (EfﬁcientNetB0, with 4 M parameters) as convolutional encoder.
Revised: (EfﬁcientNetB0, with 4 M parameters) as convolutional encoder.

Original: On AudioSet, we also experiment with a state-of-the-art CNN encoder (Kong et al., 2019), with 81 M parameters.
Revised: On AudioSet, we also experiment with a state-of-the-art CNN14 encoder (Kong et al., 2019), with 81 M parameters.

Original: A head is a single linear layer with a number of outputs equal to the number of classes.
Revised: A head is a single linear layer with a number of outputs equal to the number of classes.

Original: In the multi-task setting we use a different head for each of the target tasks, all sharing the same encoder.
Revised: In the multi-task setting we use a different head for each of the target tasks, all sharing the same encoder.

Original: The input signal sampled at F s = 16 kHz is passed through the frontend which feeds into the convolutional encoder.
Revised: The input signal sampled at F s = 16 kHz is passed through the frontend which feeds into the convolutional encoder.

Original: As baseline, we use a log-compressed mel-ﬁlterbank with 40 channels, computed over windows of 25 ms with a stride of 10 ms .
Revised: As baseline, we use a log-compressed mel-ﬁlterbank with 40 channels, computed over windows of 25 ms with a stride of 10 ms .

Original: For a fair comparison, both LEAF and the learnable baselines also have N = 40 ﬁlters, each with W = 401 coefﬁcients ( ≈ 25 ms at 16 kHz ).
Revised: For a fair comparison, both LEAF and the learnable baselines also have N = 40 ﬁlters, each with W = 401 coefﬁcients ( ≈ 25 ms at 16 kHz ).

Original: The learnable pooling is computed over 401 samples with a stride of 160 samples ( 10 ms at 16 kHz ), giving the same output dimension as mel-ﬁlterbanks.
Revised: The learnable pooling is computed over 401 samples with a stride of 160 samples ( 10 ms at 16 kHz ), giving the same output dimension as mel-ﬁlterbanks.

Original: On AudioSet we use 64 channels instead of 40 as (Kong et al., 2019) observed improvements from using 64 mel-ﬁlters.
Revised: On AudioSet we use 64 channels instead of 40 as Kong et al. (2019) observed improvements from using 64 mel-ﬁlters.

Original: To address the variable length of the input sequences, we train on randomly sampled 1 second windows.
Revised: To address the variable length of the input sequences, we train on randomly sampled 1 second windows.

Original: We train with ADAM (Kingma & Ba, 2014) and a learning rate of 10 − 4 for 1 M batches, with
Revised: We train with ADAM (Kingma & Ba, 2014) and a learning rate of 10 − 4 for 1 M batches, with

Original: [Table] [Table]
Revised: [Table] [Table]

Original: batch size 256.
Revised: batch size 256.

Original: For Audioset experiments, we train with mixup (Zhang et al., 2017) and SpecAugment (Park et al., 2019).
Revised: For Audioset experiments, we train with mixup (Zhang et al., 2017) and SpecAugment (Park et al., 2019).

Original: During evaluation, we consider the full-length sequences, splitting them into consecutive non-overlapping 1 second windows and averaging the output logits over windows.
Revised: During evaluation, we consider the full-length sequences, splitting them into consecutive non-overlapping 1 second windows and averaging the output logits over windows.

Original: 4. S INGLE - TASK AUDIO CLASSIFICATION
Revised: 4. S INGLE - TASK AUDIO CLASSIFICATION

Original: We train independent single-task supervised models on 8 distinct classiﬁcation problems: acoustic scene classiﬁcation on TUT (Heittola et al., 2018), birdsong detection (Stowell et al., 2018), emotion recognition on Crema-D
Revised: We train independent single-task supervised models on 8 distinct classiﬁcation problems: acoustic scene classiﬁcation on TUT (Heittola et al., 2018), birdsong detection (Stowell et al., 2018), emotion recognition on Crema-D

Original: (Cao et al., 2014), speaker identiﬁcation on VoxCeleb (Nagrani et al., 2017), musical instrument and pitch detection on NSynth (Engel et al., 2017), keyword spotting on Speech
Revised: (Cao et al., 2014), speaker identiﬁcation on VoxCeleb (Nagrani et al., 2017), musical instrument and pitch detection on NSynth (Engel et al., 2017), keyword spotting on Speech

Original: Commands (Warden, 2018), and language identiﬁcation on VoxForge (Revay & Teschke, 2019).
Revised: Commands (Warden, 2018), and language identiﬁcation on VoxForge (Revay & Teschke, 2019).

Original: A summary of the datasets used in our experiments is illustrated in Table A.1.
Revised: A summary of the datasets used in our experiments is illustrated in Table A.1.

Original: Table 1 reports the results for each task, with 95% conﬁdence intervals representing the uncertainty due to the limited test sample size.
Revised: Table 1 reports the results for each task, with 95% conﬁdence intervals representing the uncertainty due to the limited test sample size.

Original: No class rebalancing is applied, neither during training, nor during testing.
Revised: No class rebalancing is applied, neither during training, nor during testing.

Original: On average, we observe that LEAF outperforms all alternatives.
Revised: On average, we observe that LEAF outperforms all alternatives.

Original: When considering results for each individual tasks, we observe that LEAF outperforms or matches the accuracy of other frontends, with the notable exception of SincNet on Voxceleb.
Revised: When considering results for each individual tasks, we observe that LEAF outperforms or matches the accuracy of other frontends, with the notable exception of SincNet on Voxceleb.

Original: This is consistent with the fact that SincNet was originally proposed for speaker identiﬁcation (Ravanelli & Bengio, 2018) and illustrates the importance of evaluating over a wide range of tasks.
Revised: This is consistent with the fact that SincNet was originally proposed for speaker identiﬁcation (Ravanelli & Bengio, 2018) and illustrates the importance of evaluating over a wide range of tasks.

Original: To evaluate the robustness of our results with respect to the choice of the datasets, we apply a statistical bootstrap (Efron & Tibshirani, 1993) to compute the non-parametric distribution of the difference between the accuracy obtained with LEAF and each of the other frontends, when sampling datasets with replacement among the eight datasets.
Revised: To evaluate the robustness of our results with respect to the choice of the datasets, we apply a statistical bootstrap (Efron & Tibshirani, 1993) to compute the non-parametric distribution of the difference between the accuracy obtained with LEAF and each of the other frontends, when sampling datasets with replacement among the eight datasets.

Original: We test the null hypothesis that the mean of the difference is zero and measure the following one-sided p-values: p Mel < 10 − 5 , p TD-fbanks < 10 − 5 , p SincNet = 0 .
Revised: We test the null hypothesis that the mean of the difference is zero and measure the following one-sided p-values: p Mel < 10 − 5 , p TD-fbanks < 10 − 5 , p SincNet = 0 .

Original: Figure A. illustrates the corresponding bootstrap distribution, showing the statistical signiﬁcance of our results with respect to the choice of the datasets.
Revised: Figure A.1 illustrates the corresponding bootstrap distribution, showing the statistical signiﬁcance of our results with respect to the choice of the datasets.

Original: We dive deeper in the comparison between the mel-ﬁlterbanks frontend and LEAF by studying the effect of the compression component (log, PCEN, sPCEN).
Revised: In Table 2, we study the effect of the compression function.

Original: Table 2 shows that PCEN improves signiﬁcantly over log-compression on both mel- and learned ﬁlterbanks.
Revised: PCEN improves signiﬁcantly over log-compression on both mel- and learned ﬁlterbanks.

Original: The proposed LEAF frontend adopts sPCEN, which gives the best performance.

Original: We also observe that for any choice of the compression function, the learnable frontend matches or outperforms the corresponding mel-ﬁlterbank.
Revised: We also observe that for any choice of the compression function, the learnable frontend matches or outperforms the corresponding mel-ﬁlterbank.

Original: 4. M ULTI - TASK AUDIO CLASSIFICATION
Revised: 4. M ULTI - TASK AUDIO CLASSIFICATION

Original: To learn a general-purpose frontend that generalizes across tasks, we train a multi-task model that uses the same LEAF parameters and encoder for all tasks, with a task-speciﬁc linear head.More [Table]
Revised: To learn a general-purpose frontend that generalizes across tasks, we train a multi-task model that uses the same LEAF parameters and encoder for all tasks, with a task-speciﬁc linear head.

Original: [Table] LEAF [Table] - speciﬁcally, a single network with K heads ( h θ 1 , . .
Revised: More speciﬁcally, a single network with K heads ( h θ 1 , . .

Original: , h θ K ) is trained on mini-batches, uniformly sampled from the K datasets.
Revised: , h θ K ) is trained on mini-batches, uniformly sampled from the K datasets.

Original: A mini-batch of size B can be represented as ( x k
Revised: A mini-batch of size B can be represented as ( x k

Original: i
Revised: i

Original: i , y k i i ) i
Revised: i , y k i i ) i

Original: =1 ..B , where k i is the associated task the example has been sampled from.
Revised: =1 ..B , where k i is the associated task the example has been sampled from.

Original: The multi-task loss function is now computed on a mini-batch as the sum of the individual loss functions:
Revised: The multi-task loss function is now computed on a mini-batch as the sum of the individual loss functions:

Revised: [Table]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (7) where θ and ψ represent the shared parameters of the encoder and frontend respectively, θ k , k = 1 , . .
Revised: where θ and ψ represent the shared parameters of the encoder and frontend respectively, θ k , k = 1 , . .

Original: , K the task-speciﬁc parameters of the heads
Revised: , K the task-speciﬁc parameters of the heads

Original: h θ k ( · ) , and δ is the Kronecker delta function.
Revised: h θ k ( · ) , and δ is the Kronecker delta function.

Original: We use the same set of tasks described in Section 4.1 with one exception, replacing VoxCeleb with Librispeech as a representative speaker identiﬁcation task.
Revised: We use the same set of tasks described in Section 4.1 with one exception, replacing VoxCeleb with Librispeech for speaker identiﬁcation.

Original: As illustrated in Table A.1, VoxCeleb has much more classes ( ≈ 1200 ) than any other task.
Revised: As illustrated in Table A.1, VoxCeleb has much more classes ( ≈ 1200 ) than any other task.

Original: This creates an imbalance in the number of parameters in the heads, making training signiﬁcantly slower and reducing the accuracy on all other tasks, regardless of the frontend.
Revised: This creates an imbalance in the number of parameters in the heads, making training signiﬁcantly slower and reducing the accuracy on all other tasks, regardless of the frontend.

Original: Table 3 reports the accuracy obtained on each task, for all the frontends.
Revised: Table 3 reports the accuracy obtained on each task, for all the frontends.

Original: LEAF shows the best overall performance while matching or outperforming all other methods on every tasks.
Revised: LEAF shows the best overall performance while matching or outperforming all other methods on every tasks.

Original: We repeated theanalysis using the bootstrap  to evaluate the statistical signiﬁcance of these results with respect to the choice of the datasets and observed the following p-values: p Mel = 0 . 048 , p TD-fbanks < 10 − 5 , p SincNet = 10 − 5 .Figure A.2 illustrates the corresponding bootstrap distribution.
Revised: We repeated the bootstrap analysis (Figure A.2) to evaluate the statistical signiﬁcance of these results with respect to the choice of the datasets and observed the following p-values: p Mel = 0 . 048 , p TD-fbanks < 10 − 5 , p SincNet = 10 − 5 .

Original: To the best of our knowledge, it is the ﬁrst time a learnable frontend is shown to outperform mel-ﬁlterbanks over several tasks with a unique parametrization.
Revised: To the best of our knowledge, it is the ﬁrst time a learnable frontend is shown to outperform mel-ﬁlterbanks over several tasks with a unique parametrization.

Original: 4. M ULTI - LABEL CLASSIFICATION ON A UDIO S ET
Revised: 4. M ULTI - LABEL CLASSIFICATION ON A UDIO S ET

Original: Table 4 shows the performance  of the different frontends on Audioset (Gemmeke et al., 2017), a large-scale multi-label dataset of sounds of all categories, described by an ontology of 527 classes.
Revised: Table 4 shows the performance (averaged over three runs) of the different frontends on Audioset (Gemmeke et al., 2017), a large-scale multi-label dataset of sounds of all categories, described by an ontology of 527 classes.

Original: Audioset examples are weakly labelled with one or more positive labels, so we evaluate with the standard metric of this dataset, the balanced d-prime, a non-linear transformation of the AUC averaged uniformly across all classes.
Revised: Audioset examples are weakly labelled with one or more positive labels, so we evaluate with the standard metric of this dataset, the balanced d-prime, a non-linear transformation of the AUC averaged uniformly across all classes.

Original: EfﬁcientNetB0 trained with LEAF achieves a d-prime of 2.63, outperforming the level achieved when using mel-ﬁlterbanks (d-prime: 2.58) or other frontends.
Revised: EfﬁcientNetB0 trained with LEAF achieves a d-prime of 2.63, outperforming the level achieved when using mel-ﬁlterbanks (d-prime: 2.61) or other frontends.

Original: We compare this result with the state-of-the-art PANN model (Kong et al., 2019), which reports a d- prime of 2.73 when trained on mel-ﬁlterbanks.
Revised: We compare this result with the state-of-the-art PANN model (Kong et al., 2019), which reports a d-prime of 2.73 when trained on mel-ﬁlterbanks.

Original: Kong et al.
Revised: Kong et al.

Original: (2019) also train a learnable “Wavegram” frontend made of 9 layers of 1D- and 2D- convolutions, for a total of 300 k parameters, reporting a d-prime equal to 2.61.
Revised: (2019) also train a learnable “Wavegram” frontend made of 9 layers of 1D- and 2D- convolutions, for a total of 300 k parameters, reporting a d-prime equal to 2.61.

Original: EfﬁcientNetB0 on LEAF thus outperforms this system, while having a frontend with 670x fewer parameters, and a considerably smaller encoder.
Revised: EfﬁcientNetB0 on LEAF thus outperforms this system, while having a frontend with 670x fewer parameters, and a considerably smaller encoder.

Original: To conﬁrm these results, we also train EfﬁcientNetB0 on the Wavegram and get similar ﬁndings.
Revised: To conﬁrm these results, we also train EfﬁcientNetB0 on the Wavegram and get similar ﬁndings.

Original: All these ﬁndings are replicated when replacing EfﬁcientNetB0 with our implementation of CNN14: even though the gap between LEAF and baselines reduces as the scores get higher, CNN14 on LEAF matches the current state-ofthe-art on AudioSet.
Revised: All these ﬁndings are replicated when replacing EfﬁcientNetB0 with our implementation of CNN14: even though the gap between LEAF and baselines reduces as the scores get higher, CNN14 on LEAF matches the current state-of-the-art on AudioSet.

Original: To conclude these experiments and show that individual components of LEAF can beneﬁt other frontends, we replace the max-pooling of SincNet with our Gaussian lowpass ﬁlter, and LayerNorm (Ba et al., 2016) with the proposed sPCEN.
Revised: To conclude these experiments and show that individual components of LEAF can beneﬁt other frontends, we replace the max-pooling of SincNet with our Gaussian lowpass ﬁlter, and LayerNorm (Ba et al., 2016) with the proposed sPCEN.

Original: This version, named “SincNet+” in Table 4, signiﬁcantly outperforms the original version, regardless of the encoder.
Revised: This version, named “SincNet+” in Table 4, signiﬁcantly outperforms the original version.

Original: 4. A NALYSIS OF LEARNED FILTERS , POOLING AND COMPRESSION
Revised: 4. A NALYSIS OF LEARNED FILTERS , POOLING AND COMPRESSION

Original: Figure A.3 illustrates the center frequencies learned by the Gabor ﬁltering layer of LEAF on AudioSet, and compares them to those of mel-ﬁlterbanks.
Revised: Figure A.3 illustrates the center frequencies learned by the Gabor ﬁltering layer of LEAF on AudioSet, and compares them to those of mel-ﬁlterbanks.

Original: At a high level, these ﬁlters do not deviate much from their mel-scale initialization.
Revised: At a high level, these ﬁlters do not deviate much from their mel-scale initialization.

Original: On the one hand, this indicates that the mel-scale is a strong initialization, a result consistent with previous work (Sainath et al., 2015; Zeghidour et al., 2018b).
Revised: On the one hand, this indicates that the mel-scale is a strong initialization, a result consistent with previous work (Sainath et al., 2015; Zeghidour et al., 2018b).

Original: On the other hand, there are differences at both ends of the range, with LEAF covering a wider range of frequencies.
Revised: On the other hand, there are differences at both ends of the range, with LEAF covering a wider range of frequencies.

Original: For example, the lowest frequency ﬁlter is centered around 60Hz, as opposed to 100Hz for mel-ﬁlterbanks.
Revised: For example, the lowest frequency ﬁlter is centered around 60 Hz , as opposed to 100 Hz for mel-ﬁlterbanks.

Original: We believe that is one of the reasons that explain the out-performance of LEAF on Audioset, as it focuses on a more appropriate frequency range to represent the underlying audio events.
Revised: We believe that is one of the reasons that explain the out-performance of LEAF on Audioset, as it focuses on a more appropriate frequency range to represent the underlying audio events.

Original: Figure A.4 shows the learned Gaussian lowpass ﬁlters at convergence.
Revised: Figure A.4 shows the learned Gaussian lowpass ﬁlters at convergence.

Original: We see that they all deviate towards a larger frequency bandwidth (characterized by a smaller standard deviation in the time domain), and that each ﬁlter has a different bandwidth, which conﬁrms the utility of using depthwise pooling.
Revised: We see that they all deviate towards a larger frequency bandwidth (characterized by a smaller standard deviation in the time domain), and that each ﬁlter has a different bandwidth, which conﬁrms the utility of using depthwise pooling.

Original: Figure A.5 shows the learned exponents r n (initialized at 2 . 0 ) and smoothing coefﬁcients s n (initialized at 0 . 04 ) of sPCEN, ordered by ﬁlter index.
Revised: Figure A.5 shows the learned exponents r n (initialized at 2 . 0 ) and smoothing coefﬁcients s n (initialized at 0 . 04 ) of sPCEN, ordered by ﬁlter index.

Original: The exponents are spread in [1 . 9 , 2 . 6] , which shows the importance of learning per-channel exponents, as well as the appropriate choice of using a ﬁxed cubic root in previous work (Lyons & Paliwal, 2008), as an alternative to log-compression.
Revised: The exponents are spread in [1 . 9 , 2 . 6] , which shows the importance of learning per-channel exponents, as well as the appropriate choice of using a ﬁxed cubic root in previous work (Lyons & Paliwal, 2008), as an alternative to log-compression.

Original: Learning per-channel smoothing coefﬁcients also allows deviating from the initialization.
Revised: Learning per-channel smoothing coefﬁcients also allows deviating from the initialization.

Original: Interestingly, most ﬁlters keep a low coefﬁcient (slowly moving average), except for the ﬁlter with the highest frequency, which has a signiﬁcantly faster moving average ( s ≈ 0 . 15 ).
Revised: Interestingly, most ﬁlters keep a low coefﬁcient (slowly moving average), except for the ﬁlter with the highest frequency, which has a signiﬁcantly faster moving average ( s ≈ 0 . 15 ).

Original: 4. R OBUSTNESS TO NOISE
Revised: 4. R OBUSTNESS TO NOISE

Original: We compare the robustness of LEAF and mel-ﬁlterbanks to synthetic noise, on the Speech Commands dataset.
Revised: We compare the robustness of LEAF and mel-ﬁlterbanks to synthetic noise, on the Speech Commands dataset.

Original: To do so, we artiﬁcially add Gaussian noise to the waveforms both during training and evaluation, with different gains to obtain a Signal-to-Noise Ratio (SNR) from + inf (no noise) to − 5 dB. Figure A.6 shows that while performance (averaged over three runs) degrades for all models as SNR decreases, LEAF is more resilient than mel-ﬁlterbanks.
Revised: To do so, we artiﬁcially add Gaussian noise to the waveforms both during training and evaluation, with different gains to obtain a Signal-to-Noise Ratio (SNR) from + inf (no noise) to − 5 dB . Figure A.6 shows that while performance (averaged over three runs) degrades for all models as SNR decreases, LEAF is more resilient than mel-ﬁlterbanks.

Original: In particular, when using a logarithmic compression, the Gabor 1-D convolution and Gaussian pooling of LEAF maintain a signiﬁcantly higher accuracy than mel-ﬁlterbanks.
Revised: In particular, when using a logarithmic compression, the Gabor 1-D convolution and Gaussian pooling of LEAF maintain a signiﬁcantly higher accuracy than mel-ﬁlterbanks.

Original: Using PCEN has an even higher impact on performance when using mel-ﬁlterbanks, the best results being obtained with LEAF+PCEN.
Revised: Using PCEN has an even higher impact on performance when using mel-ﬁlterbanks, the best results being obtained with LEAF + PCEN.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we introduce LEAF, a fully learnable frontend for audio classiﬁcation as an alternative to using handcrafted mel-ﬁlterbanks.
Revised: In this work we introduce LEAF, a fully learnable frontend for audio classiﬁcation as an alternative to using handcrafted mel-ﬁlterbanks.

Original: We demonstrate over a large range of tasks that our model is a good drop-in replacement to these features with no adjustment to the task at hand, and can even learn a single set of parameters for general-purpose audio classiﬁcation while outperforming previously proposed learnable frontends.
Revised: We demonstrate over a large range of tasks that our model is a good drop-in replacement to these features with no adjustment to the task at hand, and can even learn a single set of parameters for general-purpose audio classiﬁcation while outperforming previously proposed learnable frontends.

Original: In future work, we will move yet a step forward in removing handcrafted biases from the model.
Revised: In future work, we will move yet a step forward in removing handcrafted biases from the model.

Original: In particular, our model still relies on an underlying convolutional architecture, with ﬁxed ﬁlter length and stride.
Revised: In particular, our model still relies on an underlying convolutional architecture, with ﬁxed ﬁlter length and stride.

Original: Learning these important parameters directly from data would allow for an easier generalization across tasks with various sampling rates and frequency content.
Revised: Learning these important parameters directly from data would allow for an easier generalization across tasks with various sampling rates and frequency content.

Original: Moreover, we believe that the general principle of learning to ﬁlter, pool and compress can beneﬁt the analysis of non-audio signals, such as seismic data or physiological recordings.
Revised: Moreover, we believe that the general principle of learning to ﬁlter, pool and compress can beneﬁt the analysis of non-audio signals, such as seismic data or physiological recordings.

Revised:  A CKNOWLEDGMENTS

Revised: Authors thank Dick Lyon, Vincent Lostanlen, Matt Harvey, and Alex Park for helpful discussions.

Revised: Authors are also thankful to Julie Thomas for helping with the design of Figure 1.

Revised: Finally, authors thank the reviewers of ICLR 2021 for their feedback that helped improving the quality of this work.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: F r e q u e n c y
Revised: F r e q u e n c y

Original: [ H z ]
Revised: [ H z ]

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Learned depthwise Gaussian lowpass filters init
Revised: filters init

Original: [Figure]
Revised: [Figure]

Original: PCEN smooth coefficients       2.
Revised: PCEN smooth coefficients       2.

Original: PCEN root coefficients
Revised: PCEN [Figure]

Original: SNR
Revised: SNR

Original: [Figure]
Revised: [Figure]

Revised: cc

Original: [Figure]
Revised: [Figure]

Original: cc u r a c y
Revised: c y

Original: [Figure]
Revised: [Figure]