L EARNING THE SPECTROGRAM TEMPORAL RESOLUTION FOR AUDIO CLASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The audio spectrogram is a time-frequency representation that has been widely used for audio classification.
The temporal resolution of a spectrogram depends on hop size.
Previous works generally assume the hop size should be a constant value such as ten milliseconds.
However, a fixed hop size or resolution is not always optimal for different types of sound.
This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution learning to improve the performance of audio classification models.
Given a spectrogram calculated with a fixed hop size, DiffRes merges non-essential time frames while preserving important frames.
DiffRes acts as a “drop-in” module between an audio spectrogram and a classifier, and can be end-to-end optimized.
We evaluate DiffRes on the melspectrogram, followed by state-of-the-art classifier backbones, and apply it to five different subtasks.
Compared with using the fixed-resolution mel-spectrogram, the DiffRes-based method can achieve the same or better classification accuracy with at least 25% fewer temporal dimensions on the feature level, which alleviates the computational cost at the same time.
Starting from a high-temporal-resolution spectrogram such as one-millisecond hop size, we show that DiffRes can improve classification accuracy with the same computational complexity.
I NTRODUCTION
Audio classification refers to a series of tasks that assign labels to an audio clip.
Those tasks include audio tagging (Kong et al., 2020), speech keyword classfication (Kim et al., 2021), and music genres classification (Castellon et al., 2021).
The input to an audio classification system is usually a one-dimensional audio waveform, which can be represented by discrete samples.
Although there are methods using time-domain samples as features (Kong et al., 2020; Luo & Mesgarani, 2018; Lee et al., 2017), the majority of studies on audio classification convert the waveform into a spectrogram as the input feature (Gong et al., 2021b;a).
Spectrogram is usually calculated by the Fourier transform (Champeney & Champeney, 1987), which is applied in short waveform chunks multiplied by a windowing function, resulting in a two-dimensional time-frequency representation.
According to the Gabor’s uncertainty principle (Gabor, 1946), there is always a trade-off between time and frequency resolutions.
To achieve desired resolution on the temporal dimension, it is a common practice (Kong et al., 2021; Liu et al., 2022) to apply a fixed hop size between windows to capture the dynamics between adjacent frames.
In this case, the spectrogram has a fixed temporal resolution, which we will refer to simply as resolution.
Using a fixed resolution is not necessarily optimal for an audio classification model.
Intuitively, the resolution should depend on the temporal pattern: fast-changing signals are supposed to have high resolution, while relatively steady signals or blank signals may not need the same high resolution for the best accuracy.
For example, Figure 1 shows that by increasing resolution, more details appear in the spectrogram of Alarm Clock while the pattern of Siren stays mostly the same. This indicates the finer details in high-resolution Siren may not essentially contribute to the classification accuracy.
There are plenty of studies on learning a suitable frequency resolution with a similar spirit (Stevens et al., 1937; Sainath et al., 2013; Ravanelli & Bengio, 2018b; Zeghidour et al., 2021), but the work on learning temporal resolution is still rare (Ilyashenko et al., 2019).
Most previous studies focus on exploring the effects of different resolutions.
Ferraro et al. (2021) experiments on music tagging with coarse-resolution spectrograms, and observes a similar performance can be maintained while being much faster to compute.
Recently, (Kazakos et al., 2021) proposes a two- stream architecture that process both fine-grained and coarse-resolution spectrogram and shows the state-of-the-art result on VGG-Sound (Chen et al., 2020) and
{+EPIC-KITCHENS-
 (Damen et al., 2020).+} In addition, our pilot study shows the optimal resolution is not the same for different types of sound (see Figure 10a in Appendix A.1).
The pilot study shows that when increasing resolution, the improvement on different types of sound is not consistent, in which some of them even degrade with a higher resolution.
This motivates us to design a method that can learn the optimal resolution.
[Figure]
Besides, the potential of the high-resolution spectrogram, e.g., with one milliseconds (ms) hop size, is still unclear.
Some popular choices of hop size including 10 ms (B¨ock et al., 2012; Kong et al., 2020;
Gong et al., 2021a) and 12 . 5 ms (Shen et al., 2018; Rybakov et al., 2022)
Previous studies (Kong et al., 2020; Ferraro et al., 2021) show classification performance can be steadily improved with the increase of resolution.
One remaining question is: Can even finer resolution improve the performance?
We conduct a pilot study for this question on a limited-vocabulary speech recognition task with hop sizes smaller than 10 ms (see Figure 10b in Appendix A.1).
We noticed that accuracy can still be improved with smaller hop size, at a cost of increased computational complexity.
This indicates there is still useful information in the higher temporal resolution.
In this work, we believe that we are the first to demonstrate learning temporal resolution on the audio spectrogram. We show that learning temporal resolution leads to efficiency and accuracy improvements over the fixed-resolution spectrogram.
We propose a lightweight algorithm, DiffRes, that makes spectrogram resolution differentiable during model optimization.
DiffRes can be used as a “drop-in” module after spectrogram calculation and optimized end-to-end with the downstream task.
The output of DiffRes is a time-frequency representation with varying resolution, which is achieved by adaptively merging the time steps of a fixed-resolution spectrogram.
The adaptive temporal resolution alleviates the spectrogram temporal redundancy and can speed up computation during training and inference.
We perform experiments on five different audio tasks, including the largest dataset AudioSet (Gemmeke et al., 2017).
DiffRes shows clear improvements on all tasks over the fixed-resolution mel-spectrogram baseline and other learnable front-ends (Zeghidour et al., 2021; Ravanelli & Bengio, 2018b; Zeghidour et al., 2018).
Compared with the fixed-resolution spectrogram, we show that using DiffRes can achieve a temporal dimension reduction of at least 25 % with the same or better audio classification accuracy.
On high-resolution spectrogram, we also show that DiffRes can improve classifier performance without increasing the feature temporal dimensions.
Our code is publicly available 1 .
L EARNING T EMPORAL R ESOLUTION WITH D IFF R ES
We provide an overview of DiffRes-based audio classification in Section 2.1.
We introduce the detailed formulation of DiffRes and the end-to-end optimization in Section 2.2.1, 2.2.2, and 2.
. 2. O VERVIEW
Let x ∈ R L denote a one-dimensional audio time waveform, where L is the number of audio samples.
An audio classification system can be divided into a feature extraction stage and a classification stage.
In the feature extraction stage, the audio waveform will be processed by a function
Q l,h : R L → R F × T , which maps the time waveform into a two-dimensional time-frequency representation X , such as a mel-spectrogram, where X : ,τ = ( X 1 ,τ , ..., X F,τ ) is the τ -th frame.
Here, T and F stand for the time and frequency dimensions of the extracted representation.
We also refer to the representation along the temporal dimensions as frames.
We use l and h to denote window length and hop size, respectively. Usually T ∝ Lh . We define the temporal resolution 1 h by frame
[Figure]
i c i e n t N e t
C l a ss i f i e r [Figure] per second (FPS), which denotes the number of frames in one second.
In the classification stage, X will be processed by a classification model G θ parameterized by θ .
The output of G θ is the label predictions ˆ y , in which ˆ y i denotes the probability of class i .
Given the paired training data ( x , y ) ∈ D , where y denotes the one-hot vector for ground-truth labels, the optimization of the classification system can be formulated as
[Equation]
where L is a loss function such as cross entropy (De Boer et al., 2005).
Figure 2 show an overview of performing classification with DiffRes.
DiffRes is a “drop-in” module between X and G θ focusing on learning the optimal temporal resolution with a learnable function F ϕ : R F × T → R F × t , where t is the parameter denoting the target output time dimensions of DiffRes, and ϕ is the learnable parameters.
DiffRes formulates F ϕ with two steps: i) estimating the importance of each time frame with a learnable model H ϕ :
X → s , where s is a 1 × T shape row vector; and ii) warping frames based on a frame warping algorithm, the warping is performed along a single direction on the temporal dimension.
We introduce the details of these two steps in Section 2.2.1 and Section 2.2.2.
We define the dimension reduction rate δ of DiffRes by δ =
( T − t ) /T .
Usually, δ ≤ 1 and t ≤ T because the temporal resolution of the DiffRes output is either coarser or equal to that of X .
Given the same T , a larger δ means fewer temporal dimensions t in the output of DiffRes, and usually less computation is needed for G θ .
Similar to Equation 1, F ϕ can be end-to-end optimized by
[Equation]
2. D IFFERENTIABLE TEMPORAL RESOLUTION MODELING
Figure 2 illustrates how DiffRes performs adaptive temporal resolution learning.
We introduce frame importance estimation in Section 2.2.1, and introduce the warp matrix construction function α ( · ) and frame warping function β ( · ) in Section 2.2.2.
Figure 4 illustrates an example of DiffRes on the mel-spectrogram.
2.2. F RAME IMPORTANCE ESTIMATION
Conv1d
[Figure]
BatchNorm1d
LeakyReLU
[Figure]
Conv1d
[Figure]
BatchNorm1d
LeakyReLU
[Figure]
+
We design a frame importance estimation module H ϕ to decide the proportion of each frame that needs to be kept in the output.
The frame importance estimation module will output a row vector s ′ with shape 1 × T , where the element s ′ τ is the importance score of the τ -th time frame X : ,τ .
The frame importance estimation can be denoted as
[Equation]
(3) where s ′ is the row vector of importance scores, and σ is the sigmoid function (Han & Moraga, 1995).
A higher value in s ′ τ indicates the τ -th frame is important for classification.
We apply the sigmoid function to stabilize training by limiting the values in s ′ between zero and one.
We implement H ϕ with a stack of one-dimensional convolutional neural networks (CNNs) (Fukushima & Miyake, 1982; LeCun et al., 1989).
Specifically, H ϕ is a stack of five one-dimensional convolutional blocks ( ResConv1D ). We design the ResConv1D block following other CNN based methods (Srinivasamurthy, 2018; Liu et al., 2020).
As shown in Figure 3, each ResConv1D has two layers of one-dimensional CNN with batch normalization (Ioffe & Szegedy, 2015) and leaky rectified linear unit activation functions (Xu et al., 2015). We apply residual connection (He et al.,
[Figure] 2016) for easier training of the deep architecture (Zaeemzadeh et al., 2020). Each CNN layer is zero-padded to ensure the temporal dimension does not change (LeCun et al., 2015). We use exponentially decreasing channel numbers to reduce the computation (see details in Table 6).
In the next frame warping step (Section 2.2.2), elements in the importance score will represent the proportion of each input frame that contributes to an output frame.
Therefore, we perform rescale operation on s ′ , resulting in an s that satisfies s ∈ [0 , 1] 1 × T and (cid:80) Tk =1 s k ≤ t .
The rescale operation can be denoted as ˇ s =
[Equation]
) , where ˇ s is an intermediate variable that may contain elements greater than one, max denotes the maximum operation.
To quantify how active H ϕ is trying to distinguish between important and less important frames, we also design a measurement, activeness ρ , which is calculated by the standard derivation of the non-empty frames, given by
[Equation]
where S active is the set of indices of non-empty frames, ϵ is a small value, energy( · ) is a function that calculate the root-mean-square energy (Law & Rennie, 2015) of a frame in the spectrogram, and | S | denotes the size of set S .
We use δ to unify the value of ρ for easier comparison between different δ settings.
The activeness ρ can be used as an indicator of how DiffRes behaves during training.
A higher ρ indicates the model is more active at learning the frame importance.
A lower ρ such as zero indicates learning nothing.
We will discuss the learning process of DiffRes with ρ in Section 3.3.
2.2. T EMPORAL F RAME W ARPING
We perform temporal frame warping based on s and X to calculate a representation with adaptive temporal resolution.
Generally, the temporal frame warping algorithm can be denoted by W = α ( s ) and
O = β ( X , W ) , where α ( · ) is a function that convert s into a warp matrix W with shape t × T , and β ( · ) is a function that applies W to X to calculate the warpped feature O .
Elements in W such as W i,j denote the contribution of the j -th input frame X : ,j to the i -th output frame O : ,i .
We will introduce the realization of α ( · ) and β ( · ) in the following sections. Figure 11 in Appendix A. provides an example of the warp matrix construction function.
Warp matrix construction Function α ( · ) calculates the warp matrix W with s by:
[Equation]
where we calculate the cumulative sum of s to decide which output frame each input frame will be warped into.
The warp matrix W here can be directly used for frame warping function β ( · ) .
However, we found the DiffRes can learn more actively if the warp matrix satisfies (cid:80) tj =1 W j, : = 1 , which means each output frame is assigned an equal amount of total warp weights on input frames.
We further process W to meet this requirement with Algorithm 1 in Appendix A.4.
We also provide a vectorized version of Algorithm 1 that can be run efficiently on GPUs (see Appendix A.5).
Frame warping Function β ( · ) performs frame warping based on the warp matrix W .
The i -th output frame is calculated with X and the i -th row of W , given by
[Equation]
[Equation] where A : R 1 × T → R stands for the frame aggregation function such as averaging, O is the final output feature with shape F × t .
Resolution encoding The final output O does not contain the resolution information at each time step, which is crucial information for the classifier.
Since the temporal resolution can be represented with W , we construct a resolution encoding with W in parallel with frame warping.
Firstly, we construct a positional encoding matrix E with shape F × T , using the similar method described in Vaswani et al.
Each column of E represents a positional encoding of a time step.
Then we calculate the resolution encoding by E = EW ⊤ , where W ⊤ stands for the transpose of W . The shape of the resolution encoding is F × t .
Both E and O are concatenated on the channel dimension as the classifier input feature.
2. O PTIMIZATION
We propose a guide loss to provide guidance for DiffRes on learning frame importance.
Since we do not know the ground truth frame importance, we cannot directly optimize s .
We introduce L guide as an inductive bias (Mitchell, 1980) to the system based on the assumption that an empty frame should have a low importance score.
Specifically, we propose the guide loss by
[Equation]
where S empty is a set of time indexes that have low energy, and λ is a constant threshold.
Given that the output of DiffRes has fewer temporal dimensions than X , the DiffRes layer forms an information bottleneck (Tishby et al., 2000; Shwartz-Ziv & Tishby, 2017) that encourages DiffRes to assign a higher score to important frames.
We analyze the information bottleneck effect of DiffRes in Section 3.3.
The parameter λ is a threshold for the guide loss to take effect.
This threshold can alleviate the modeling bias toward energy.
For example, if λ = 0 , the importance scores of empty frames are strongly regularized, and the model will also tend to predict low importance scores for lower energy frames, which may contain useful information.
L bce is the standard binary cross entropy loss function (Shannon, 2001) for classification, given by Equation 8, where ˆ y is the label prediction and N is the total number of classes.
[Equation]
( y i log(ˆ y i )
+ (1 − y i log(1 − ˆ y i ))) , (8) The loss function of the DiffRes-based audio classification system includes our proposed guide loss L guide and the binary cross entropy loss L bce , given by L =
L guide + L bce .
E XPERIMENTS
We focus on evaluating DiffRes on the mel-spectrogram, which is one of the most popular features used by state-of-the-art systems (Chen et al., 2022;
Gong et al., 2022; Verbitskiy et al., 2022; Koutini et al., 2021).
We evaluate DiffRes on five different tasks and datasets (see Table 5 in Appendix A.3), including audio tagging on AudioSet (Gemmeke et al., 2017) and FSD50K (Fonseca et al., 2021), environmental sound classification on ESC50 (Piczak, 2015), limited-vocabulary speech recognition on SpeechCommands (Warden, 2018), and music instrument classification on NSynth (Engel et al., 2017).
All the datasets are resampled at a sampling rate of 16 kHz.
Following the evaluation protocol in the previous works (Zeghidour et al., 2021;
Riad et al., 2021; Kong et al., 2020; Gong et al., 2021b), we report the mean average precision (mAP) as the main evaluation metric on AudioSet and FSD50K, and report classification accuracy (ACC) on other datasets.
In all experiments, we use the same architecture as used by Gong et al.
(2021b), which is an EfficientNet-B2 (Tan & Le, 2019) with four attention heads ( 13 .
6 M parameters).
We reload the ImageNet pretrained weights for EfficientNet-B2 in a similar way to Gong et al. (2021a;b).
We apply random specaugmentation (Park et al., 2019) with up to 37 . 5 % and 50 % masking on the frequency and time dimension following Gong et al. (2021b).
We also apply mixup (Zhang et al., 2017) augmentation with 0 . 5 probability.
All experiments are repeated three times with different seeds to reduce randomness.
We also report the standard derivation of the repeated trails along with the averaged result.
We train the DiffRes layer with λ = 0 .
5 and ϵ = 1 × 10 − 4 .
For the frame aggregation function A (see Equation 6), we use both the max and mean operations, whose outputs are concatenated with the resolution encoding E on the channel dimension as the input feature to the classifier.
The frame importance estimation module we used in this paper is a stack of five ResConv1D (see Table 6 in [Table] Appendix A.3) with a total 82 371 of parameters.
We calculate the mel-spectrogram with a Hanning window, 25 ms window length, 10 ms hop size, and 128 mel-filterbanks by default.
We list our detailed hyperparameters setting in Table 4 (Appendix A.3).
We also encourage readers to read through Figure 12 and Figure 13 in Appendix A.6 for examples of DiffRes on the mel-spectrogram.
3. A DAPTIVELY COMPRESS THE TEMPORAL DIMENSION
Compression of mel-spectrogram temporal dimension can lead to a considerable speed up on training and inference (see Figure 5), thus it is a valuable topic for efficient classification (Huang & Leanos, 2018) and on-device application (Choi et al., 2022).
The experiments in this section aim to evaluate the effectiveness of DiffRes in compressing temporal dimensions and maintaining precision.
We compare DiffRes with three temporal dimension reduction methods: i) Change hop size (CHSize) reduces the temporal dimension by enlarging the hop size.
The output of CHSize has a fixed resolution and may lose information between output frames. ; ii) AvgPool is a method that performs average pooling on a 100 FPS spectrogram to reduce the temporal dimensions.
AvgPool also has a fixed resolution, but it can aggregate information between output frames by pooling; iii) ConvAvgPool is the setting that the 100 FPS mel-spectrogram will be processed by a stack of
ResConv1D (mentioned in Section 2.2.1), followed by an average pooling for dimension reduction.
ConvAvgPool has a total of 493 824 parameters and we provide the detailed structure in Table 6 (Appendix A.3).
Based on a learnable network, ConvAvgPool has the potential of learning more suitable features and temporal resolution implicitly.
Baseline comparisons Table 1 shows our experimental result.
The baseline of this experiment is performed on mel-spectrogram without temporal compression (i.e., 100 FPS) and the baseline result is shown under each task name.
When reducing 25 % of the temporal dimension (i.e., 75 FPS), the proposed method can even considerably improve the baseline performance on most datasets, except on speech recognition tasks where we maintain the same performance.
We assume the improvement comes from the data augmentation effect of DiffRes, which means divergent temporal compression on the same data at different training steps.
With a 50 FPS, four out of five datasets can maintain comparable performance.
With only 25 FPS, the proposed method can still improve the FSD50K tagging and music instrument classification tasks, which indicates the high temporal redundancy in these datasets.
Our proposed method also significantly outperforms other temporal dimension reduction baselines.
With fixed resolution and fewer FPS, the performance of CHSize degrades more notably.
AvgPool can outperform CHSize by aggregating more information between output frames.
Although ConvAvgPool has an extra learnable neural network, it does not show significant improvements compared with AvgPool.
ConvAvgPool even has an inferior performance on FSD50K and environmental sound classification tasks.
This indicates employing a neural network for feature learning is not always beneficial.
Also, by comparison, the interpretability of our proposed method is much better than ConvAvgPool (see Figure 9).
On variable-length audio data Note that the proposed method even improves the mAP performance by 1 . 3 % with only 25 FPS on the FSD50K dataset.
This is because the audio clip durations in the FSD50K have a high variance (see Table 5).
In previous studies (Gong et al., 2021a;b; Kong et al., 2020), a common practice is padding the audio data into the same duration in batched training and inference, which introduces a considerable amount of temporal redundancy in the data with a significantly slower speed.
By comparison, DiffRes can unify the audio feature shape regardless of their durations.
Model optimization becomes more efficient with DiffRes.
As a result, the proposed method can maintain an mAP of 55 .
6 ± 0 .
2 on the FSD50K, which is comparable to the baseline, with only 15 FPS and 28 % of the original training time.
This result shows that DiffRes provides a new mind map for future work on classifying variable-length audio files.
3. L EARNING WITH SMALLER HOP SIZE
Previous studies have observed that a higher resolution spectrogram can improve audio classification accuracy (Kong et al., 2020; Ferraro et al., 2021).
However, a hop size smaller than 10 ms has not been widely explored.
This is partly because the computation becomes heavier for a smaller hop size.
For example, with 1 ms hop size (i.e., 1000 FPS), the time and space complexity for an EfficientNet classifier will be 10 times heavier than with a common 10 ms hop size.
Since DiffRes can control the temporal dimension size, namely FPS, working on a small hop size spectrogram becomes computationally friendly.
Table 2 shows model performance can be considerably improved with smaller hop sizes.
AudioSet and environment sound dataset achieve the best performance on 6 ms and 1 ms hop size, and other tasks benefit most from 3 ms hop sizes.
In later experiments, we will use these best hop size settings on each dataset. [Table]
Comparing with other learnable front-ends The DiffRes is learnable, so the Mel+DiffRes setting as a whole can be viewed as a learnable front-end.
Table 3 compares our proposed method with SOTA learnable front-ends, our best setting is denoted as Mel+DiffRes (Best) [Table] which achieves the best result on all datasets.
For a fair comparison, we control the experiment setup to be consistent with Zeghidour et al.
(2021) in Mel+DiffRes.
Specifically, we change the backbone to EfficientNetB0 ( 5 . 3 M parameters) without ImageNet pretraining.
We also remove spec-augment and mixup, except in AudioSet
[Table]
and change our Mel bins from 128 to [Table] except in the AudioSet experiment where we change to 64.
The result shows Mel+DiffRes can outperform SOTA learnable frontend (Zeghidour et al., 2021; Ravanelli & Bengio, 2018b; Zeghidour et al., 2018) by a large margin. [Table] [Figure] h p u t p e r s e c o n d ( s )
[Figure] [Figure] [Figure] Computational cost We assess the one-second throughput of different front-ends on various FPS settings to compare their computational efficiency.
We control the FPS of Mel and LEAF by average pooling.
The computation time is measured between inputting waveform and outputting label prediction (with EfficientNet-B2).
We use 128 filters in LEAF (Zeghidour et al., 2021) for a fair comparison with 128 mel-filterbanks in Mel and DiffRes.
As shown in Figure 5, our proposed DiffRes only introduces marginal computational cost compared with Mel.
The state-of-the-art learnable front-end, LEAF, is about four times slower than our proposed method.
3. A NALYSIS FOR THE LEARNING OF D IFF R ES
Learning activeness DiffRes does not explicitly learn the optimal frame importance score because the ground truth frame importance is not available.
Instead, DiffRes is optimized with the guidance of guide loss L guide (Equation 7), which is a strong assumption we introduced to the model.
Figure 6 shows the trajectories of the DiffRes learning activeness (defined in Section 2.2.1) during the optimization with different FPS settings on the speech recognition task in Table 1.
According to the final converged value, DiffRes with a smaller FPS tend to be more active at learning frame importance.
This is intuitive since smaller FPS leads to more information bottleneck effects (Saxe et al., 2019) in DiffRes.
With a 25 FPS, the activeness even keeps increasing with more training steps, indicating the active learning of DiffRes.
Figure 7 shows the guide loss curve during training with different FPS settings.
Intuitively, when the FPS is small, a model needs to preserve more non-empty frames and fewer empty frames for better accuracy.
This assumption is aligned with our experiment result, which shows the model tends to have a lower guide loss with a smaller FPS.
Data augmentation and regularization effect As reflected in the curve of ρ and L guide in Figureand 7, DiffRes is optimized along with the classifier during training.
Hence DiffRes may produce different outputs for the same training data at different epochs.
This is equivalent to performing data augmentation on the audio data.
We suppose this is the main reason for the improved performance shown in Table 1.
Also, DiffRes reduces the sparsity of the audio feature by adaptive temporal compression.
This is equivalent to performing an implicit regularization (Neyshabur, 2017; Arora et al., 2019) on the feature level, which is beneficial for the system efficiency.
Ablation studies We perform the ablation study on AudioSet (Gemmeke et al., 2017), the largest audio dataset by far. The ablation study is performed on the mel-spectrogram with 10 ms hop size and a 75 % DiffRes dimension reduction rate (i.e., 25 FPS). Figure 8 shows the value of guide loss, activeness, and mean average precision (mAP) with different training steps and ablation setups. As the red solid line shows, if we remove the guide loss, the DiffRes activeness becomes very high, but the mAP becomes worse.
Meanwhile, the value of guide loss increases gradually, which is counter-intuitive because the empty frames should not have high important scores.
This indicates guide loss is an effective prior knowledge we can introduce to the system. As the dotted green line shows, if we remove the resolution encoding, the curve of guide loss and activeness almost show no changes, while the mAP degrades 1 . 15 %.
This indicates resolution encoding is crucial information for the classifier.
If we remove the max aggregation function (see the dotted black line), both the activeness and the mAP have a notable degradation. As shown by the green solid line, if we do not apply Algorithm 1, the DiffRes activeness will converge to a small value, which means DiffRes is not working active enough, indicating Algorithm 1 is crucial for DiffRes to take effects. We also provide a weakness analysis of DiffRes in Appendix A.2. [Figure]
R ELATED W ORKS
Neural-network based methods have been successfully applied on audio classification and achieved state-of-the-art performance, such as the pretrained audio neural networks (PANNs) (Kong et al., 2020), pretraining, sampling, labeling, and aggregation based audio tagging (PSLA)
(Gong et al., 2021b), and audio spectrogram transformer (AST) (Gong et al., 2021a).
We will cover two related topics on audio classification in the following sections: i) learnable audio front-ends; and ii) learning feature resolution in neural networks.
Learnable audio front-ends In recent years, learning acoustic features from waveform using trainable audio front-ends has attracted increasing interest from researchers.
Sainath et al.
(2013) introduced one of the earliest works that propose to jointly learn the parameter of filter banks with a speech recognition model.
Later, SincNet (Ravanelli & Bengio, 2018b) proposes to learn a set of bandpass filters on the waveform and has shown success on speaker recognition (Ravanelli & Bengio, 2018b;a).
Most recently, (Zeghidour et al., 2021) proposes to learn bandpass, and lowpass filtering as well as per-channel compression (Wang et al., 2017) simultaneously in the audio front-end and shows consistent improvement in audio classification.
Different from existing work on learnable audio front-ends, which mostly focus on the frequency dimension, our objective is learning the optimal temporal resolution.
We show that our method can outperform existing audio front-ends for audio classification on both accuracy and computation efficiency (see Table 3 and Figure 5).
Note that our proposed method can also be applied after most learnable front-ends (Zeghidour et al., 2021), which will be our future direction.
Learning feature resolution with neural networks One recent work on learning feature resolution for audio classification is DiffStride (Riad et al., 2021), which learns stride in convolutional neural network (CNN) in a differentiable way and outperforms previous methods using fixed stride settings.
By comparison, DiffStride needs to be applied in each CNN layer and can only learn a single fixed stride setting, while DiffRes is a one-layer lightweight algorithm and can personalize the best temporal resolution for each audio during inference.
Recently, Gazneli et al. (2022) proposed to use a stack of one-dimension-CNN blocks to downsample the audio waveform before the audio classification backbone network, e.g., Transformer, which can learn temporal resolution implicitly for audio classification.
In contrast, DiffRes can explicitly learn temporal resolution on the feature level with similar interpretability as mel-spectrogram.
C ONCLUSIONS
In this paper, we introduce DiffRes, a “drop-in” differentiable temporal resolution learning module that can be applied between audio spectrogram and downstream tasks.
We demonstrate over a large range of tasks that DiffRes can improve or maintain similar performance with 25 % to 75 % reduction on temporal dimensions, and can also efficiently utilize the information in high-resolution spectrograms to improve accuracy.
In future work, we will move forward to evaluate DiffRes on different kinds of time-frequency representations with more sophisticated frame importance prediction models.
Also, we believe the idea of DiffRes can work on other time series data as well for learning optimal temporal resolutions, such as video or seismic data.
R EPRODUCIBILITY STATEMENT
We believe that we have covered the details of our proposed method with figures (see Figuresto 13), algorithms (see Algorithm 1 to 3) and mathematical equations (see Equation 2 to 8).
As shown in Table 5 and 4, the hyperparameters are provided as exhaustive as we can, both in the main text and the appendix.
Moreover, we fully open-source our code at https://anonymous.
4open.science/r/diffres-8F22 to facilitate reproducibility.
A. W EAKNESS ANALYSIS
Table 2 shows DiffRes does not improve the model performance on 1 ms setting on most datasets. This may be due to the insufficient receptive field of the convolutions in DiffRes, which is aroundtime steps. By comparison, the temporal dimension of X on AudioSet is t = 3333 and t =
10with 3 ms and 1 ms hop size, respectively.
DiffRes may not effectively capture the useful information with only 41 temporal receptive field in this case.
Future work will address this problem by designing the resolution prediction model with a large receptive field.
Also, although we use the same λ setting in the guide loss (see Equation 7) in all the experiments, our experiment shows a proper tuning of λ can still affect the performance, which might require manual parameter tuning
[Table]
A. D ATASET AND EXPERIMENT DETAILS
[Table]
[Table]
[Table]
A. A LGORITHM FORMULATION OF D IFF R ES
Inputs
1 i ← j ← s ←▷ i, j for indexing, s store the sum value in each row.
2 while i < t and j < T do if W i,j > 0 then s ← s + W i,j ;
▷ Add up the weights in row i .
j ← j + 1; ▷ Move to the next column in row i .
else j sum
4 if training then
1 s 1 × T ←− Norm( σ ( H ϕ ( X )));
▷ Calculate frame importance (see Equation 3)
2 W t × T ←− α ( s );
▷ Construct a warp matrix (see Equation 5 and Algorithm 1)
3 O F × t ←− β ( X , W );
▷ Warping frames with the warp matrix (see Equation 6)
4 E F × t ←− EW ⊤ ;
E A. F AST IMPLEMENTATION OF A LGORITHM We provide a pure matrix operation based version of Algorithm 1 for efficient model optimization on GPUs.
Given the output of Equation 5, W 0 , we will process it into a matrix W with the same shape so that (cid:80) ti =1 W i, : = 1 and (cid:80) t,Ti =1 ,j =1 W i,j ≤ t .
Figure 11 provides an example of the fast implementation discussed in this section.
[Figure]
First, we calculate the cumulative sum of the distance between one and the total input weight assigned to each output frame, given by
[Equation]
(9) where 1 t denotes the all-one column vector with shape t .
For each row in W 0 , only the first nonnegative element and the first zero after the last non-negative element need update (see Algorithm 1).
And we locate those elements by performing the following convolution: convolving (no padding)
M with a kernel K of shape t × 2 , given by
[Equation]
where sgn( · ) stands for a sign function.
We pad a column of zero on the first row of the output Q .
The output Q ∈ { 0 , 1 , − 1 } is a matrix with shape t × T .
Then we calculate the value that needs updates with P and Q , given by
[Equation]
where P 1: t − 1 =
( P ij ) i ∈ [1: t − 1] ,j ∈ [1 ,t ] denotes slicing of the matrix row, 0 T is the all-zero row vector with length T , and ⊙ denotes element-wise multiplication.
Note that here the start index of a matrix is one.
U and V store the values that need addition and subtraction for each element in W 0 .
The final warp matrix is calculated by
[Equation]
[Figure]
[Figure]