S PEECH -MLP: A SIMPLE MLP ARCHITECTURE FOR SPEECH PROCESSING
Anonymous authors
Paper under double-blind review
A BSTRACT
Transformers have shown outstanding performance in recent years, achieving state-of-the-art results in speech processing tasks such as speech recognition, speech synthesis and speech enhancement.
In this paper, we show that, despite their success, such complex models are not needed for some important speech related tasks, which can be solved with much simpler and compact models.
Thus, we propose a multi-layer perceptron (MLP) architecture, namely speech-MLP, useful for extracting information from speech signals.
The model splits feature channels into non-overlapped chunks and processes each chunk individually.
These chunks are then merged together and further processed to consolidate the output.
By setting different numbers of chunks and focusing on different contextual window sizes, speech-MLP learns multiscale local temporal dependency.
The proposed model is successfully evaluated on two tasks: keyword spotting and speech enhancement.
In our experiments, two benchmark datasets are adopted for keyword spotting (Google speech command V2-35 and LibriWords) and one dataset (VoiceBank) for speech enhancement.
In all experiments, speech-MLP surpassed the transformer-based solutions, achieving better performance with fewer parameters lower GFLOPS.
Such results indicate that more complex models, such as transformers, are oftentimes not necessary for speech processing tasks.
Hence, simpler and more compact models should always be considered as an alternative, specially in resource-constrained scenarios.
I NTRODUCTION
As in many machine learning disciplines, speech processing is embracing more and more complex models, where transformer (Vaswani et al., 2017) is a particular example.
It was ﬁrst proposed to tackle machine translation, and afterwards was successfully applied to multiple research ﬁelds such as natural language processing (NLP)
(Devlin et al., 2018) and computer vision (CV) (Dosovitskiy et al., 2020).
The core of the transformer model is a self-attention mechanism, by which any two elements in a sequence can interact with each other, hence capturing long-range dependency.
Considering that speech signals are naturally temporal-dependent, researchers in the speech community recently explored transformer-based models in multiple speech processing tasks, and remarkable performance was reported in speech recognition (Dong et al., 2018; Karita et al., 2019; Huang et al., 2020), speech enhancement (SE)
(Kim et al., 2020; Fu et al., 2020), keyword spotting (KWS) (Berg et al., 2021; Vygon & Mikhaylovskiy, 2021) and speech synthesis (Li et al., 2019).
Recently, the conformer architecture, which combines convolution and self-attention, achieved excellent success in speech processing tasks and attracted much attention Gulati et al.
In this paper, we ask the following question: Do we need complex models such as transformers for certain speech processing tasks?
This question is closely related to the principle of ‘parsimony of explanations’, a.k.a., Occam’s razor (Walsh, 1979).
According to this principle, if there is any possibility, we should seek the models that can represent the data with the least complexity (Rasmussen & Ghahramani, 2001; Blumer et al., 1987).
However, in the public benchmark tests, complex and elaborately designed models are often ranked higher, due to the better reported performance.
For example, the KWS benchmark on Google speech command 1 and the SE benchmark on VoiceBank+DEMAND 2 , transformer-based models are among the top ranks.
Although the good performance is celebrating, the increased model complexity implies potential over-tuning and over-explanation, the risk that the Occam’s razor principle intends to avoid.
We, therefore, attempt to discover the simplest neural architecture, that is powerful enough to achieve comparable performance as the best existing models, in particular transformers, while eliminating unnecessary complexity.
Our design is based on domain knowledge, in particular, three properties of speech signals: (1) temporal invariance , (2) frequency asymmetry , and (3) short-term dependency (Huang et al., 2001; Benesty et al., 2008; Furui, 2018).
Based on these knowledge, we build the speech-MLP, a simple multi-layer perceptron (MLP) architecture, shown in Fig.
Besides the normalization components, the architecture involves simple linear transformations only.
The core of the architecture is the Split & Glue layer, which splits the channel dimension into multiple chunks, processes each chunk separately, and ﬁnally merges the processed chunks in order to attain the output.
Speech-MLP processes each time frames independently (compatible to temporal invariance), and the splitting & gluing procedure allows different treatments for different frequency bands (compatible to frequency asymmetry), and involves the local context of multiple scales (compatible to short-term dependency),
We tested the model on two speech processing tasks: keyword spotting with the Google speech command V2-35 and Libriword benchmark datasets; and speech enhancement with the VoiceBank benchmark dataset.
Results showed that on both tasks the proposed speech-MLP outperforms complex models, in particular models based on transformers.
Such results demonstrate that by utilizing domain knowledge and employing appropriate normalization techniques, it is possible to design simple yet powerful models.
In some cases, these simple models even beat complex models on open benchmarks, where complex models are more likely to obtain good performance by careful tuning.
In summary, we proposed Speech-MLP, a simple yet effective neural model to represent speech signal.
On the KWS and SE tasks, we demonstrated that the simple model can achieve performance comparable to or even better than transformers with less parameters and inference time. Our work shows that by taking domain-knowledge into account, it is possible to remove unnecessary complexity (e.g., modeling for the long-range dependency in KWS and SE) in model design, as advocated by the Occam’s razor.
R ELATED W ORK
Recent research has shown that a simple model can be as effective as complex and task speciﬁc models such as transformers in some important tasks.
In (Tolstikhin et al., 2021), for example, the authors proposed a simple architecture for vision, namely MLP-Mixer.
The model receives a sequence of image patches and performs channel-wise and patch-wise linear projection alternatively and iteratively.
Without using convolutions or self-attention, the Mixer architecture separates the per-location (channel-mixing) and cross-location (token-mixing) operations
(Tolstikhin et al., 2021).
While the channel-mixing MLPs enable communication between different channels, the token-mixing MLPs allow communication between different spatial locations (tokens).
Tested on image classiﬁcation benchmarks, MLP-Mixer achieved performance comparable to SOTA models, in particular the vision transformer model (Tolstikhin et al., 2021).
In another recent work (Liu et al., 2021), the authors investigated the need of the self-attention mechanism in transformers, proposing an alternative MLP-based architecture, namely gMLP.
The model, based on MLP layers with gating, consists of a stack of L identical blocks. Each block comprises a normalization layer, a channel projection, followed by an activation function and a spatial gating unit, followed by another channel projection (Liu et al., 2021). It achieves similar performance when compared to the vision transformer model (Touvron et al., 2021b), being 3 % more accurate than the aforementioned MLP-mixter model with 66 % fewer parameters.
The model was also successful on language modeling in the BERT setup (Liu et al., 2021), minimizing perplexity as well as Transformers.
The authors also found that perplexity reduction was more inﬂuenced by the model capacity than by the attention mechanism.
Inspired by vision transformers (Touvron et al., 2021b)(Dosovitskiy et al., 2020), in (Touvron et al., 2021a), the authors apply the skip connection technique from ResNet’s to MLP layers and propose the so-called Residual Multi-Layer Perceptrons (ResMLP).
The model receives non-overlapping image patches, typically 16 × 16. These patches go through a linear transformation in order to attain d-dimensional embeddings. The embeddings are then fed to a sequence of ResMLP blocks to produce a set of d-dimensional output embeddings.
An average pooling is applied on the ddimension output vector to represent the image, a linear classiﬁer is used then to predict the label associated with the image (Touvron et al., 2021a). Differently from Mixer-MLP, gMLP and ResMLP, CycleMLP can process inputs of arbitrary resolution with linear computational complexity as its receptive ﬁelds are enlarged for context aggregation (Chen et al., 2021). The model is based on Cycle Fully-Connected Layer (Cycle FC), serving as a generic, plug-and-play transformer-free architecture. Results show CycleMLP outperforming existing MLP-like models on ImageNet classiﬁcation, achieving good performance on object detection, instance segmentation and semantic segmentation (Chen et al., 2021).
The aforementioned research highlights that, despite their success, convolution and self-attention mechanisms are not mandatory for some CV and NLP tasks, and can be replaced by simpler layers such as MLP with a customized design.
Although typical convolution operations are not used by these MLP solutions (but rather 1 × 1 convolution as pointed out in (Chen et al., 2021) and (Tolstikhin et al., 2021)), these MLP approaches are inspired by CNN architectures for computer vision related tasks.
Their building block, nonetheless, is similar and based on applying linear transformation on spatial locations and feature channels. Although inspired by these new MLP architectures, speech-MLP focuses on speech signals rather than images. This implies in processing different input resolutions given the nature of the input signal.
The split & glue layer is very similar to a separable CNN (Chen et al., 2018), if we regard the frame-independent processing as 1-D convolution in time. In particular, it is essentially a group-wised CNN (Romero et al., 2020) with different kernels for each group. However, from the perspective of feature learning, the entire split& glue is an MLP if our focus is a particular frame (within a context). That is why a 1-D convolution is often called a time-delay neural net (TDNN)
(Waibel et al., 1989). We follow this convention and name our structure as speech-MLP. A key motivation of the speech-MLP structure is to respect the properties of speech signals. It should be emphasized that almost all successful techniques in speech processing take these properties into account, for instance the hidden Markov model (HMM) assumes short-term dependency (Rabiner &
Juang, 1986), TDNN assumes temporal invariance (Waibel et al., 1989), and frequency asymmetry is explicitly implemented in the famous MFCC feature (Mermelstein, 1976). In this paper, the role of knowledge of speech signals is to help remove unnecessary complexity, i.e., seeking the minimum structure that make reﬂect these basic properties. Finally, MLP is not new in speech processing; in fact the neural models used in early days in speech processing are all general MLPs, e.g., (Bourlard & Morgan, 2012). Speech-MLP is a special designed MLP, by taking the properties of speech signals into account.
M ETHODOLOGY
Our model, referred to as speech-MLP, is presented in Figure 1.
Note that for a given speech waveform, a sequence of acoustic features, denoted by X = { x 1 , x 2 , ..., x n } , are ﬁrst extracted.
These features are then fed into N stacked speech-MLP blocks and the output of the last speech-MLP block is a speech representation that needs to undergo task-speciﬁc layers in order to perform speciﬁc tasks, such as the ones addressed in this study: SE and KWS.
Inside of each speech-MLP block, there are three components: (1) a linear transformation for a pre-projection of the extracted acoustic features; (2) a Split & Glue layer for processing the projected acoustic features while addressing frequency asymmetry and temporal dependency, and (3) another linear transformation for post-projection of the ﬁnal representation.
Two residual connections are also adopted to encourage gradient propagation.
The ﬁrst one maps the input features onto the output of the last linear transformation (i.e., the output of the post-projection operation).
The second residual connection maps the output of the ﬁrst linear transformation (i.e., the output of the pre-projection operation) onto the output of the Split & Glue layer.
Note that normalization tech- niques are also applied to regulate the feature distribution (by layer norm) and temporal variance (by instance norm).
In the next section, we give more details on the Split & Glue layer, followed by a discussion on the normalization methods adopted in this work.
3. S PLIT & G LUE
Figure 2 depicts how the Split & Glue layer operates.
The sequence of acoustic features is denoted by X ∈ R H × T , with T and H being, respectively, the length and the number of channels of the input sequence.
The ﬁrst step is to split X into K non-overlapping chunks , as illustrated in both
Figure 1 and Figure 2.
The split referred to as X → { X 1 , .., X k , .., X K } , is performed along the channel dimension.
In our experiments, the channel dimension of each chunk is considered the same, leading to X k ∈ R H/K × T .
For each chunk, X k , a context expansion is then performed through the so-called unfolding operations.
This results in context-expanded chunks, denoted by
X k w ∈ R w k H/K × T , where w k is the size of the context window induced by the unfolding operation.
Note that the number of chunks K and the window size w k can be arbitrarily selected for each chunk.
This ﬂexibility allows us to represent multi-scale contexts by adopting different window sizes for different chunks.
In Figure 2, for instance, the input channels are split into two chunks, and the window sizes are set to 3 and 5 , respectively.
This leads to the model learning from small and large contexts simultaneously.
The unfolded chunk X kw is projected by a linear transformation, leading to a new representation for the initial chunk,
Y k ∈ R
ˆ H × T , where ˆ H could be set arbitrary and is called the number of Glue channels .
We highlight that the linear transformation used in the above chunk-wise operation is shared across all the time steps for a single chunk, and each time frame is processed independently.
This setting reduces the number of parameters and is compatible with the temporal invariance property of speech signals.
Nevertheless, different weight parameters are adopted for different chunks, to provide sufﬁcient ﬂexibility.
Finally, all the learned speech representations, Y i , are concatenated along the channel dimension, forming a glued feature matrix Y G =
{ Y 1 , Y 2 , ..., Y K } .
Following, another linear transformation is applied in order to obtain the output feature Y ∈ R H × T .
Again, the linear transformation is shared across all the time steps, to reﬂect temporal invariance.
The reader can refer to Algorithmfor a more formal description of these steps.
Input Sequence: X ∈ R H
×
T : sequence of acoustic features of T frames and H dimensions
Input Parameter: w = { w 0 , w 1 , ..., w K } : window sizes of the K chunks
Input Parameter: p = { p 0 , p 1 , ..., p K } : padding deﬁnition for the K chunks
Input Parameter: s : stride in context expansion
Output Y ∈ R H
× T : sequence of output features of T frames and H dimensions
Ensure:
H % K =
{ X 1 , ..., X K }
= chunk ( X, H, K ) (cid:46) Split X to K pieces on the channel dimension for k in range( K ) do
X kw = unfold
X k , w k , p k , s ) (cid:46) Context expansion by unfolding
Y k
= W kA
X kw + b kA
(cid:46)
Linear projection A for each chunk, where W kA =
[ ˆ H, w k × H/K ] end for
Y G =
[ Y 0 ; Y 1 , ..., Y K ] (cid:46)
Concatenate Y
k along channel dimension
Y G = GELU ( Y G )
Y = W B Y + b B (cid:46)
Linear projection B to glue the chunks, where W B =
[ H, K × ˆ H ]
N ORMALIZATIONS
Normalization plays an important role in our speech-MLP model.
We employed two normalization approaches: (1) layer normalization (LN)
(Ba et al., 2016) and (2) instance normalization (IN) (Ulyanov et al., 2016).
Layer normalization is applied across the channel dimension at each time step.
Thus, it computes statistics (mean and variance) on each column of X ∈ R H × T , and then uses these statistics to normalize the elements in the same column.
With this normalization technique, the distribution of the feature vector at each time step is regularized.
Instance normalization is used to perform per-channel normalization.
That is, the statistics are computed on each row of X ∈ R H × T and applied across the time steps to normalize the elements of each row.
Thus, the temporal variation of each channel is normalized.
Note that IN extends the conventional cepstral mean normalization (CMN) approach (Liu et al., 1993), by normalizing not only acoustic features, but also features produced by any hidden layer.
Empirically, we found that IN was only effective for the SE task while the LN was more important for the KWS task.
Therefore, we apply LN only for KWS and IN for SE.
E XPERIMENTS
[Table] We evaluate the proposed speech-MLP model in two speech processing tasks: speech enhancement and keyword spotting.
In this section, we introduce these tasks and their respective datasets, used in our experiments, followed by experimental settings, experimental results [Table] and the ablation study.
4. K EYWORD SPOTTING
Keyword spotting aims at detecting predeﬁned words in speech utterances (Sz¨oke et al., 2005;
Mamou et al., 2007; Wang, 2010; Mandal et al., 2014).
In our experiments, we explore two KWS datasets: (1) the Google speech commands V2 dataset (Warden, 2018), and (2) the LibriWords (Vygon & Mikhaylovskiy, 2021).
The Google speech commands V2 dataset (here, referred to as V2-35 ) consists of 105 , 829 utterances of 35 words, recorded by 2,618 speakers.
The training, validation and test sets contain 84 , 843 , 11 , 005 and 9 , 981 utterances respectively.
The LibriWords dataset, larger and more complex, is derived from 1000-hours of English speech from the LibriSpeech dataset (Panayotov et al., 2015).
Signal-to-word alignments were generated using the Montreal
Forced Aligner (McAuliffe et al., 2017) and are available in (Lugosch et al., 2019).
The averaged duration of the keywords are 0 .
28 seconds.
The provider deﬁned four benchmark tests, based on the
[Table]
number of target keywords: LW-10 , LW-100 , LW-1000 and LW-10000 , where the target keywords are 10 , 100 , 1 k and 10 k respectively.
More details on this dataset are presented in Appendix.
4.1. S ETTINGS
We used the same architecture in all the KWS tasks, except that the dimension of the output layer was adapted to the number of keywords, as shown in Table 1.
Note that we set the window size w to be { 3 , 7 , 9 , 11 } .
This allows us to exploit multi-scale contexts.
Additionally, we set the stride to be 1 and appropriately set the padding list p to ensure that all the expanded features are in the same length and equal to that of the input feature.
Prior to the feature extraction step, each speech recording is resampled to 16 kHz.
Then, 40dimensional Mel-Frequency Cepstral Coefﬁcients (MFCC) are attained as the acoustic features.
The MFCC features are then projected to 128-dimensional feature vector by a linear layer and then forwarded to 4 speech-MLP blocks.
The output features are then passed through a linear layer, and a max-pooling operation collects the information across time steps.
Finally, a linear layer and a softmax activation are employed in order to attain the posterior probabilities that the input speech belongs to each keyword.
For regularization we used SpecAugment (Park et al., 2019), dropout (Baldi & Sadowski, 2013), and label smoothing (M¨uller et al., 2019) were used to prevent overﬁtting.
Two model architectures have been veriﬁed in all the experiments, a 180k small model denoted by
Speech-MLP-S and a 480k large model denoted by Speech-MLP-L .
The two models are different in the number of channels of the hidden layer (i.e., after the pre-projection) and the channels within the Split & Glue block (i.e., channels after Linear A in Fig. 2), as shown in Table 1.
For the experiments on the Google speech commands dataset, we applied the following data augmentation techniques: time shifting, audio re-sampling, and noise perturbation, as in (Berg et al., 2021; Vygon & Mikhaylovskiy, 2021).
After augmentation, the data was increased to 10 times the size of V2-35 .
We set the batch size to be 512 and trained the model for 100 epochs on 8 cards V
Nvidia GPU.
For the experiments on the LibriWords, the batch size was set to 1024, and we trained the model for 20 epochs on 2 cards V100 Nvidia GPU which showed to be enough for this dataset.
The training schemes were set differently simply because Libriwords is huge and long-term training is not economic.
The performance of the proposed model is compared to three benchmarks.
The ﬁrst one referred to as Att-RNN, is a CNN-LSTM architecture with the attention mechanism introduced in (de Andrade et al., 2018).
The model has approximately 202k trainable parameters and attains reasonable performance.
Another recent solution, based on a transformer architecture is adopted as the second benchmark (Berg et al., 2021).
We refer to this benchmark as KWT-KD for when the authors used knowledge distillation and KWT-NoKD for when knowledge distillation is not used.
Res15 (Vygon & Mikhaylovskiy, 2021), another recent work based on ResNet reports high performance on both V2-35 and Libriwords.
The authors reported results with two conﬁgurations, one trained by cross entropy (Res15-CE) and the other based on triple loss (Res15-TL).
We use them as the third benchmark.
4.1. R ESULTS
Table 2 presents the results of the benchmarks discussed in the previous section and the performance of the proposed Speech-MLP under the two conﬁgurations discussed previously.
It can be observed that the speech-MLP-S and speech-MLP-L outperform all the benchmarks for all tasks.
Note that the small version of speech-MLP, which contains less than half of parameters of its large version, can still maintain reasonable performance, providing higher accuracy than the benchmarks.
The performance of our solution on the Libriword dataset is even more signiﬁcant.
It outperforms Res15CE and Res15-TL while being able to maintain performance across all LibriWord dataset sizes.
Our conjecture is that by the knowledge-driven design, we can use the parameters more efﬁciently, which allows for the use of smaller models to handle large-scale tasks.
4.1. A BLATION STUDY
To investigate how each module impacts the performance of speech-MLP, we conducted an ablation study.
We particularly focus on the chunk splitting, specially the number of chunks and the context window of each chunk.
They are the only hyperparameters that we need to design in speech-MLP, by using domain knowledge.
The results are reported in Table 3.
It can be observed that the setting for the number of chunks and the context window does matter.
A longer context window is clearly beneﬁcial, and setting different context windows for different chunks can further improve the performance.
This conﬁrms our conjecture that contextual information is important for representing speech signals, and exploiting multi-scale contextual information is especially important.
An interesting comparison is between the Speech-MLP-S model with window { 3 , 7 , 9 , 11 } and the
Speech-MLP-L model with window { 1 } .
The parameters of the two models are comparable, but the latter model does not involve any chunk splitting and context expansion.
The clear advantage of the
Speech-MLP-S model demonstrated that the performance improvement with larger and multi-scale context windows (ref.
performance of Speech-MLP-S or Speech-MLP-L with different windows) is due to the newly designed Split& Glue structure, rather than the increase in parameters.
This in turn demonstrated the value of domain knowledge: if we can exploit it appropriately, it is possible to design very parsimonious models.
[Table]
[Table]
[Table]
S
[Table] E [Table]
Speech enhancement, which aims at inferring clean speech from a corrupted version (Benesty et al., 2006; Loizou, 2007; Das et al., 2020)
[Table]
is another fundamental task used to evaluate our model.
We choose the Voicebank+Demand datasetValentini-Botinhao et al.
(2016) to perform the SE test.
It contains clean speech signals from the Voicebank dataset, includes 28 speakers for training andspeakers for testing.
Noise signals of 40 types from the DEMAND atabase Thiemann et al.
(2013) were selected and were mixed into the clean speech.
After the mixing, the training set and testing set involve 11,572 and 824 clips respectively.
We split the training utterances into segments ofseconds without overlap.
This resulted into 17,989 training samples, each sampling consisting of a noise corrupted segment and the corresponding clean segment.
The goal of SE is to learn a mapping function that converts a noisy segment to a clean segment.
4.2. S ETTINGS
The architecture of our SE model is shown in Table 1.
As input, the model receives a 257dimensional log-magnitude spectrum.
The extracted features are ﬁrst projected by a linear layer and reduced to 256-dimensional feature vector, which are then forwarded to 10 stacked speechMLP blocks.
The output from the last speech-MLP block is re-projected to 257-dimensional feature vector.
After a hard-sigmoid function Courbariaux et al.
(2015), the value of the output units correspond to the ratio masks on the 257-dimensional input log-magnitude spectrum.
The clean speech signal is estimated by applying the ratio masks onto the noisy spectrum and reusing the noisy phase.
More details of the settings can be found in Appendix.
The performance of the proposed model is compared to six benchmarks.
Note that we focus on models trained without extra data, or extra models for knowledge distillation.
The reader can ﬁnd details on these enhancement methods in the references presented in Table 4.
Following the convention on this test set, we report the results of four metrics: PESQ, BAK, SIG and OVL (Hu & Loizou, 2007).
4.2. R ESULTS
The results are shown in Table 4, where we choose 6 baseline systems for comparison.
Among these systems, T-GAS (Kim et al., 2020) is based on a transformer model.
Similar to speech-MLP, the authors of T-GAS also noticed the importance of local context and designed an annealing approach to encourage attention on neighbour frames.
However the attention is still global in nature, and the improvement with T-GAS was still attributed to the capacity of transformers in learning (not so) long-range dependency.
Note that the size of the T-GAS model was not reported in the original paper, so we made an estimation according to the structure description.
The results shown in Table 4 demonstrated that our speech-MLP model outperformed all the six baselines.
In particular, without modeling any long-range dependency, it outperformed T-GSA.
This comparative results challenge the assumption that the better performance of T-GSA over other base- lines is due to its capacity of capturing long-range dependence in speech.
Moreover, the model size of Speech-MLP is much smaller than T-GSA, and due to the concise architecture, the training is simple and fast.
It provides a strong support for our argument that complex models are not necessarily the best, and a knowledge-based model may easily beat complex models with parsimonious parameters.
C ONCLUSIONS
In this paper, we propose the speech-MLP model, a simple MLP architecture for speech processing tasks.
Our main motivation was to ﬁnd a compact solution that eliminates unnecessary complexity while being able to capture essential information from speech signals.
By utilizing domain knowledge of speech, we designed a simple yet effective structure that involves only linear transform and normalization.
The main ingredient is a split & glue structure, which splits input features into multiple chunks and makes them accounting for different contexts.
This knowledge-based design reﬂects several properties of speech signals, including temporal variance, frequency symmetry, and short-term dependency.
The experimental results on keyword spotting and speech enhancement demonstrated that speech-MLP is highly effective: with much less parameters and computation, it can beat larger and more elaborately designed models including transformers.
Although very promising, the speech-MLP has not been fully evaluated yet on tasks that require long-range knowledge such as speech recognition.
Our primary results on the Librispeech benchmark show that speech-MLP can achieve comparable performance as Jasper (a full CNN architecture) on Librispeech, but cannot beat models based on transformer and conformer. This indicates that Speech-MLP is not very suitable for tasks requiring long-range dependency modeling. Nevertheless, it can still be used as the feature encoder, and model long-range dependency using temporal models such as RNN and Transformer.
Much work remains, for example, how to design a better chunking and context; how to make the model even smaller (e.g., removing unnecessary residual connections); how to trade off the complexity in chunks and in depth. The ultimate goal is to design a light-weighted, sufﬁciently powerful and generalizable component for speech feature extraction.
R EPRODUCIBILITY S TATEMENT
We made the following efforts to ensure that the results reported in the paper can be reproduced by other researchers.
• We will release the code on github, so everyone can download • The datasets used in this paper are all publicly available for researchers • We documented the required python environment and provided a step-by-step guidance for the reproduction • We ﬁxed the random seed in the code, so that others can reproduce our result exactly.
A A PPENDIX A: D ETAILS OF KWS EXPERIMENT
In this section, we present the details of the KWS experiment.
We start with the system architecture, followed by the data preparation.
We then present the training methods and the hyperparameters used in the experiments.
[Table]
A. S YSTEM A RCHITECTURE
Prior to feature extraction [Table] speech signals are resampled to 16 kHz if needed.
Then, we use librosato extract a 40-dimensional MFCC features.
The parameters used to extract these features are presented in Table 5.
Global mean and variance is also applied to normalize the extracted features.
These statistics are calculated using the respective training set of each task.
After that, the features are fed into the model shown in Figure
[Table]
Speciﬁcally, a linear transformation (Linear 0) operates on the normalized MFCC features, projecting them to 128-dimensional embeddings.
These embeddings are then forwarded to stacked SpeechMLP blocks (4 blocks in our KWS study) to extract multiscale contextual representations.
For each speech utterance, the last Speech-MLP block outputs a sequence of context-rich representations, and then a max pooling operation is adopted to aggregate this sequence to a single utterance-level representation.
This representation is then passed to a 128 × 128 linear transformation and a GELU nonlinear activation function.
It is then further processed by a 128 × M linear transformation and a softmax nonlinear activation, where M is the number of keywords.
The ﬁnal output of the above process is a vector that represents the posterior probabilities that the original speech utterance belongs to each keyword.
A. D ATA P REPARATION A.2. G OOGLE S PEECH COMMANDS
The google speech commands V2-35 contains 35 classes.
The data can be obtained at the provider’s website 5 .
There are 84 , 843 training samples in total, with strictly no overlapping between training, validation and test sets.
tensorflow.org/ /speech_commands_v0.02.tar.gz
Data augmentation techniques have been used to increase the training data by 9 times.
Combined with the original data, we have 848 , 430 training samples in total.
We ﬁxed the random seed to be 59185 when producing the augmented samples.
Following are the augmentation strategies adopted in this work:
• Noise perturbation: the noise perturbation script provided by the organizer of the DNS challenge is used to add background noise to clean speech 6 .
The SNR factor is randomly sampled from [5 , 10 , 15] with equal probabilities; • Time shifting: time shifting is applied in the time domain.
It shifts the waveform by a timeshift factor t sampled from [ − T, T ] .
In our experiments we set T = 100 .
When t < 0 , the waveform is shifted left by t samples and t zeros are padded to the right side.
When t > 0 , the waveform is shifted right by t samples and t zeros are padded to left side; •
Resampling: the resample function from scipy (scipy.signal.resample) is used to perform resampling augmentation, which changes the sampling rate slightly.
Speciﬁcally, given a parameter R , a resampling factor r is drawn from [1 − R, 1+ R ] , and the augmented sample is obtained by changing the sampling rate to r × 16000 .
R is set to 0 .
15 in our experiments.
After the above augmentation, the original speech and the augmented speech are further corrupted by SpecAug (Park et al., 2019).
The setting of SpecAug is shown in Table 5.
Note that SpecAug does not enlarge the dataset.
A.2. L IBRIWORDS
The LirbriWords dataset is a larger and more complex dataset.
The samples are extracted from the json ﬁles provided by the providers 7 .
We follow the task deﬁnition of the dataset provider, and the details are given below.
• LibriWords 10 ( LW-10 ): this task contains 10 keywords, including “the”, “and”, “of”, “to”, “a”, “in”, “he”, “I”, “that”, and “was”.
There are 1 , 750 k samples in total, and they are spit to a training set ( 1 , 400 k), a validation set ( 262 , 512 ) and a test set ( 87 , 501 ).
• LibriWords 100 ( LW-100 ): a more challenging task that contains 100 keywords.
There are 1 , 512 k training samples, 189 , 010 validation samples and 188 , 968 samples, totalling 1 , 890 k samples.
• LibriWords 1000 ( LW-1000 ): with increased difﬁcult, this task contains 1000 keywords.
The training set involves 2 , 178 k samples, and the validation set and the test set contain 272 , 329 samples and 271 , 858 samples respectively.
• LibriWords 10000 ( LW-10000 ): The most challenging task presents 9998 keywords.
The training set contains 2 , 719 k training samples, 339 , 849 validation samples and 335 ,test samples.
Given the large number of samples, data augmentation was not required for this task.
We only performed SpecAug (Park et al., 2019) based on the settings presented in Table 5.
A. T RAINING PARAMETERS
The parameters used during training are speciﬁed in Table 5.
Further details are presented below.
• the cross entropy between the model prediction and the ground truth is used as loss function; • The optimizer used in all the experiments is AdamW.
The initial learning rate is set to 0.01, and cosine annealing is applied to adjust the learning rate from 0.01 to 0.0001; • Dropout is applied onto the residual connections within the speech-MLP block, with the dropout rate set to 0.1; • Label smoothing is employed to prevent the over-conﬁdence problem.
The smoothing factor is set to 0 . 1 ; •
In the V2-35 experiment, the models are trained for 100 epochs and 10 epochs warmup is applied,
In the LibriWords experiment, the models are trained for 20 epochs without warmup; •
In both the experiments, the model of each epoch is evaluated on the evaluation set, and the checkpoint that performs the best on the validation set is saved to report the performance on test set; • We ﬁx the random seed to be 123 in all the experiments, for the sake of reproducibility.
B A PPENDIX B: D ETAILS OF SE EXPERIMENT B. S YSTEM A RCHITECTURE
The model architecture has been presented in Figure 4.
The primary goal is to learn a mapping function that converts noisy magnitude spectrum to clean magnitude spectrum.
The model output predicts the soft ratio masks, that can be applied to the noisy magnitude spectrum to estimate
the
[Table]
mangitude spectrum of the clean speech.
Combining the denoised magnitude spectrum and the phase spectrum of the original noisy speech, one can attain the denoised waveform by inverse STFT.
More speciﬁcally, 257-dimensional log-magnitude spectrum is ﬁrstly extracted from the noisy speech as the acoustic features, following the conﬁguration shown in Table 6.
Then a linear layer follows and transfers the input features to 256-dimensional vectors PreX .
The transformed feature vectors are then forwarded to 10 Speech-MLP blocks, and the output from the last block, denoted by
PostX , involves multiscale contextual information.
Afterwards, a residual connection adds PreX and PostX together, and instance normalization is applied to regulate temporal variance.
Finally, another linear transform and a non-linear HardSigmoid activation projects the normalized feature to a masking space where the dimensionality is the same as the input feature, corresponding the ratio mask M ∈ [0 , 1] on the noisy magnitude spectrum.
B. L OSS F UNCTIONS
The loss function of our model is computed based on the discrepancy between the denoised speech
X d and the clean speech X c .
The entire loss consists of two parts: (1) the distance on powercomposed magnitude spectrum, denoted by L mag , and (2) the distance on power-compressed STFT, denoted by L stft .
We use a single frame to demonstrate this computation, where the real loss should compute the average of L on all the frames.
[Equation]
B. T RAINING PARAMETERS
The parameters for model training are summarized in Table 6.
Speciﬁcally, the model was trained for 1000 epochs using the adamw optimizer.
The initial learning was set to 0.01, and a cosine annealling learning scheduler was used to adjust the learning rate from 0.01 to 0.0001 in 3000 steps.
Warmup was applied and involved 30 epochs .
The model was evaluated on the evaluation set every epoch, and the best checkpoint (in terms of PESQ) on the evaluation set was saved.
The results are reported in terms of four metrics: PESQ, BAK, SIG and OVL (Hu & Loizou, 2007).