A UDIO
L OTTERY
: S PEECH R ECOGNITION
M ADE
U LTRA -L IGHTWEIGHT , T RANSFERABLE , AND N OISE -
R
OBUST
Anonymous authors
Paper under double-blind review
A BSTRACT
Lightweight speech recognition models have seen explosive demands owing to a growing amount of speech-interactive features on mobile devices.
Since designing such systems from scratch is non-trivial, practitioners typically choose to compress large (pre-trained) speech models.
Recently, lottery ticket hypothesis reveals the existence of highly sparse subnetworks that can be trained in isolation without sacriﬁcing the performance of the full models.
In this paper, we investigate the tantalizing possibility of using lottery ticket hypothesis to discover lightweight speech recognition models, that are (1) robust to various noise existing in speech; (2) transferable to ﬁt the open-world personalization; and 3) compatible with structured sparsity.
We conducted extensive experiments on CTC, RNN-Transducer, and Transformer models, and veriﬁed the existence of highly sparse “winning tickets” that can match the full model performance across those backbones.
We obtained winning tickets that have less than 20% of full model weights on all backbones, while the most lightweight one only keeps 4.4% weights.
Those winning tickets generalize to structured sparsity with no performance loss, and transfer exceptionally from large source datasets to various target datasets.
Perhaps most surprisingly, when the training utterances have high background noises, the winning tickets even substantially outperform the full models , showing the extra bonus of noise robustness by inducing sparsity.
All our codes will be released.
I NTRODUCTION
End-to-end automatic speech recognition (ASR) (Wang et al., 2019; Hannun et al., 2014; Graves, 2012; Chorowski et al., 2015; Dong et al., 2018a) has become an indispensable technology in consumer-interactive devices (e.g., smartphones, smart speakers, tablets) over the past few years (He et al., 2019; Cohen, 2008; Schalkwyk et al., 2010).
Conventional on-device ASR system usually requires the involvement of servers, i.e., streaming the audio to servers and then streaming the results back to the devices.
By contrast, recent studies (He et al., 2019; McGraw et al., 2016;
Kim et al., 2019; Sim et al., 2019; Park et al., 2018; Waibel et al., 2003; Arık et al., 2017) have spurred the success of ASR models fully run on devices, which can be advantageous in terms of computational resources, latency, and user data privacy.
Developing on-device ASR models are challenging since the computational resources (e.g., CPU, memory, battery) are typically very limited.
A standard design approach to ﬁt ASR model in budget is through applying various neural network compression techniques to the larger ASR models, such as network pruning (Takeda et al., 2017; Shangguan et al., 2019; Gao et al., 2020), knowledge distillation (Li et al., 2018a), and parameter quantization (He et al., 2019; Sainath et al., 2020).
However, there always exists a trade-off between computational efﬁciency and the model performance, and the efﬁciency improvements are usually at the cost of word error rate (WER).
In most prior works, we observed a non-negligible degradation of WER in compressed models.
A new horizon emerges with the discovery of lottery tickets hypothesis (LTH) (Frankle & Carbin, 2019b).
LTH empirically demonstrated the existence of highly sparse matching subnetworks (i.e., winning tickets ) in full dense networks, that can be independently trained from scratch to match or even surpass the performance of the latter.
LTH was widely observed in various models in computer vision (Liu et al., 2019b; Evci et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al.,
2020a
; You et al., 2020; Ma et al., 2021; Girish et al., 2020) and natural language processing (Gale et al., 2019; Yu et al., 2020; Prasanna et al., 2020; Chen et al., 2020b;c).
Yet to our best knowledge, it has not been studied nor utilized in the realm of speech processing and recognition
This paper presents the ﬁrst investigation on LTH for developing on-device ASR models.
Despite the rich literature of LTH in vision and language, a practically useful winning ticket for real-world ASR would demand two unique properties: transferability , noise robustness .
• As one persistent challenges of ASR, each individual has a different voice and speaking style.
Unlike text or images, whose data are much more “standardized”, the spoken word varies greatly based on regional dialects, speed, emphasis, even social class and gender.
Therefore, scaling up any ASR system has always been a signiﬁcant obstacle, since the testing utterances may have very different distribution from the training utterances.
That has made transferability a crucial demand for ASR in the open world.
• In an ideal world, one would have to speak very clearly, slowly, and in an environment with no background noise, for the sounds being recognized: that unfortunately will not happen in the practice.
In the real-world ASR applications, noise robustness is becoming another crucially demanded technological factor since ASR is now expected to work in much more difﬁcult acoustic environments than in the past (Li et al., 2014).
For example, the recognition of speech recorded by distant microphones is challenged by acoustic interference such as noise, reverberation and interference speakers (Kinoshita et al., 2020; Haeb-Umbach et al., 2019; Kinoshita et al., 2013).
Even in the standard ASR benchmark such as LibriSpeech (Panayotov et al., 2015), there are signiﬁcant background noise in its “clean” subset (Zen et al., 2019).
More importantly, prior LTH studies mostly use unstructured sparsity during model pruning.
However, designing chips that speeds up unstructured
sparse networks are much more complex than those for structured sparsity (e.g., Block-Sparse GPU Kernels (Gray et al., 2017)).
Together with the two unique gaps, they make ASR no less challenging than visual recognition or text understanding, if not more.
They account for the prior arts’ difﬁculty to maintain a satisfactory balance between model efﬁciency and recognition performance; and similarly, they question the applicability of LTH in ASR.
We provide an afﬁrmative, positive answer of LTH in ASR.
As the subject of study, we choose most commonly used ASR architectures in both research and products over the past few years:
) CNN+RNN with connectionist temporal
classiﬁcation (CTC) (Graves et al., 2006); 2) RNNTransducer (Graves, 2012); and 3) Convolution-augmented Transformer (Gulati et al., 2020).
We conducted extensive LTH experiments using these backbones on three popular corpora: TEDLIUM (Rousseau et al., 2012), Common Voice (Ardila et al., 2020), and LibriSpeech (Panayotov et al., 2015).
Unlike most of existing LTH studies that only approached to explaining and demonstrating the correctness of LTH theory, in this work, we make the ﬁrst attempt to apply LTH to real-world use cases.
Namely, we investigate three unique properties that were rarely studied in previous LTH research but are key to user-interactive ASR devices, bringing new insights to both LTH theory and lightweight ASR research.
Our main contributions are outlined below:
• We for the ﬁrst time reveal the existence of winning tickets in the context of ASR by answering three research questions regarding LTH theory.
The most lightweight winning tickets from CTC, RNN-Transducer, and Conformer backbones only possess 21.0%, 10.7%, and 8.6% remaining nonzero weights, respectively.
We also show that winning tickets signiﬁcantly outperform other state-of-the-art network pruning and knowledge distillation methods.
• We are the ﬁrst to explore the use of structured sparsity (i.e., block sparsity (Narang et al., 2017b)) in LTH, and successfully found highly sparse winning tickets (e
g., 4.4% remaining weights) that have no performance degradation compared to using unstructured sparsity.
• Winning tickets have exceptional transferability across different datasets, which are notably better than full models.
Also, the winning tickets identiﬁed from large source datasets transfer better.
In the presence of various levels of background noise, the winning tickets consistently achieve signiﬁcantly better WERs than full models.
That indicates stronger noise robustness might be an extra bonus of inducing sparsity, and a missing gem by previous LTH works.
R ELATED W ORKS
End-to-End Automatic Speech Recognition.
Previous mainstream ASR systems are mostly based on hidden Markov model (HMM)-Gaussian mixture model (GMM) or HMM-Deep neural network
(DNN).
These systems can achieve descent performance.
However, such system is usually composed of modules (e.g., acoustic model, language model, lexicon) that are needed to be trained separately, which makes it hard to be optimize globally in products (Wang et al., 2019).
End-to-end ASR can directly transcribe an input audio sequence to a token (e.g., grapheme or phoneme) sequence.
Current end-to-end ASR frameworks can be generally categorized into three types: CTC-based (Hannun et al., 2014;
Amodei et al., 2016; Graves & Jaitly, 2014; Miao et al., 2015; Eyben et al., 2009), RNN-transducer (Graves, 2012; Graves et al., 2013; Rao et al., 2017; Dong et al., 2018b), sequence-to-sequence (Seq2Seq) model (Chorowski et al., 2015; Bahdanau et al., 2016;
Chan et al., 2016; Zhang et al., 2017; Chiu et al., 2018; Prabhavalkar et al., 2018), and Transformer model (Gulati et al., 2020; Dong et al., 2018a;
Wang et al., 2020b; Baevski et al., 2020).
CTC (Graves et al., 2006) is essentially a loss function, which maximizes the probability of all the paths that correspond to the ground-truth token sequence, with an augmented blank token “-” indicating no output.
It avoids the need of segmental alignment/labels in training utterances, which makes tons of speech materials usable without additional annotating effort and thus fully exploits the modeling capacity of DNNs.
RNN-transducer is composed of an encoder for the input audio sequence, a prediction network to model the interdependencies in between the output token sequence, and a joint network to align the input and output sequence and produce the prediction.
Seq2Seq model usually has an encoder-decoder structure, where the encoder maps the input audio sequence to a hidden representation sequence and the decoder autogressively decodes the output token sequence.
An attention mechanism (Chorowski et al., 2015) is trained to learn the alignment between the input and output sequences.
Transformer models are similar to Seq2seq models but use multi-head attention (MHA) (Vaswani et al., 2017) layers for encoders and decoders, which has been proven to achieve the state-of-the-art ASR performance.
Lottery Tickets Hypothesis.
The recently emerged LTH (Frankle & Carbin, 2019b) deviates from the common wisdom of after-training pruning (Han et al., 2015), and demonstrates the existence of highly sparse subnetworks that are independently trainable from scratch, called winning tickets .
Once trained, they are capable of matching or even surpassing the performance of their full models.
Follow-up efforts (Renda et al., 2020; Frankle et al., 2020b) introduce the effective weight rewinding techniques to scale up LTH to large networks on large-scale datasets.
LTH draws wide attention from various deep learning ﬁelds, and has been studied in image classiﬁcation (Liu et al., 2019b;
Evci et al., 2019; Frankle et al., 2020c; Savarese et al., 2020; Wang et al., 2020a; You et al., 2020; Ma et al., 2021;
Chen et al., 2020a), natural language processing (Gale et al., 2019; Yu et al., 2020; Prasanna et al., 2020; Chen et al., 2020b;c), object detection (Girish et al., 2020), generative adversarial networks (Chen et al., 2021d; Kalibhat et al., 2020;
Chen et al., 2021a), graph neural networks (Chen et al., 2021b), reinforcement learning (Yu et al., 2020), and life-long learning (Chen et al., 2021c).
Several pioneer works (Mehta, 2019; Morcos et al., 2019; Desai et al., 2019; Chen et al., 2020b;a) also investigate LTH transferability across datasets and downstream tasks.
Yet to the best of our knowledge, LTH in speech models remains untouched – and that would not be a trivial extension for three aspects of reasons.
On the task level, ASR has unique demands for individual user transferrability and noise robustness, which has been explained previously.
On the model level, compared to CV models, speech models are mostly based on RNN backbones (Hannun et al., 2014; Chan et al., 2016), which contain recursive computational graphs and are notoriously unstable to train.
That makes the pruning of RNN-based models challenging too (Zhang & Stadie, 2019), and off-the-shelf methods developed for pruning CNNs are often found ineffective or even inferior to random pruning, when applied to RNNs.
Pruning methods customized for RNNs typically call for special sparse structures or stability regularizations (Narang et al., 2017a;b; Kliegl et al., 2017; Wen et al., 2018; Zhang & Stadie, 2019), and it is hence unclear whether IMP would remain stable and effective for RNN-based models in ASR.
On the data level, compared to NLP models, the sequence lengths of speech signals are usually signiﬁcantly larger than word embeddings (e.g., the spectrogram sequence length of a 10-second speech utterance extracted with 10ms shift is 1,000; the number of words in a sentence is usually between 10 and 50), which also inevitably costs higher computational complexity.
Therefore, ﬁnding sparse subnetworks that can maintain the full model performance for ASR models is practically meaningful yet highly non-trivial.
P RELIMINARIES AND S ETUPS
Backbone Network.
We investigate three ASR backbone networks that are widely used in both academia and productions: 1) CNN+LSTM (Amodei et al., 2016) model with CTC loss (Graves et al., 2006); 2) RNN-Transducer (Graves, 2012); and 3) Convolution-augmented Transformer (Conformer) (Gulati et al., 2020).
Please see Appendix A.1 for details of the backbones.
Datasets, Training, and Evaluations.
We conducted experiments on three commonly used ASR corpora:
TED-LIUM (Rousseau et al., 2012) (118 hours), Common Voice (Ardila et al., 2020) ( hours), and LibriSpeech (Panayotov et al., 2015) (960 hours).
Note that LibriSpeech has two test sets: test-clean – little noise inutterances; test-other – considerable noise in utterances.
We test the
LibriSpeech model on the two test sets hereinafter.
During training, we set the batch size to 32 and an initial learning rate to 0 .
0003 , which is annealed down by a factor of 1 . 1 at the end of each epoch.
All the models were trained for 16 epochs.
To evaluate the performance, we consider two measurements:
• Word Error Rate.
WER is the standard metric measuring the accuracy of ASR models.
WER
is computed as: WER =
( S + I + D ) /N , where S , I , D , and N denote the number of substitutions, insertions, deletions, and the total number of words, respectively.
• Number of Parameters.
The number of parameters measures the complexity of a model.
In our case,
since all subnetworks were pruned from the full models, we use the percentage of Remaining Weights as an alternative measurement.
We deﬁne Sparsity as Sparsity (%) = 100% − Remaining Weights (%) .
Subnetworks.
For a dense model f ( x ; θ ) , its subnetworks can be derived as f ( x ; m (cid:12) θ ) with a binary pruning mask m ∈ { 0 , 1 } d , where (cid:12) is the element-wise product and d is the dimension of pruneable model parameters.
We use A D t ( f ( x ; θ )) to represent the training algorithm
(e.g.,
Adam (Kingma & Ba, 2017) with grid searched hyperparameters) that trains a network f ( x ; θ ) on a dataset D (e.g., LibriSpeech) for t iterations.
Let θ 0 be the random initialized network weights.
Subnetwork Evaluation.
To measure the generalization ability of obtained subnetworks, we deﬁne
E D ( A D t ( f ( x ; θ ))) as the evaluation function of model f returned from A D t on the dataset D .
Then, we further introduce:
(cid:63) Matching subnetworks.
Following the deﬁnition in (Frankle et al., 2020a; Chen et al., 2020b;a), a subnetwork f ( x ; m (cid:12) θ ) is matching if it satisﬁes the following condition that indicates
matching subnetworks
achieve
no worse performance
than
its dense counterpart
under the
same training
algorithm A D t and evaluation metric E D : E D (cid:0) A D t ( f ( x ; m (cid:12) θ )) (cid:1) ≥ E D (cid:0)
A D t ( f ( x ; θ 0 )) (cid:1) .
(cid:63) Winning ticket.
f ( x ; m (cid:12) θ ) is a winning ticket for A D t , if it is ( i ) a matching subnetwork and ( ii ) θ
= θ 0 for A D t .
(cid:63)
Transferable
Winning ticket.
A subnetwork f ( x ; m (cid:12) θ ) is transferable to target datasets {D i } Ni =
if and only if it is a winning ticket for each A D i t i .
The subnetwork f ( x ; m (cid:12) θ ) is derived from the source dataset D s / ∈
{D i } Ni =1 .
Pruning Method for Subnetwork Searching.
Iterative weight magnitude pruning (IMP) is the widely used algorithm in previous LTH literature (Frankle & Carbin, 2019a; Frankle et al., 2020a;
Chen et al., 2020b).
To identify subnetworks f ( x ; m (cid:12) θ ) , IMP performs following three steps: (1) training a unpruned dense network to completion on a dataset D (i.e., applying A D t ); (2) eliminating a portion of insigniﬁcant weights with the globally smallest magnitudes (Han et al., 2015; Renda et al., 2020) so that the model only has s i % of weights remaining (i.e., the sparsity); (3) rewinding model weights to θ ( θ = θ 0 , the original random initialization; or θ = θ pre , the weights from a pre-trained model) and ﬁnetuning the subnetwork to converge by leveraging A D t .
Note that steps (2) and (3) usually needs to be iteratively repeated for several rounds for ﬁnding highly competitive winning tickets.
In all experiments, we set s i %
=
(1 − 0 .
8 i ) × 100% , where i is the number of iterations.
T HE E XISTENCE OF W INNING T ICKETS IN S PEECH R ECOGNITION
In this section, we explore the existence of winning tickets in the three ASR backbones.
Namely, we would like to answer the following research questions from an empirical perspective:
• RQ1: Can we ﬁnd winning tickets f ( x ; m IMP (cid:12) θ ) for speech recognition model using IMP?
How much do they improve model complexity?
• RQ2: Do winning tickets identiﬁed by IMP have less complexities or better performance, compared to random pruning/random tickets and other compression methods?
• RQ3: Instead of using randomized weights θ 0 as the initialization of IMP, does it improve the performance of winning tickets if we use weights θ pre from a pre-trained model?
[Table] [Table]
RQ1: Does winning tickets exist in speech recognition models?
To answer the questions, we conducted experiments on three backbones and three datasets.
For each trial, we ﬁrst run IMP to extract a binary pruning mask at each sparsity.
Then, we generate one subnetwork at each sparsity by applying the corresponding mask to the model and reset the weights to the original random initialization θ 0 .
Finally, we train each subnetwork and computes their WER on the test set to determine if they are winning tickets.
All the training hyperparameters in training a subnetwork are the same as those in training the full model.
As shown in Table 1, winning tickets can be identiﬁed on all three backbones.
The most lightweight winning tickets on CTC, RNN-Transducer, and Conformer have 21.0%, 10.7%, and 8.6% remaining weights, respectively.
In addition, we noticed that the RNN-Transducer and Conformer subnetworks at the extreme sparsity have less percentage of remaining weights than CTC subnetworks, likely due to these models being more over-parameterized (CTC: 86.62M parameters; RNN-Transducer: 132.23M; Conformer: 118.40M).
These results show that, for a ﬁxed dataset, the winning tickets extracted from larger models are sparser.
Similarly, winning tickets can also be identiﬁed on all three datasets, as shown in Table 2.
We also found that the sparsity of a winning ticket is correlated to the size of the dataset.
For example, TEDLIUM has a relatively small size (118 hours) compared to CommonVoice (582 hours) and LibriSpeech (960 hours).
Accordingly, the remaining weights of TED-LIUM winning ticket are signiﬁcantly lower than that of CommonVoice and LibriSpeech winning tickets.
A possible explanation is models become relatively more overparameterized for smaller training sets, which allows them to be more amenable for sparsiﬁcation (Li et al., 2020).
Similar observations can be found in (Chen et al., 2020b).
From Table 1 and 2, another interesting ﬁnding is that the subnetworks with low sparsity (remain most of the weights) always achieve preferable performance than the full model.
We also provided a visualization of the outputs from the full model, the most sparse subnetwork, and the best performing subnetwork in Figure 1 (see Appendix A.4 for more examples.).
Similarly, we observed larger performance improvement on smaller datasets, possibly also due to the over parameterization issue.
Results indicate that LTH is also a potential research direction in improving overall ASR performance.
RQ2: Does IMP winning tickets have lower complexity or better performance than random pruning/tickets and other compression methods?
As suggested in previous studies (Frankle &
Carbin, 2019a; Chen et al., 2020b; 2021d; 2020a), the two key aspects for a winning ticket to achieve the desired performance are: 1) initial weight θ , and 2) mask generated from IMP m IMP .
In this subsection, we test if this argument hold in ASR winning tickets.
To achieve this, we compare the winning tickets against two baseline pruning approaches: random pruning and random tickets.
The subnetworks identiﬁed with random pruning are initialized with θ
but the masks are randomly generated m RAND .
By contrast, the subnetworks identiﬁed with random tickets are randomly initialized θ (cid:48)
but the masks are generated by IMP m IMP .
As a result, with these two comparisons, we can explore if the two aspects are necessary to a winning ticket.
We tested the three pruning approaches on TED-LIUM dataset.
Figure 2 shows the WER curves of the three approaches.
From the ﬁgure, we found that random pruning and random ticket could ﬁnd winning tickets, but the extreme remaining weights are much higher than the one identiﬁed by IMP (IMP: 4.4%, Random Pruning: 26.2%, Random Ticket: 10.7%).
Random pruning can roughly retain its performance when the remaining weights are larger than 26.2%.
However, the performance drops dramatically as the remaining weights continue decreasing.
Random ticket achieves very similar performance to IMP when the remaining weights are larger than 10.7%, but it degrades much faster than IMP after this point.
Results veriﬁed the previous statement in ASR models that both initial weight θ and mask generated from IMP m IMP are necessary for winning tickets.
Additionally, the comparison between random pruning and random ticket suggests that m IMP is more important than θ for winning tickets.
[Table]
Additionally, we compared our proposed approach against four state-of-the-art neural network compression baselines: 1) Standard Pruning (Han et al., 2015;
Blalock et al., 2020; Shangguan et al., 2019); 2) TutorNet (Yoon et al., 2021); 3) MLKD + multi-teacher
(Li et al., 2021); 4) Sequence-level KD (Takashima et al., 2018).
Baseline 1 is the most commonly used and best performing network pruning approach, which iteratively prunes the lowest magnitude weights and train the network until reaching the target sparsity (without any rewinding).
Baseline 2, 3, and 4 are three recent knowledge distillation approaches.
As shown in Table 3, our winning ticket at extreme sparsity (Proposed ext ) achieves superior WER than all the baselines, while using less parameters.
RQ3: Can IMP ﬁnd better subnetworks by initializing from a pre-trained model?
Training
ASR model from a good initialization usually results in more satisfactory performance (Jaitly et al., 2012), which is commonly achieved by ﬁnetuning a pre-trained model in practice.
Being aware of this, we would like to investigate if IMP can ﬁnd better subnetworks for ASR by initializing with weights from a pre-trained model θ = θ pre .
To verify this, we ran IMP on CNN+LSTM backbone with TED-LIUM dataset, where the weights were initialized from either LibriSpeech pre-trained model θ pre = θ Libri or CommonVoice pre-trained model θ pre = θ CV .
WER curves of the subnetworks in this experiment are shown in Figure 3.
Both initializing from θ Libri and θ CV signiﬁcantly improves the performance of the subnetworks at any sparsity.
More importantly, we found a rapid WER increase when the remaining weights are less than 4.4% for
IMP initialized from θ 0 .
In contrast, the WERs of subnetworks in θ Libri and θ CV systems degrades much slower when the remaining weights are extremely low, indicating that the subnetworks that are identiﬁed from pre-trained models can utilize the parameters more efﬁciently.
A possible reason is that the ﬂatter loss surfaces of the pre-trained models (Liu et al., 2019a) make it more traceable for compression, leading to highly quality subnetworks, as also evidenced in (Chen et al., 2020b;a).
[Table]
Table 4 shows quantitative results of these systems.
Initializing from θ Libri achieves the best performance among all three systems.
The
WERs of subnetworks initialized from either θ Libri or θ CV at different sparsities varies marginally, which makes it hard to ﬁnd the winning ticket at extreme sparsity.
Therefore, we show the WER of all the system with 4.4% remaining weights instead, corresponding to the extreme sparsity of the subnetworks initialized with θ 0 .
Although WER 4 .
4% of θ Libri and θ CV systems are equal or higher than WERs of the corresponding full models, they are still signiﬁcantly lower than θ 0 system.
Additionally, the remaining weights of best performing subnetworks in θ Libri and θ CV systems are higher than θ 0 , however, when reducing the amount of remaining weights, the
WERs only have minimal degradations.
Summary.
We conducted extensive experiments to answer the three research questions about the existence of winning tickets in three ASR backbones.
First, our results veriﬁed the existence of winning tickets in CTC, RNN-Transducer, and Conformer models, even at high sparsity (e.g., 4.4% remaining weights).
Second, we compared IMP with random pruning/tickets and other state-of-the-art network compression methods.
The results suggest that matching subnetworks extracted by IMP signiﬁcantly exceed those extracted by random pruning and random tickets in all measurements, which corroborates that both binary pruning mask and weight initialization are indispensable in ﬁnding the winning tickets.
Our approach also signiﬁcantly outperforms other compression approaches, achieving state-of-the-art performance on ASR model compression.
Lastly, we explored the use of weights from pre-trained models to initialize IMP.
We found that, in this way, IMP can identify more effective winning tickets compared to random initialization, and more importantly, these winning tickets have a considerably higher parameter efﬁciency.
These results collectively advocate the profound beneﬁts that LTH can bring to both server-side and on-device ASR.
T OWARDS P RACTICAL ASR WITH W INNING T ICKETS : S TUDYING S TRUCTURED S PARSITY , T RANSFERABLITY , AND N OISE R OBUSTNESS
After we proved the existence of winning tickets in ASR models, three more properties remain to be veriﬁed: structured sparsity, transferability, and noise robustness, which are key to ASR applications.
Study of structured sparsity As we mentioned earlier, designing chips that speeds up unstructured sparse networks are much more complex than those for structured sparsity (e.g., Block-Sparse GPU
Kernels (Gray et al., 2017)).
Therefore, it is critical to verify if we can also identify winning tickets using structured pruning.
However, the exploration of structured pruning in prior LTH studies is very limited.
(You et al., 2020) is only able to identify limited winning tickets with structured sparsity at low sparsity levels ( 70% remaining weights), which again emphasizes the difﬁculty in ﬁnding computation-friendly sparse patterns.
We also show the results with unstructured sparsity as a reference
[Table]
In this experiment, we applied block sparsity (Narang et al., 2017b; Shangguan et al., 2019; Wu et al., 2021) with 1 × 4 block to subnetwork searching, and then evaluated the subnetworks to see if they can match the full model performance.
Results in Table 5 show that using block sparsity during subnetwork searching process lead to comparable results to using unstructured pruning.
Also, we visualized the weights of winning ticket models that were discovered with unstructured sparsity and block sparsity, as shown in Figure 4.
We can clearly observe 1 × 4 block patterns from the weight matrix pruned with block sparsity.
Study of Transferability In practical scenarios, the testing utterances are directly recorded from users in the wild, which may have very different distributions from the training utterances.
A common way to address this issue is through speaker adaptation (Gauvain & Lee, 1994; Leggetter & Woodland, 1995; Woodland, 2001; Liao, 2013;
Li et al., 2018b; Meng et al., 2019; Weninger et al., 2019), which is usually achieved by ﬁnetuning a pre-trained model on utterances whose distributions are similar to test utterances directly, or with additional loss terms (Li et al., 2018b; Meng et al., 2019; Ding et al., 2020).
In the context of winning tickets, these sparse architectures have to be transferable to new datasets so that we are able to perform speaker adaptation on these models.
To examine the transferability, we conducted the following experiment: First, we identify subnetworks f ( x ; m IMP (cid:12) θ 0 ) on a source dataset D s at different sparsities using IMP.
Then, we re-train each subnetwork on a target dataset D t and evaluate their performance.
We tested the transferability between TED-LIUM, CommonVoice, and LibriSpeech datasets.
These datasets are different in terms of recording scenario, noise level, speaker coverage, training set size, etc., and therefore, their utterances have very difference distributions.
For example, TED-LIUM is created using TED talks, CommonVoice is composed of volunteer’s voice recorded through website or mobile apps, and
LibriSpeech is extracted from LibriVox audio books.
Figure 5 shows the performance of subnetworks transferring to the three target datasets.
Results on the three datasets consistently suggest that the winning tickets are transferable across different datasets.
When the remaining weights are larger than around 26.2%, the transferring tickets can generally replicate the performance of the winning tickets identiﬁed on the target datasets.
However, at extremely high sparsity, the performance of transferring tickets degrades faster than the winning tickets identiﬁed on the target datasets, corresponding to the observations from previous studies (Chen et al., 2020c; Morcos et al., 2019; Chen et al., 2021d).
For example, Table 9 in Appendix A. shows WER and remaining weight of CommonVoice and LibriSpeech winning tickets transferring to TED-LIUM dataset at extreme sparsity.
Although the remaining weights of transferring winning tickets are higher than TED-LIUM winning ticket, they are still signiﬁcantly lower than the full model.
We also noticed is that the winning tickets identiﬁed from a larger dataset usually have better transferability, which is in accordance with (Morcos et al., 2019).
Study of
Noise Robustness
The training/adaptation speech utterances are mostly collected from users, which are usually recorded from uncontrolled environments with notable background noise.
Even in standard ASR benchmarks such as LibriSpeech (Panayotov et al., 2015), there are signiﬁcant background noise even in its “clean” subset (Zen et al., 2019).
To test the noise robustness of winning tickets, we conducted an experiment on TED-LIUM dataset.
Namely, we re-trained the winning tickets identiﬁed from TED-LIUM, CommonVoice, and LibriSpeech on resynthesized TED-
LIUM dataset with background noise.
We used DESED dataset (Serizel et al., 2020) as the noise source, which consists of various background sounds in domestic environments such as bell, animals, running water, speech, vacuum cleaners
[Table]
etc.
During resynthesis, the noise audio is added to the original speech utterance with a random noise level, which is drawn from a uniform distribution from [0 , N max ] .
In addition, we considered three different noise levels: N max = 0 (no noise), N max = 0 .(low noise), N max = 0 .
5 (high noise)
[Table] [Table]
Table 6 shows WERs on TED-LIUM test set of the re-trained winning tickets.
First, we found the performance of the full model is dramatically susceptible to background noise.
When the noise level is increase from N max = 0 to
N max = 0 . 5 , WER increases from 15.93% to 38.21%.
In contrast, WERs of TED
-LIUM
win-
ning ticket
at
different
noise level
are
very close,
indicating that
the sparse subnetwork is sub-
staintially more robust
to
noise.
Although the winning tickets from CommonVoice and LibriSpeech cannot reach the WER of TED-LIUM
winning tickets on noise data, their WERs are still signiﬁcantly lower than full model.
More essentially, we can adapt transfer tickets to any target users without the need of re-ﬁnding winning tickets, which is more feasible for practical scenarios.
Our ﬁndings coincide with the ones in (Ye et al., 2019; Gui et al., 2019) that an appropriate sparsity connectivity severs as implicit regularization for network training, which produces ameliorated generalization ability on shifted data distributions such as noisy or perturbed data.
Summary
In this section, we examined the structured sparsity, noise robustness, and trasferability of the winning tickets.
Throughout the studies, we found that winning tickets can generalize to structured sparsity with no performance degradation.
In addition, winning tickets identiﬁed from source datasets can achieve matching performance on target datasets, which veriﬁes the transferability.
Lastly, winning tickets (identiﬁed from either target dataset or source datasets) are signiﬁcantly more robust to noise compared to full models, especially when the noise level is high.
These results jointly demonstrate the eligibility and beneﬁts of winning tickets in on-device ASR applications.
C ONCLUSIONS AND F UTURE W ORKS .
In this work, we examine and leverage lottery ticket hypothesis in speech recognition for the ﬁrst time.
Our extensive results show that the winning tickets are not only ultra-lightweight, but also highly transferable and (even more) noise-robust, compared to the full models.
These results collectively propound the use of LTH into ASR models, bringing new insights to both LTH theory and portable
ASR research.
The future works to this study is to evaluate the actual hardware beneﬁts of our found winning tickets.
For example, Block-Sparse GPU Kernels (Gray et al., 2017) can achieve over 5x speed up with over 80% block sparse network, and XNNPACK (Elsen et al., 2020) can speed up networks at 70%-90% unstructured sparsity on smartphone processors, which applies to our found winning tickets.
In addition, we also would like to generalize LTH compression paradigm to other speech tasks such as text-to-speech synthesis (Wang et al., 2017; Shen et al., 2018) and voice-to-voice conversion
(Jia et al., 2019; Biadsy et al., 2019; Zhao et al., 2019).
These new directions require further customized studies and point to new opportunities for LTH in speech research.
R EPRODUCIBILITY S TATEMENT
To ensure reproducibility, we the Machine Learning Reproducibility Checklist v2.0, Apr.2020 (Pineau et al., 2021).
• For all models and algorithms presented, – A clear description of the mathematical settings, algorithm, and/or model.
We clearly describe all of the settings, formulations, and algorithms in Section 3.
– A clear explanation of any assumptions.
We do not make assumptions.
– An analysis of the complexity (time, space, sample size) of any algorithm.
We use the number of parameters as the spatial complexity of our models, as shown in Table 1.
• For any theoretical claim , – A clear statement of the claim.
We do not make theoretical claims.
– A complete proof of the claim.
We do not make theoretical claims.
• For all datasets used, check if you include: – The relevant statistics, such as number of examples.
We used publicly available speech corpora in all of our experiments, and we cite the papers of them following the authors instructions, and we provide related statistics in Section 3.
– The details of train/validation/test splits.
We use the default train/validation/test splits of the three public datasets, as described in Section 3.
– An explanation of any data that were excluded, and all pre-processing step.
We did not exclude any data.
Our pre-processing steps are explained in A.1.
– A link to a downloadable version of the dataset or simulation environment.
All the datasets are publicly available, and their downloadable links can be easily found at their original papers that we cite in Section 3.
– For new data collected,a complete description of the data collection process, such as instructions to annotators and methods for quality control.
We do not collect or release new datasets.
• For all shared code related to this work, check if you include: – All of our code will be released after the paper was peer-reviewed, as we mentioned in Abstract.
We also include the key frameworks and libraries that we used during implementation in Appendix A.2.
• For all reported experimental results , check if you include: – The range of hyper-parameters considered, method to select the best hyperparameter conﬁguration, and speciﬁcation of all hyper-parameters used to generate results.
We provide all details of the hyper-parameter we used in in Sectionand Appendix A.1.
We did not perform any hyper-parameter tuning in this work since the datasets we used are with large scale, and performing hyper-parameter tuning on large scale datasets are extremely time- and resource-consuming.
– The exact number of training and evaluation runs.
One run is conducted for each experiments.
– A clear deﬁnition of the speciﬁc measure or statistics used to report results.
We use the word error rate (WER) as the performance metric and the remaining weights (RW) as the complexity metric on test sets, – A description of results with central tendency (e.g. mean) & variation (e.g. error bars).
We do not report error bars.
– The average runtime for each result, or estimated energy cost.
We do not report the running time or energy cost.
– A description of the computing infrastructure used.
All detailed descriptions are presented in Section 3 and Appendix A.1.
A A PPENDIX A. D ETAILS IN BACKBONE ARCHITECTURES
Data pre-processing.
The input to all the backbone networks is a sequence of 161-dimensional magnitude spectrogram (160 magnitude points plus 1 DC component), extracted with 20ms window, 10ms shift, and 320-point fast fourier transform.
The ground-truth labels are represented bygraphemes, including 26 English characters, space, and apostrophe symbol (an extra blank token “-” is also used for CTC backbone).
CTC.
The network is composed of two convolutional layers and ﬁve recurrent layers, followed by a ﬁnal fully-connected layer.
The ﬁrst convolutional layer has 32 41 × 11 (in frequency × time) kernels, with 2 × 2 stride.
The second convolutional layer has 32 21 × 11 kernels, with 2 × 1 stride.
For each convlutional layers, tanh is used as nonlinearity.
Following these, there are ﬁve bi-directional RNN layers, each of them having 1,024 units.
A ﬁnal fully-connected layer with softmax operator outputs a probability distribution over characters.
The total number of parameters in the full model is 86.62M.
RNN-Transducer.
In RNN-Transducer, we set the encoder and decoder to have 5 and 1 bidirectionalLSTM layers, each of them having 1,024 units.
The total number of parameters in the full model is 132.23M.
Conformer.
In Conformer, we used the same hyper-parameters as the Conformer(L) in [4].
The model has 17 encoder layers, each of them is composed of 8-head convolution augmented attention.
The convolution kernel has a shape of 1 × 32, and the fully-connected layers in attention is 512dimensional.
The decoder layer is the same as RNN-Transducer, with 1,024 units.
The total number of parameters in the full model is 118.40M.
A. M ORE IMPLEMENTATIONAL DETAILS
As we mentioned in abstract, we will open-source all of the code once the paper is peer-reveiwed.
Here, we provide key references and tools that we used during our implementation.
We implement neural network training using PyTorch library (Paszke et al., 2019).
Our implementation of CTC backbone is based on:
• https://github.com/SeanNaren/deepspeech.pytorch
Our implementation of RNN-Transduce and Conformer is based on:
• https://github.com/openspeech-team/openspeech
We used PyTorch pruning library for unstructured sparsity:
• https://pytorch.org/docs/stable/generated/torch.nn.utils.
prune.global_unstructured.html
We implemented pruning functions for structured sparsity based on:
• https://pytorch.org/docs/stable/generated/torch.nn.utils.
prune.RandomStructured.html • https://github.com/openai/blocksparse • https://www.tensorflow.org/model_optimization/guide/pruning
A. E VALUATING WINNING TICKETS OF THREE ASR BACKBONES ON L IBRI S PEECH TEST - OTHER SUBSET
The performance of the three backbones on LibriSpeech test-other subset is shown in Table 7.
[Table]
[Figure]
M ORE E XAMPLES OF THE M ODEL O UTPUTS
We provided a couple more examples of the outputs from full model, winning tickets at extreme sparsity and best performing winning tickets on TED-LIUM test utterances in Figure 6.
The examples in the ﬁrst row show the cases where the winning tickets avoid the errors appeared in the full model outputs.
Meanwhile, we also noticed a few cases where winning tickets produce erroneous recognition results, as shown in the second and the third rows.
Most of these utterances have either a very fast speaking rate or unclear pronunciations of words.
Therefore, neither full model nor winning tickets reasonably transcribe the utterances in challenging cases, showing the limited capability of the models.
A. C OMPARISON TO OTHER
TEST - OTHER SUBSET
Comparison between the proposed approach and state-of-the-art distillation and pruning methods on
LibriSpeech test-other subset is shown in Table 8.
Sequence-level KD approach (Takashima et al., 2018) does not report their evaluation results on test-other subset, and therefore, we use “N/A” in the table.
[Table]
M ORE R ESULTS IN T RANSFERRING S TUDY
We have shown the WER curves of the transferring winning tickets transferred in Figure 5.
Here, we included the WER and RW at the extreme sparsity in Table 9, Table10 and Table
[Table]
respectively.
The transferring tickets to both TED-LIUM and CommonVoice have a extreme remaining weights of 32.8%, and those to LibriSpeech have a extreme remaining weights of 32.8% compared to the full models.
Although they are not as sparse as the winning tickets identiﬁed from the target datasets, the transferring tickets are still much more efﬁcient than full models.
These results are in consensus with our ﬁndings in Section 5. [Table] [Table] [Table]
A. E XTRA NEW RESULTS A.7. R ESULTS ON THE EXISTENCE OF THE THREE BACKBONES ON TED-LIUM AND C OMMON V OICE
[Table]
[Table]
.7. R UN TIME EVALUATIONS
[Table]
[Table]
[Table]
[Table]
. U PDATED NOISE ROBUSTNESS RESULTS
[Table]
B LOCK SPARSITY EXPERIMENTS WITH LARGER BLOCK SIZE
[Table]
A
P SEUDO - CODE OF LTH PRUNING ALGORITHM A.7. F URTHER EXPLANATION OF BLOCK SPARSITY . In terms of block sparsity, it prunes a block of parameters (e.g., 1x4 block) instead of just one parameter, which has the minimal magnitude.
Unstructured pruning could be thought as block sparsity with 1x1 block.
A.7. E XPLANATION OF THE CHOICE OF SPARSITY LEVELS
In our current setting, we prune out the 20% of remaining weights that have the lowest magnitude.
The reason for choosing the value to be 20% is based on considerations of model performance and computation resources.
When the amount of weights pruned at each iteration becomes larger, less iterations will be needed.
However, it is more likely to have larger performance regressions, since the pruned weights could be useful in the subnetwork at higher sparsity.
On the other hand, if the amount of weight pruned at each iteration is too small, the number of iterations required would be very large, which is extremely resource demanding.
As a result, we feel 20% is a good one considering both factors.