SPIRAL: S ELF - SUPERVISED P ERTURBATIONI NVARIANT R EPRESENTATION L EARNING FOR S PEECH P RE -T RAINING
A BSTRACT
We introduce a new approach for speech pre-training named SPIRAL which works by learning denoising representation of perturbed data in a teacher-student framework.
Speciﬁcally, given a speech utterance, we ﬁrst feed the utterance to a teacher network to obtain corresponding representation.
Then the same utterance is perturbed and fed to a student network.
The student network is trained to output representation resembling that of the teacher.
At the same time, the teacher network is updated as moving average of student’s weights over training steps.
In order to prevent representation collapse, we apply an in-utterance contrastive loss as pre-training objective and impose position randomization on the input to the teacher.
SPIRAL achieves competitive or better results compared to state-of-theart speech pre-training method wav2vec 2.0, with signiﬁcant reduction of training cost (80% for B ASE model, 65% for L ARGE model).
Furthermore, we address the problem of noise-robustness that is critical to real-world speech applications.
We propose multi-condition pre-training by perturbing the student’s input with various types of additive noise.
We demonstrate that multi-condition pre-trained SPIRAL models are more robust to noisy speech (9.0% - 13.3% relative word error rate reduction on real noisy test data), compared to applying multi-condition training solely in the ﬁne-tuning stage.
Source code is available 1 .
INTRODUCTION
Industrial-scale automatic speech recognition (ASR) systems are usually trained with ten-thousands of hours of hand-transcribed speech data (Galvez et al., 2021).
However, labeling speech data is expensive and time-consuming, especially for languages with small speaker populations, or for speciﬁc domains (e.g., legal, ﬁnancial, scientiﬁc).
Recently, methods of utilizing unlabeled speech data to improve speech recognition system have achieved remarkable progress.
Amongst them, self-training (Manohar et al., 2015; Kahn et al., 2020a; Synnaeve et al., 2020a; Chen et al., 2020b; Xu et al., 2020; Park et al., 2020b; Xiao et al., 2021), also known as pseudo-labeling , starts by training an ASR model with labeled speech data, which is referred to as teacher model.
Then the teacher model, usually combined with a language model (LM), is used to produce pseudo-labels for unlabeled speech data.
Finally, the labeled data and the pseudo-labeled data are combined to train a new model, which is referred to as student model.
The process is repeated by taking the student model as the teacher in next iteration.
Another line of work is speech pre-training (van den Oord et al., 2019; Chung & Glass, 2020; Wang et al., 2020; Baevski et al., 2020b; Liu et al., 2020).
Pre-training learns speech representation from unlabeled data in a self-supervised way.
The pre-trained model is then ﬁne-tuned on the labeled data.
Self-training and pre-training are complementary as shown in recent work (Xu et al., 2021;
Zhang et al., 2020).
In this paper, we introduce a new speech pre-training method which works by learning denoising representation of perturbed data with the teacher-student framework, named as Self-supervised
Perturbation-Invariant Representation Learning (SPIRAL).
Compared to state-of-the-art speech pretraining methods such as wav2vec 2.0 (Baevski et al., 2020b) and HuBERT (Hsu et al., 2021), our method allows end-to-end training with a single contrastive loss, and without relying on discrete unit discovery techniques such as vector quantization (J´egou et al., 2011; Baevski et al., 2020a;b) or iterative clustering process (Hsu et al., 2021).
We apply multi-condition training with SPIRAL (Seltzer et al., 2013;
Ko et al., 2015) to improve noise-robustness for the downstream speech tasks.
SPIRAL is motivated by the observation that human tolerates speech perturbations or distortions fairly well.
For example, people can communicate effectively in a noisy environment, or over a distorted telephone channel.
Therefore, we hypothesize that by learning representation invariant to perturbation, the model will learn high-level representation which can enhance speech applications.
To learn perturbation-invariant representation in a self-supervised way, we employ a teacher-student framework similar to Tarvainen & Valpola (2017).
During pre-training, given a speech utterance, we guide the student network which consumes the perturbed utterance to learn from the teacher network which consumes the clean utterance.
The student is trained to produce denoised representation of the perturbed utterance similar to teacher’s representation of the clean utterance.
Meanwhile, the teacher, which shares the same model architecture with student, is updated as moving average of the student’s weights over past training steps.
We apply the in-utterance contrastive loss to avoid model collapse to trivial constant representation (Chopra et al., 2005).
As speech utterance are sequential data, there is another possible trivial solution which we call positional collapse .
Positional collapse occurs when the student “cheats” by exploiting position correlation in teacher’s representation to minimize the loss, while ignoring the content of the input utterance.
To prevent positional collapse, we propose position randomization by adding random number of paddings on both sides of input utterance to the teacher.
Large-scale speech pre-training is computationally demanding.
To reduce computation cost, we adopt a gradual down-sampling strategy in SPIRAL model, which has been veriﬁed effective in speech recognition literatures with negligible performance degradation (Peddinti et al., 2018; Han et al., 2020b; Huang et al., 2020).
We also speculate that aggressive down-sampling helps to remove redundancy in speech.
To evaluate the effectiveness of SPIRAL, we conduct experiments on LibriSpeech and Libri-Light datasets.
By training a small convolutional classiﬁer on the representation of a frozen SPIRAL model, we can achieve WER of 3.5% and 6.4% on Librispeech test-clean and test-other respectively.
SPIRAL achieves competitive or better results compared to state-of-the-art speech pre-training methods, while being much more training-efﬁcient.
We also demonstrate that multi-condition pretrained SPIRAL are more robust to noisy speech with 9.0% - 13.3% relative word error rate (WER) reduction on real noisy test data from ChiME-3 (Barker et al., 2015), compared to the model applying multi-condition training solely in ﬁne-tuning stage.
R ELATED WORK
Mean Teacher (MT) (Tarvainen & Valpola, 2017) proposes using a student network to learn from a teacher network which is the moving average version of the student in the semi-supervised learning setting.
The authors apply a supervised loss for labeled data and a consistency loss between teacher and student predictions for unlabeled data.
However, direct application of MT to self-supervised learning leads to representation collapse (Grill et al., 2020).
Noisy student training (NST) (Xie et al., 2020; Park et al., 2020b) is a self-training method.
NST demonstrates the importance of the aggressive injection of noise into the student.
Although not emphasized, no noise is injected into pseudo-labeling process of the teacher.
We consider our work as an extension of self-training approach to the self-supervised learning regime.
Instead of using the teacher to provide pseudo-labels, we utilize the teacher for pseudo-reference representation.
Denoising autoencoders (Vincent et al., 2008) learn to recover a clean input from a corrupted version.
However, speech data contain redundancy which is irrelevant to some speech applications such as speech recognition.
Previous work (Baevski et al., 2019) shows that speech pre-training by recovering masked input speech features is not effective.
In SPIRAL, we instead enforce latent representation of a corrupted input to resemble that of the corresponding clean input.
Bootstrap Your Own Latent (BYOL) (Grill et al., 2020) is a self-supervised image representation learning method.
The method is based on a teacher-student framework similar to MT.
The authors refer to student network as online network and teacher network as target network.
They observe that naive application of MT to self-supervised learning leads to trivial constant representation.
They prevent the representation collapse by appending a predictor to the student network.
The theory behind is under investigation (Chen & He, 2021; Tian et al., 2021).
Our method draws inspirations from BYOL and shares the similar architecture, but there are crucial differences.
Instead of learning a single global representation for an image as in BYOL, SPIRAL learns a sequence of representation for an utterance.
We aim for sequence applications such as speech recognition.
In our preliminary experiments, we observe that appending a predictor to student network is not sufﬁcient to prevent trivial constant representation for sequential representation learning.
We use in-utterance contrastive loss (Baevski et al., 2020b)
combined with input position randomization to successfully avoid representation collapse.
We still keep the predictor in SPIRAL, but only for the sake of performance improvement from our observation.
Another difference is that BYOL does not perform representation denoising.
BYOL applies perturbation, which they call augmentation, to both the inputs of the teacher and the student.
We demonstrate that representation denoising is crucial for speech pre-training.
When perturbation is applied to the teacher’s input, the effectiveness of speech pre-training degrades drastically.
Wav2vec 2.0 (Baevski et al., 2020b) is a self-supervised speech representation learning method which belongs to the masked prediction family.
Masked prediction methods are effective for text pre-training
(Devlin et al., 2019), but not for speech pre-training when naively applied (Baevski et al., 2019).
The reason is that speech data contains redundancy such as speaker information, pronunciation variations, which are irrelevant to the semantic meaning of the utterance.
To overcome this problem, wav2vec 2.0 perform masking in intermediate latent space and performs target discretization with a differentiable quantization scheme.
However, quantization leads to a more complex model by introducing additional hyper-parameters and an additional diversity loss.
SPIRAL does not utilize quantization, and still achieves competitive performance compared to wav2vec 2.0.
We hypothesize that aggressive down-sampling and learning by matching output representation may help to remove redundancy from the learned representation.
We leave the investigation of whether target discretization could further improve SPIRAL for future work.
Liang et al.
(2018) demonstrates that under the supervised learning setting, enforcing noise-invariant representation by penalizing difference between clean and noisy data improves ASR model accuracy.
METHOD 3. S ELF - SUPERVISED P ERTURBATION -I NVARIANT R EPRESENTATION L EARNING (SPIRAL)
Figure 1 shows the diagram of SPIRAL in the pre-training stage, where we use two neural networks, a student F θ and a teacher F θ (cid:48) .
The weights of the teacher θ (cid:48) is the moving average of the weights of the student θ .
At step t , the weights of the teacher θ (cid:48) t are updated as
[Equation]
(1) where α t determines the rate of weight updates.
Given a speech utterance X =
( x 1 , . .
, x T ) of length T , the student takes a perturbed version ˜ X
= s ( X ) =
(˜ x 1 , . . .
, ˜ x T ) as input where s ( · ) is a perturbation function.
The output of the student is a representation sequence Z = F ( ˜ X ; θ ) =
( z 1 , . . .
, z T ) .
The teacher takes the same utterance without perturbation as input and output another representation sequence Z (cid:48)
=
F ( X ; θ (cid:48) )
=
( z (cid:48) 1 , . . .
, z (cid:48) T ) .
For each representation z i ∈ Z , the student is trained to match the teacher’s representation z (cid:48)
i at the same position amongst k distracting samples.
The distracting samples are randomly drawn from other positions of the same utterance in Z (cid:48) , which is found to be more effective than samples drawn from an entire batch of utterances (Baevski et al., 2020b).
The in-utterance contrastive loss is deﬁned following Sohn (2016);
Wu et al.
(2018) as,
[Equation]
[Figure] where φ ( a , b ) = a T b / (cid:107) a (cid:107)(cid:107) b (cid:107) is cosine similarity, D i is the set of indices of distractors for the i -th position, and κ is the temperature parameter.
However, applying in-utterance contrastive loss could cause a kind of representation collapse which we refer to as positional collapse.
Contrastive candidates are sampled based on their positions in utterances.
When a teacher’s representation z (cid:48)
i is correlated with its position i (e.g., correlation introduced by positional encoding in Transformer), the student could exploit this correlation to generate its representation z i solely based on the position index i , while ignoring content of the input.
In this case, the model does not learn meaningful representation of the input content.
Therefore, we prevent positional collapse by randomizing positions of teacher’s representation.
In particular, we add random number of padding data at both ends of the input to the teacher to randomly shift the position information for each output representation z (cid:48) i .
The student thereby is unable to exploit the spurious position information to minimize the contrastive loss.
Note that when calculating the contrastive loss, we exclude the corresponding representation of the padded data.
3. M ODEL ARCHITECTURE
[Figure]
but without the predictor. As illustrated in Figure 2, student F θ is composed of an encoder f ( · ) , a projection head g ( · ) (Chen et al., 2020a) and an optional predictor q ( · ) (Grill et al., 2020), i.e., F θ =
( f ◦ g ◦ q )( · ; θ ) .
The teacher
F θ (cid:48) has the same structure expect that it has no predictor, F θ (cid:48)
=
( f ◦ g )( · ; θ (cid:48) ) .
The encoder consists of two blocks.
In each block, we ﬁrst apply temporal convolutions to perform down-sampling, followed by Transformer (Vaswani et al., 2017) with convolutional relative position encoding (Baevski et al., 2020b).
Each convolution is followed by layer normalization (LN)
(Ba et al., 2016) and ReLU.
For the projection head, we apply a simple linear layer.
The predictor consists of two layers of temporal convolution and a linear layer.
The convolutions are followed by batch normalization (BN) (Ioffe &
Szegedy, 2015) and ReLU.
During pre-training, we add computation noise to both the student and the teacher by applying dropout (Srivastava et al., 2014) and LayerDrop (Fan et al., 2020) in Transformer.
We use the same dropout and LayerDrop rates for the student and the teacher.
3. A DAPTIVE S PEC A UGMENT
We apply adaptive SpecAugment similar to Park et al.
(2020a)
as the primary perturbation method.
Along either time or frequency dimension, we sample uniformly a certain proportion p of all timesteps to be start indices and mask the subsequent consecutive L time-steps.
The masked time-steps is ﬁlled with zeros along frequency dimension.
Along time dimension, we use Gaussian noise as masking values to avoid numerical problems for LN (Park et al., 2020a).
3. M ULTI - CONDITION P RE - TRAINING
For noise-robust pre-training with SPIRAL, we perturb input of the student with various types of additive noise.
We consider this technique as an implementation of multi-condition training (MCT) (Seltzer et al., 2013) in self-supervised setting.
Speciﬁcally, for each input utterance to the student, we sample noise clips from a noise dataset, and mix the noise clips with the whole utterance by addition in time-domain.
We ﬁrst uniformly sample a signal-to-noise ratio (SNR) from a predeﬁned range for each utterance.
Then we scale the noise volume according to the required SNR.
In our preliminary experiments, we found that applying additive noise alone as perturbation degrades performance.
Therefore, we apply additive noise perturbation together with adaptive SpecAugment.
3. M ODEL F INE - TUNING
After pre-training, we take the encoder from the student model in SPIRAL and add a randomly initialized convolutional classiﬁer on top of it.
The convolutional classiﬁer is composed of two layers of convolution, followed by LN and ReLU, and a linear output projection.
The convolution ﬁlters consist of 512 channels with kernel width of 5.
We ﬁne-tune the model with connectionist temporal classiﬁcation (CTC) (Graves et al., 2006) objective for speech recognition.
We use 1024 subwords as output units.
The sub-words are generated from training transcripts of LibriSpeech with SentencePiece (Kudo & Richardson, 2018).
We further investigate SPIRAL’s ability to learn high-level representation of speech during pretraining.
In addition to whole-model ﬁne-tuning, we apply frozen ﬁne-tuning.
We freeze the pretrained parameters and only ﬁne-tune the convolutional classiﬁer which can only perform local classiﬁcation due to limited receptive ﬁeld.
EXPERIMENTAL SETUP 4. D ATA
For pre-training, we use the 960-hour training data (ignoring the labels) from LibriSpeech (Panayotov et al., 2015)(LS-960), or 60k-hour unlabeled audio data from Libri-Light (Kahn et al., 2020b)
(LL-60K).
For Libri-Light, we segment the data using ofﬁcial tools with a threshold of 16s, resulting in 46.9k hours of data.
The two datasets are both derived from English audiobooks from LibriVox project 2 .
For ASR ﬁne-tuning, we apply 100-hour subset (train-clean-100) as low-resource labeled data and entire LS-960 with labels as high-resource labeled data, both from LibriSpeech.
For multi-condition training, we use the noise dataset from Reddy et al.
The dataset consists of 181 hours of noise data with about 150 noise types and 70,000 clips.
We shufﬂe and split the noise data with a ratio of 8:1:1, which are used for training, synthesizing noisy dev-sets and synthetic noisy [Table] [Table] test-sets (results in Appendix A.2) respectively.
SNRs of speech mixtures are set from 0 to 30 dB.
We evaluate on real noisy data test set from CHiME-3 (Barker et al., 2015), which is comprised of speech data recorded in real noisy environments (bus, cafe, pedestrian area, and street junction).
The data are recorded with a microphone array composed of multiple microphone channels located at different positions of a tablet, and a close-talking microphone.
4. T RAINING SETUPS
We apply 128-dimensional log-mel ﬁlterbank extracted with 20 ms window and 10 ms stride as the input acoustic feature.
We experiment with B ASE model and L ARGE model conﬁgurations as shown in Table 1.
The numbers of parameters are comparable to wav2vec 2.0 B ASE and L ARGE models correspondingly.
For SpecAugment, we set p = 0 .
025 and L = 20 for time-dimension mask, and p = 0 .
02 and L = 20 for frequency-dimension mask.
In pre-training, we optimize with Adam (Kingma & Ba, 2015) optimizer, warming up the learning rate for the ﬁrst 8% of updates to a peak of 3e-3.
Then the learning rate decays to 0 with a cosine schedule.
The moving average update rate α t of teacher’s weight also follows a cosine schedule (Grill et al., 2020).
We increase α t from 0 .
995 to 1 .
and from 0 .
990 to 0 .
999 for B ASE and L ARGE models respectively.
We train the B ASE model with batch size of 24 per GPU for 200k steps on 16 V100 GPUs, which takes about 1.3 days.
For the L ARGE model, we train with batch size of 20 per GPU for 500k steps on 32 V100 GPUs, which takes about 7.25 days.
As shown in Table 2, there is a signiﬁcant reduction of training cost (GPU days) compared to wav2vec 2.0 (Baevski et al., 2020b).
SPIRAL requires 80% and 65% less training cost for B ASE and L ARGE respectively.
Note that mix-precision training is not applied for SPIRAL yet.
For ﬁne-tuning, we optimize with Adam and a tri-state rate schedule where the learning rate is warmed up for the ﬁrst 10% of updates to 3e-5, held constant for the next 40% and then linearly decayed to zero following Baevski et al.
(2020b).
We ﬁne-tune B ASE and L ARGE with batch size of 14 and 18 per GPU respectively on 8 GPUs for 80k steps on train-clean-100.
We ﬁne-tune L ARGE with batch size of 10 per GPU on 16 GPUs for 320k steps on LS-960.
We apply SpecAugment for whole-model ﬁne-tuning but not for frozen ﬁne-tuning.
For multi-condition pre-training and [Table] ﬁne-tuning, we randomly perturb each utterance with additive noise with 50% probability before applying SpecAugment.
SNR is uniformly sampled from 0-30 dB.
4. L ANGUAGE M ODEL AND D ECODING
We use a word-level Transformer LM (Baevski & Auli, 2019) trained on Librispeech LM corpus which is identical to Synnaeve
et al.
(2020b).
For low-resource ASR setting, we also evaluate
SPIRAL B ASE with the ofﬁcial LibriSpeech 4-gram LM. We observe that models ﬁne-tuned with subword units performs worse than models ﬁne-tuned with character units when decoding with word-level LM.
Therefore, we apply character-based models for LM decoding, which is the same setting as wav2vec 2.0. The results of LM decoding with subword-based models are available in
Appendix A.1. As output frame rate of pre-trained SPIRAL encoder is low (80ms), the output sequence may be too short for character units. To reuse the pre-trained encoder, we devise an upsampling strategy for the SPIRAL encoder output in ﬁne-tuning stage. We apply a 1-D convolution layer to project the original encoder output of dimension d into a vector of dimension 4 d .
At each time-step, we reshape the projected output vector from (1 , 4 d ) to (4 , d ) .
The frame rate now becomes 20ms.
Then we feed the upsampled outputs to convolutional classiﬁer.
We perform random search for decoding parameters and choose the best parameters according to performance on dev-other with beam 50.
The ﬁnal test performance is measured with beam 500.
We use the beam search decoder of Pratap et al.
RESULTS 5. E VALUATION UNDER L OW -R ESOURCE AND H IGH -R ESOURCE L ABELED D ATA S ETTINGS
We ﬁrst evaluate our method under a low-resource ASR setting in which we ﬁne-tune the models with 100-hour LibriSpeech data (train-clean-100).
The results are shown in Table 3.
We evaluate a B ASE model pre-trained with 960-hour LibriSpeech (LS-960) and a L ARGE model pre-trained with Libri-Light (LL-60K).
The frozen B ASE model performs well, achieving a WER of 13.0% [Table] SPIRAL [Table]
on test-other, which is on par with wav2vec 2.0 B ASE .
This suggests that SPIRAL indeed learns meaningful high-level representations in a self-supervised way.
When we ﬁne-tune the whole B ASE model, the model achieves WER of 5.4% and 11.2% on test-clean and test-other respectively, outperforming wav2vec 2.0 B ASE with 11.5% and 15.8% relative WER reduction.
When decoding with
Transformer LM, the B ASE model achieves WER of 2.7% and 6.1% on test-clean and test-other respectively.
The results are on par with wav2vec 2.0 B ASE .
The SPIRAL L ARGE model consists of more parameters and is pre-trained with more data.
The model achieves WER of 2.2% and 4.3% on test-clean and test-other respectively.
The signiﬁcant improvement of L ARGE over B ASE demonstrates the scalability of SPIRAL.
The results of SPIRAL L ARGE are competitive to wav2vec 2.0 L ARGE .
This is encouraging, as SPIRAL L ARGE only takes 35% of training cost of wav2vec 2.0 L ARGE .
We further evaluate SPIRAL L ARGE pre-trained with Libri-Light (LL-60K) under a high-resource
ASR setting with 960-hour LS-960 as ﬁne-tuning data.
As shown in Table 4, the L ARGE model achieves WER of 1.8% and 3.5% on test-clean and test-other respectively, which are on par with the wav2vec 2.0 L ARGE model.
We note that the supervised models and the noisy student model (Park et al., 2020b) in Table 4 are autoregressive models.
Our models are ﬁne-tuned with CTC objective which is non-autoregressive and generally inferior to autoregressive models.
We use CTC objective for its simplicity and comparability to previous speech pre-training methods.
We consider SPIRAL as a preferred alternative to wav2vec 2.0 given that SPIRAL only requires 20% − 35% computation cost of wav2vec 2.0.
We expect further efﬁciency improvement when we implement mix-precision training for SPIRAL.
5. N OISE - ROBUST PRE - TRAINING
To evaluate noise-robustness of the pre-trained models, we compare the effects of applying multicondition training (MCT) in pre-training or ﬁne-tuning stages of SPIRAL.
The results are shown in
Table 5.
The vanilla SPIRAL B ASE model and wav2vec 2.0 B ASE model deteriorate with signiﬁcantly higher WER on noisy test data.
On real noisy test speech data in CHiME-3 for different microphone channels (ch), SPIRAL with multi-condition pre-training signiﬁcantly improves speech recognition performance.
Compared to the model applying MCT solely in ﬁne-tuning, applying MCT both in pre-training and ﬁne-tuning achieves 12.4%, 13.3% and 9.0% relative WER reduction for ch 1, 5 and 2 respectively.
There is smaller performance improvement of 3.8% relative WER reduction for ch 0, which is a closetalking microphone with the highest SNR.
We note that ch 2 faces backwards to the speaker.
SNR
[Table]
of the recordings from ch 2 is the lowest, leading to high WER.
We note that other pre-training methods including wav2vec
[Table]
0 may beneﬁt from multi-condition training, which are worth for further investigation.
5. A BLATIONS 5.3. I NPUT P ERTURBATION AND COMPUTATION NOISE OF TEACHER
SPIRAL learns denoised representation of perturbed data.
By default, we only apply perturbation to the input of the student.
An alternative method is to perturb both inputs of the teacher and the student, and optimize consistency between their representations (Grill et al., 2020; Chen & He, 2021).
We conduct experiments to evaluate the effects of perturbing the input and adding computation noise (dropout and LayerDrop) to the teacher.
The results are shown in Table 8 in Appendix A.3.
The results suggest that applying SpecAugment to teacher’s input degrades performance signiﬁcantly.
Performance degradation decreases but is still signiﬁcant with lower ratio and width of the masks.
This supports the necessity of representation denoising, and our view of SPIRAL as an extension of self-training in which teacher network are fed with clean input.
The results also support applying computation noise to teacher during pre-training.
There is a 15.9% relative WER reduction with computation noise.
This may be linked to Gal & Ghahramani (2016).
5.3. E FFECTS OF PREDICTOR AND PROJECTION HEAD
We do ablation studies to understand the role of predictor and projection head in SPIRAL.
The results are shown in Table 9 in Appendix A.3.
When removing the predictor from the student, we observe performance degradation, but representation collapse does not happen.
In the architectures relying on predictor to prevent collapse (Grill et al., 2020; Chen & He, 2021), applying batch normalization (BN) in the predictor is essential.
While in SPIRAL, we observe that BN in the predictor can be replaced by layer normalization (LN) with a small performance degradation.
When the predictor is removed, we observe performance improvement by applying a convolutional projection head.
The convolutional projection head is composed of a temporal convolution layer with LN and ReLU, and a linear layer.
But applying convolutional projection head to the model with a predictor, there is no further performance improvement.
This suggests that convolutional projection head and predictor play a similar role in SPIRAL, and they are not complementary.
CONCLUSION
We presented SPIRAL, a new approach to speech pre-training by learning denoising representation of perturbed data with a teacher-student framework.
SPIRAL can learn high-level speech representation in self-supervised way.
Training a small convolutional classiﬁer on frozen representation of SPIRAL achieves WER of 3.5% and 6.4% on Librispeech test-clean and test-other respectively.
We show that SPIRAL achieves competitive or better results compared to state-of-the-art speech pre- training methods, with signiﬁcant reduction of training cost.
We investigate multi -condition pre-training and demonstrates that multi -condition pre-training is more effective than solely apply ing multi -condition training in the ﬁne tuning stage.
We presume SPIRAL as a general pre training method, which can apply to other modalities such as images and text.
We leave it for future work
A.
O UTPUT UNITS COMPARISON FOR L ANGUAGE M ODEL D ECODING
We evaluate decoding performance of different combinations of SPIRAL output units and Transformer LM.
The subword-level LM is trained on the same Librispeech LM corpus and shares the same 1024 subwords as the SPIRAL model ﬁne-tuned with subword units. We compare SPIRAL B ASE pre- We report WER (%) on Librispeech dev/test [Table]
SPIRAL B ASE
LS-
[Table]
SPIRAL B ASE
LS-
[Table]
SPIRAL B ASE
LS-
[Table]
SPIRAL B ASE [Table] [Table] SPIRAL B ASE
LSchar
[Table]
[Table]
[Table]
SPIRAL L ARGE
LL-60k
[Table]
[Table]
SPIRAL L ARGE
LL-60k char3.
SPIRAL L ARGE
LL-60k
[Table]
[Table]
SPIRAL L ARGE
LL-60k
[Table]
SPIRAL L ARGE
LL-60k char word 1.
High-resource
SPIRAL L ARGE
LL-60k subword2.
SPIRAL L ARGE
LL-60k char2.
SPIRAL L ARGE
LL-60k subword word 1.
SPIRAL L ARGE
LL-60k subword subword 1.
SPIRAL L ARGE
LL-60k char word 1.
A. P ERFORMANCE ON SYNTHETIC NOISY DATASET
On the synthetic noisy dataset (NS-Librispeech) with matched SNR range (0-30 dB) of the training data, SPIRAL pre-trained and ﬁne-tuned with MCT is more effective than applying MCT solely in ﬁne-tuning.
We observe 5.3% and 9.1% relative WER reduction on synthetic noisy test-clean and test-other sets respectively. [Table] SPIRAL [Table]
SPIRAL -
[Table]
SPIRAL (cid:88) - 5.
SPIRAL (cid:88) (cid:88) 5.
A. R ESULTS OF ABLATION STUDIES
Here are the results of ablation studies discussed in Section 5.3.
[Table]
[Table]