A SK 2M ASK : G UIDED D ATA S ELECTION FOR M ASKED S PEECH M ODELING
Anonymous authors
Paper under double-blind review
A BSTRACT
Masked speech modeling (MSM) methods such as wav2vec2 or w2v-BERT learn representations over speech frames which are randomly masked within an utterance.
While these methods improve performance of Automatic Speech Recognition (ASR) systems, they have one major limitation.
They treat all unsupervised speech samples with equal weight, which hinders learning as not all samples have relevant information to learn meaningful representations.
In this work, we address this limitation.
We propose ask2mask (ATM), a novel approach to focus on speciﬁc samples during MSM pre-training.
ATM employs an external ASR model or scorer to weight unsupervised input samples in two different ways: 1) A ﬁnegrained data selection is performed by masking over the highly conﬁdent input frames as chosen by the scorer.
This allows the model to learn meaningful representations.
2) ATM is further extended to focus at utterance-level by weighting the ﬁnal MSM loss with the utterance-level conﬁdence score.
We conduct ﬁne-tuning experiments on two well-benchmarked corpora:
LibriSpeech (matching the pretraining data) and Commonvoice, TED-LIUM, AMI and CHiME-6 (not matching the pre-training data).
The results substantiate the efﬁcacy of ATM on signiﬁcantly improving the recognition performance under mismatched conditions (up to 11.6% relative over published results and upto 4.46% relative over our internal baseline) while still yielding modest improvements under matched conditions.
I NTRODUCTION
Self-training and self-supervised training techniques rely on huge amounts of unlabeled speech or text data for better generalization.
The self-training techniques such as pseudo-labeling
Scudder, 1965; Kahn et al., 2020) and student-teacher training (Park et al., 2020) have shown promising improvements by incorporating the data selection process.
This data selection step removes pseudolabels with less conﬁdence as denoted by the teacher model before feeding the input to a student model.
Xu et al.
(2021) shows that self-training and self-supervised training are complementary to each other and also show that self-supervised models act as good initialization for self-training techniques.
Self-supervised training (Hinton & Zemel, 1994) is a representation learning approach which implicitly learns patterns in the data without relying on explicit labels.
Masked speech modeling (MSM) is the recent and successful self-supervised learning technique, thanks to the advent of BERT (Devlin et al., 2018) in NLP which inspired learning speech representations from masked inputs.
MSM techniques such as wav2vec2 (Baevski et al., 2020), HuBERT (Hsu et al., 2021b) and w2v-BERT (Chung et al., 2021) have shown considerable gains across various down-stream speech tasks and have become the go-to models for ASR.
Unfortunately, MSM does not have a data selection scheme to discard the irrelevant input samples and instead imposes burden on the training criterion to learn the relevance of the input samples in learning meaningful representations.
Hsu et al.
(2021a) noticed the impact of not selecting relevant data from the huge amounts of unsupervised data during pre-training by showing degradation in ASR performance when ﬁne-tuned to a target dataset with limited data.
To mitigate this constraint, Chan et al.
(2021) introduced substantially more ﬁne-tuning data related to the target dataset but did not achieve satisfactory results.
Hsu et al.
(2021a) attempted to solve this issue by heuristically selecting the data from a closed set of unsupervised speech databases or by pooling in data relevant to target dataset along with the existing pre-training dataset.
However, this data selection approach is not done within the existing pre-training dataset and it is not completely empirically motivated.
In this study, in order to break the above limitation of the MSM techniques, we propose a simple strategy named ask2mask
ATM
) to incorporate data selection within a chosen pretraining dataset.
• In ATM, the masking is done over the input samples or speech frames with higher conﬁdence as determined by the scorer.
This is contrary to the random selection of frames to be masked in conventional MSM models.
We hypothesize that this guided selection of frames to be masked allows the model to focus on the frames which can provide meaningful representations.
The scoring model used in this work is necessarily a speech recognition model trained on small amount of data and provides frame-level conﬁdence for each input.
• The ATM technique is further extended to exploit the conﬁdence values provided by the scorer by directly using them to re-weight the MSM loss.
We denote this approach as ATM with loss scaling (ATM+S).
It allows the model training to focus on certain utterances by down scaling the utterances with low-conﬁdences.
Similar to our work based on masking with external guidance, there is work in NLP that also beneﬁt by incorporating masking with knowledge.
In Sun et al.
(2019), masking is done at phrase-level segments in BERT and has shown to learn semantic dependencies.
In Wang et al.
(2019), phonetic knowledge is injected to mask over phonetic segments to perform spectral augmentation.
Our ATM approach is primarily motivated based on the recent work by Vesel´y et al.
(2017) on semi-supervised learning of conventional ASR systems which shows that performing data selection at frame-level or token-level on unsupervised data provides better performance.
The importance of pruning out the input samples at frame-level has been studied in Ferreira et al.
(2021) to improve both classiﬁcation and regression tasks.
Few works on unsupervised learning also highlight the importance of weighting the data based on its conﬁdence (Wessel et al., 2001;
Ren et al., 2020;
Coleman et al., 2019).
We hypothesize that ATM can leverage the effect of data selection within a particular training corpus to further enhance the recognition performance of MSM techniques.
To summarize, our contributions are listed as follows:
• Novelty :
To the extent of our knowledge, ATM is the ﬁrst approach to incorporate a withincorpus data selection strategy in MSM.
We also show that data selection can be simply performed inside MSM by guided selection of frames to be masked using a scorer model.
• Technical contributions :
We provide two simple strategies to incorporate data selection into MSM pretraining by applying the conﬁdence of the scorer: 1) choosing the data at framelevel by applying guided masking 2) soft weighting the data at utterance level by scaling the MSM loss of each utterance with its corresponding conﬁdence score.
ATM is designed to be compatible to all MSM based pre-training techniques.
• Empirical study : Analysis is done to ﬁnd an optimal masking percentage for ATM and we highlight the effectiveness of ATM across varying masking percentages.
The importance of masking frames with high conﬁdence is substantiated by empirically comparing it with masking low conﬁdent frames and random frames respectively.
Experiments are performed on AMI data which is from a distinct condition compared to Libri-light corpus used for MSM based pretraining.
The results conﬁrm the importance of ATM by improving the recognition performance on evaluation sets of AMI by a signiﬁcant margin.
P RELIMINARIES ON M ASKED SPEECH M ODELING (MSM)
In this section, we formally deﬁne the masked speech modeling (MSM) technique and brief primary instantiations including wav2vec2 and w2v-BERT.
The technique can be formulated by deﬁning input speech sequence X
=
[ x 1 , x 2 , ..., x T (cid:48) ] , where x t is the log Mel-ﬁlterbank feature vector at time t .
X is sent to the feature encoder Φ to obtain the encoded representations E =
Φ( X ) .
The feature encoder contains convolutional layers performing subsampling at a factor of 4 and reducing the total number of frames of an utterance from T (cid:48) to T to get E =
[ e 1 , e 2 , ..., e T ] .
E is then sent to two parallel modules
2. M ASKING
The idea behind masking input samples and predicting them was initially proposed in BERT (Devlin et al., 2018) and later adopted to speech (Baevski et al., 2020) with modiﬁcations to suit the characteristics of speech input.
The masking is done over sets of frames or blocks b 1 , b 2 , ..., b K and accommodates overlap between blocks.
Here K is the number of masked blocks in a randomly masked encoded sequence ˜ E .
The importance of block masking is motivated by the improvements observed in Span-BERT by Joshi et al.
(2020) and ERNIE (Sun et al., 2019).
The block b k =
[ i k , c ] , where i k is the starting index of the masked block and c is the corresponding right context size denoting the number of consecutive speech frames.
Here i k are randomly sampled from a uniform distribution.
It has been empirically observed by Baevski et al.
(2020) that 49% of the frames are masked and c = 10 is chosen as the golden ratio to attain best representation during pre-training.
2. Q UANTIZER
Gumbel-softmax quantizer component Ψ is used to get quantized representations which act as targets for wav2vec2 and w2v-BERT models.
These quantized representations align to phonetic units as described in Baevski et al.
Each quantized vector is of L dimensions which denote the number of targets or codes used in a codebook.
Each incoming input E is projected to L dimensions within the quantizer before applying the Gumbel-softmax.
2. C ONTEXT N ETWORK AND MSM L OSS
Wav2vec2-conformer :
In this model type, the unmasked sequence E is sent to Ψ to get Q = Ψ( E ) , where Q =
[ q 1 , q 2 , .., q T ] as described in Baevski et al.
The masked sequence ˜ E is fed to the context network Ω which contains conformer blocks to learn contextual representations from the input.
C =
Ω( E ) denotes the output of the context network.
The contrastive loss L ctr ( c j , q j ) objective is computed between the quantized representation q j and context network output c j ∈ C for all masked time instances j ∈ J .
Diversity loss L div is computed as an auxiliary objective in wav2vec2 to force the model to choose diverse codes in the quantization codebook. Detailed description of L div is in Baevski et al.
The ﬁnal training objective is denoted as:
[Equation]
where L ctr =
(cid:80)
Jj =1 L ctr ( c j , q j ) .
HuBERT-conformer : This is another variant of wav2vec2-conformer model with two major differences: 1) Targets are k-means cluster ids which are computed over a small portion of input 2)
Cross-entropy loss L ce (ˆ y j , y j ) is computed between the prediction of the context network ˆ y j and the k-means cluster id target y j .
W2V-BERT : This model marries the concept of wav2vec2 and BERT model by including an additional context network Λ containing conformer blocks in addition to Ω as in wav2vec2.
The Λ receives the output of the Ω and strives to further learn reﬁned contextual information to get
H = Λ( C ) .
The targets of w2v-BERT y j is computed by taking an argmax over the codebook dimensions L of quantized representations q j,l as:
[Equation]
Finally, the cross-entropy loss L ce (ˆ y j , y j ) is computed between the prediction ˆ y j = softmax( h j ) and the target y j over the masked time instances J .
The ﬁnal training objective L wb =
L ce + L wv is a combination of cross-entropy loss and wav2vec2 loss.
A block diagrammatic overview of the above MSM architectures are available in appendix A.7.
A SK 2M ASK (ATM)
The primary reason to employ pre-training models is to exploit the abundantly available unsupervised data for improving ASR under limited availability of supervised data.
While the MSM models such as wav2vec2 and w2v-BERT described in Section 2 exploit the unsupervised data, they treat each data with equal weight for computing the ﬁnal objective.
Instead, we generate a score s t for each encoded frame e t .
This is used to select relevant data in a ﬁne-grained manner during masking for computing the loss objective.
3. M ETHODOLOGY
For each encoded feature frame e t ∈ E , the scorer emits probabilities p ( v t = l | E ); l ∈ L of the frame belonging to a particular label.
The scorer model is a CTC based frame-synchronous ASR model separately trained with a limited amount of data.
Our initial intuition was to chose the scorer’s training data to match the target data condition, however our empirical analysis in (cf.
Section 5.3) shows that the performance is agnostic to the scorer model’s training data.
Finally, the conﬁdence score s t of the frame is deﬁned as the maximum probability across all labels:
[Equation]
We sample K masking start indices { i 1 , .., i k } with probabilities:
[Equation]
That is, we sample beginning frames with probability proportional to the scores of each frame.
The indicator function δ t/ ∈{ i 1 .,,i k − 1 } ensures that we sample without replacement.
This is the key difference between ATM and the random masking in prior works as described in Section 2.1.
Prior works uniformly sample the start indices of each masking block b 1: K , while the ATM uses the probability distribution induced by the scorer.
K is determined by the percentage of frames to be masked.
We hypothesize that frames with maximum conﬁdence from an external scoring model will be 1) easiest to learn using an MSM training criteria and 2) most informative in for pretraining to facilitate ﬁne-tuning.
Conversely, the lowest conﬁdence frames, those more confusable to an external scoring model, will be the least reliably learned by MSM and least informative for pretraining.
The resulting frames are sent as input to the MSM architecture and the ﬁnal loss objective L is determined by either of the MSM objectives L wv or L wb described in Sections 2.3.
This modiﬁed training objective allows the model to focus on learning from gradients calculated from the frames with high conﬁdences.
3. ATM WITH MSM L OSS S CALING (ATM+S)
The ATM loss is computed over the frames with high conﬁdence performing a ﬁne-grained data selection within a u th speech sequence X u .
Utterances with higher average frame conﬁdence as measured by the scorer are accorded higher value than those with more confused frames.
To perform data selection at a coarser utterance level, conﬁdence scores s u are computed as:
[Equation]
For simplicity, we denote s = s u and the MSM loss computed over each masked frame is scaled by s to impose the importance of a particular utterance u .
The ﬁnal training objective L (cid:48) atm of a particular speech sequence is denoted as:
[Equation]
3. P ROBABILITY AS CONFIDENCE MEASURE IN ATM
The ATM uses probability as a simple form of conﬁdence measure to each frame.
The analysis of conﬁdence measures for semi-supervision in ASR has been done in Vesel´y et al.
and they show that posterior probability acts as a reliable conﬁdence measure for frame, word and sentence based data selection.
They also perform an extensive analysis on using the posteriors for hybrid ASR systems. Based on the motivation from this work we chose to use softmax probability directly as our conﬁdence score. A similar observation has been noted in Ferreira et al.
(2021), where the usage of softmax probability directly as a conﬁdence measure has been applied to select relevant data samples during training.
We also experimented with Entropy and exponential scaling or log scaling on softmax probabilities as conﬁdence measure, but it did not fetch advantage over simple usage of probability.
E XPERIMENTAL S ETUP
All experiments including pre-training and ﬁne-tuning are performed using 80 dimensional log Melﬁlterbank features computed over the sampled 16kHz audio.
Datasets (such as AMI) contains wideband audio and are downsampled to 16kHz.
We evaluate with the test-other (LibriSpeech partition) to show the importance of ATM on matched data conditions, while IHM-eval and SDM-eval (AMI partitions) is used to validate the model under mismatched conditions.
4. D ATASETS USED
Pretraining (PT) : Libri-light (LL-60k) dataset contains 60k hours of unlabeled speech and is used to pre-train all MSM models.
LL-60k is the most widely used large unsupervised speech corpus for various PT techniques.
Each input speech sequence is constructed by ﬁrst randomly selecting 32seconds segments from the original utterance.
From these segments, a contiguous 32 second region is extracted from a random starting point on-the-ﬂy during MSM PT as described in Zhang et al.
(2020b).
Finetuning (FT) : Different target datasets including 1) 100 & 960 hours of Librispeech (LS-100 &
LS-960)
(Panayotov et al., 2015).
2) 100 hours of AMI and 3) speechstew (approx.
5k hours)
(Chan et al., 2021) are used to perform our FT experiments.
Each dataset used is speciﬁc to a certain target data condition, for instance LS-960 is closely matches the LL-60k, AMI dataset is distinct from the LL-60k condition and it contains speech from two kinds of microphones ( i ) Independent head microphone (IHM).
( ii ) single distant microphone (SDM).
SpeechStew is composed of datasets chosen from multiple conditions to create a mixed domain aggregate corpus.
Details of its processing are described in Chan et al.
Evaluation : We hypothesize that evaluation over AMI using IHM-eval and SDM-eval reveals the effectiveness of ATM in providing informative samples for better representation learning.
We also evaluate using evaluation sets from Tedlium and Common voice as their training counter parts are used in SpeechStew based FT.
Finally, we also evaluate using CHiME-6 Watanabe et al. (2020) without using any FT data from CHiME-6 training set to compare the performance of ATM on completely unseen target dataset.
Scorer training data : A CTC (Graves et al., 2006) based conformer model with 100M parameters is trained on LS-100 (“LS-scorer”).
A similar model is also trained on AMI (“AMI-scorer”).
Wordpiece model (WPM) with 1024 tokens are used as labels for training the scorer models.
All the results in this paper use “LS-scorer” besides the comparison Section 5.3.
4. MSM ARCHITECTURE
Wav2vec2-conformer : This is a wav2vec2 with conformer based context network which ﬁrst encodes the ﬁlterbank features using two 2D convolutional layers with strides (2,2).
Model has 100M/600M parameters and is denoted as “w2v2-conformer-L/XL”.
HuBERT-conformer-L/XL is similar to w2v2-conformer-L/XL - it differs in using the k-means based quantizer with 1024 targets and computes the cross-entropy loss as described in Hsu et al.
(2021b).
The “L/XL” size models contains context network Ω 12/24 conformer layers with 8 attention heads and 1024 hidden dimensions.
W2v-BERT : W2v-BERT is explored using two model sizes: one with 100M parameters denoted as “w2v-BERT-L” and containing 2 conformer layers in context net Ω and 4 conformer layers in Λ .
A 600M parameter model is denoted as “w2v-BERT-XL” contains 8 conformer layers in Ω andconformer layers in Λ .
Each conformer block contains 1024 hidden dimensions with 8 attention heads, kernel size of 5 with local context of 128.
The remaining architecture is identical to the conﬁguration deﬁned in Chung et al.
4. PT AND FT CONFIGURATION
The models L/XL are trained with a global batch size of 512/2048 on 64/256 Google TPU V cores for 2-4 days respectively.
Adam optimizer is used with a learning rate schedule (Section 5. of Vaswani et al.
with 2e-3 as peak learning rate and 25k warmup steps.
The model training conﬁguration follows similar procedure as described in Zhang et al.
(2020b).
The FT is done by employing the context network from the PT model by adding a ﬁnal layer with 1024 WPM units learnt using the RNN-T objective function (Zhang et al., 2020a).
The FT is done on w2v-BERT-XL, w2v2-conformer-XL and HuBERT-conformer-XL after 400k PT model updates.
The w2v-BERT-L model is FT after 900k PT model updates.
w2v-BERT-L is used to initially perform wide range of analysis and hyper-parameter optimization on ATM.
w2v-BERT-XL is ﬁnally used to compare the results of ATM across existing works in literature.
w2v2-conformer-XL and HuBERT-conformer-XL are also used in our experiments.
All these models are trained with the same conﬁguration as in Zhang et al.
(2020b).
ATM ANALYSIS
The empirical study on ATM is done primarily using w2v-BERT-L since this generates the best
WER performance across similarly sized models (cf.
Figure 2).
The pre-trained models are ﬁnetuned with either LS-100 or AMI.
The resulting ﬁnetuned models are evaluated on IHM and SDM evaluation sets to understand the domain generalization aspect of ATM.
Librispeech evaluation sets are used in unison to study how ATM behaves under matching domain condition.
These experiments are performed with using the loss scaling (it will be discussed in Section 5.5).
5. M ASKING PERCENTAGES
[Figure]
shows substantial The number of masked frames within an utterance plays a key role in masked input learning and in this study, we vary the masking percentages from 30% to 50% to determine the best percentage for ATM approach.
Previous works on wav2vec2 (Baevski et al.
(2020)) showed that masking 49% of the frames is ideal for 30 second utterance and this has been followed subsequent works such as HuBERT and w2vBERT.
In case of ATM, this can differ as the frames selected are of higher conﬁdences.
Figure 1 shows that ATM achieves its “sweet spot” with 40% masking for both IHM-eval and testother set.
Interestingly ATM’s performance is stable across large variations in masking rates with relatively good performance with masking rate as low as 30%.
This is a signiﬁcant difference from the uniform sampling of prior work which suffers signiﬁcant drop in performance as the masking rate goes below 40%.
The result indicates that masking the right set of frames, which ATM aims to do, is able to promote more stable performance.
For instance,
ATM achieves a %WER of 12.65 with 33% masking and 12.52 with 40% masking on IHMeval respectively as shown in Figure 1.
The recognition performance on test-other and IHM-eval improves over baseline from 8.86% to 8.79% and 13.38% to 12.52% respectively by using ATM.
5. ATM MASKING STRATEGIES
[Table]
The default setting of ATM is chosen based on a hypothesis that those frames that are scored with high-conﬁdence from an external scoring model will be most useful as candidates for
MSM pretraining.
This hypothesis is interrogated in this section by analysing the impact of choosing the frames with low conﬁdences or equal mix of both high and low conﬁdence frames (Mixed).
For masking low conﬁdent frames, we modify the score in Equation 3 as:
[Equation]
We evaluated these three masking strategies of
ATM on both IHM and SDM evaluation sets.
Table 1 shows the comparison between these sampling strategies.
We observe that masking high conﬁdent frames are consistently better than masking the low conﬁdence counterparts.
In fact, “Low” conﬁdent frames perform worse than the baseline with random masking.
Finally, we observe that performance of “Mixed” falls between that of “High” and “Low”.
The “Mixed” strategy is similar to random masking, as both high and low conﬁdence frames are selected.
This similarity is also reﬂected in comparable performance between “Mixed” and random masking.
These results provide support for our initial hypothesis that masking frames with high conﬁdence leads to better pre-training.
5. H OW TO CHOOSE THE SCORING MODEL
test and test-other, while the FT is done on AMI to evaluate using IHM-eval and SDM-eval
Evalset
[Table]
The scorer used in this work is a speech recognition model (100M parameters) trained in a supervised fashion.
The scorer is chosen based on the target downstream task and in addition to this, the scorer needs to be frame-synchronous to provide conﬁdence for every frame in a speech sequence.
In this work, we use a framesynchronous ASR system as the scorer by employing the connectionist temporal classiﬁcation (CTC) objective.
The CTC is preferred over the RNN-T by analysing the reliability of the frame-level predictions.
To analyse the importance of the supervised data used to train the scorer, we trained two scorer models: LS-scorer and AMI-scorer are CTC models trained with LS-100 and AMI dataset respectively.
The AMI-scorer outperforms on SDM-eval by improving the %WER from 27.34 to 27.00.
Surprisingly, our results on Table 2, shows that the results on IHM-eval using an LS-scorer are comparable to the AMI-scorer.
Evaluation on test and test-other shows that LS-scorer is better than AMI-scorer on both sets.
Based on these observations, we choose the LS-scorer as the universal scoring model for all ATM based pre-trained models regardless of the target domain (eg: AMI) used in our experiments.
Table 2 shows that although matching the scorer to the target domain improves the performance, the difference is not signiﬁcant.
5. C ONSISTENCY ACROSS DIFFERENT ARCHITECTURES
Figure 2 shows that ATM consistently outperforms on both IHM-eval and SDM-eval across multiple MSM architectures including wav2vec2 and HuBERT.
In the case of IHM-eval, ATM attains a relative improvement of 9% over w2v2-conformer-L, 4% relative improvement over
HuBERT-conformer-L and 5% relative gain over w2v-BERT-L baseline models respectively.
W2v2conformer-L using ATM obtained 6.2% relative improvement over its baseline counterpart and
HuBERT-conformer-L with ATM attained 7.9% rel.
improvement over HuBERT-L baseline on
SDM-eval respectively.
On the other hand w2v-BERT-L baseline is better compared to w2v2conformer-L and HuBERT-conformer-L on both IHM-eval and SDM-eval by achieving 12.52% and 27.34% WER respectively. % W E R on I H M - e v a l    w2v2-cfr-L HuBERT-cfr-L w2v-BERT-L
Baseline ATM % W E R on S D M - e v a l     w2v2-cfr-L
HuBERT-cfr-L w2v-BERT-L
Baseline ATM 5. ATM WITH L OSS SCALING (ATM+S) A NALYSIS ATM training can incorporate utterance-level weighting by scaling the MSM loss obtained using w2v-BERT-L models with the utterance-level conﬁdence score according to Equation 6.
We evalu-
[Table]
ate the value of utterance-level loss scaling by re-weighting utterances in the context of both baseline MSM (i.e., without ATM frame selection) and ATM (ATM+S).
These results are in Table 3.
Re-weighting utterances by scaling the MSM loss with the conﬁdence score on baseline model is denoted as “Baseline+S” and on ATM is labeled as “ATM+S”.
MSM loss scaling is effective even without ATM; baseline+S improves over baseline on both IHM-eval and SDM-eval.
Moreover,
ATM+S improves over ATM on SDM-eval by attaining 27.19% WER while showing degradation on IHM-eval.
This shows that ATM+S is effective on very hard evaluation task such as SDM-eval compared to IHM-eval.
On the IHM-eval test set, the impact of MSM loss scaling is observed over the Baseline MSM without ATM.
We hypothesize that ATM+S may not able to provide improvement on IHM-eval as ATM already incorporates optimally incorporates scorer information on this task.
R ESULTS
In this Section, XL models are used to compare the importance of ATM on LS-960, AMI and
SpeechStew.
These three datasets show the effect of ATM on diverse conditions with a much larger model.
Results are compared with appropriate prior work.
Table 4 shows that ATM and ATM+S improves over dev-other while on dev set there was improvement only using ATM and not ATM+S.
Although the ATM and ATM+S does not show improvement on test and test-other, matches the very strong baseline.
Considering the similarity between LS-960 and PT data, ATM manages to provide gains without hurting the performance across all Librispeech evaluation sets.
This validates our argument that MSM models are better without any data selection when trained under matched data condition but can beneﬁt under mismatched conditions.
Table 5 presents the results of ATM on AMI by comparing it with w2v2-conformer-XL baseline and w2v-BERT-XL baselines.
We include w2v2-conformer-XL to further test the consistency of ATM on XL models when evaluated on harder tasks.
ATM+S and ATM observes consistent gains over baseline on both IHM-eval and SDM-eval when trained with XL models.
However, ATM+S did not [Table] [Table] demonstrate improvement on IHM-eval using w2v-BERT-XL.
6 analyses the effect of ATM [Table] [Table] et
[Table]
[Table]
[Table] ATM [Table] and ATM+S on multiple evaluation sets such as Commonvoice, Tedlium, AMI and CHiME-6.
These four sets are chosen based on the mismatch range from minimum to maximum and for instance,
Commonvoice has the minimum mismatch with Libri-light data, while CHiME-6 has the maximum mismatch. The state-of-the-art results published in Chan et al. (2021) are obtained by choosing the best Conformer model supervisedly trained with multiple datasets such as AMI, CommonVoice, Broadcast News, Librispeech, Switchboard/Fischer, TED-LIUM and Wall Street Journal.
Note that the training data did not include the CHiME-6 data. The authors in Chan et al. (2021) show that simply training an ASR with lots of data leads to best results compared to the wav2vec2 ﬁnetuned model. Their best results are denoted in table 6 and will be used to compare with our best ATM results.
Our baseline w2v-BERT-XL attained better results over the published w2v2-conformer-XL and
Speechstew results.
In Commonvoice and CHiME-6, the baseline attained 7.4% and 2.9% relative improvement over Speechstew respectively. However, by including our ATM and ATM+S with w2v-BERT-XL, there was consistent improvement across all range of mismatched domains. For instance, ATM+S attains 5.76% relative improvement on CHiME-6 over the Speechstew.
This result clearly justiﬁes that selection of reasonable input samples during pre-training reduces the necessity of having ﬁnetuning data from the same domain to improve performance. To further substantiate this, the results on AMI show a 4.6% relative improvement on AMI-SDM over Speechstew which is of different domain compared to pre-training domain. In case of minimal mismatch domain such as Commonvoice, the ATM attained 11.6% relative improvement over Speechstew.
These observations show that ATM and ATM+S demonstrate their effectiveness to generalize to unseen and challenging speech recognition conditions.
C ONCLUSION
In this work, we introduce ask2mask (ATM) to perform data selection over unsupervised samples for MSM based pre-training to focus on relevant information and learn meaningful representations.
ATM achieves 21.0% WER on mismatched AMI SDM set with guided masking and a 20.7% WER is obtained by including loss scaling (ATM+S).
We empirically show that ATM is more robust to changes in masking percentage compared to random masking.
as typically used in MSM.
Our results substantiate the importance of learning from high conﬁdent frames by attaining improvements across multiple evaluation sets.
An important aspect of ATM approach is its ﬂexibility to incorporate into any MSM pretraining techniques and ATM+S can also be easily adopted into self-supervised pre-training methods.
In our future work, we wish to apply ATM over pretraining data containing data from multiple domains (Hsu et al., 2021a;
Likhomanenko et al., 2020) to achieve further improvements.
We also consider two future enhancements to ATM: (1) Joint training of the scorer model with MSM model by simultaneous training on supervised and unsupervised data.
(2) Perform active learning by sharing the parameters of MSM with the scorer once the MSM is well trained.
E THICS S TATEMENT
While we are unaware of any speciﬁc bias in Ask2Mask, it is possible that bias within the SpeechStew and LibriLight training corpora themselves may introduce bias to the resulting ASR model.
Ask2Mask uses an ASR model trained on labeled data as a scoring model identify regions for masking; any bias in this scoring model (trained on LibriSpeech-100 or AMI) could also impact the ﬁne-tuning process, leading to bias in the ﬁnal model.
Without being aware of speciﬁc biasing effects, it is unclear how, if at all, biases in the scoring model and downstream ASR model may interact with each other.
Automatic Speech Recognition (ASR) can exacerbate security and privacy issues by facilitating the search and analysis of speech, though in and of itself ASR does not pose a security nor privacy issue.
R EPRODUCIBILITY S TATEMENT
All of the corpora used for 1) training the scorer model, and 2) pretraining and ﬁnetune the ASR models, as well as 3) all evaluations are publicly available and use standard published partitions.
We aim to describe Ask2Mask and Ask2Mask with scaling comprehensively in the text to facilitate reproducibility.
Implementation of major components, Conformer RNN-T models, wav2vec 2.
and CTC training are available within in public, open-source libraries including Nvidia-nemo (link) and ESPNet (link).
This supports the reproducibility of Ask2Mask and the results reported in this paper within multiple frameworks.
Additional details to promote reproducibility are provided in the
Appendix.
A A PPENDIX A. D ETERMINE THE BEST PT AND FT CHECKPOINT ( CKPT .)
Selecting the best PT ckpt.
to perform the ﬁne-tuning is cumbersome as it involves ﬁnding the best
PT ckpt.
based on the validation accuracy on quantized targets and also run ﬁne-tuning experiments on multiple PT ckpts.
Figure 3 shows that ATM starts to show improved performance after 250k
PT ckpt.
On contrary to our belief that ATM might require less FT, both baseline and ATM requires same number of FT iterations to achieve the best performance.
A. A NALYSIS OF SCORER BASED ON %WER ON PT DATA (1 HOUR OF L IBRI - LIGHT )
We further analyse the impact of the recognition performance of the scorer on ATM, by choosing three categories of scorers such as bad, better and best.
The 10 hours of supervised libri-light (LL10h) data is used to evaluate the performance of these scorers and the results are as follows:
Initial model checkpoint (ckpt.).
LS-scorer attains 51.8% WER and AMI-scorer achieves 72.7% at this ckpt respectively.
Intermediate model ckpt.
Evaluating LS-scorer at this ckpt.
, achieves 45.8% and
AMI-scorer achieves 57.1% WER.
Final model ckpt.
chosen after convergence.
The LS-scorer gains 35.5% WER and AMI-scorer attains 46.8% by evaluating at this ckpt.
Figure 4 shows a more
[Figure]
challenging scenario by testing the ability of ATM using scorers from different domains with varying range of performances.
A LS-scorer with 51.8%WER on LL-1hr performs well on both IHM-eval and SDM-eval, while the AMI-scorer with 46.8% WER slightly improves over the above mentioned
LS-scorer.
The key aspect in selecting better scorer is to choose ckpt.
either from the category which achieves %WER within the range of 50-60 on the small subset of pre-training data.
A. A NALYSIS OF ATM ON L IBRISPEECH
Our ATM analysis is also done on w2v-BERT-L to validate the ATM on LS-100 and is present in table 7.
ATM attained 4.3% and rel.
imp.
on test-other over the baseline model.
Apart from that there is a slight change in test set which can be treated as noise.
On both dev-other and test we did not ﬁnd any improvement and instead there was slight degradation in performance. [Table]
To further evaluate the impact of increasing the model parameters from “L” size to “XL” size, we
FT on LS-100 using MSM models with XL size and the results are in table 8.
We did not ﬁnd any consistency in the performance across the evaluation sets using any of the MSM architectures.
Slight gains are observed on test or dev or dev-other using w2v2-conformer-XL.
Once the baseline in w2vBERT-XL gets better, ATM did not achieve gains on test-other.
This scenario can be explained due to effectiveness of MSM pre-training under matched condition and can perform well without any ncessary data selection approach.
A. ATM+S ANALYSIS ON VALIDATION DATA DURING PT
The effect of ATM and ATM+S is analysed by plotting the validation scores on dev-other during pre-training.
The ﬁrst plot in ﬁgure 5 shows that the contrastive loss improves over the baseline with the aid of ATM and is further enhanced with ATM+S.
The second plot shows the number of unique [Table] codes used from the quantizer codebook.
Analysing this plot helps us to understand if the validation loss or accuracy is improved by just using less % of unique codes which will affect the performance at FT.
Among the 1024 codes, 94%-95% are used by both ATM and ATM+S.
This is similar to the % unique codes used by the baseline model and conﬁrms that improvement of ATM and ATM+S is not by choosing smaller set of unique codes.
The third plot shows that the MSM accuracy of
ATM and ATM+S improves over baseline model.
ATM+S shows that re-weighting each utterance is complementary to the ATM.
A. C OMPARISON BETWEEN FRAME - LEVEL AND UTTERANCE - LEVEL L OSS SCALING
ATM+S performs MSM loss scaling using the utterance-level conﬁdences which performs focus on each utterance at a coarse level.
We also experimented with scaling with frame-level conﬁdence scores.
Our experiments showed that scaling all utterances with frame-level conﬁdence hurts the model performance.
To solve this issue, we randomly selected utterances which participate in frame scaling.
Scaling 10% utterances resulted in better performance and the results are shown in Table
[Table] [Table]
A. R ESULTS ON C OMMONVOICE WITH PUNCTUATION NORMALIZATION
Some previous work, including Likhomanenko et al.
(2020), report Commonvoice results by scoring with a normalization process that removes punctuation.
For comparison, we do the same in Table 10.
[Table]
[Figure]
B LOCK DIAGRAMMATIC VIEW OF MSM ARCHITECTURES
[Figure]
BottleneckFeatures
[Figure] [Figure]