Original: Continuous Soft Pseudo-Labeling in ASR
Revised: Continuous Soft Pseudo-Labeling in ASR

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Continuous pseudo-labeling (PL) algorithms such as slimIPL have recentlyemerged as a powerful strategy for semi-supervised learning in speech recognition.
Revised: Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently emerged as a powerful strategy for semi-supervised learning in speech recognition.

Original: In contrast with earlier strategies that alternated between training a modeland generating pseudo-labels (PLs) with it, here PLs are generated in end-to-endmanner as training proceeds, improving training speed and the accuracy of thefinal model.
Revised: In contrast with earlier strategies that alternated between training a model and generating pseudo-labels (PLs) with it, here PLs are generated in end-to-end manner as training proceeds, improving training speed and the accuracy of the final model.

Original: PL shares a common theme with teacher-student models such asdistillation in that a teacher model generates targets that need to be mimicked bythe student model being trained.
Revised: PL shares a common theme with teacher-student models such as distillation in that a teacher model generates targets that need to be mimicked by the student model being trained.

Original: However, interestingly, PL strategies in generaluse hard-labels, whereas distillation uses the distribution over labels as the targetto mimic.
Revised: However, interestingly, PL strategies in general use hard-labels, whereas distillation uses the distribution over labels as the target to mimic.

Original: Inspired by distillation we expect that specifying the whole distribution(aka soft-labels) over sequences as the target for unlabeled data, instead of a singlebest pass pseudo-labeled transcript (hard-labels) should improve PL performanceand convergence.
Revised: Inspired by distillation we expect that specifying the whole distribution (aka soft-labels) over sequences as the target for unlabeled data, instead of a single best pass pseudo-labeled transcript (hard-labels) should improve PL performance and convergence.

Original: Surprisingly and unexpectedly, we find that soft-labels targetscan lead to training divergence, with the model collapsing to a degenerate tokendistribution per frame.
Revised: Surprisingly and unexpectedly, we find that soft-labels targets can lead to training divergence, with the model collapsing to a degenerate token distribution per frame.

Original: We hypothesize that the reason this does not happen withhard-labels is that training loss on hard-labels imposes sequence-level consistencythat keeps the model from collapsing to the degenerate solution.
Revised: We hypothesize that the reason this does not happen with hard-labels is that training loss on hard-labels imposes sequence-level consistency that keeps the model from collapsing to the degenerate solution.

Original: In this paper, weshow several experiments that support this hypothesis, and experiment with severalregularization approaches that can ameliorate the degenerate collapse when usingsoft-labels.
Revised: In this paper, we show several experiments that support this hypothesis, and experiment with several regularization approaches that can ameliorate the degenerate collapse when using soft-labels.

Original: These approaches can bring the accuracy of soft-labels closer to thatof hard-labels, and while they are unable to outperform them yet, they serve as auseful framework for further improvements.
Revised: These approaches can bring the accuracy of soft-labels closer to that of hard-labels, and while they are unable to outperform them yet, they serve as a useful framework for further improvements.

Original:  Introduction and Related Works
Revised:  Introduction and Related Works

Original: The Deep Learning community has recently made significant progress in semi-supervised learningvia self-training or pseudo-labeling (PL) [1] in vision, speech and other domains [2, 3, 4, 5, 6, 7, 8,9, 10, 11].
Revised: The Deep Learning community has recently made significant progress in semi-supervised learning via self-training or pseudo-labeling (PL) [1] in vision, speech and other domains [2, 3, 4, 5, 6, 7, 8, 9, 10, 11].

Original: In this paper we focus on self-training in the context of automatic speech recognition(ASR).
Revised: In this paper we focus on self-training in the context of automatic speech recognition (ASR).

Original: ASR presents some unique challenges in self-training that are not seen in some other tasksstemming from their sequential nature.
Revised: ASR presents some unique challenges in self-training that are not seen in some other tasks stemming from their sequential nature.

Original: Inputs and targets have different lengths and targets are notproduced with the same regularity, compared to the inputs.
Revised: Inputs and targets have different lengths and targets are not produced with the same regularity, compared to the inputs.

Original: Further, ASR models are often coupledwith language models, that can be additionally used to aid the pseudo-labeling process.
Revised: Further, ASR models are often coupled with language models, that can be additionally used to aid the pseudo-labeling process.

Original: In semi-supervised learning [12] we have access to labeled and unlabeled data, where the unlabeleddata is typically much larger, since it is often easier to find unlabeled data, compared to labeleddata.
Revised: In semi-supervised learning [12] we have access to labeled and unlabeled data, where the unlabeled data is typically much larger, since it is often easier to find unlabeled data, compared to labeled data.

Original: PL, in its simplest formulation, employs a base model trained with labeled data which is laterused as a “teacher” to generate predictions on the unlabeled data.
Revised: PL, in its simplest formulation, employs a base model trained with labeled data which is later used as a “teacher” to generate predictions on the unlabeled data.

Original: The unlabeled data is converted to“pseudo-labeled” data through the addition of these teacher generated targets.
Revised: The unlabeled data is converted to “pseudo-labeled” data through the addition of these teacher generated targets.

Original: A student model is thentrained on the combination of labeled and pseudo-labeled data to yield a final model.
Revised: A student model is then trained on the combination of labeled and pseudo-labeled data to yield a final model.

Original: PL with ASR has recently been applied to three different types of end-to-end models – Connectionist
Revised: PL with ASR has recently been applied to three different types of end-to-end models – Connectionist

Original: Temporal Classification (CTC) [13], Transducer [14], and sequence-to-sequence [15].
Revised: Temporal Classification (CTC) [13], Transducer [14], and sequence-to-sequence [15].

Original: To achieve state-of-the-art results, PLs are generated from the models using an external language model (LM)either with beam-search decoding [16, 6, 17] or through shallow fusion [18, 19, 20, 21].
Revised: To achieve state-of-the-art results, PLs are generated from the models using an external language model (LM) either with beam-search decoding [16, 6, 17] or through shallow fusion [18, 19, 20, 21].

Original: Unlike invision, here PLs are sequences of tokens, such as characters 1 .
Revised: Unlike in vision, here PLs are sequences of tokens, such as characters 1 .

Original: With this setting, however, acousticmodels tend to overfit to the text training set of the LM used for PL [6, 20, 11].
Revised: With this setting, however, acoustic models tend to overfit to the text training set of the LM used for PL [6, 20, 11].

Original: Recent approaches on PL in ASR have attempted to rectify this problem through continuous training [22, 5], in which the model that is being trained is the same model that is used to generate PLs,and PLs are generated without using an LM.
Revised: Recent approaches on PL in ASR have attempted to rectify this problem through continuous training [22, 5], in which the model that is being trained is the same model that is used to generate PLs, and PLs are generated without using an LM.

Original: The main challenge for continuous PL is stability, whichwas reported independently by several authors [7, 9, 10, 11] and resolved partially by introducingeither a caching mechanism [7] which kept an evolving cache of PL data or by using an Exponential
Revised: The main challenge for continuous PL is stability, which was reported independently by several authors [7, 9, 10, 11] and resolved partially by introducing either a caching mechanism [7] which kept an evolving cache of PL data or by using an Exponential

Original: Moving Average (EMA) of the model being trained to generate the PLs [8, 9, 10, 11].
Revised: Moving Average (EMA) of the model being trained to generate the PLs [8, 9, 10, 11].

Original: This continuoustraining greatly simplifies the pipeline, improves convergence speed and achieves state-of-the-artcompared to other semi-supervised approaches [7].
Revised: This continuous training greatly simplifies the pipeline, improves convergence speed and achieves state-of-the-art compared to other semi-supervised approaches [7].

Original: Interestingly, the above strategies in general use one (or a small number) of PLs from the teacher totrain the student.
Revised: Interestingly, the above strategies in general use one (or a small number) of PLs from the teacher to train the student.

Original: By contrast distillation [23] and soft-labeling [3, 4] approaches use the predicteddistribution over labels from the teacher as the target to mimic.
Revised: By contrast distillation [23] and soft-labeling [3, 4] approaches use the predicted distribution over labels from the teacher as the target to mimic.

Original: Inspired by these works we expectthat specifying the whole distribution (aka soft-labels) over sequences as the target for unlabeleddata should improve performance and convergence.
Revised: Inspired by these works we expect that specifying the whole distribution (aka soft-labels) over sequences as the target for unlabeled data should improve performance and convergence.

Original: However, with CTC-based models for labeleddata we find experimentally that training with soft-labels leads to models where the distributionconverges to a degenerate solution, e.g. emitting only the blank token (this only happens in continuouspseudo-labeling, while teacher-student exhibit stable soft-labeling, see Appendix E.2).
Revised: However, with CTC-based models for labeled data we find experimentally that training with soft-labels leads to models where the distribution converges to a degenerate solution, e.g. emitting only the blank token (this only happens in continuous pseudo-labeling, while teacher-student exhibit stable soft-labeling, see Appendix E.2).

Original: By contrast,hard-labels are more stable and collapse to a degenerate solution is not observed.
Revised: By contrast, hard-labels are more stable and collapse to a degenerate solution is not observed.

Original: We believe thisis because structured CTC loss used for hard-labels imposes sequence-level consistency while losson soft-labels (cross-entropy or L 2 ) does not have this property, could destroy the sequence-levelrepresentation and thus easily converge to the degenerate solution, quite bad for generalization.
Revised: We believe this is because structured CTC loss used for hard-labels imposes sequence-level consistency while loss on soft-labels (cross-entropy or L 2 ) does not have this property, could destroy the sequence-level representation and thus easily converge to the degenerate solution, quite bad for generalization.

Original: Weapply different regularization approaches to try and prevent degenerate solutions such as entropyregularization, smoothing of the soft PLs, sampling from the teacher distribution [24, 25], blendingthe soft PLs with hard PLs and PLs filtering.
Revised: We apply different regularization approaches to try and prevent degenerate solutions such as entropy regularization, smoothing of the soft-labels, sampling from the teacher distribution [24, 25], blending the soft-labels with hard-labels and PLs filtering.

Original: These approaches can bring the accuracy of soft-labelscloser to that of hard-labels, and while they are unable to outperform them yet, they serve as a usefulframework for further improvements.
Revised: These approaches can bring the accuracy of softlabels closer to that of hard-labels, and while they are unable to outperform them yet, they serve as a useful framework for further improvements.

Original:  Background on Pseudo-Labeling and slimIPL
Revised:  Background on Pseudo-Labeling and slimIPL

Original: Let L = { x i , y i } and U = { x j } be the labeled and unlabeled datasets respectively.
Revised: Let L = { x i , y i } and U = { x j } be the labeled and unlabeled datasets respectively.

Original: We consider a
Revised: We consider a

Original: PL approach where an acoustic model
Revised: PL approach where an acoustic model

Original: A ( x ; θ ) with model parameters θ is continuously trained on acombination of L and a pseudo-labelled set derived from U .
Revised: A ( x ; θ ) with model parameters θ is continuously trained on a combination of L and a pseudo-labelled set derived from U .

Original: The model is trained by minimizing aloss L ( θ ) =
Revised: The model is trained by minimizing a loss L ( θ ) =

Original: L L ( θ )+ λ L U ( θ ) where λ ∈ R + is a tunable hyper-parameter controlling the importanceof unlabeled data.
Revised: L L ( θ )+ λ L U ( θ ) where λ ∈ R + is a tunable hyper-parameter controlling the importance of unlabeled data.

Original: The loss for labeled data is defined as L L ( θ ) =
Revised: The loss for labeled data is defined as L L ( θ ) =

Original: − E ( x , y ) ∼ L log p θ ( y | x ) , wherep θ ( y | x ) is the conditional distribution defined by model A ( x ; θ ) .
Revised: − E ( x , y ) ∼ L log p θ ( y | x ) , where p θ ( y | x ) is the conditional distribution defined by model A ( x ; θ ) .

Original: The loss for unlabeled data isdefined as L U ( θ ) =
Revised: The loss for unlabeled data is defined as L U ( θ ) =

Original: − E x ∼
Revised: − E x ∼

Original: U log p θ (ˆ y | x ) , where ˆ y is the PL transcription for a data point generatedusing the model being trained and a prior LM.
Revised: U log p θ (ˆ y | x ) , where ˆ y is the PL transcription for a data point generated using the model being trained and a prior LM.

Original: Specifically,
Revised: Specifically,

Original: [Equation]
Revised: [Equation]

Original:  where p lm ( y ) is an LM, and α ≥ 0 is a hyper-parameter controlling the reliance on the LM.
Revised: (1) where p lm ( y ) is an LM, and α ≥ 0 is a hyper-parameter controlling the reliance on the LM.

Original: Theabove minimization is usually approximated with a beam-search decoding algorithm [19].
Revised: The above minimization is usually approximated with a beam-search decoding algorithm [19].

Original: In recentworks it was found that α > 0 gives stable training but can lead to over-fitting to the language modelused [6] while α = 0 can lead to training instability and divergence [7].
Revised: In recent works it was found that α > 0 gives stable training but can lead to over-fitting to the language model used [6] while α = 0 can lead to training instability and divergence [7].

Original: To resolve these issues an LM free ( α = 0 ) continuous PL approach called slimIPL was recently proposed for ASR [7].
Revised: To resolve these issues an LM free ( α = 0 ) continuous PL approach called slimIPL was recently proposed for ASR [7].

Original: It also optimizes further the speed of the PL generation: instead of Eq.
Revised: It also optimizes further the speed of the PL generation: instead of Eq.

Original: (1) with α = 0 which defineshard-transcriptions, slimIPL uses hard-alignment (defined in Eq.
Revised: (1) with α = 0 which defines hard-transcriptions, slimIPL uses hard-alignment defined in Eq.

Original: Models trained with the CTCloss output distributions over token set w (including  auxiliary blank token) for every output frame,or timestamp, t = 1 , T as p t θ ( w | x ) , w ∈ w , where T is the total number of frames in the sample.
Revised: Models trained with the CTC loss output distributions over token set w (including an auxiliary blank token) for every output frame, or timestamp, t = 1 , T as p t θ ( w | x ) , w ∈ w , where T is the total number of frames in the sample.

Original: The CTC lossthus maximizes the probability  of a transcription y across all possible alignments 2 a 
Revised: The CTC loss maximizes the probability p θ ( y | x ) of a transcription y across all possible alignments 2 a = ( a 1 , . .

Original: = ( a 1 , . . . , a T ) which lead to y : (cid:80)
Revised: . , a T ) which lead to y assuming factorization p θ ( y | x ) =

Original:  a → y (cid:81) Tt =1 p t θ ( a t | x )→ max .
Revised: (cid:80) a → y (cid:81) Tt =1 p t θ ( a t | x ) .

Original: In order to get PLs generatedby Eq.
Revised: In order to get PLs generated by Eq.

Original: (1) ( α = 0 ) we would need to compute the total probability of the transcription y by summingprobabilities of every alignment a → y and then taking the most probable y .
Revised: (1) ( α = 0 ) we would need to compute the total probability of the transcription y by summing probabilities of every alignment a → y and then taking the most probable y .

Original: This is intractable butbeam search could be used in this case in order to approximate the argmax search.
Revised: This is intractable but beam search could be used in this case in order to approximate the argmax search.

Original: With slimIPL, theauthors decided to use a simpler approach – hard-alignment (or hard-path), obtained with:
Revised: With slimIPL, the authors decided to use a simpler approach – hard-alignment (or hard-path), obtained with:

Original: [Equation]
Revised: [Equation]

Original: where for every frame the most probable token is ˆ a i .
Revised: where for every frame the most probable token is ˆ a i .

Original: From there, the final pseudo-label transcriptionˆ y is obtained through the transformation denoted by G , which removes repeated tokens as well asthe CTC blank token.
Revised: From there, the final pseudo-label transcription ˆ y is obtained through the transformation denoted by G , which removes repeated tokens as well as the CTC blank token 3 .

Original: This is equivalent to minimizing Eq.
Revised: This is equivalent to minimizing Eq.

Original: (1) with a (lexicon-free) beam searchconstrained to a beam size of 1.
Revised: (1) with a (lexicon-free) beam search constrained to a beam size of 1.

Original: A schematic overview of slimIPL is presented in Appendix A Algorithm 1.
Revised: A schematic overview of slimIPL is presented in Appendix A Algorithm 1.

Original: slimIPL exploits adynamic cache, which cleverly facilitates access to PLs from previous model states (this can beviewed as model ensemble for PL generation).
Revised: slimIPL exploits a dynamic cache, which cleverly facilitates access to PLs from previous model states (this can be viewed as model ensemble for PL generation).

Original: This dynamic cache was proposed in order to stabilizethe optimization and to avoid sudden model divergence due to collapsing to empty transcriptionpredictions.
Revised: This dynamic cache was proposed in order to stabilize the optimization and to avoid sudden model divergence due to collapsing to empty transcription predictions.

Original: slimIPL has several hyperparameters: (i) λ , (ii) M , the training step at which PLgeneration begins, (iii) C , the dynamic cache size and (iv) p , the probability of updating the cache.
Revised: slimIPL has several hyper-parameters: (i) λ , (ii) M , the training step at which PL generation begins, (iii) C , the dynamic cache size and (iv) p , the probability of updating the cache.

Original: Our goal in the next sections is to investigate how “soft”-labeling can further improve slimIPL as webelieve it may contain richer information compared to hard-labels.
Revised: Our goal in the next sections is to investigate how “soft”-labeling can further improve slimIPL as we believe it may contain richer information compared to hard-labels.

Original: Interestingly, ”soft“-labeling iscomputationally cheap, similar to the cost implied by hard-paths obtained from Eq.
Revised: Interestingly, “soft”-labeling is computationally cheap, similar to the cost implied by hard-paths obtained from Eq.

Original: (2), and muchcheaper than with pseudo-labels which would be obtained by minimizing Eq.
Revised: (2), and much cheaper than with pseudo-labels which would be obtained by minimizing Eq.

Original: (1) (via a beam search).
Revised: (1) (via a beam search)

Revised: but requires more memory.

Original: We start our analysis from the best models found in slimIPL work [7].
Revised: We start our analysis from the best models found in slimIPL work [7].

Original:  Experimental Setup and Methods
Revised:  Experimental Setup and Methods

Original: Inspired by results on soft-labeling in vision [3, 4] and knowledge distillation [23] our empiricalhypothesis is the following: exploiting the whole distribution over sequences should be more powerfulfor PL and bring benefit in either faster convergence and / or lower WER at the end of training.
Revised: Inspired by results on soft-labeling in vision [3, 4] and knowledge distillation [23] our empirical hypothesis is the following: exploiting the whole distribution over sequences should be more powerful for PL and bring benefit in either faster convergence and / or lower error at the end of training.

Original: Inthis section we describe the data, the training setup and methods used to test this hypothesis.
Revised: In this section we describe the data, the training setup and methods used to test this hypothesis.

Original: Data In this paper we use the LibriSpeech dataset [26] and the train-10h subset of Libri-Light [27].
Revised: We use the LibriSpeech dataset [26] and the train-10h subset of Libri-Light [27].

Original: We use the train-clean-360 and train-other-500 standard LibriSpeech subsets as unlabeled data andconsider either 10h ( train-10h ) or 100h ( train-clean-100 ) as the labeled data.
Revised: We use the train-clean-360 and train-other-500 standard LibriSpeech subsets as unlabeled data and consider either 10h ( train-10h ) or 100h ( train-clean-100 ) as the labeled data.

Original: The standard setsdev-clean and dev-other are used to tune all hyperparameters and to select the best models.
Revised: The standard sets dev-clean and dev-other are used to tune all hyper-parameters and to select the best models.

Original: The setstest-clean and test-other are used to report final token (TER) and word (WER) error rates.
Revised: The sets test-clean and test-other are used to report final token (TER) and word (WER) error rates.

Original: Training setup We closely follow slimIPL experiments [7] with the same hyper-parameters,including transformer model, CTC loss and character tokens (see Appendix B).
Revised: We closely follow slimIPL experiments [7] with the same hyper-parameters, including transformer model, CTC loss and character tokens (see Appendix B).

Original: We make onlyfew changes: a) to speed up training and decrease memory usage we get rid of relative positionalembedding [28] and use CAPE [29] instead; b) we train with larger batch per GPU but on 8 GPUs.
Revised: We make only few changes: a) to speed up training and decrease memory usage we get rid of relative positional embedding [28] and use CAPE [29] instead; b) we train with larger batch per GPU but on 8 GPUs.

Original: Inour experiments we only vary C , p and the type of PL (soft-labels vs hard-alignment), while keepingeverything else fixed.
Revised: In our experiments we only vary C , p and the type of PL (soft-labels vs hard-alignment), while keeping everything else fixed.

Original: We run experiments with 3 seeds and we report mean and standard deviation asthe final performance.
Revised: We run experiments with 3 seeds and we report mean and standard deviation as the final performance.

Original: Soft-labeling slimIPL
Revised: Soft-labeling slimIPL

Original: Even though speech recognition involves a sequence-to-sequence alignmentproblem, we are going to perform “soft”-labeling on a per-frame basis.
Revised: Even though speech recognition involves a sequence-to-sequence alignment problem, we are going to perform “soft”-labeling on a per-frame basis.

Original: In that respect, the soft-labelstrategy is similar than what is commonly used for non-sequential data.
Revised: In that respect, the soft-label strategy is similar than what is commonly used for non-sequential data.

Original: Formally, let’s denotelogits as z tw =
Revised: Formally, let’s denote logits as z tw =

Original: log p t θ ( w | x ) and PL logits ˆ z tw = log p t θ ′ ( w | x ) for x ∈ U .
Revised: log p t θ ( w | x ) and PL logits ˆ z tw = log p t θ ′ ( w | x ) for x ∈ U .

Original: We also denote asq tw ( x ; τ ) =
Revised: We also denote as q tw ( x ; τ ) =

Revised: e ztw/τ

Original: [Equation]
Revised: [Equation]

Original:  e ztw
Revised: w ′ e ztw

Original: [Equation] [Equation]
Revised: /τ

Original: and ˆ q tw ( x ; τ ) =
Revised: and ˆ q tw ( x ; τ ) =

Original: e ˆ ztw/τ (cid:80) w ′ e
Revised: e ˆ ztw/τ (cid:80) w ′ e

Original: ˆ ztw ′
Revised: ˆ ztw ′

Original: /τ .
Revised: /τ .

Original: Then the loss function on unlabeled data x ∈ Uis defined in a similar way than the distillation loss [23]: either as cross-entropy (CE) with temperatureτ > 0 or as a L 2 regression (equivalent to CE when τ = ∞ , see [23]):
Revised: Then the loss function on unlabeled data x ∈ U is defined in a similar way than the distillation loss [23]: either as cross-entropy (CE) with temperature τ > 0 or as a L 2 regression (equivalent to CE when τ = ∞ , see [23]):

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: In order to keep the same λ (ratio between labeled and unlabeled data loss, introduced in Section 2)for both hard-labels and soft-labels experiments
Revised: In order to keep the same λ (ratio between labeled and unlabeled data losses, introduced in Section 2) for both hard-labels and soft-labels experiments

Original: [Figure]
Revised: [Figure]

Original: we add a re-scaling factor β > 0 in Eq.
Revised: we add a re-scaling factor β > 0 in Eq.

Original: [Figure]
Revised: [Figure]

Original: whichprovides a way to keep gradients computed for L U and L U,ce / L U,L 2 at the same scale 4 .
Revised: which provides a way to keep gradients computed for L U and L U,ce / L U,L 2 at the same scale 4 .

Original:  Empirical Study and Discussion
Revised:  Empirical Study and Discussion

Original: Training with soft-labels is unstable First experiments with both soft-labels losses reveal sometraining instability, when using the same cache size C and probability p than with hard-labels:training diverges, with WER increasing on validation sets as well.
Revised: Training with soft-labels is unstable First experiments with both soft-labels losses reveal some training instability, when using the same cache size C and probability p than with hard-labels: training diverges, with WER increasing on validation sets as well.

Original: The WER between PLs and theircorresponding golden transcription also diverges.
Revised: The WER between PLs and their corresponding golden transcription also diverges.

Original: By increasing the cache size by 10x (from 100to 1k) and lowering its probability (0.1) the training is however stabilized.
Revised: By increasing the cache size by 10x (from 100 to 1k) and lowering its probability (0.1) the training is however stabilized (Figure 1).

Original: We performed similarexperiments with EMA [8, 11] and observed the same trend: the EMA decay factor needs to beincreased (from 0.999 to 0.9999/0.99995) compared to the one used with hard-alignments.
Revised: We performed similar experiments with EMA [8, 11] and observed the same trend: the EMA decay factor needs to be increased (from 0.999 to 0.9999/0.99995) compared to the one used with hard-alignments.

Original: Previouslyit was reported that EMA and slimIPL are robust to hyper-parameters and that using a small cache ora larger value of p would still lead to stable training, and similar WER results.
Revised: Previously it was reported that EMA and slimIPL are robust to hyper-parameters and that using a small cache or a larger value of p would still lead to stable training, and similar WER results.

Original: All our experimentsdemonstrate that this is not the case with soft-labeling.
Revised: All our experiments demonstrate that this is not the case with soft-labeling.

Original: Moreover, the best soft-labeling performingmodel ( τ = 10 and β = 1 ) has worse WER performance than when using hard-alignment, as shownin Table 1 5 .
Revised: Moreover, the best soft-labeling performing model ( τ = 10 and β = 1 ) has worse WER performance than when using hard-alignment, as shown in Table 1 5 .

Original: The same 1k cache for hard-alignment gives worse WER performance than with 100cache (as PLs are then more slowly updated), but it is still better than with soft-labels.
Revised: The same 1k cache for hard-alignment gives worse WER performance than with 100 cache (as PLs are then more slowly updated), but it is still better than with soft-labels.

Original: Soft-labeling easily collapses to blank To understand instability issues in soft-labeling, we trackhow the PLs of a subset of the unlabeled samples evolve over thetraining, see Appendix C Figure 8(top), in the case of soft-labeling training with C = 100 , p = 0 . 
Revised: Soft-labeling easily collapses to blank To understand instability issues in soft-labeling, we track how the PLs of a subset of the unlabeled samples evolve over the training with C = 100 , p = 0 . [Figure] Appendix C Figure 8 (top).

Original: Interestingly, only several (23) tokens dominate every frame during the training.
Revised: Interestingly, only several (2-3) tokens dominate every frame during the training.

Original: When the model diverges,  the blank tokenprobability increases over all frames, ultimately taking over the whole distribution mass of soft-labels.
Revised: When the model diverges, e.g. the blank token probability increases over all frames, ultimately taking over the whole distribution mass of soft-labels.

Original: [Figure] level transitions are noisy.
Revised: Hard-labels frame-level transitions are noisy.

Original: Sequence-level ones are smooth.
Revised: Sequence-level ones are smooth.

Original: Comparedto soft-labels, hard-path dynamics (Appendix C Figure 6) exhibit non-smooth transition betweenalignments.
Revised: Compared to soft-labels, hard-path dynamics (Appendix C Figure 6) exhibit non-smooth transition between alignments.

Original: Instead of looking at each sample individually, we summarize the evolution of the samesubset of unlabeled samples in Figure 4 (for hard-labels) and Figure 5 (for soft-labels).
Revised: Instead of looking at each sample individually, we summarize the evolution of the same subset of unlabeled samples in Figure 4 (for hard-labels) and Figure 5 (for soft-labels).

Original: We trackthe average distance between the PLs generated for the same sample, for two consecutive iterations.
Revised: We track the average distance between the PLs generated for the same sample, for two consecutive iterations.

Original: We report the frame-level Levenshtein distance of these consecutive PLs, after converting them intohard-alignements ˆ a (whether the PLs were soft or hard).
Revised: We report the frame-level Levenshtein distance of these consecutive PLs, after converting them into hard-alignments ˆ a (whether the PLs were soft or hard).

Original: We also report the Levenshtein distanceat the sequence level ˆ y .
Revised: We also report the Levenshtein distance at the sequence level ˆ y .

Original: When tracking individual samples, we observe more transition noise inhard-labels, than for soft-labels.
Revised: When tracking individual samples, we observe more transition noise in hard-labels, than for soft-labels.

Original: However, at the sequence level, hard-labels PLs transitions aresignificantly smoother as the training goes.
Revised: However, at the sequence level, hard-labels PLs transitions are significantly smoother as the training goes.

Original: Sequence-level PLs may lead to better WER
Revised: Sequence-level PLs may lead to better WER

Original: We now consider Eq.
Revised: We now consider Eq.

Original: (1) with beam sizes from10 to 100 to understand if a richer sequence-level representation can be helpful, by turning off the
Revised: (1) with beam sizes from 10 to 500 to understand if a richer sequence-level representation can be helpful, by turning off the

Original: LM ( α = 0 ).
Revised: LM ( α = 0 ).

Original: Figure 2 top shows that beam and hard-paths perform similarly.
Revised: Figure 2 top shows that beam and hard-path perform similarly.

Original: Moreover, the WERbetween PLs generated from beam search and PLs generated from hard-paths becomes smaller than2% (right plot) very quickly (after 30k updates).
Revised: Moreover, the WER between PLs generated from beam search and PLs generated from hard-path becomes smaller than 2% (right plot) very quickly (after 30k updates).

Original: However, Figure 2 bottom, shows that if we switchto using only 10h of supervision, we observe a 1.5-2% absolute decrease in WER from increasing thebeam size to 50-100, while the PLs from beam and hard-alignments are still similar (WER=1-3%).
Revised: However, Figure 2 bottom, shows that if we switch to using only 10h of supervision, we observe a 1.5-2% absolute decrease in WER from increasing the beam size to 50-100, while the PLs from beam and hard-path are still similar (WER is 1-3%).

Original: Based on the above observations we decided to investigate several ways to improve soft-labeling:regularizing the soft-label distribution in the hope to avoid blank collapses, sampling as a way toregularize soft-labels transitions, and finally combining soft and hard-labels.
Revised: Based on the above observations we decided to investigate several ways to improve soft-labeling: regularizing the soft-label distribution in the hope to avoid blank collapses, sampling as a way to regularize soft-labels transitions, and finally combining soft and hard-labels.

Original: 4.1 CTC blank regularization does not solve instability issue
Revised: 4.1 CTC blank regularization does not solve instability issue

Original: To resolve the issue of collapsing to blank tokens we start with different regularization schemes toconstrain the predictive distribution to be close to a specified prior distribution.
Revised: To resolve the issue of collapsing to blank tokens we start with different regularization schemes to constrain the predictive distribution to be close to a specified prior distribution.

Original: All variants we triedslow down divergence, but were unable to improve on our best slimIPL results.
Revised: All variants we tried slow down divergence, but were unable to improve on our best slimIPL results (Figure 1).

Original: The different priordistribution we tried are as follows:
Revised: The different prior distribution we tried are as follows:

Original: Entropy regularization: This prior attempts to maximize the entropy of the predictive distribution.
Revised: Entropy regularization: This prior attempts to maximize the entropy of the predictive distribution.

Original: Sequence-level prior : The average predicted probability vector across all time steps for a sampleis regularized to be close to (using KL-divergence) either (i) a uniform prior across tokens, or (ii) atoken distribution computed from train-clean-100 text.
Revised: Sequence-level prior : The average predicted probability vector across all time steps for a sample is regularized to be close to (using KL-divergence) either (i) a uniform prior across tokens, or (ii) a token distribution computed from train-clean-100 text.

Original: Since text does not have a blank token, theprior for the blank token is estimated by a supervised model trained on train-clean-100 .
Revised: Since text does not have a blank token, the prior for the blank token is estimated by a supervised model trained on train-clean-100 .

Original: Frame-level prior : Each frame is regularized to either match a target blank probability, or thepredictions and/or targets as label-smoothed by averaging with a uniform distribution with weight γ ,which is varied from 0.01 to 0.1.
Revised: Frame-level prior : Each frame is regularized to either match a target blank probability, or the predictions and/or targets as label-smoothed by averaging with a uniform distribution with weight γ , [Figure] which is varied from 0.01 to 0.1.

Original: Logits averaging : Logits from neighboring frames are smoothed with window w = 1 · · · 10 frames.[Figure]
Revised: Logits averaging : Logits from neighboring frames are smoothed with window w = 1 · · · 10 frames.

Original: 4.2 Sampling as stable soft-labeling
Revised: 4.2 Sampling as stable soft-labeling

Original: Next, we investigate how sampling [24, 25] from the model performs because sampling encapsulatesthe uncertainty in the predictions.
Revised: Next, we investigate how sampling [24, 25] from the model performs because sampling encapsulates the uncertainty in the predictions.

Original: We generate ˆ y =
Revised: We generate ˆ y =

Original: G (ˆ a 1 , . .
Revised: G (ˆ a 1 , . .

Original: , ˆ a T ) where ˆ a t ∼ p t θ ( w | x ) for t = 1 , T .
Revised: , ˆ a T ) where ˆ a t ∼ p t θ ( w | x ) for t = 1 , T .

Original: A temperature τ is varied from 0.0001 to 10 to change the peakiness of the distribution [Figure] and CTC lossis used on PL data.
Revised: A temperature τ is varied from 0.0001 to 10 to change the peakiness of the distribution, and CTC loss is used on PL data.

Original: For τ → 0 sampling corresponds to the hard-path while τ → ∞ corresponds tothe uniform distribution over tokens 6 .
Revised: For τ → 0 sampling corresponds to the hard-path while τ → ∞ corresponds to the uniform distribution over tokens 6 .

Original: We can see from Figure 3 that sampling converges to hard-pathin the end (top) while in limited supervision (bottom) only τ
Revised: We can see from Figure 3 that sampling converges to hard-path in the end (top) while in limited supervision (bottom) only τ

Original: = 1 does not.
Revised: = 1 does not.

Original: As with hard-transcriptionwe observe that sampling is similar to hard-path for 100h and is better for 10h of supervision.
Revised: As with hard-transcription we observe that sampling is similar to hard-path for 100h and is better for 10h of supervision.

Original: Unlikesoft-labels, sampled PLs are stable like the hard-path PLs.
Revised: Unlike soft-labels, sampled PLs are stable like the hard-path PLs.

Original: 4.3 Blending soft-labels with hard-labels as regularization
Revised: 4.3 Blending soft-labels with hard-labels as regularization

Original: We observed and discussed above, when models are trained with hard-alignments, hard-transcriptionsor sampling, their frame-level predictions vary a lot from step to step during training.
Revised: We observed and discussed above, when models are trained with hard-alignments, hard-transcriptions or sampling, their frame-level predictions vary a lot from step to step during training.

Original: In contrastsoft-labels do not display such rapid changes in the predictive distribution (compare Appendix C
Revised: In contrast soft-labels do not display such rapid changes in the predictive distribution (compare Appendix C

Original: Figure 4 vs Figure 5).
Revised: Figure 4 vs Figure 5).

Original: We hypothesize that soft-labeling loss is trivial to optimize to degeneratesolutions and the model over-fits to it easily while memorizing the entire labeled set (see Appendix D).
Revised: We hypothesize that soft-labeling loss is trivial to optimize to degenerate solutions and the model over-fits to it easily while memorizing the entire labeled set (see Appendix D).

Original: Meanwhile, hard PL versions provide sequence level targets which might prevent a collapse.
Revised: Meanwhile, hard PL versions provide sequence level targets which might prevent a collapse.

Original: So wecombine hard-path PL and soft-PL using δ L U + (1 − δ )
Revised: So we combine hard-path PL and soft-PL using δ L U + (1 − δ )

Original: L U,ce , δ = { 0 . 01 , 0 . 1 , 0 . 5 , 0 . 9 , 0 . 99 } .
Revised: L U,ce , δ = { 0 . 01 , 0 . 1 , 0 . 5 , 0 . 9 , 0 . 99 } .

Original: Thisis the most effective way to stabilize training: e.g. see Appendix E.3 Figure 14 with τ = 1 , C = 100 .
Revised: This is the most effective way to stabilize training: e.g. see Appendix E.3 Figure 15 with τ = 1 , C = 100 .

Original: Further, with this blend models can be trained with 10h of labeled data which was not possible withsoft-labels alone (see Table 1).
Revised: Further, with this blend models can be trained with 10h of labeled data which was not possible with soft-labels alone (see Table 1).

Original: The strategy also works better than the other soft-PL variants (Table 1)while converging 1.5-2x faster (see Figure 1).
Revised: The strategy also works better than the other soft-PL variants (Table 1) while converging 1.5-2x faster (see Figure 1).

Original: However some issues still remain as it slowly starts todiverge at the very end of training, and the results are slightly worse than hard-path.
Revised: However some issues still remain as it slowly starts to diverge at the very end of training, and the results are slightly worse than hard-path.

Original:  Conclusion
Revised:  Conclusion

Original: At first glance, a change from hard-labels to soft-labels for continuous pseudo-labeling in ASRseemed simple and obvious, but our experiments revealed entirely unexpected results.
Revised: At first glance, a change from hard-labels to soft-labels for continuous pseudo-labeling in ASR seemed simple and obvious, but our experiments revealed entirely unexpected results.

Original: Soft-labelingwas found to quickly drive the PLs that are generated continuously to trivial, degenerate solutionswhile memorizing the entire labeled data at the same time.
Revised: Soft-labeling was found to quickly drive the PLs that are generated continuously to trivial, degenerate solutions while memorizing the entire labeled data at the same time.

Original: We found that different regularizationapproaches help  improve stability of soft-label training but are unable to match the accuracy ofhard-label trained models.
Revised: We found that different regularization approaches help to improve stability of soft-label training but are unable to match the accuracy of hard-label trained models.

Original: Hard-transcription and sampling from the distribution on the other handshow stability and improved results in lower supervision setting.
Revised: Hard-transcription and sampling from the distribution on the other hand show stability and improved results in lower supervision setting.

Original: We found that combining bothhard-alignment PLs with CTC loss and soft-label PLs with cross-entropy loss on unsupervised dataleads to improved results.
Revised: We found that combining both hard-alignment PLs with CTC loss and soft-label PLs with cross-entropy loss on unsupervised data leads to improved results.

Original: This almost matches hard-alignment and speeds up convergence by 1.5-2x.
Revised: This almost matches hard-alignment and speeds up convergence by 1.5-2x.

Original: B Empirical setup
Revised: A slimIPL Algorithm Overview

Original: {+Result: Acoustic model A ( x ; θ ) For random x ∈ U generate ˆ y = PL ( A inference ( x ; θ k )) and C ← C (cid:83) { x , ˆ y } ; θ k +
 =+}
Revised: {+Result: Acoustic model A ( x ; θ ) For random x ∈ U generate ˆ y = PL ( A inference ( x ; θ k )) and C ← C (cid:83) { x , ˆ y } ; θ k +
 =+}

Revised: θ k − η k ∇L L ( A (˜ x ; θ k ) , y ) ,

Revised: ( x , y ) ∈ L ; endrepeatif rand (0 , 1) < N L / ( N L + N U ) then

Original: {+Take random ( x , y ) ∈ L and θ k +
+}
Revised: {+Take random ( x , y ) ∈ L and θ k +
+}

Revised: =

Revised: θ k − η k ∇L L ( A (˜ x ; θ k ) , y ) else

Original: {+Take random b = ( x , y ) ∈ C and θ k +
+}
Revised: {+Take random b = ( x , y ) ∈ C and θ k +
+}

Revised: =

Revised: θ k − η k ∇L U

Revised: ( A ( ˜ x ; θ k ) , y ) ; if rand (0 , 1) <

Revised: p then

Revised: For random x ′ ∈ U generate ˆ y ′ =

Revised: PL ( A inference ( x ; θ k )) and C ← C \ b (cid:83) { x ′ , ˆ y ′ } end end k ← k + 1 ; until convergence or maximum iterations are reached ;

Revised: [Table]

Revised: B Empirical Setup

Original: We keep the original 16kHz sampling rate from LibriSpeech, and compute log-mel filterbanks with80 coefficients for a 25ms sliding window, strided by 10ms.
Revised: We keep the original 16kHz sampling rate from LibriSpeech, and compute log-mel filterbanks with 80 coefficients for a 25ms sliding window, strided by 10ms.

Original: All features are normalized to havezero mean and unit variance per input sequence before feeding them into the acoustic model.
Revised: All features are normalized to have zero mean and unit variance per input sequence before feeding them into the acoustic model.

Original: Weclosely follow the architecture and training pipeline from slimIPL, including same hyper-parameters,
Revised: We closely follow the architecture and training pipeline from slimIPL, including same hyper-parameters,

Original: CTC loss, transformer-based architecture, token set (26 English alphabet letters augmented with theapostrophe and a word boundary token) , SpecAugment [31] parameters, Adagrad optimizer [32] anddynamic batching.
Revised: CTC loss, transformer-based architecture, token set (26 English alphabet letters augmented with the apostrophe and a word boundary token), SpecAugment [31] parameters, Adagrad optimizer [32] and dynamic batching.

Original: We make only few changes: a) to speed up the training and decrease memoryusage, we replace relative positional embedding [28] by CAPE positional embedding [29] (only aglobal shift of 30s is used); b) we train all models on 8 A100 40GB GPUs with dynamic batching of∼ 290 audio per GPU, and FP32 tensor core computations for up to 500k updates.
Revised: We make only few changes: a) to speed up the training and decrease memory usage, we replace relative positional embedding [28] by CAPE positional embedding [29] (only a global shift of 30s is used); b) we train all models on 8 A100 40GB GPUs with dynamic batching of ∼ 290 audio per GPU, and FP32 tensor core computations for up to 500k updates.

Original: In all experimentswe only vary the PL type and few slimIPL hyper-parameters (cache size C and cache probabilityupdate p ), while keeping everything else fixed.
Revised: In all experiments we only vary the PL type and few slimIPL hyper-parameters (cache size C and cache probability update p ), while keeping everything else fixed.

Original: Most of experiments are run with 3 different seeds,and we report mean and standard deviation as the final performance.
Revised: Most of experiments are run with 3 different seeds, and we report mean and standard deviation as the final performance.

Original: We start with models where
Revised: We start with models where

Original: C = 100 , M = 20 , 000 , N U
Revised: C = 100 , M = 20 , 000 , N U

Original: = 3 , p = 0 .
Revised: = 3 , p = 0 .

Original: 1 for 100h and C = 1000 , M = 60 , 000 , N U = 10 , p = 0 . 1for 10h of supervision.
Revised: 1 for 100h and C = 1000 , M = 60 , 000 , N U = 10 , p = 0 .for 10h of supervision.

Revised: C Frame-Level and Sequence-Level Evolution in PLs

Original: [Figure]
Revised: [Figure]

Original: To further analyse convergence to a degenerate solution for soft-labels
Revised: j [Figure] [Figure] ( x

Revised: [Figure]

Revised: [Figure]

Revised: h a r

Original: [Figure]
Revised: [Figure]

Revised: L ( x

Original:  we keep tracking of PLsfor the same subset of unlabeled samples as training goes.
Revised: [Figure] j ) [Figure] To further analyse convergence to a degenerate solution for soft-labels, we keep tracking of PLs for the same subset of unlabeled samples as training goes (200 samples).

Original: We then compute the average distancebetween the PLs generated for the same sample, for two consecutive iterations.
Revised: We then compute the average distance between the PLs generated for the same sample, for two consecutive iterations.

Original: We report the framelevel Levenshtein distance of these consecutive PLs, after converting them into hard-alignementsˆ a (whether the PLs were soft or hard).
Revised: We report the frame-level Levenshtein distance of these consecutive PLs, after converting them into hard-alignments ˆ a (whether the PLs were soft or hard).

Original: Here we consider also top-2 and top-3 distances: we checkwhether ˆ a k ′ t is in top-2/top-3 tokens set of p t θ k ( w | x ) , where k and k ′ are two consecutive iterations,and k < k ′ .
Revised: Here we consider also top-2 distance: we check whether ˆ a jt is in top-2 tokens set of p t θ k ( w | x ) , where k and j are two consecutive iterations, and k < j .

Original: We also report the Levenshtein distance at the sequence level ˆ y .
Revised: We also report the Levenshtein distance at the sequence level ˆ y .

Original: Results (for cache
Revised: Results (for cache

Original: C = 100 and p = 0 . 1 ) are shown for hard-path, beam and sampling in Figure 4, and for differentvariants of soft-labels with regularization in Figure 5.
Revised: C = 100 and p = 0 . 1 ) are shown for hard-path, beam and sampling in Figure 4, and for different variants of soft-labels with regularization in Figure 5.

Original: When tracking individual samples, we observemore transition noise in hard-labels, than for soft-labels.
Revised: When tracking individual samples, we observe more transition noise in hard-labels, than for soft-labels.

Original: However, at the sequence level, hard-labels
Revised: However, at the sequence level, hard-labels

Original: PLs transitions are significantly smoother as the training goes.
Revised: PLs transitions are significantly smoother as the training goes.

Original: Moreover, for models convergedto a degenerate solution (two last rows in Figure 5) we observe first sequence-level and later alsoframe-level consistency destruction between PLs for two consecutive iterations.
Revised: Moreover, for models converged to a degenerate solution (Figure 5) we observe first sequence-level and later also frame-level consistency destruction between PLs for two consecutive iterations.

Original: Below in Figures 6,7,8,9,10,11 we show detailed per-frame token distribution evolution duringtraining with 100h labeled data, C = 100 , p = 0 . 1 .
Revised: Below in Figures 6,7,8,9,10,11 we show detailed per-frame token distribution evolution during training with 100h labeled data, C = 100 , p = 0 . 1 .

Original: Top part of these Figures displays evolutionin training of output token distribution per frame for one particular random sample from unlabeledset U .
Revised: Top part of these Figures displays evolution in training of output token distribution per frame for one particular random sample from unlabeled set U .

Original: Every sub-figure is a time frame t of the sample.

Original: For every sub-figure: x -axis is a trainingstep when we trained on the sample, y -axis is the probability of a token in range [0 , 1] .

Original: CTC blanktoken is denoted by #.

Original: Bottom left part shows total probability of top-3 tokens per frame for set

Original: unlabeled samples averaged across training steps, while bottom right percentage blanktoken training.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] [Figure] t=0, space t=1, # t=2, # t=3, # t=4, # t=5, a t=6, n t=7, d t=8, space t=9, space t=10, space t=11, h t=12, # t=13, a t=14, n t=15, # t=16, d t=17, s t=18, s t=19, o t=20, m t=21, e t=22, space t=23, space t=24, l t=25, i t=26, # t=27, # t=28, v t=29, e t=30, r t=31, r t=32, # t=33, y t=34, # t=35, space t=36, space t=37, # t=38, c t=39, a t=40, a t=41, m t=42, e t=43, e t=44, space t=45, space t=46, # t=47, f t=48, o t=49, # t=50, r t=51, # t=52, w t=53, w t=54, a t=55, r t=56, r t=57, d t=58, d t=59, # t=60, space t=61, space t=62, a t=63, n t=64, d t=65, d t=66, space t=67, space t=68, r t=69, e t=70, e t=71, s t=72, s t=73, p t=74, e t=75, c t=76, # t=77, t t=78, t t=79, f t=80, u t=81, # t=82, l t=83, # t=84, l t=85, # t=86, y t=87, space t=88, t t=89, t t=90, o t=91, u t=92, c t=93, h t=94, e t=95, e t=96, d t=97, space t=98, space t=99, h t=100, i t=101, s t=102, space t=103, h t=104, h t=105, a t=106, a t=107, # t=108, t t=109, t t=110, # t=111, space t=112, space t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, t t=133, h t=134, e t=135, space t=136, space t=137, c t=138, a t=139, r t=140, # t=141, # t=142, r t=143, i t=144, a t=145, g t=146, e t=147, space t=148, i t=149, s t=150, s t=151, space t=152, h t=153, e t=154, r t=155, e t=156, space t=157, space t=158, m t=159, y t=160, # t=161, # t=162, space t=163, l t=164, a t=165, # t=166, # t=167, d t=168, i t=169, # t=170, # t=171, space t=172, space t=173, # t=174, # t=175, h t=176, # t=177, e t=178, space t=179, space t=180, s t=181, # t=182, a t=183, i t=184, # t=185, d t=186, space t=187, space t=188, space t=189, # t=190, # t=191, # t=192, t t=193, t t=194, a t=195, # t=196, k t=197, k t=198, i t=199, i t=200, n t=201, g t=202, g t=203, space t=204, space t=205, f t=206, r t=207, r t=208, o t=209, m t=210, space t=211, space t=212, h t=213, e t=214, r t=215, r t=216, space t=217, space t=218, t t=219, t t=220, h t=221, e t=222, space t=223, w t=224, w t=225, r t=226, a t=227, # t=228, # t=229, p t=230, # t=231, s t=232, s t=233, space t=234, space t=235, a t=236, n t=237, # t=238, d t=239, space t=240, space t=241, space t=242, s t=243, s t=244, m t=245, # t=246, a t=247, l t=248, # t=249, # t=250, l t=251, l t=252, space t=253, space t=254, # t=255, a t=256, r t=257, r t=258, # t=259, t t=260, # t=261, i t=262, c t=263, c t=264, l t=265, e t=266, e t=267, s t=268, space t=269, space t=270, space t=271, s t=272, h t=273, e t=274, space t=275, space t=276, w t=277, a t=278, s t=279, s t=280, space t=281, space t=282, c t=283, a t=284, r t=285, # t=286, # t=287, r t=288, y t=289, y t=290, i t=291, i t=292, n t=293, n t=294, g t=295, space t=296, space t=297, space t=298, # t=299, # t=300, # t=301, #
Revised: Every sub-figure is a time frame t of t=0, space t=1, # t=2, # t=3, # t=4, # t=5, a t=6, n t=7, d t=8, space t=9, space t=10, space t=11, h t=12, # t=13, a t=14, n t=15, # t=16, d t=17, s t=18, s t=19, o t=20, m t=21, e t=22, space t=23, space t=24, l t=25, i t=26, # t=27, # t=28, v t=29, e t=30, r t=31, r t=32, # t=33, y t=34, # t=35, space t=36, space t=37, # t=38, c t=39, a t=40, a t=41, m t=42, e t=43, e t=44, space t=45, space t=46, # t=47, f t=48, o t=49, # t=50, r t=51, # t=52, w t=53, w t=54, a t=55, r t=56, r t=57, d t=58, d t=59, # t=60, space t=61, space t=62, a t=63, n t=64, d t=65, d t=66, space t=67, space t=68, r t=69, e t=70, e t=71, s t=72, s t=73, p t=74, e t=75, c t=76, # t=77, t t=78, t t=79, f t=80, u t=81, # t=82, l t=83, # t=84, l t=85, # t=86, y t=87, space t=88, t t=89, t t=90, o t=91, u t=92, c t=93, h t=94, e t=95, e t=96, d t=97, space t=98, space t=99, h t=100, i t=101, s t=102, space t=103, h t=104, h t=105, a t=106, a t=107, # t=108, t t=109, t t=110, # t=111, space t=112, space t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, t t=133, h t=134, e t=135, space t=136, space t=137, c t=138, a t=139, r t=140, # t=141, # t=142, r t=143, i t=144, a t=145, g t=146, e t=147, space t=148, i t=149, s t=150, s t=151, space t=152, h t=153, e t=154, r t=155, e t=156, space t=157, space t=158, m t=159, y t=160, # t=161, # t=162, space t=163, l t=164, a t=165, # t=166, # t=167, d t=168, i t=169, # t=170, # t=171, space t=172, space t=173, # t=174, # t=175, h t=176, # t=177, e t=178, space t=179, space t=180, s t=181, # t=182, a t=183, i t=184, # t=185, d t=186, space t=187, space t=188, space t=189, # t=190, # t=191, # t=192, t t=193, t t=194, a t=195, # t=196, k t=197, k t=198, i t=199, i t=200, n t=201, g t=202, g t=203, space t=204, space t=205, f t=206, r t=207, r t=208, o t=209, m t=210, space t=211, space t=212, h t=213, e t=214, r t=215, r t=216, space t=217, space t=218, t t=219, t t=220, h t=221, e t=222, space t=223, w t=224, w t=225, r t=226, a t=227, # t=228, # t=229, p t=230, # t=231, s t=232, s t=233, space t=234, space t=235, a t=236, n t=237, # t=238, d t=239, space t=240, space t=241, space t=242, s t=243, s t=244, m t=245, # t=246, a t=247, l t=248, # t=249, # t=250, l t=251, l t=252, space t=253, space t=254, # t=255, a t=256, r t=257, r t=258, # t=259, t t=260, # t=261, i t=262, c t=263, c t=264, l t=265, e t=266, e t=267, s t=268, space t=269, space t=270, space t=271, s t=272, h t=273, e t=274, space t=275, space t=276, w t=277, a t=278, s t=279, s t=280, space t=281, space t=282, c t=283, a t=284, r t=285, # t=286, # t=287, r t=288, y t=289, y t=290, i t=291, i t=292, n t=293, n t=294, g t=295, space t=296, space t=297, space t=298, # t=299, # t=300, # t=301, #

Original: [Figure]
Revised: [Figure]

Original: p l e ' s t o pt o k e n s t o t a l p r o b a b ili t y
Revised: p l e ' s t o pt o k e n s t o t a l p r o b a b ili t y

Original: [Figure]
Revised: [Figure]

Original: t=0, space t=1, # t=2, # t=3, # t=4, a t=5, n t=6, n t=7, d t=8, space t=9, space t=10, h t=11, h t=12, a t=13, # t=14, n t=15, d t=16, # t=17, s t=18, # t=19, o t=20, m t=21, e t=22, space t=23, space t=24, l t=25, i t=26, # t=27, # t=28, v t=29, e t=30, r t=31, r t=32, # t=33, y t=34, # t=35, space t=36, space t=37, # t=38, c t=39, a t=40, m t=41, # t=42, e t=43, space t=44, space t=45, space t=46, f t=47, f t=48, o t=49, # t=50, r t=51, # t=52, w t=53, a t=54, a t=55, r t=56, r t=57, r t=58, d t=59, s t=60, s t=61, space t=62, a t=63, a t=64, n t=65, d t=66, space t=67, space t=68, r t=69, r t=70, e t=71, s t=72, # t=73, p t=74, e t=75, c t=76, # t=77, t t=78, # t=79, f t=80, u t=81, # t=82, l t=83, # t=84, l t=85, y t=86, y t=87, space t=88, t t=89, o t=90, u t=91, # t=92, c t=93, h t=94, # t=95, e t=96, d t=97, space t=98, h t=99, h t=100, i t=101, s t=102, space t=103, space t=104, h t=105, a t=106, # t=107, # t=108, t t=109, t t=110, space t=111, space t=112, space t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, t t=133, h t=134, e t=135, space t=136, c t=137, a t=138, r t=139, # t=140, # t=141, r t=142, i t=143, i t=144, a t=145, g t=146, e t=147, space t=148, space t=149, i t=150, s t=151, space t=152, h t=153, e t=154, r t=155, e t=156, space t=157, space t=158, m t=159, y t=160, # t=161, space t=162, space t=163, l t=164, a t=165, # t=166, # t=167, d t=168, # t=169, y t=170, # t=171, space t=172, space t=173, space t=174, # t=175, h t=176, e t=177, # t=178, space t=179, space t=180, s t=181, # t=182, a t=183, i t=184, # t=185, d t=186, d t=187, space t=188, space t=189, # t=190, # t=191, # t=192, t t=193, # t=194, a t=195, # t=196, k t=197, # t=198, i t=199, # t=200, n t=201, g t=202, space t=203, space t=204, space t=205, f t=206, r t=207, o t=208, # t=209, m t=210, space t=211, space t=212, h t=213, e t=214, r t=215, r t=216, space t=217, space t=218, # t=219, t t=220, h t=221, e t=222, space t=223, w t=224, w t=225, r t=226, a t=227, # t=228, # t=229, p t=230, p t=231, # t=232, s t=233, s t=234, space t=235, # t=236, a t=237, n t=238, d t=239, # t=240, space t=241, space t=242, # t=243, s t=244, m t=245, m t=246, a t=247, l t=248, # t=249, # t=250, l t=251, # t=252, space t=253, space t=254, # t=255, a t=256, # t=257, r t=258, # t=259, t t=260, # t=261, i t=262, c t=263, c t=264, l t=265, e t=266, e t=267, s t=268, s t=269, space t=270, # t=271, s t=272, h t=273, e t=274, space t=275, space t=276, w t=277, a t=278, s t=279, # t=280, space t=281, c t=282, # t=283, a t=284, r t=285, # t=286, # t=287, r t=288, y t=289, y t=290, # t=291, i t=292, n t=293, g t=294, space t=295, space t=296,
Revised: t=0, space t=1, # t=2, # t=3, # t=4, a t=5, n t=6, n t=7, d t=8, space t=9, space t=10, h t=11, h t=12, a t=13, # t=14, n t=15, d t=16, # t=17, s t=18, # t=19, o t=20, m t=21, e t=22, space t=23, space t=24, l t=25, i t=26, # t=27, # t=28, v t=29, e t=30, r t=31, r t=32, # t=33, y t=34, # t=35, space t=36, space t=37, # t=38, c t=39, a t=40, m t=41, # t=42, e t=43, space t=44, space t=45, space t=46, f t=47, f t=48, o t=49, # t=50, r t=51, # t=52, w t=53, a t=54, a t=55, r t=56, r t=57, r t=58, d t=59, s t=60, s t=61, space t=62, a t=63, a t=64, n t=65, d t=66, space t=67, space t=68, r t=69, r t=70, e t=71, s t=72, # t=73, p t=74, e t=75, c t=76, # t=77, t t=78, # t=79, f t=80, u t=81, # t=82, l t=83, # t=84, l t=85, y t=86, y t=87, space t=88, t t=89, o t=90, u t=91, # t=92, c t=93, h t=94, # t=95, e t=96, d t=97, space t=98, h t=99, h t=100, i t=101, s t=102, space t=103, space t=104, h t=105, a t=106, # t=107, # t=108, t t=109, t t=110, space t=111, space t=112, space t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, t t=133, h t=134, e t=135, space t=136, c t=137, a t=138, r t=139, # t=140, # t=141, r t=142, i t=143, i t=144, a t=145, g t=146, e t=147, space t=148, space t=149, i t=150, s t=151, space t=152, h t=153, e t=154, r t=155, e t=156, space t=157, space t=158, m t=159, y t=160, # t=161, space t=162, space t=163, l t=164, a t=165, # t=166, # t=167, d t=168, # t=169, y t=170, # t=171, space t=172, space t=173, space t=174, # t=175, h t=176, e t=177, # t=178, space t=179, space t=180, s t=181, # t=182, a t=183, i t=184, # t=185, d t=186, d t=187, space t=188, space t=189, # t=190, # t=191, # t=192, t t=193, # t=194, a t=195, # t=196, k t=197, # t=198, i t=199, # t=200, n t=201, g t=202, space t=203, space t=204, space t=205, f t=206, r t=207, o t=208, # t=209, m t=210, space t=211, space t=212, h t=213, e t=214, r t=215, r t=216, space t=217, space t=218, # t=219, t t=220, h t=221, e t=222, space t=223, w t=224, w t=225, r t=226, a t=227, # t=228, # t=229, p t=230, p t=231, # t=232, s t=233, s t=234, space t=235, # t=236, a t=237, n t=238, d t=239, # t=240, space t=241, space t=242, # t=243, s t=244, m t=245, m t=246, a t=247, l t=248, # t=249, # t=250, l t=251, # t=252, space t=253, space t=254, # t=255, a t=256, # t=257, r t=258, # t=259, t t=260, # t=261, i t=262, c t=263, c t=264, l t=265, e t=266, e t=267, s t=268, s t=269, space t=270, # t=271, s t=272, h t=273, e t=274, space t=275, space t=276, w t=277, a t=278, s t=279, # t=280, space t=281, c t=282, # t=283, a t=284, r t=285, # t=286, # t=287, r t=288, y t=289, y t=290, # t=291, i t=292, n t=293, g t=294, space t=295, space t=296,

Original: [Figure]
Revised: [Figure]

Original: t=297, space t=298, # t=299, # t=300, # t=301, #
Revised: t=297, space t=298, # t=299, # t=300, # t=301, #

Original: [Figure]
Revised: [Figure]

Original: p l e ' s t o pt o k e n s t o t a l p r o b a b ili t y
Revised: p l e ' s t o pt o k e n s t o t a l p r o b a b ili t y

Original: [Figure]
Revised: [Figure]

Original: t=0, space t=1, # t=2, # t=3, a t=4, a t=5, a t=6, a t=7, d t=8, space t=9, space t=10, space t=11, # t=12, # t=13, # t=14, space t=15, space t=16, # t=17, # t=18, s t=19, o t=20, o t=21, e t=22, space t=23, space t=24,
Revised: t=0, space t=1, # t=2, # t=3, a t=4, a t=5, a t=6, a t=7, d t=8, space t=9, space t=10, space t=11, # t=12, # t=13, # t=14, space t=15, space t=16, # t=17, # t=18, s t=19, o t=20, o t=21, e t=22, space t=23, space t=24,

Original: i t=25, i t=26, i t=27, # t=28, space t=29, space t=30, space t=31, # t=32, # t=33, # t=34, # t=35, # t=36, # t=37, # t=38, # t=39, a t=40, a t=41, a t=42, # t=43, space t=44, space t=45, # t=46, # t=47, # t=48, o t=49, o t=50, o t=51, # t=52, a t=53, a t=54, a t=55, a t=56, d t=57, space t=58, space t=59, space t=60, space t=61, # t=62, space t=63, e t=64, e t=65, s t=66, s t=67, s t=68, s t=69, s t=70, s t=71, s t=72, p t=73, p t=74, e t=75, c t=76, c t=77, c t=78, t t=79, space t=80, space t=81, u t=82, f t=83, u t=84, # t=85, # t=86, space t=87, space t=88, space t=89, t t=90, u t=91, u t=92, u t=93, h t=94, space t=95, space t=96, space t=97, # t=98, h t=99, h t=100, s t=101, s t=102, space t=103, h t=104, a t=105, a t=106, t t=107, t t=108, space t=109, space t=110, # t=111, # t=112, # t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, t t=133, t t=134, h t=135, e t=136, e t=137, space t=138, a t=139, a t=140, a t=141, a t=142, space t=143, space t=144, space t=145, s t=146, s t=147, space t=148, t t=149, h t=150, e t=151, e t=152, space t=153, space t=154, a t=155, a t=156, # t=157, # t=158, # t=159, space t=160, space t=161, space t=162, # t=163, # t=164, # t=165, # t=166, a t=167, a t=168, a t=169, # t=170, # t=171, d t=172, y t=173, y t=174, # t=175, space t=
Revised: i t=25, i t=26, i t=27, # t=28, space t=29, space t=30, space t=31, # t=32, # t=33, # t=34, # t=35, # t=36, # t=37, # t=38, # t=39, a t=40, a t=41, a t=42, # t=43, space t=44, space t=45, # t=46, # t=47, # t=48, o t=49, o t=50, o t=51, # t=52, a t=53, a t=54, a t=55, a t=56, d t=57, space t=58, space t=59, space t=60, space t=61, # t=62, space t=63, e t=64, e t=65, s t=66, s t=67, s t=68, s t=69, s t=70, s t=71, s t=72, p t=73, p t=74, e t=75, c t=76, c t=77, c t=78, t t=79, space t=80, space t=81, u t=82, f t=83, u t=84, # t=85, # t=86, space t=87, space t=88, space t=89, t t=90, u t=91, u t=92, u t=93, h t=94, space t=95, space t=96, space t=97, # t=98, h t=99, h t=100, s t=101, s t=102, space t=103, h t=104, a t=105, a t=106, t t=107, t t=108, space t=109, space t=110, # t=111, # t=112, # t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, t t=133, t t=134, h t=135, e t=136, e t=137, space t=138, a t=139, a t=140, a t=141, a t=142, space t=143, space t=144, space t=145, s t=146, s t=147, space t=148, t t=149, h t=150, e t=151, e t=152, space t=153, space t=154, a t=155, a t=156, # t=157, # t=158, # t=159, space t=160, space t=161, space t=162, # t=163, # t=164, # t=165, # t=166, a t=167, a t=168, a t=169, # t=170, # t=171, d t=172, y t=173, y t=174, # t=175, space t=

Original: [Table]
Revised: [Table]

Original: , space t=177, space t=178, space t=179, space t=180, # t=181, s t=182, a t=183, a t=184, # t=185, d t=186, space t=187, space t=188, # t=189, # t=190, # t=191, # t=192, a t=193, a t=194, a t=195, # t=196, space t=197, # t=198, # t=199, i t=200, i t=201, i t=202, g t=203, g t=204, f t=205, f t=206, o t=207, o t=208, o t=209, space t=210, space t=211, e t=212, e t=213, # t=214, # t=215, space t=216, space t=217, space t=218, h t=219, # t=220, # t=221, space t=222, space t=223, a t=224, a t=225, p t=226, p t=227, p t=228, s t=229, # t=230, # t=231, # t=232, # t=233, a t=234, a t=235, a t=236, a t=237, n t=238, space t=239, space t=240, space t=241, # t=242, # t=243, # t=244, # t=245, s t=246, s t=247, a t=248, a t=249, # t=250, space t=251, a t=252, a t=253, a t=254, r t=255, t t=256, i t=257, i t=258, # t=259, c t=260, l t=261, l t=262, l t=263, e t=264, # t=265, s t=266, s t=267, # t=268, # t=269, # t=270, # t=271, # t=272, h t=273, h t=274, # t=275, # t=276, space t=277, space t=278, a t=279, a t=280, a t=281, # t=282, space t=283, space t=284, space t=285, # t=286, c t=287, r t=288, r t=289, y t=290, # t=291, i t=292, i t=293, space t=294, # t=295, # t=296, # t=297, # t=298, # t=299, # t=300, # t=301, #
Revised: , space t=177, space t=178, space t=179, space t=180, # t=181, s t=182, a t=183, a t=184, # t=185, d t=186, space t=187, space t=188, # t=189, # t=190, # t=191, # t=192, a t=193, a t=194, a t=195, # t=196, space t=197, # t=198, # t=199, i t=200, i t=201, i t=202, g t=203, g t=204, f t=205, f t=206, o t=207, o t=208, o t=209, space t=210, space t=211, e t=212, e t=213, # t=214, # t=215, space t=216, space t=217, space t=218, h t=219, # t=220, # t=221, space t=222, space t=223, a t=224, a t=225, p t=226, p t=227, p t=228, s t=229, # t=230, # t=231, # t=232, # t=233, a t=234, a t=235, a t=236, a t=237, n t=238, space t=239, space t=240, space t=241, # t=242, # t=243, # t=244, # t=245, s t=246, s t=247, a t=248, a t=249, # t=250, space t=251, a t=252, a t=253, a t=254, r t=255, t t=256, i t=257, i t=258, # t=259, c t=260, l t=261, l t=262, l t=263, e t=264, # t=265, s t=266, s t=267, # t=268, # t=269, # t=270, # t=271, # t=272, h t=273, h t=274, # t=275, # t=276, space t=277, space t=278, a t=279, a t=280, a t=281, # t=282, space t=283, space t=284, space t=285, # t=286, c t=287, r t=288, r t=289, y t=290, # t=291, i t=292, i t=293, space t=294, # t=295, # t=296, # t=297, # t=298, # t=299, # t=300, # t=301, #

Original: [Figure]
Revised: [Figure]

Original: l e ' s t o p
Revised: l e ' s t o p

Original: [Figure]
Revised: [Figure]

Original: t o k e n s t o t a l p r o b a b ili t
Revised: t o k e n s t o t a l p r o b a b ili t

Original: y
Revised: y

Original: [Figure]
Revised: [Figure]

Original: t=0, space t=1, a t=2, a t=3, n t=4, d t=5, space t=6, h t=7, a t=8, n t=9, d t=10, s t=11, o t=12, m t=13, e t=14, e t=15, space t=16, l t=17, i t=18, v t=19, e t=20, r t=21, y t=22, space t=23, c t=24, a t=25, a t=26, m t=27, e t=28, space t=29, f t=30, o t=31, r t=32, w t=33, w t=34, a t=35, r t=36, d t=37, s t=38, space t=39, # t=40, # t=41, # t=42, # t=43, # t=44, # t=45, # t=46, # t=47, # t=48, # t=49, # t=50, # t=51, a t=52, n t=53, d t=54, space t=55, r t=56, e t=57, s t=58, s t=59, p t=60, e t=61, c t=62, t t=63, f t=64, u t=65, u t=66, l t=67, # t=68, l t=69, y t=70, space t=71, t t=72, o t=73, o t=74, u t=75, c t=76, h t=77, e t=78, d t=79, space t=80, # t=81, # t=82, # t=83, # t=84, # t=85, # t=86, # t=87, h t=88, i t=89, s t=90, s t=91, space t=92, h t=93, a t=94, t t=95, t t=96, space t=97, # t=98, # t=99, # t=100, # t=101, # t=102, # t=103, # t=104, # t=105, # t=106, # t=107, # t=108, # t=109, # t=110, # t=111, # t=112, # t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, t t=124, h t=125, e t=126, space t=127, c t=128, a t=129, r t=130, r t=131, # t=132, r t=133, i t=134, a t=135, g t=136, e t=137, space t=138, space t=139, i t=140, s t=141, space t=142, h t=143, e t=144, r t=145, r t=146, e t=147, space t=148, a t=149, space t=150, space t=151, l t=152, a t=153, d t=154, y t=155, space t=156, # t=157, # t=158, # t=159, # t=160, # t=161, # t=162, # t=163, # t=164, # t=165, # t=166, h t=167, e t=168, space t=169, s t=170, a t=171, i t=172, d t=173, space t=174, t t=175, a t=176, k t=177, i t=178, n t=179, g t=180, space t=181, f t=182, r t=183, o t=184, m t=185, space t=186, h t=187, e t=188, r t=189, r t=190, t t=191, h t=192, e t=193, space t=194, space t=195, r t=196, a t=197, p t=198, # t=199, # t=200, # t=201, # t=202, # t=203, # t=204, # t=205, # t=206, # t=207, # t=208, # t=209, # t=210, # t=211, # t=212, # t=213, # t=214, # t=215, # t=216, # t=217, # t=218, # t=219, # t=220, # t=221, # t=222, # t=223, # t=224, # t=225, # t=226, # t=227, # t=228, # t=229, # t=230, # t=231, # t=232, # t=233, # t=234, # t=235, # t=236, # t=237, # t=238, # t=239, # t=240, # t=241, # t=242, # t=243, # t=244, # t=245, # t=246, # t=247, # t=248, # t=249, # t=250, # t=251, # t=252, # t=253, # t=254, # t=255, # t=256, # t=257, # t=258, # t=259, # t=260, # t=261, # t=262, # t=263, # t=264, # t=265, # t=266, # t=267, # t=268, # t=269, # t=270, # t=271, # t=272, # t=273, # t=274, # t=275, # t=276, # t=277, # t=278, # t=279, # t=280, # t=281, # t=282, # t=283, # t=284, # t=285, # t=286, # t=287, # t=288, # t=289, # t=290, # t=291, # t=292, # t=293, # t=294, # t=295, # t=296, # t=297, # t=298, # t=299, # t=300, # t=301, #
Revised: t=0, space t=1, a t=2, a t=3, n t=4, d t=5, space t=6, h t=7, a t=8, n t=9, d t=10, s t=11, o t=12, m t=13, e t=14, e t=15, space t=16, l t=17, i t=18, v t=19, e t=20, r t=21, y t=22, space t=23, c t=24, a t=25, a t=26, m t=27, e t=28, space t=29, f t=30, o t=31, r t=32, w t=33, w t=34, a t=35, r t=36, d t=37, s t=38, space t=39, # t=40, # t=41, # t=42, # t=43, # t=44, # t=45, # t=46, # t=47, # t=48, # t=49, # t=50, # t=51, a t=52, n t=53, d t=54, space t=55, r t=56, e t=57, s t=58, s t=59, p t=60, e t=61, c t=62, t t=63, f t=64, u t=65, u t=66, l t=67, # t=68, l t=69, y t=70, space t=71, t t=72, o t=73, o t=74, u t=75, c t=76, h t=77, e t=78, d t=79, space t=80, # t=81, # t=82, # t=83, # t=84, # t=85, # t=86, # t=87, h t=88, i t=89, s t=90, s t=91, space t=92, h t=93, a t=94, t t=95, t t=96, space t=97, # t=98, # t=99, # t=100, # t=101, # t=102, # t=103, # t=104, # t=105, # t=106, # t=107, # t=108, # t=109, # t=110, # t=111, # t=112, # t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, t t=124, h t=125, e t=126, space t=127, c t=128, a t=129, r t=130, r t=131, # t=132, r t=133, i t=134, a t=135, g t=136, e t=137, space t=138, space t=139, i t=140, s t=141, space t=142, h t=143, e t=144, r t=145, r t=146, e t=147, space t=148, a t=149, space t=150, space t=151, l t=152, a t=153, d t=154, y t=155, space t=156, # t=157, # t=158, # t=159, # t=160, # t=161, # t=162, # t=163, # t=164, # t=165, # t=166, h t=167, e t=168, space t=169, s t=170, a t=171, i t=172, d t=173, space t=174, t t=175, a t=176, k t=177, i t=178, n t=179, g t=180, space t=181, f t=182, r t=183, o t=184, m t=185, space t=186, h t=187, e t=188, r t=189, r t=190, t t=191, h t=192, e t=193, space t=194, space t=195, r t=196, a t=197, p t=198, # t=199, # t=200, # t=201, # t=202, # t=203, # t=204, # t=205, # t=206, # t=207, # t=208, # t=209, # t=210, # t=211, # t=212, # t=213, # t=214, # t=215, # t=216, # t=217, # t=218, # t=219, # t=220, # t=221, # t=222, # t=223, # t=224, # t=225, # t=226, # t=227, # t=228, # t=229, # t=230, # t=231, # t=232, # t=233, # t=234, # t=235, # t=236, # t=237, # t=238, # t=239, # t=240, # t=241, # t=242, # t=243, # t=244, # t=245, # t=246, # t=247, # t=248, # t=249, # t=250, # t=251, # t=252, # t=253, # t=254, # t=255, # t=256, # t=257, # t=258, # t=259, # t=260, # t=261, # t=262, # t=263, # t=264, # t=265, # t=266, # t=267, # t=268, # t=269, # t=270, # t=271, # t=272, # t=273, # t=274, # t=275, # t=276, # t=277, # t=278, # t=279, # t=280, # t=281, # t=282, # t=283, # t=284, # t=285, # t=286, # t=287, # t=288, # t=289, # t=290, # t=291, # t=292, # t=293, # t=294, # t=295, # t=296, # t=297, # t=298, # t=299, # t=300, # t=301, #

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: l p r o b a b ili t y
Revised: l p r o b a b ili t y

Original: [Figure]
Revised: [Figure]

Original: t=0, # t=1, # t=2, # t=3, # t=4, # t=5, # t=6, # t=7, # t=8, # t=9, # t=10, # t=11, # t=12, # t=13, # t=14, # t=15, # t=16, # t=17, # t=18, # t=19, # t=20, # t=21, # t=22, # t=23, # t=24, # t=25, # t=26, # t=27, # t=28, # t=29, # t=30, # t=31, # t=32, # t=33, # t=34, # t=35, # t=36, # t=37, # t=38, # t=39, # t=40, # t=41, # t=42, # t=43, # t=44, # t=45, # t=46, # t=47, # t=48, # t=49, # t=50, # t=51, # t=52, # t=53, # t=54, # t=55, # t=56, # t=57, # t=58, # t=59, # t=60, # t=61, # t=62, # t=63, # t=64, # t=65, # t=66, # t=67, # t=68, # t=69, # t=70, # t=71, # t=72, # t=73, # t=74, # t=75, # t=76, # t=77, # t=78, # t=79, # t=80, # t=81, # t=82, # t=83, # t=84, # t=85, # t=86, # t=87, # t=88, # t=89, # t=90, # t=91, # t=92, # t=93, # t=94, # t=95, # t=96, # t=97, # t=98, # t=99, # t=100, # t=101, # t=102, # t=103, # t=104, # t=105, # t=106, # t=107, # t=108, # t=109, # t=110, # t=111, # t=112, # t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, # t=133, # t=134, # t=135, # t=136, # t=137, # t=138, # t=139, # t=140, # t=141, # t=142, # t=143, # t=144, # t=145, # t=146, # t=147, # t=148, # t=149, # t=150, # t=151, # t=152, # t=153, # t=154, # t=155, # t=156, # t=157, # t=158, # t=159, # t=160, # t=161, # t=162, # t=163, # t=164, # t=165, # t=166, # t=167, # t=168, # t=169, # t=170, # t=171, # t=172, # t=173, # t=174, # t=175, # t=176, # t=177, # t=178, # t=179, # t=180, # t=181, # t=182, # t=183, # t=184, # t=185, # t=186, # t=187, # t=188, # t=189, # t=190, # t=191, # t=192, # t=193, # t=194, # t=195, # t=196, # t=197, # t=198, # t=199, # t=200, # t=201, # t=202, # t=203, # t=204, # t=205, # t=206, # t=207, # t=208, # t=209, # t=210, # t=211, # t=212, # t=213, # t=214, # t=215, # t=216, # t=217, # t=218, # t=219, # t=220, # t=221, # t=222, # t=223, # t=224, # t=225, # t=226, # t=227, # t=228, # t=229, # t=230, # t=231, # t=232, # t=233, # t=234, # t=235, # t=236, # t=237, # t=238, # t=239, # t=240, # t=241, # t=242, # t=243, # t=244, # t=245, # t=246, # t=247, # t=248, # t=249, # t=250, # t=251, # t=252, # t=253, # t=254, # t=255, # t=256, # t=257, # t=258, # t=259, # t=260, # t=261, # t=262, # t=263, # t=264, # t=265, # t=266, # t=267, # t=268, # t=269, # t=270, # t=271, # t=272, # t=273, # t=274, # t=275, # t=276, # t=277, # t=278, # t=279, # t=280, # t=281, # t=282, # t=283, # t=284, # t=285, # t=286, # t=287, # t=288, # t=289, # t=290, # t=291, # t=292, # t=293, # t=294, # t=295, # t=296, # t=297, # t=298, # t=299, # t=300, # t=301, #
Revised: t=0, # t=1, # t=2, # t=3, # t=4, # t=5, # t=6, # t=7, # t=8, # t=9, # t=10, # t=11, # t=12, # t=13, # t=14, # t=15, # t=16, # t=17, # t=18, # t=19, # t=20, # t=21, # t=22, # t=23, # t=24, # t=25, # t=26, # t=27, # t=28, # t=29, # t=30, # t=31, # t=32, # t=33, # t=34, # t=35, # t=36, # t=37, # t=38, # t=39, # t=40, # t=41, # t=42, # t=43, # t=44, # t=45, # t=46, # t=47, # t=48, # t=49, # t=50, # t=51, # t=52, # t=53, # t=54, # t=55, # t=56, # t=57, # t=58, # t=59, # t=60, # t=61, # t=62, # t=63, # t=64, # t=65, # t=66, # t=67, # t=68, # t=69, # t=70, # t=71, # t=72, # t=73, # t=74, # t=75, # t=76, # t=77, # t=78, # t=79, # t=80, # t=81, # t=82, # t=83, # t=84, # t=85, # t=86, # t=87, # t=88, # t=89, # t=90, # t=91, # t=92, # t=93, # t=94, # t=95, # t=96, # t=97, # t=98, # t=99, # t=100, # t=101, # t=102, # t=103, # t=104, # t=105, # t=106, # t=107, # t=108, # t=109, # t=110, # t=111, # t=112, # t=113, # t=114, # t=115, # t=116, # t=117, # t=118, # t=119, # t=120, # t=121, # t=122, # t=123, # t=124, # t=125, # t=126, # t=127, # t=128, # t=129, # t=130, # t=131, # t=132, # t=133, # t=134, # t=135, # t=136, # t=137, # t=138, # t=139, # t=140, # t=141, # t=142, # t=143, # t=144, # t=145, # t=146, # t=147, # t=148, # t=149, # t=150, # t=151, # t=152, # t=153, # t=154, # t=155, # t=156, # t=157, # t=158, # t=159, # t=160, # t=161, # t=162, # t=163, # t=164, # t=165, # t=166, # t=167, # t=168, # t=169, # t=170, # t=171, # t=172, # t=173, # t=174, # t=175, # t=176, # t=177, # t=178, # t=179, # t=180, # t=181, # t=182, # t=183, # t=184, # t=185, # t=186, # t=187, # t=188, # t=189, # t=190, # t=191, # t=192, # t=193, # t=194, # t=195, # t=196, # t=197, # t=198, # t=199, # t=200, # t=201, # t=202, # t=203, # t=204, # t=205, # t=206, # t=207, # t=208, # t=209, # t=210, # t=211, # t=212, # t=213, # t=214, # t=215, # t=216, # t=217, # t=218, # t=219, # t=220, # t=221, # t=222, # t=223, # t=224, # t=225, # t=226, # t=227, # t=228, # t=229, # t=230, # t=231, # t=232, # t=233, # t=234, # t=235, # t=236, # t=237, # t=238, # t=239, # t=240, # t=241, # t=242, # t=243, # t=244, # t=245, # t=246, # t=247, # t=248, # t=249, # t=250, # t=251, # t=252, # t=253, # t=254, # t=255, # t=256, # t=257, # t=258, # t=259, # t=260, # t=261, # t=262, # t=263, # t=264, # t=265, # t=266, # t=267, # t=268, # t=269, # t=270, # t=271, # t=272, # t=273, # t=274, # t=275, # t=276, # t=277, # t=278, # t=279, # t=280, # t=281, # t=282, # t=283, # t=284, # t=285, # t=286, # t=287, # t=288, # t=289, # t=290, # t=291, # t=292, # t=293, # t=294, # t=295, # t=296, # t=297, # t=298, # t=299, # t=300, # t=301, #

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t o k
Revised: t o k

Original: [Figure]
Revised: [Figure]

Original: n s t o t a l p r
Revised: n s t o t a l p r

Original: [Figure]
Revised: [Figure]

Original: b a b ili t y
Revised: b a b ili t y

Original: [Figure]
Revised: [Figure]

Original: t=0, ' t=1, ' t=2, ' t=3, ' t=4, ' t=5, ' t=6, ' t=7, ' t=8, ' t=9, ' t=10, ' t=11, ' t=12, ' t=13, ' t=14, ' t=15, ' t=16, ' t=17, ' t=18, ' t=19, ' t=20, ' t=21, ' t=22, ' t=23, ' t=24, ' t=25, ' t=26, ' t=27, ' t=28, ' t=29, ' t=30, ' t=31, ' t=32, ' t=33, ' t=34, ' t=35, ' t=36, ' t=37, ' t=38, ' t=39, ' t=40, ' t=41, ' t=42, ' t=43, ' t=44, ' t=45, ' t=46, ' t=47, ' t=48, ' t=49, ' t=50, ' t=51, ' t=52, ' t=53, ' t=54, ' t=55, ' t=56, ' t=57, ' t=58, ' t=59, ' t=60, ' t=61, ' t=62, ' t=63, ' t=64, ' t=65, ' t=66, ' t=67, ' t=68, ' t=69, ' t=70, ' t=71, ' t=72, ' t=73, ' t=74, ' t=75, ' t=76, ' t=77, ' t=78, ' t=79, ' t=80, ' t=81, ' t=82, ' t=83, ' t=84, ' t=85, ' t=86, ' t=87, ' t=88, ' t=89, ' t=90, ' t=91, ' t=92, ' t=93, ' t=94, ' t=95, ' t=96, ' t=97, ' t=98, ' t=99, ' t=100, ' t=101, ' t=102, ' t=103, ' t=104, ' t=105, ' t=106, ' t=107, ' t=108, ' t=109, ' t=110, ' t=111, ' t=112, ' t=113, ' t=114, ' t=115, ' t=116, ' t=117, ' t=118, ' t=119, ' t=120, ' t=121, ' t=122, ' t=123, ' t=124, ' t=125, ' t=126, ' t=127, ' t=128, ' t=129, ' t=130, ' t=131, ' t=132, ' t=133, ' t=134, ' t=135, ' t=136, ' t=137, ' t=138, ' t=139, ' t=140, ' t=141, ' t=142, ' t=143, ' t=144, ' t=145, ' t=146, ' t=147, ' t=148, ' t=149, ' t=150, ' t=151, ' t=152, ' t=153, ' t=154, ' t=155, ' t=156, ' t=157, ' t=158, ' t=159, ' t=160, ' t=161, ' t=162, ' t=163, ' t=164, ' t=165, ' t=166, ' t=167, ' t=168, ' t=169, ' t=170, ' t=171, ' t=172, ' t=173, ' t=174, ' t=175, ' t=176, ' t=177, ' t=178, ' t=179, ' t=180, ' t=181, ' t=182, ' t=183, ' t=184, ' t=185, ' t=186, ' t=187, ' t=188, ' t=189, ' t=190, ' t=191, ' t=192, ' t=193, ' t=194, ' t=195, ' t=196, ' t=197, ' t=198, ' t=199, ' t=200, ' t=201, ' t=202, ' t=203, ' t=204, ' t=205, ' t=206, ' t=207, ' t=208, ' t=209, ' t=210, ' t=211, ' t=212, ' t=213, ' t=214, ' t=215, ' t=216, ' t=217, ' t=218, ' t=219, ' t=220, ' t=221, ' t=222, ' t=223, ' t=224, ' t=225, ' t=226, ' t=227, ' t=228, ' t=229, ' t=230, ' t=231, ' t=232, ' t=233, ' t=234, ' t=235, ' t=236, ' t=237, ' t=238, ' t=239, ' t=240, ' t=241, ' t=242, ' t=243, ' t=244, ' t=245, ' t=246, ' t=247, ' t=248, ' t=249, ' t=250, ' t=251, ' t=252, ' t=253, ' t=254, ' t=255, ' t=256, ' t=257, ' t=258, ' t=259, ' t=260, ' t=261, ' t=262, ' t=263, ' t=264, ' t=265, ' t=266, ' t=267, ' t=268, ' t=269, ' t=270, ' t=271, ' t=272, ' t=273, ' t=274, ' t=275, ' t=276, ' t=277, ' t=278, ' t=279, ' t=280, ' t=281, ' t=282, ' t=283, ' t=284, ' t=285, ' t=286, ' t=287, ' t=288, ' t=289, ' t=290, ' t=291, ' t=292, ' t=293, ' t=294, ' t=295, ' t=296, ' t=297, ' t=298, ' t=299, ' t=300, ' t=301, '
Revised: t=0, ' t=1, ' t=2, ' t=3, ' t=4, ' t=5, ' t=6, ' t=7, ' t=8, ' t=9, ' t=10, ' t=11, ' t=12, ' t=13, ' t=14, ' t=15, ' t=16, ' t=17, ' t=18, ' t=19, ' t=20, ' t=21, ' t=22, ' t=23, ' t=24, ' t=25, ' t=26, ' t=27, ' t=28, ' t=29, ' t=30, ' t=31, ' t=32, ' t=33, ' t=34, ' t=35, ' t=36, ' t=37, ' t=38, ' t=39, ' t=40, ' t=41, ' t=42, ' t=43, ' t=44, ' t=45, ' t=46, ' t=47, ' t=48, ' t=49, ' t=50, ' t=51, ' t=52, ' t=53, ' t=54, ' t=55, ' t=56, ' t=57, ' t=58, ' t=59, ' t=60, ' t=61, ' t=62, ' t=63, ' t=64, ' t=65, ' t=66, ' t=67, ' t=68, ' t=69, ' t=70, ' t=71, ' t=72, ' t=73, ' t=74, ' t=75, ' t=76, ' t=77, ' t=78, ' t=79, ' t=80, ' t=81, ' t=82, ' t=83, ' t=84, ' t=85, ' t=86, ' t=87, ' t=88, ' t=89, ' t=90, ' t=91, ' t=92, ' t=93, ' t=94, ' t=95, ' t=96, ' t=97, ' t=98, ' t=99, ' t=100, ' t=101, ' t=102, ' t=103, ' t=104, ' t=105, ' t=106, ' t=107, ' t=108, ' t=109, ' t=110, ' t=111, ' t=112, ' t=113, ' t=114, ' t=115, ' t=116, ' t=117, ' t=118, ' t=119, ' t=120, ' t=121, ' t=122, ' t=123, ' t=124, ' t=125, ' t=126, ' t=127, ' t=128, ' t=129, ' t=130, ' t=131, ' t=132, ' t=133, ' t=134, ' t=135, ' t=136, ' t=137, ' t=138, ' t=139, ' t=140, ' t=141, ' t=142, ' t=143, ' t=144, ' t=145, ' t=146, ' t=147, ' t=148, ' t=149, ' t=150, ' t=151, ' t=152, ' t=153, ' t=154, ' t=155, ' t=156, ' t=157, ' t=158, ' t=159, ' t=160, ' t=161, ' t=162, ' t=163, ' t=164, ' t=165, ' t=166, ' t=167, ' t=168, ' t=169, ' t=170, ' t=171, ' t=172, ' t=173, ' t=174, ' t=175, ' t=176, ' t=177, ' t=178, ' t=179, ' t=180, ' t=181, ' t=182, ' t=183, ' t=184, ' t=185, ' t=186, ' t=187, ' t=188, ' t=189, ' t=190, ' t=191, ' t=192, ' t=193, ' t=194, ' t=195, ' t=196, ' t=197, ' t=198, ' t=199, ' t=200, ' t=201, ' t=202, ' t=203, ' t=204, ' t=205, ' t=206, ' t=207, ' t=208, ' t=209, ' t=210, ' t=211, ' t=212, ' t=213, ' t=214, ' t=215, ' t=216, ' t=217, ' t=218, ' t=219, ' t=220, ' t=221, ' t=222, ' t=223, ' t=224, ' t=225, ' t=226, ' t=227, ' t=228, ' t=229, ' t=230, ' t=231, ' t=232, ' t=233, ' t=234, ' t=235, ' t=236, ' t=237, ' t=238, ' t=239, ' t=240, ' t=241, ' t=242, ' t=243, ' t=244, ' t=245, ' t=246, ' t=247, ' t=248, ' t=249, ' t=250, ' t=251, ' t=252, ' t=253, ' t=254, ' t=255, ' t=256, ' t=257, ' t=258, ' t=259, ' t=260, ' t=261, ' t=262, ' t=263, ' t=264, ' t=265, ' t=266, ' t=267, ' t=268, ' t=269, ' t=270, ' t=271, ' t=272, ' t=273, ' t=274, ' t=275, ' t=276, ' t=277, ' t=278, ' t=279, ' t=280, ' t=281, ' t=282, ' t=283, ' t=284, ' t=285, ' t=286, ' t=287, ' t=288, ' t=289, ' t=290, ' t=291, ' t=292, ' t=293, ' t=294, ' t=295, ' t=296, ' t=297, ' t=298, ' t=299, ' t=300, ' t=301, '

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: t o k
Revised: t o k

Original: [Figure]
Revised: [Figure]

Original: n s t
Revised: n s t

Original: [Figure]
Revised: [Figure]

Original: t a l p r
Revised: t a l p r

Original: [Figure]
Revised: [Figure]

Original: b a b ili t y
Revised: b a b ili t y

Original: [Figure]
Revised: [Figure]

Original: D Collapsing to degenerate solution and labeled set memorization
Revised: D Collapsing to a Degenerate Solution and Labeled Set Memorization

Original: For some models where the WER between hard-alignment and golden transcription increases, wekeep training for a very long time.
Revised: For some models where the WER between hard-alignment and golden transcription increases, we keep training for a very long time.

Original: In all cases, while we observe different dynamics, all modelsconverge to some degenerate solution, while memorizing the labeled training set.
Revised: In all cases, while we observe different dynamics, all models converge to some degenerate solution, while memorizing the labeled training set.

Original: Convergenceto degenerate solution can be very slow and may take a huge number of training steps.
Revised: Convergence to a degenerate solution can be very slow and may take a huge number of training steps.

Original: For allexperiments we observe in some cases tendency to converge to blank prediction only (Figure 10) orto e.g. uniform distribution (Figure 11).
Revised: For all experiments we observe in some cases tendency to converge to blank prediction only (Figure 10) or to e.g. uniform distribution (Figure 11).

Original: In case of convergence to these trivial solutions we observethat the training loss on pseudo-labeled data becomes zero while the loss on labeled data is non-zero(and decreasing), as shown in Figure 12.
Revised: In case of convergence to these trivial solutions we observe that the training loss on pseudo-labeled data becomes zero while the loss on labeled data is non-zero (and decreasing), as shown in Figure 12.

Original: When we look at the model’s generated transcriptions, wesee empty transcriptions for unlabeled data and close to ground truth (WER is a few percentages) forlabeled data.
Revised: When we look at the model’s generated transcriptions, we see empty transcriptions for unlabeled data and close to ground truth (WER is a few percentages) for labeled data.

Original: This indicates that the model is in the regime of labeled data memorization.
Revised: This indicates that the model is in the regime of labeled data memorization

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E AblationsE.1 Excluding labeled data from PL phase
Revised: E Ablations E.1 Labeled Data Exclusion from PL Phase

Original: To confirm experimentally that there is no optimization problem (leading to trivial solutions) comingfrom the joint usage of different losses (CTC for labeled data, cross-entropy for unlabeled data), we  stop to train on labeled data as soon as the cache is filled.
Revised: To confirm experimentally that there is no optimization problem (leading to trivial solutions) coming from the joint usage of different losses (CTC for labeled data, cross-entropy for unlabeled data) we [Figure] stop to train on labeled data as soon as the cache is filled.

Original: With a cache size set to 100, we againobserve model divergences with soft-labels, while cache size of 1000 is able to stabilize training.
Revised: With a cache size set to 100, we again observe model divergences with soft-labels, while cache size of 1000 is able to stabilize training.

Original: Both cases have worse performance than their hard-path counterpart with the same hyper-parameters.
Revised: Both cases have worse performance than their hard-path counterpart with the same hyper-parameters.

Original: E.2 Teacher-student training provides stable soft-labeling
Revised: E.2 Teacher-Student Training as a Stable Soft-Labeling

Original: We tried also to disentangle continuous PL from soft-labels, and performed ablations with teacherstudent PL and soft-labels.
Revised: We tried also to disentangle continuous PL from soft-labels, and performed ablations with teacherstudent PL and soft-labels.

Original: We first train a supervised only model on 100h of LibriSpeech (gets 18%
Revised: We first train a supervised only model on 100h of LibriSpeech (gets 18%

Original: WER on dev-other ), use it to generate soft-labels for all unlabeled data and then train a new modelfrom scratch on joint labeled and pseudo-labeled data.
Revised: WER on dev-other ), use it to generate soft-labels for all unlabeled data and then train a new model from scratch on joint labeled and pseudo-labeled data.

Original: This model reduces WER on dev-other to14.1-14.3% WER for different values of τ and β while being very stable in training.
Revised: This model reduces WER on dev-other to 14.1-14.3% WER for different values of τ and β while being very stable in training.

Original: We perform thenthe second round of teacher-student training using the latter model as a new teacher.
Revised: We perform then the second round of teacher-student training using the latter model as a new teacher.

Original: Training a newmodel from scratch gives on dev-other 12.8-13.4% WER being stable again.
Revised: Training a new model from scratch gives on dev-other 12.8-13.4% WER being stable again.

Original: Hence, we conclude thatunstable training with soft-labels may come from continuous training dynamics.
Revised: Hence, we conclude that unstable training with soft-labels may come from continuous training dynamics.

Original: It is still an openquestion how to make continuous pseudo-labeling stable with soft-labels.
Revised: It is still an open question how to make continuous pseudo-labeling stable with soft-labels

Original: E.3 Blending soft-labels with hard-labels as regularization

Original: [Figure]
Revised: [Figure]

Original: i n i n g l
Revised: E.3 Regularization: Blending Soft-Labels with Hard-Labels

Original: [Figure]
Revised: [Figure]

Original: E.4 Sequence level consistency PL filtering
Revised: E.4 Sequence-Level Consistency: Pseudo-Labels Filtering

Revised: As we discovered issues with sequence-level consistency for soft-labels (Figure 4 vs Figure 5) for continuous training we tried regularization based on simple filtering pseudo-labeled data which

Original: [Figure]
Revised: [Figure]

Original: As we discovered issues with sequence-level consistency for soft-labels (Figure 4 vs Figure 5) forcontinuous training we tried regularization based on simple filtering pseudo-labeled data whichbreaks this consistency: we measure Levenshtein distance (on the token level) between hard-path transcriptions of current pseudo-label for a sample x ∈ U and of previously used pseudo-label tomake a training step of the same sample; if the distance is larger than some threshold we filter asample from the unlabeled batch for the current update step.
Revised: breaks this consistency: we measure Levenshtein distance (on the token level) between hard-path transcriptions of current pseudo-label for a sample x ∈ U and of previously used pseudo-label to make a training step of the same sample; if the distance is larger than some threshold we filter a sample from the unlabeled batch for the current update step.

Original: Our preliminary experiments show thatthis filtering is able to stabilize the training with a proper threshold, see Figure 15, however it reachesquite bad performance yet.
Revised: Our preliminary experiments show that this filtering is able to stabilize the training with a proper threshold, see Figure 16, however it reaches quite bad performance yet.