Original: Star Temporal Classification: Sequence Modeling with Partially Labeled Data
Revised: Star Temporal Classification: Sequence Modeling with Partially Labeled Data

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We develop an algorithm which can learn from partially labeled and unsegmentedsequential data.
Revised: We develop an algorithm which can learn from partially labeled and unsegmentedsequential data.

Original: Most sequential loss functions, such as Connectionist Temporal
Revised: Most sequential loss functions, such as Connectionist Temporal

Original: Classification (CTC), break down when many labels are missing.
Revised: Classification (CTC), break down when many labels are missing.

Original: We addressthis problem with Star Temporal Classification (STC) which uses a special startoken to allow alignments which include all possible tokens whenever a tokencould be missing.
Revised: We address thisproblem with Star Temporal Classification (STC) which uses a special star tokento allow alignments which include all possible tokens whenever a token could bemissing.

Original: We express STC as the composition of weighted finite-statetransducers (WFSTs) and use GTN (a framework for automatic differentiation with WFSTs) to compute gradients.
Revised: We express STC as the composition of weighted finite-state transducers(WFSTs) and use GTN (a framework for automatic differentiation with WFSTs)to compute gradients.

Original: We perform extensive experiments on automaticspeech recognition.
Revised: We perform extensive experiments on automatic speechrecognition.

Original: These experiments show that STC can recover most of theperformance of supervised baseline  when up to 70% of the labels are missing.
Revised: These experiments show that STC can close the performance gap withsupervised baseline to about 1% WER when up to 70% of the labels are missing.

Original: We also perform experiments in handwriting recognition to show that our methodeasily applies to other sequence classification tasks
Revised: We also perform experiments in handwriting recognition to show that our methodeasily applies to other sequence classification tasks

Original:  Introduction
Revised:  Introduction

Original: Applications of machine learning in settings with little or no labeled data are growing rapidly.
Revised: Applications of machine learning in settings with little or no labeled data are growing rapidly.

Original: Muchprior work in temporal classification focuses on semi-supervised or self-supervised learning [6, 37].
Revised: Muchprior work in temporal classification focuses on semi-supervised or self-supervised learning [6, 37].

Original: Instead, we focus on weakly supervised learning, in which the learner receives incomplete labelsfor the examples in the data set.
Revised: Instead, we focus on weakly supervised learning, in which the learner receives incomplete labelsfor the examples in the data set.

Original: Examples include multi-instance learning [1, 44], and partial-labellearning [18, 9, 25].
Revised: Examples include multi-instance learning [1, 44], and partial-labellearning [18, 9, 25].

Original: In our case, we assume that each example in the training set is partially labeled,as in Figure 1: neither the number of missing words nor their positions in the label sequence areknown in advance.
Revised: In our case, we assume that each example in the training set is partially labeled,as in Figure 1: neither the number of missing words nor their positions in the label sequence areknown in advance.

Original: [Figure]
Revised: [Figure]

Original: Partial labeling problem could arise in many practical scenarios.
Revised: Partial labeling problem could arise in many practical scenarios.

Original: In semi-supervised learning ortransfer learning of sequence labeling tasks, we can only consider high confidence tokens in the labelsequence and discard rest of the tokens.
Revised: In semi-supervised learning ortransfer learning of sequence labeling tasks, we can only consider high confidence tokens in the labelsequence and discard rest of the tokens.

Original: These labels with incomplete tokens can be treated as partiallabels.
Revised: These labels with incomplete tokens can be treated as partiallabels.

Original: When transcribing a dataset, we can randomly label parts of the dataset and use them as partiallabels, at a much lower cost than transcribing the full dataset.
Revised: When transcribing a dataset, we can randomly label parts of the dataset and use them as partiallabels, at a much lower cost than transcribing the full dataset.

Original: We can also consider semi-automatic ways to label a dataset.
Revised: We can also consider semi-automaticways to label a dataset.

Original: Restoring damaged texts [2, 3] where some of the writing is illegible could be considered as a partial labeling problem.
Revised: Restoring damaged texts [2, 3] where some of the writing is illegible could be considered as a partial labeling problem.

Original: We also often face partial alignment problem [5] in practice where beginning and ending tokens are missing.
Revised: We also often face partial alignment problem [5] in practice where beginning and ending tokens are missing.

Original: This is a sub-problem of the more general version ofpartial labeling problem that we solve in this work.
Revised: This is a sub-problem of the more general version ofpartial labeling problem that we solve in this work.

Original: To solve this weak supervision problem we develop the Star Temporal Classification (STC) lossfunction.
Revised: To solve this weak supervision problem we develop the Star Temporal Classification (STC) lossfunction.

Original: The STC algorithm explicitly accounts for the possibility of missing labels in the outputsequence at any position, through a special star token.
Revised: The STC algorithm explicitly accounts for the possibility of missing labels in the outputsequence at any position, through a special star token.

Original: For the example of Figure 1, the STClabel sequence is * from * airplane * the * a * spider * (where * can be anything butthe following label token).
Revised: For the example of Figure 1, the STClabel sequence is * from * airplane * the * a * spider * (where * can be anything butthe following label token).

Original: We show how to simply implement STC in the weighted finite-statetransducer (WFST) framework.
Revised: We show how to simply implement STC in the weighted finite-statetransducer (WFST) framework.

Original: Moreover, through the use of automatic differentiation with WFSTs,gradients of the loss function with respect to the inputs are computed automatically, allowing STC tobe used with standard neural network training pipelines.
Revised: Moreover, through the use of automatic differentiation with WFSTs,gradients of the loss function with respect to the inputs are computed automatically, allowing STC tobe used with standard neural network training pipelines.

Original: For practical applications, a naive implementation of STC using WFSTs is intractably slow.
Revised: For practical applications, a naive implementation of STC using WFSTs is intractably slow.

Original: Wepropose several optimizations to improve the efficiency of the STC loss.
Revised: Wepropose several optimizations to improve the efficiency of the STC loss.

Original: The star token we useresults in much smaller graphs since it combines many arcs into one.
Revised: The star token we useresults in much smaller graphs since it combines many arcs into one.

Original: The star token also enables usto substantially reduce the amount of memory transfers between the GPU, on which the network iscomputed, and the CPU, on which the STC algorithm is evaluated.
Revised: The star token also enables usto substantially reduce the amount of memory transfers between the GPU, on which the network iscomputed, and the CPU, on which the STC algorithm is evaluated.

Original: These optimizations result in onlya minor increase in model training time when using the STC loss over a CTC baseline.
Revised: These optimizations result in onlya minor increase in model training time when using the STC loss over a CTC baseline.

Original: The STC loss can be applied to most sequence transduction tasks including speech recognition, handwriting recognition, machine translation, and action recognition from videos.
Revised: The STC loss can be applied to most sequence transduction tasks including speech recognition, handwriting recognition, machine translation, and action recognition from videos.

Original: Here we demonstratethe effectiveness of STC on both automatic speech recognition and offline handwriting recognition.
Revised: Here we demonstratethe effectiveness of STC on both automatic speech recognition and offline handwriting recognition.

Original: In speech recognition, with up to 70% of the labels missing, STC is able to achieve low word errorrates.
Revised: In speech recognition, with up to 70% of the labels missing, STC is able to achieve similar (about 1%

Revised: WER difference) performance as supervised baseline .

Original: Similarly in handwriting recognition STC can achieve low character error rates with up to 50%of the labels missing.
Revised: Similarly in handwriting recognition STCcan achieve low character error rates with up to 50% of the labels missing.

Original: In both cases STC yields dramatic gains over a baseline system which does notexplicitly handle missing labels.
Revised: In both cases STC yieldsdramatic gains over a baseline system which does not explicitly handle missing labels.

Original: To summarize, the main contributions of this work are:
Revised: To summarize, the main contributions of this work are:

Original: • We introduce STC, a novel temporal classification algorithm which explicitly handles anarbitrary number of missing labels with an indeterminate location.•
Revised: • We introduce STC, a novel temporal classification algorithm which explicitly handles anarbitrary number of missing labels with an indeterminate location.

Original:  We simplify the implementation of STC by using WFSTs and automatic differentiation.
Revised: • We simplify the implementation of STC by using WFSTs and automatic differentiation.•

Original: • We describe key optimizations for STC enabling it to be used with minimal increase intraining time.•
Revised: We describe key optimizations for STC enabling it to be used with minimal increase intraining time.•

Original: We show in practical speech and handwriting recognition tasks that STC yields low wordand character error rates with up to 70% of the labels missing.
Revised: We show in practical speech and handwriting recognition tasks that STC yields low wordand character error rates with up to 70% of the labels missing.

Original:  Related Work
Revised:  Related Work

Original: Many generalizations and variations of CTC have been proposed.
Revised: Many generalizations and variations of CTC have been proposed.

Original: Graph-based temporal classification(GTC)
Revised: Graph-based temporal classification(GTC)

Original: [32] generalizes CTC to allow for an n -best list of possible labels.
Revised: [32] generalizes CTC to allow for an n -best list of possible labels.

Original: [40] also extends CTC forusing with for multiple labels.
Revised: [40] also extends CTC forusing with for multiple labels.

Original: [23] propose selfless-CTC which disallows self-loops for non-blankoutput tokens.
Revised: [23] propose selfless-CTC which disallows self-loops for non-blankoutput tokens.

Original: Unlike these alternatives, STC allows for an arbitrary number of missing tokensanywhere in the label for a given example.
Revised: Unlike these alternatives, STC allows for an arbitrary number of missing tokensanywhere in the label for a given example.

Original: A recent work, concurrent with this work, proposes a wildcard version of CTC [5].
Revised: A recent work, concurrent with this work, proposes a wildcard version of CTC [5].

Original: However, in that work the wild cards, or missing tokens, can only be at thebeginning or end of the label.
Revised: However, in that work the wild cards, or missing tokens, can only be at thebeginning or end of the label.

Original: Extended Connectionist Temporal Classification (ECTC) [17] was developed for weakly supervisedvideo action labeling.
Revised: Extended Connectionist Temporal Classification (ECTC) [17] was developed for weakly supervisedvideo action labeling.

Original: However, the weak supervision in this case refers to the lack of a segmentation,which is the common paradigm for CTC.
Revised: However, the weak supervision in this case refers to the lack of a segmentation,which is the common paradigm for CTC.

Original: The extensions to CTC are intended to handle the muchhigher ratio of input video frames to output action labels.
Revised: The extensions to CTC are intended to handle the muchhigher ratio of input video frames to output action labels.

Original: [11] develop a sequence level loss functionwhich can learn from noisy labels.
Revised: [11] develop a sequence level loss functionwhich can learn from noisy labels.

Original: They extend the auto segmentation criterion [7] to include a noisemodel which explicitly handles up to one missing label between tokens in the output.
Revised: They extend the auto segmentation criterion [7] to include a noisemodel which explicitly handles up to one missing label between tokens in the output.

Original: Unlike thatwork, STC extends the more commonly used CTC loss and allows for any number of missing tokens.
Revised: Unlike thatwork, STC extends the more commonly used CTC loss and allows for any number of missing tokens.

Original: The idea of expressing CTC as the composition of WFSTs is well known [27, 23, 41].
Revised: The idea of expressing CTC as the composition of WFSTs is well known [27, 23, 41].

Original: However,recently developed WFST-based frameworks which support automatic differentiation [15, 19] makethe development of variations of the CTC loss much simpler.
Revised: However,recently developed WFST-based frameworks which support automatic differentiation [15, 19] makethe development of variations of the CTC loss much simpler.

Original: WFSTs have been modified by to take into account of noisy transcripts [16, 31, 4] by composingthe WFST graphs of word sequences.
Revised: WFSTs have been modified by to take into account of noisy transcripts [16, 31, 4] by composingthe WFST graphs of word sequences.

Original: However, in this work the WFST composition is between significantly larger WFST graphs (label sequence and neural network model output sequence).
Revised: However, in this work the WFST composition is between significantly larger WFST graphs (label sequence and neural network model output sequence).

Original: Also,the composition is performed during the training of the neural network which makes it a morechallenging engineering problem.
Revised: Also,the composition is performed during the training of the neural network which makes it a morechallenging engineering problem.

Original:  Method3.1 Problem Description
Revised:  Method3.1 Problem Description

Original: We consider the problem of temporal classification [20], which predicts an output sequence y =[
Revised: We consider the problem of temporal classification [20], which predicts an output sequence y =[

Original: y 1 , ...., y U
Revised: y 1 , ...., y U

Original: ] ∈
Revised: ] ∈

Original: A 1 × U , where A is a fixed alphabet of possible output tokens, from an unsegmentedinput sequence x = [ x 1 , ..., x T ] ∈ R d × T , where each x i is a d -dimensional feature vector.
Revised: A 1 × U , where A is a fixed alphabet of possible output tokens, from an unsegmentedinput sequence x = [ x 1 , ..., x T ] ∈ R d × T , where each x i is a d -dimensional feature vector.

Original: We alsoassume that the length of the output U is less than or equal to that of the input, T .
Revised: We alsoassume that the length of the output U is less than or equal to that of the input, T .

Original: A partially labeledoutput sequence ˜ y of length ˜ U , such that ( ˜ U ≤ U ), is formed by removing zero or more elementsfrom the true output sequence y .
Revised: A partially labeledoutput sequence ˜ y of length ˜ U , such that ( ˜ U ≤ U ), is formed by removing zero or more elementsfrom the true output sequence y .

Original: Given a training set D ≜ { x i , ˜ y
Revised: Given a training set D ≜ { x i , ˜ y

Original: i : i = 1 , . . .
Revised: i : i = 1 , . . .

Original: , N } consisting ofinput sequences x i =
Revised: , N } consisting ofinput sequences x i =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x T i ] and partially labeled output sequences ˜ y
Revised: , x T i ] and partially labeled output sequences ˜ y

Original: i =
Revised: i =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y ˜ U
Revised: , y ˜ U

Original: i ] ourgoal is to learn temporal models which can predict true output sequences y
Revised: i ] ourgoal is to learn temporal models which can predict true output sequences y

Original: i
Revised: i

Original: We consider this aweakly supervised temporal classification problem as the learning algorithm only has access to asubset of the true output labels during training.
Revised: We consider this aweakly supervised temporal classification problem as the learning algorithm only has access to asubset of the true output labels during training.

Original: 3.2 Connectionist Temporal Classification (CTC)
Revised: 3.2 Connectionist Temporal Classification (CTC)

Original: Connectionist Temporal Classification (CTC) [13] is a widely used loss function for training sequencemodels where the alignment of input sequence with the target labels is not known in advance.
Revised: Connectionist Temporal Classification (CTC) [13] is a widely used loss function for training sequencemodels where the alignment of input sequence with the target labels is not known in advance.

Original: Consider an input sequence x
Revised: Consider an input sequence x

Original: = [ x 1 , . .
Revised: = [ x 1 , . .

Original: , x T ] and target sequence y =
Revised: , x T ] and target sequence y =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y U ] ∈
Revised: , y U ] ∈

Original: A 1 × U , where
Revised: A 1 × U , where

Original: A is a finite alphabet of possible output tokens and U ≤ T .
Revised: A is a finite alphabet of possible output tokens and U ≤ T .

Original: The CTC loss uses a special blank token,⟨ b ⟩ , to represents frames which do not correspond to an output token.
Revised: The CTC loss uses a special blank token,⟨ b ⟩ , to represents frames which do not correspond to an output token.

Original: An alignment between theinput x and output y in CTC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ]
Revised: An alignment between theinput x and output y in CTC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ]

Original: whereπ t ∈
Revised: whereπ t ∈

Original: A ∪ { blank } .
Revised: A ∪ { blank } .

Original: The CTC collapse function B maps an alignment to an output, B ( π ) =
Revised: The CTC collapse function B maps an alignment to an output, B ( π ) =

Original: y .
Revised: y .

Original: Thefunction B removes all but one of any consecutively repeated tokens and then removes blank tokense.g.
Revised: Thefunction B removes all but one of any consecutively repeated tokens and then removes blank tokense.g.

Original: , (a, b, blank , blank , b, b, blank , a) (cid:55)→ (a, b, b, a).
Revised: , (a, b, blank , blank , b, b, blank , a) (cid:55)→ (a, b, b, a).

Original: Given the input x , each output is independent ofall other outputs, hence the probability of an alignment P ( π | x ) is:
Revised: Given the input x , each output is independent ofall other outputs, hence the probability of an alignment P ( π | x ) is:

Original: [Equation]
Revised: [Equation]

Original: There can be many possible alignments π for a given x and y pair.
Revised: There can be many possible alignments π for a given x and y pair.

Original: The CTC loss computes thenegative log probability of y given x by marginalizing over all possible alignments:
Revised: The CTC loss computes thenegative log probability of y given x by marginalizing over all possible alignments:

Original: [Equation]
Revised: [Equation]

Original: The sum over all alignments can be computed efficiently with dynamic programming as described by[13].
Revised: The sum over all alignments can be computed efficiently with dynamic programming as described by[13].

Original: The CTC loss can be computed purely with WFST 1 operations as described in Figure 2.
Revised: The CTC loss can be computed purely with WFST 1 operations as described in Figure 2.

Original: Thecomposition of the emission graph (Figure 2a) which has arc weights weights corresponding tolog P ( π t | x ) and the label graph (Figure 2b) is shown in Figure 2c.
Revised: Thecomposition of the emission graph (Figure 2a) which has arc weights weights corresponding tolog P ( π t | x ) and the label graph (Figure 2b) is shown in Figure 2c.

Original: Each path from a start to a finalstate in Figure 2c is a valid CTC alignment.
Revised: Each path from a start to a finalstate in Figure 2c is a valid CTC alignment.

Original: The negation of the forward score of the composedgraph gives the CTC loss.
Revised: The negation of the forward score of the composedgraph gives the CTC loss.

Original: These operations are differentiable and gradients can easily be propagatedbackward from the loss to a neural network to train the model.
Revised: These operations are differentiable and gradients can easily be propagatedbackward from the loss to a neural network to train the model.

Original: Constructing the CTC loss from simpler WFST graphs also simplifies the implementation of variationsof CTC.
Revised: Constructing the CTC loss from simpler WFST graphs also simplifies the implementation of variationsof CTC.

Original: For example, the variations of [32] only require changes to the label graph Y ctc .
Revised: For example, the variations of [32] only require changes to the label graph Y ctc .

Original: However,when efficiency is a primary concern, custom GPU kernels can be faster than a generic implementationwith WFSTs.
Revised: However,when efficiency is a primary concern, custom GPU kernels can be faster than a generic implementationwith WFSTs.

Original: [Figure]
Revised: [Figure]

Original: 3.3 Star Temporal Classification (STC)
Revised: 3.3 Star Temporal Classification (STC)

Original: The label graph of CTC, L ctc (Figure 2b), constructed from partial labels does not allow for the truetarget as a possibility.
Revised: The label graph of CTC, L ctc (Figure 2b), constructed from partial labels does not allow for the truetarget as a possibility.

Original: The STC algorithm addresses this problem by allowing for zero or more tokensfrom the alphabet between any two tokens in the partial label.
Revised: The STC algorithm addresses this problem by allowing for zero or more tokensfrom the alphabet between any two tokens in the partial label.

Original: Like CTC, an alignment between theinput x and output y in STC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ]
Revised: Like CTC, an alignment between theinput x and output y in STC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ]

Original: whereπ t ∈
Revised: whereπ t ∈

Original: A ∪ { blank } and y is a partial label of B ′ ( π ) .
Revised: A ∪ { blank } and y is a partial label of B ′ ( π ) .

Original: Unlike CTC, the STC collapse function B ′ ( π )only removes blank tokens.
Revised: Unlike CTC, the STC collapse function B ′ ( π )only removes blank tokens.

Original: In other words, we do not allow self-loops on non blank tokens in the
Revised: In other words, we do not allow self-loops on non blank tokens in the

Original: STC label graph.
Revised: STC label graph.

Original: This enables us to use a token insertion penalty as discussed in Section 3.3.1.
Revised: This enables us to use a token insertion penalty as discussed in Section 3.3.1.a/w b/w c/w d/w → ⟨ * ⟩ /w (

Original: [Figure]
Revised: [Figure]

Original: a b c , d } to star token,
Revised: Collapsing to ⟨∗⟩ token w5 = logsumexp(w1,w2,w3,w4) a/w b/w c/w d/w → ⟨ * ⟩ [Figure]

Original: s ⟩
Revised: , ⟨∗⟩

Original: To make the computation of the STC loss efficient, STC uses a special star token, ⟨ s ⟩ , whichrepresents every token in the alphabet.
Revised: To make the computation of the STC loss efficient, STC uses a special star token, ⟨∗⟩ , whichrepresents every token in the alphabet.

Original: It can be used to collapse the arcs between any two nodesin the WFST graph as shown in Figure 3a.
Revised: It can be used to collapse the arcs between any two nodesin the WFST graph as shown in Figure 3a.

Original: We also define ⟨ s ⟩\ t =
Revised: We also define ⟨∗⟩\ t =

Original: { y : y ∈ A ; y ̸ = t } which is therelative complement of t in A .
Revised: { y : y ∈ A ; y ̸ = t } which is therelative complement of t in A .

Original: We use this in the label graph of STC, L stc , to avoid counting thesame alignment multiple times.
Revised: We use this in the label graph of STC, L stc , to avoid counting thesame alignment multiple times.

Original: The STC label graph, L stc , for the partial label ( a, b, c ) is shown in
Revised: The STC label graph, L stc , for the partial label ( a, b, c ) is shown in

Original: Figure 4.
Revised: Figure 4.

Original: The graph allows any alignment with zero or more tokens in between any two tokens of thegiven partial label sequence.
Revised: The graph allows any alignment with zero or more tokens in between any two tokens of thegiven partial label sequence.

Original: We also manually add the ⟨ s ⟩ and ⟨ s ⟩\ t tokens to the emission graph E X with weights correspondingto their log-probabilities  given by:
Revised: We also manually add the ⟨∗⟩ and ⟨∗⟩\ t tokens to the emission graph E X with weights correspondingto their log-probabilities for each timestep t given by:

Original: [Equation]
Revised: [Equation]

Original: ⟨ b ⟩ /  a/  ⟨ s ⟩ \a/ λ 2 ⟨ b ⟩ / 3 b/ ⟨ b ⟩ / a/0 ⟨ s ⟩ \a/ λ ⟨ b ⟩ / b/  ⟨ s ⟩ \b/ λ 4 ⟨ b ⟩ / 5 c/0 ⟨ b ⟩ / b/ ⟨ s ⟩ \b/ λ ⟨ b ⟩ / c/  ⟨ s ⟩ \c/ λ 6 ⟨ b ⟩ / 10 ⟨ s ⟩ /0 ⟨ b ⟩ / c/ ⟨ s ⟩ \c/ λ ⟨ b ⟩ / ⟨ s ⟩ /0 ⟨ b ⟩ / ⟨ s ⟩ /
Revised: ⟨ b ⟩ /  a/  ⟨ * ⟩ \a/ λ 2 ⟨ b ⟩ / 3 b/ ⟨ b ⟩ / a/0 ⟨ * ⟩ \a/ λ ⟨ b ⟩ / b/  ⟨ * ⟩ \b/ λ 4 ⟨ b ⟩ / 5 c/0 ⟨ b ⟩ / b/ ⟨ * ⟩ \b/ λ ⟨ b ⟩ / c/  ⟨ * ⟩ \c/ λ 6 ⟨ b ⟩ / 10 ⟨ * ⟩ /0 ⟨ b ⟩ / c/ ⟨ * ⟩ \c/ λ ⟨ b ⟩ / ⟨ * ⟩ / ⟨ b ⟩ / ⟨ * ⟩ /

Original: 3.3.1 Token Insertion Penalty, λ
Revised: 3.3.1 Token Insertion Penalty, λ

Original: We noticed that model training with STC does not work if we train directly with the STC label graph,even when 10% of the labels are missing.
Revised: We noticed that model training with STC does not work if we train directly with the STC label graph,even when 10% of the labels are missing.

Original: This is because the new paths introduced in STC from
Revised: This is because the new paths introduced in STC from

Original: CTC (marked in red in Figure 4) would allow the model to produce a lot of possible output sequencesfor any given example and can confuse the model, especially during the early stages of training.
Revised: CTC (marked in red in Figure 4) would allow the model to produce a lot of possible output sequencesfor any given example and can confuse the model, especially during the early stages of training.

Original: Tocircumvent this problem, we use a parameter called token insertion penalty, λ to add a penalty whenusing lot of new tokens.
Revised: Tocircumvent this problem, we use a parameter called token insertion penalty, λ to add a penalty whenusing lot of new tokens.

Original: We define STC loss in the following way
Revised: We define STC loss in the following way

Original: [Equation]
Revised: [Equation]

Original: where P path i is the probability of an alignment path i , p ∈ (0 , 1] and k i is the number of additionalstar tokens for path i .
Revised: where P path i is the probability of an alignment path i , p ∈ (0 , 1] and k i is the number of additionalstar tokens for path i .

Original: This down weight paths which include many additional tokens.
Revised: This down weight paths which include many additional tokens.

Original: The tokeninsertion penalty is in log-space and is defined as λ = log( p ) .
Revised: The tokeninsertion penalty is in log-space and is defined as λ = log( p ) .

Original: We use a exponential decay scheme to gradually reduce the penalty as the model starts training.
Revised: We use a exponential decay scheme to gradually reduce the penalty as the model starts training.

Original: [Equation]
Revised: [Equation]

Original: where p 0 , p max , τ are hyperparameters and λ t denotes the token insertion penalty used in STC fortraining step t .
Revised: where p 0 , p max , τ are hyperparameters and λ t denotes the token insertion penalty used in STC fortraining step t .

Original: For choosing value of τ , is is useful think in terms of half-life t 1 / [-
-]
Revised: For choosing value of τ , it is useful think in terms of half-life t 1 / [-
-]2 =

Original: = τln (2) , whichis the number of time steps taken for p t to reach ( p 0 + p max ) / [-
-]{+23.3.
+}
Revised: τln (2) , which isthe number of time steps taken for p t to reach ( p 0 + p max ) / [-
-]{+23.3.
+}

Original: 3.3.2 On the importance of ⟨ s ⟩\ t token
Revised: On the importance of ⟨∗⟩\ t token

Original: The “ ⟨ s ⟩\ t " token which matches any token except the token, t is necessary to not count the same pathmultiple times.
Revised: The ⟨∗⟩\ t " token which matches any token except the token, t is necessary to not count the same pathmultiple times.

Original: We illustrate it through the two possible STC label graphs from Figure 5 .
Revised: We illustrate it through the two possible STC label graphs from Figure 5 .

Original: Considerthat the partial label has just one word “the" and number of time steps, T = 3.
Revised: Considerthat the partial label has just one word "the" and number of time steps, T = 3.

Original: In Figure 5a, the path“the the cat” is counted multiple times when we use start token ⟨ s ⟩ directly [-
-]{+0 → 1 → 2 → 4 and 0 →
+}
Revised: In Figure 5a, the path“the the cat” is counted multiple times when we use start token ⟨∗⟩ directly [-
-]{+0 → 1 → 2 → 4 and 0 →
+}

Original: →1 → 2 → 4 and 0 →2 → 4 → 4.
Revised: → 4 → 4.

Original: However, if we use ⟨ s ⟩\ the token as shown in Figure 5b , we do not have this issue ofaccepting the path “the the cat” multiple times.
Revised: However, if we use ⟨∗⟩\ the token as shown in Figure 5b , we do not have this issue ofaccepting the path "the the cat" multiple times.

Original: It should also be noted that the “ ⟨ s ⟩\ t " token is only present after the token t but note before thetoken t in the label graph of STC from Figure 4.
Revised: It should also be noted that the " ⟨∗⟩\ t " token is only present after the token t but not before the tokent in the label graph of STC from Figure 4.

Original: So, the label graph would still accept all possiblelabels sequences for the given partial label.
Revised: So, the label graph would still accept all possible labelssequences for the given partial label.

Original: 3.3.3 Implementation Details
Revised: 3.3.3 Implementation Details

Original: With the recent advent of frameworks such as GTN[15], K2[19] which support autograd on WFSTs, we focus on the engineering details used by us to use these frameworks and train large-scalerecognition systems below.
Revised: With the recent advent of frameworks such as GTN[15], K2[19] which support autograd on WFSTs, we focus on the engineering details used by us to use these frameworks and train large-scalerecognition systems below.

Original: We use the CPU-based WFST algorithms from GTN [15] to implement the STC criterion.
Revised: We use the CPU-based WFST algorithms from GTN [15] to implement the STC criterion.

Original: Multiplethreads are used to compute STC loss in parallel for all of the examples in a batch.
Revised: Multiplethreads are used to compute STC loss in parallel for all of the examples in a batch.

Original: Since the neuralnetwork model is run on GPU, the emissions needed by STC must be copied to the CPU, and the
Revised: Since the neuralnetwork model is run on GPU, the emissions needed by STC must be copied to the CPU, and the

Original: STC gradients must be copied back to GPU.
Revised: STC gradients must be copied back to GPU.

Original: To reduce the amount of data transfer between the CPUand the GPU, we transfer values corresponding only to the tokens present in the partial labels and thestar tokens.
Revised: To reduce the amount of data transfer between the CPUand the GPU, we transfer values corresponding only to the tokens present in the partial labels and thestar tokens.

Original: This works since the gradients corresponding to all other tokens are zero.
Revised: This works since the gradients corresponding to all other tokens are zero.

Original: Figure 6 shows an overview of the STC training pipeline.
Revised: Figure 6 shows an overview of the STC training pipeline.

Original: The input sequence is typically passedthrough a neural network on GPU to produce a frame wise distribution (in log-space) over A∪{ blank } .
Revised: The input sequence is typically passedthrough a neural network on GPU to produce a frame wise distribution (in log-space) over A∪{ blank } .

Original: The output number of frames depend on the size of the input sequence, as well as the amount ofpadding and the stride of the neural network model architecture.
Revised: The output number of frames depend on the size of the input sequence, as well as the amount ofpadding and the stride of the neural network model architecture.

Original: We then compute the log-probabilityof star token, ⟨ s ⟩ and ⟨ s ⟩\ t tokens ∀ t ∈
Revised: We then compute the log-probabilityof star token, ⟨∗⟩ and ⟨∗⟩\ t tokens ∀ t ∈

Original: y using Equation 3.
Revised: y using Equation 3.

Original: We construct the emission graph E Xwith weights as log-probabilities and the label graph L stc on CPU.
Revised: We construct the emission graph E Xwith weights as log-probabilities and the label graph L stc on CPU.

Original: Using the WFST operations shownin Figure 2, we compute STC loss and gradients of loss with respect to arc weights (log probabilityoutput) of E X .
Revised: Using the WFST operations shownin Figure 2, we compute STC loss and gradients of loss with respect to arc weights (log probabilityoutput) of E X .

Original: These gradients are copied back to GPU and the gradients of parameters in neuralnetwork are computed using backpropagation.
Revised: These gradients are copied back to GPU and the gradients of parameters in neuralnetwork are computed using backpropagation.

Original: Then the model can be trained using standard gradientdescent methods.
Revised: Then the model can be trained using standard gradientdescent methods.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: Training Pipeline

Original:  Experimental Setup
Revised:  Experimental Setup

Original: [Figure]

Original: 4.1 Automatic Speech Recognition (ASR)
Revised: 4.1 Automatic Speech Recognition (ASR)

Original: [Figure] We use LibriSpeech [33] dataset, containing 960 hours of training audio with paired transcriptionsfor our speech recognition experiments.
Revised: We use LibriSpeech [33] dataset, containing 960 hours of training audio with paired transcriptionsfor our speech recognition experiments.

Original: The standard LibriSpeech validation sets ( dev-clean anddev-other ) are used to tune all hyperparameters, as well as to select the best models.
Revised: The standard LibriSpeech validation sets ( dev-clean anddev-other ) are used to tune all hyperparameters, as well as to select the best models.

Original: We report finalword error rate (WER) performance on the test sets ( test-clean and test-other ).
Revised: We report finalword error rate (WER) performance on the test sets ( test-clean and test-other ).

Original: We use the top 50K words (sorted by occurance frequency) from the official language model (LM)training data provided with LibriSpeech as the alphabet for training the models.
Revised: We use the top 50K words (sorted by occurance frequency) from the official language model (LM)training data provided with LibriSpeech as the alphabet for training the models.

Original: These 50K wordscover 99.04% of all the word occurances in the LM training data.
Revised: These 50K wordscover 99.04% of all the word occurances in the LM training data.

Original: The model architectures andthe STC loss are implemented with the ASR application [35] of the flashlight 2 machine-learningframework and the C++ API of GTN.
Revised: The model architectures andthe STC loss are implemented with the ASR application [35] of the flashlight 2 machine-learningframework and the C++ API of GTN.

Original: 4.1.1 Label Generation for Weakly Supervised Setup
Revised: 4.1.1 Label Generation for Weakly Supervised Setup

Original: As LibriSpeech dataset consists of fully labeled data, we need to drop the labels manually to simulatethe weakly supervised setup.
Revised: As LibriSpeech dataset consists of fully labeled data, we need to drop the labels manually to simulatethe weakly supervised setup.

Original: We quantify this using a parameter pDrop ∈ [0 , 1] , which denotesthe probability of dropping a word from the transcript label.
Revised: We quantify this using a parameter p _ drop ∈ [0 , 1] , which denotes theprobability of dropping a word from the transcript label.

Original: It can be seen that higher value ofpDrop corresponds to higher number of missing words in the transcript and vice-versa.
Revised: It can be seen that higher value of p _ dropcorresponds to higher number of missing words in the transcript and vice-versa.

Original: Figures7a-b show the histogram of percentage of the labeled words present in each training sample in LibriSpeech for different values of pDrop which closely follows a normal distribution.
Revised: Figures 7a-b showthe histogram of percentage of the labeled words present in each training sample in LibriSpeech fordifferent values of p _ drop which closely follows a normal distribution.

Original: Also, fromthe definition of pDrop , it naturally follows that pDrop = 0 corresponds to supervised training setupwhile pDrop = 1 corresponds to unsupervised training.
Revised: Also, from the definitionof p _ drop , it naturally follows that p _ drop = 0 corresponds to supervised training setup whilep _ drop = 1 corresponds to unsupervised training.

Original: To make sure our method is robust to different types of partial labels, we also test our method on twoother ways of generating the data :
Revised: To make sure our method is robust to different types of partial labels, we also test our method on twoother ways of generating the data :

Original: Randomly split the training set into p parts and use a different pDrop for each split (Figure 7c)2.
Revised: Randomly split the training set into p parts and use a different p _ drop for each split (Figure 7c)2.

Original: Randomly split the words into p parts, use a different pDrop for each split (Figure 7d)
Revised: Randomly split the words into p parts, use a different p _ drop for each split (Figure 7d)

Revised: 2 https

Original: [Figure]
Revised: [Figure]

Original: For all the new partially labeled datasets created, we prune samples which have empty transcriptionsbefore using them for training.
Revised: For all the new partially labeled datasets created, we prune samples which have empty transcriptionsbefore using them for training.

Original: 4.1.2 Pseudo-labeling
Revised: 4.1.2 Pseudo-labeling

Original: To further achieve better word error rate (WER), we generate pseudo labels (PLs) on the training setusing the word-based model trained with STC and use these PLs to train a letter-based model, via aregular CTC approach.
Revised: To further achieve better word error rate (WER), we generate pseudo labels (PLs) on the training setusing the word-based model trained with STC and use these PLs to train a letter-based model, via aregular CTC approach.

Original: The PLs are generated using rescoring as described in Section B.1.2 on thetraining set using the hyperparameters optimized on validation set.
Revised: The PLs are generated using rescoring as described in Section B.1.2 on thetraining set using the hyperparameters optimized on validation set.

Original: 4.2 Handwriting Recognition (HWR)
Revised: 4.2 Handwriting Recognition (HWR)

Original: True Label : once-and-for-all|cash|offer|of|357million|.|President pDrop = 0.1 : oc-andforallcah|offer|o|357million|.|Preident pDrop = 0.3 : once-nd-fralcasr|of|357milon|Presidnt pDrop = 0.7 : nc-llaf|7mlioPreit
Revised: True Label : once-and-for-all|cash|offer|of|357million|.|President pDrop = 0.1 : oc-andforallcah|offer|o|357million|.|Preident pDrop = 0.3 : once-nd-fralcasr|of|357milon|Presidnt pDrop = 0.7 : nc-llaf|7mlioPreit

Original: We test our approach on IAM Handwriting database [26], which is a widely used benchmark forhandwriting recognition.
Revised: We test our approach on IAM Handwriting database [26], which is a widely used benchmark forhandwriting recognition.

Original: The dataset contains 78 different characters and a white-space symbol, | .
Revised: The dataset contains 78 different characters and a white-space symbol, | .

Original: We use Aachen data splits 3 to divide the dataset into three subsets: 6,482 lines for training, 976 linesfor validation and 2,915 lines for testing.
Revised: We use Aachen data splits 3 to divide the dataset into three subsets: 6,482 lines for training, 976 linesfor validation and 2,915 lines for testing.

Original: To create weakly supervised labels, we use the same methodology using pDrop as described in4.1.1.
Revised: To create weakly supervised labels, we use the same methodology using p _ drop as described in4.1.1.

Original: However, we drop the labels at character-level instead of word-level.
Revised: However, we drop the labels at character-level instead of word-level.

Original: Figure 8 shows a trainingexample from IAM along with weakly supervised labels generated for different pDrop values.
Revised: Figure 8 shows a trainingexample from IAM along with weakly supervised labels generated for different p _ drop values.

Original: Additional implementation and training details (models, tokens, optimization, other hyperparametersand settings) for the ASR and HWR experiments are in the Appendix B.
Revised: Additional implementation and training details (models, tokens, optimization, other hyperparametersand settings) for the ASR and HWR experiments are in the Appendix B.

Original:  Results
Revised:  Results

Original: 5.1 Automatic Speech Recognition
Revised: 5.1 Automatic Speech Recognition

Original: In Table 1, we compare the WER performance of STC models for a fixed value of pDrop =0 .
Revised: In Table 1, we compare the WER performance of STC models for a fixed value of p _ drop =0 .

Original: 7 and also when pDrop is dependent on the sample or the word.
Revised: 7 and also when p _ drop is dependent on the sample or the word.

Original: We also comparethese results with the models trained directly with CTC on the partial labels.
Revised: We also comparethese results with the models trained directly with CTC on the partial labels.

Original: We can see that STC
Revised: We can see that STC

Original: [Table]
Revised: [Table]

Original: performs better than CTC in all the settings.
Revised: performs better than CTC in all the settings.

Original: We observe that performing beam search decodingand an additional second-pass rescoring with a Transformer LM can reduce the WER of the models [Table]
Revised: We observe that performing beam search decodingand an additional second-pass rescoring with a Transformer LM can reduce the WER of the modelssignificantly.

Original: significantly. We also see that pseudo labeling the training set using STC model trained on partial labels and thentraining a letter-based CTC model can further improve WER performance.
Revised: We also see that pseudo labeling the training set using STC model trained on partial labels and thentraining a letter-based CTC model can further improve WER performance.

Original: We can get competitiveresults compared with the fully supervised results.
Revised: We can get competitiveresults compared with the fully supervised results.

Original: P D ROP
Revised: P _ DROP

Original: M ETHOD
Revised: M ETHOD

Original: D EV CER
Revised: D EV CER

Original: T EST CER
Revised: T EST CER

Original: T RANSFORMER [21] 4.
Revised: T RANSFORMER [21] 4.

Original: CNN+CTC[43] 4.
Revised: CNN+CTC[43] 4.

Original: LSTM W /A TTN [28] 3.
Revised: LSTM W /A TTN [28] 3.

Original: CNN + CTC
Revised: CNN + CTC

Original: (O UR B ASELINE ) 3.
Revised: (O UR B ASELINE ) 3.

Original: CNN + CTC 5.
Revised: CNN + CTC 5.

Original: 7.7 CNN + STC 5.
Revised: 7.7 CNN + STC 5.

Original: CNN + CTC 8.
Revised: CNN + CTC 8.

Original: 11.6 CNN + STC 5. 
Revised: 11.6 CNN + STC 5. CNN + CTC

Original: [Figure]
Revised: [Figure]

Original:  CNN 
Revised: 53.6 CNN + STC

Original: [Figure]
Revised: [Figure]

Original:  CTC[Figure] [Figure] STC
Revised: CNN + CTC

Original: [Figure]
Revised: [Figure]

Original: CNN
Revised: CNN

Original: [Figure]
Revised: [Figure]

Original: CTC

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 5.2 Handwriting Recognition
Revised: 5.2 Handwriting Recognition

Original: In Table 2, we compare the CER performance of CTC and STC for various values of pDrop withgreedy decoding.
Revised: In Table 2, we compare the CER performance of CTC and STC for various values of p _ drop withgreedy decoding.

Original: We can see that using STC clearly gives better performance over CTC trainedmodels for pDrop = 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 .
Revised: We can see that using STC clearly gives better performance over CTC trainedmodels for p _ drop = 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 .

Original: The performance gap between the supervised baseline andmodels trained with partial labels using STC is larger compared to ASR experiments.
Revised: The performance gap between the supervised baseline andmodels trained with partial labels using STC is larger compared to ASR experiments.

Original: This is becausewe did not use LM decoding or pseudo-labeling steps which should help in reducing CER.
Revised: This is becausewe did not use LM decoding or pseudo-labeling steps which should help in reducing CER.

Original: Hyperparameters used for token insertion penalty, λ to train STC models for ASR, HWR experimentsare in Appendix D.
Revised: Hyperparameters used for token insertion penalty, λ to train STC models for ASR, HWR experimentsare in Appendix D.

Original: 5.3 Runtime Performance of STC
Revised: 5.3 Runtime Performance of STC

Original: In Figure 9, we compare the epoch time of STC, CTC models on LibriSpeech and IAM datasets.
Revised: In Figure 9, we compare the epoch time of STC, CTC models on LibriSpeech and IAM datasets.

Original: Fora fair comparison, we use the same model that is used to report STC results for each dataset.
Revised: Fora fair comparison, we use the same model that is used to report STC results for each dataset.

Original: Theexperiments on LibriSpeech, IAM are run on 32, 8 Nvidia 32GB V100 GPUs and uses C++, Python
Revised: Theexperiments on LibriSpeech, IAM are run on 32, 8 Nvidia 32GB V100 GPUs and uses C++, Python

Original: APIs of GTN respectively.
Revised: APIs of GTN respectively.

Original: We were able to run an epoch on LibriSpeech in about 300 seconds for the STC model and in about250 seconds for the CTC model.
Revised: We were able to run an epoch on LibriSpeech in about 300 seconds for the STC model and in about250 seconds for the CTC model.

Original: Optimizing the data transfer between CPU and GPU by onlymoving the tokens which are present in the current training sample (Section 3.3.3) is a crucial step inachieving good performance.
Revised: Optimizing the data transfer between CPU and GPU by onlymoving the tokens which are present in the current training sample (Section 3.3.3) is a crucial step inachieving good performance.

Original: Without this optimization, it would take about 5400 seconds per epochon LibriSpeech as the alphabet size, |A| of 50000 is very high.
Revised: Without this optimization, it would take about 5400 seconds per epochon LibriSpeech as the alphabet size, |A| of 50000 is very high.

Original: For IAM experiments, we did not perform the data transfer optimization for STC as |A| is only 79.
Revised: For IAM experiments, we did not perform the data transfer optimization for STC as |A| is only 79.

Original: Itcan be see that the epoch time of STC is about 25% more compared to CTC.
Revised: Itcan be see that the epoch time of STC is about 25% more compared to CTC.

Original:  Conclusion
Revised:  Conclusion

Original: In a variety of sequence labeling tasks (ASR, HWR), we show that STC enables training models withpartially labeled data and can give strong performance.
Revised: In a variety of sequence labeling tasks (ASR, HWR), we show that STC enables training modelswith partially labeled data and can give strong performance.

Original: Weakly supervised data can be collected insemi-automatic ways and can alleviate labeling of full training data, thus reducing production costs.
Revised: Weakly supervised data can be collectedin semi-automatic ways and can alleviate labeling of full training data, thus reducing productioncosts.

Original: We also show that using WFSTs with a differentiable framework is flexible and powerful tool forresearchers to solve a completely new set of problems, like the example we have shown in this paper.
Revised: We also show that using WFSTs with a differentiable framework is flexible and powerful toolfor researchers to solve a completely new set of problems, like the example we have shown in thispaper.

Revised: STC requires additional hyperparameter tuning and these vary by p _ drop .

Original: A direct potential extension is to study the application of STC to noisy labels.
Revised: A direct potentialextension is to study the application of STC to noisy labels.

Revised: The label graph can be easily modified toinclude the cases where the missing label’s position or the number of missing tokens is known inadvance.

Revised: Also, STC can be specialized to allow a particular range of missing tokens (e.g. between 2and 4 characters).

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-

Original: tions and scope?
Revised: tions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]4.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-
Revised: d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?
Revised: ing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: informa-
Revised: informa-

Original: tion or offensive content?
Revised: tion or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix: Weighted Finite-State Transducers
Revised: A Appendix: Weighted Finite-State Transducers

Original: A weighted finite-state transducer (WFST) is a generalization of a finite-state automaton (FSA) [29, 30, 39] 404 where each transition has an input label from an alphabet Σ an output label from an alphabet ∆ and scalar weightw .
Revised: A weighted finite-state transducer (WFST) is a generalization of a finite-state automaton (FSA) [29, 30, 39] 408 where each transition has an input label from an alphabet Σ an output label from an alphabet ∆ and scalar weightw .

Original: Figure 10a shows an example WFST with nodes representing states and arcs representing transitions.
Revised: Figure 10a shows an example WFST with nodes representing states and arcs representing transitions.

Original: A path 406 from an initial to a final state encodes a mapping from an input sequence i ∈
Revised: A path 410 from an initial to a final state encodes a mapping from an input sequence i ∈

Original: Σ ∗ to an output sequence o ∈ ∆ ∗
Revised: Σ ∗ to an output sequence o ∈ ∆ ∗and

Original: [Figure]
Revised: [Figure]

Original: In this work we use composition and shortest distance operations on WFSTs.
Revised: In this work we use composition and shortest distance operations on WFSTs.

Original: The composition operation 410 combines WFSTs from different modalities.
Revised: The composition operation 414 combines WFSTs from different modalities.

Original: Given two WFSTs T 1 and T 2 , if T 1 transduces a to b with weightw 1 and T 2 transduces b to c with weight w 2 , then their composition T 1 ◦ T 2 transduces a to c with weight 412 w 1 + w 2 .
Revised: Given two WFSTs T 1 and T 2 , if T 1 transduces a to b with weightw 1 and T 2 transduces b to c with weight w 2 , then their composition T 1 ◦ T 2 transduces a to c with weight 416 w 1 + w 2 .

Original: The forward score operation is the shortest distance from a start state to a final state in the log semiring.
Revised: The forward score operation is the shortest distance from a start state to a final state in the log semiring.

Original:  Given a transducer T 1 , the forward score is the log-sum-exp of the scores of all paths from any start state to any 414 final state.
Revised: 417 Given a transducer T 1 , the forward score is the log-sum-exp of the scores of all paths from any start state to any 418 final state.

Original: The example output graphs from composition and forward score operations are shown in Figure 10cand Figure 10d respectively.
Revised: The example output graphs from composition and forward score operations are shown in Figure 10cand Figure 10d respectively.

Original: A.1 Autograd with WFSTs
Revised: A.1 Autograd with WFSTs

Original: Most operations on WFST are differentiable with respect to the arc weights of the input graphs.
Revised: Most operations on WFST are differentiable with respect to the arc weights of the input graphs.

Original: This allows 418 WFSTs to be used dynamically to train neural networks.
Revised: This allows 422 WFSTs to be used dynamically to train neural networks.

Original: Frameworks like GTN [15] and k2 [19] implementautomatic differentiation with WFSTs.
Revised: Frameworks like GTN [15] and k2 [19] implementautomatic differentiation with WFSTs.

Original: For the purpose of this work, we use the GTN framework.
Revised: For the purpose of this work, we use the GTN framework.

Original: B Appendix: Additional Implementation DetailsB.1 Automatic Speech Recognition
Revised: B Appendix: Additional Implementation DetailsB.1 Automatic Speech Recognition

Original: We keep the original 16kHz sampling rate and compute log-mel filterbanks with 80 coefficients for a 25ms 423 sliding window, strided by 10ms.
Revised: We keep the original 16kHz sampling rate and compute log-mel filterbanks with 80 coefficients for a 25ms 427 sliding window, strided by 10ms.

Original: All features are normalized to have zero mean and unit variance per inputsequence before feeding them into the acoustic model.
Revised: All features are normalized to have zero mean and unit variance per inputsequence before feeding them into the acoustic model.

Original: We use SpecAugment [34] as the data augmentation 425 method for all ASR experiments.
Revised: We use SpecAugment [34] as the data augmentation 429 method for all ASR experiments.

Original: B.1.1 Model architecture
Revised: B.1.1 Model architecture

Original: The acoustic model (AM) architecture is composed of a convolutional frontend (1-D convolution with kernel- 428 width 15 and stride 8 followed by GLU activation) followed by 36 ×
Revised: The acoustic model (AM) architecture is composed of a convolutional frontend (1-D convolution with kernel- 432 width 15 and stride 8 followed by GLU activation) followed by 36 ×

Original: 4- heads Transformer blocks [38]with relative positional embedding.
Revised: 4- heads Transformer blocks [38]with relative positional embedding.

Original: The self-attention dimension is 384 and the feed-forward network (FFN) 430 dimension is 3072 in each Transformer block.
Revised: The self-attention dimension is 384 and the feed-forward network (FFN) 434 dimension is 3072 in each Transformer block.

Original: The output of the final Transformer block if followed by a Linearlayer with output dimension of 580 and a letter-to-word encoder (see Appendix C) to the output classes (word 432 vocabulary + blank ).
Revised: The output of the final Transformer block if followed by a Linearlayer with output dimension of 580 and a letter-to-word encoder (see Appendix C) to the output classes (word 436 vocabulary + blank ).

Original: For all Transformer layers, we use dropout on the self-attention and on the FFN, and layerdrop [12], dropping entire layers at the FFN level.
Revised: For all Transformer layers, we use dropout on the self-attention and on the FFN, and layerdrop [12], dropping entire layers at the FFN level.

Original: We apply LogSoftmax operation on each output frame to 434 produce a probability distribution (in log-space) over output classes.
Revised: We apply LogSoftmax operation on each output frame to 438 produce a probability distribution (in log-space) over output classes.

Original: The model consists of 70 million parametersand we use 32 × Nvidia 32GB V100 GPUs for training.
Revised: The model consists of 70 million parametersand we use 32 × Nvidia 32GB V100 GPUs for training.

Original: B.1.2 Decoding
Revised: B.1.2 Decoding

Original: Beam-Search decoder : In our experiments, we use a beam-search decoder following [8] which leverages a 438 n -gram language model to decrease the word error rate.
Revised: Beam-Search decoder : In our experiments, we use a beam-search decoder following [8] which leverages a 442 n -gram language model to decrease the word error rate.

Original: The beam-search decoder outputs a transcription ˆ y that maximizes the following objective
Revised: The beam-search decoder outputs a transcription ˆ y that maximizes the following objective

Original: [Equation]
Revised: [Equation]

Original: where log P LM ( . ) is the log-likelihood of the language model, α is the weight of the language model, β is a 441 word insertion weight and | ˆ y | is the transcription length in words.
Revised: where log P LM ( . ) is the log-likelihood of the language model, α is the weight of the language model, β is a 445 word insertion weight and | ˆ y | is the transcription length in words.

Original: The hyperparameters α and β are optimizedon the validation set.
Revised: The hyperparameters α and β are optimizedon the validation set.

Original: We use 5-gram LM trained on the official LM training data provided with LibriSpeech for beam-search decoding.
Revised: We use 5-gram LM trained on the official LM training data provided with LibriSpeech for beam-search decoding.

Original: Rescoring :
Revised: Rescoring :

Original: To further decrease the word error rate, we use the top- K hypothesis from beam search decoding 445 and perform rescoring [37] with a Transformer LM to reorder the hypotheses according to the following score:
Revised: To further decrease the word error rate, we use the top- K hypothesis from beam search decoding 449 and perform rescoring [37] with a Transformer LM to reorder the hypotheses according to the following score:

Original: [Equation]
Revised: [Equation]

Original: where log P LM ′ ( . ) is the log-likelihood of the Transformer LM, α , β are the hyperparameters as described 447 above and | ˆ y | is the transcription length in characters.
Revised: where log P LM ′ ( . ) is the log-likelihood of the Transformer LM, α , β are the hyperparameters as described 451 above and | ˆ y | is the transcription length in characters.

Original: We use the pre-trained Transformer LM on LibriSpeechfrom [24] which has a perplexity of 50 on dev-other transcripts.
Revised: We use the pre-trained Transformer LM on LibriSpeechfrom [24] which has a perplexity of 50 on dev-other transcripts.

Original: In this work, top 512 hypothesis from beam 449 search decoding are used as candidates for rescoring.
Revised: In this work, top 512 hypothesis from beam 453 search decoding are used as candidates for rescoring.

Original: B.1.3 Optimizer
Revised: B.1.3 Optimizer

Original: For training the word-based STC models, we use the Adagrad optimizer [10] with a learning rate warmup scheme 452 that increases linearly from 0 to 0 .
Revised: For training the word-based STC models, we use the Adagrad optimizer [10] with a learning rate warmup scheme 456 that increases linearly from 0 to 0 .

Original: 02 in 16000 training steps for all the experiments.
Revised: 02 in 16000 training steps for all the experiments.

Original: We halve the learning rateinitially after 400 epochs and every 200 epochs after that.
Revised: We halve the learning rateinitially after 400 epochs and every 200 epochs after that.

Original: All models are trained with dynamic batching with a 454 batch size of 240 audio sec per GPU.
Revised: All models are trained with dynamic batching with a 458 batch size of 240 audio sec per GPU.

Original: We use SpecAugment [34] with two frequency masks, and ten time maskswith maximum time mask ratio of p = 0.1, the maximum frequency bands masked by one frequency mask is 30, 456 and the maximum frames masked by the time mask is 50; time warping is not used.
Revised: We use SpecAugment [34] with two frequency masks, and ten time maskswith maximum time mask ratio of p = 0.1, the maximum frequency bands masked by one frequency mask is 30, 460 and the maximum frames masked by the time mask is 50; time warping is not used.

Original: SpecAugment is turned ononly after 32000 training steps are finished.
Revised: SpecAugment is turned ononly after 32000 training steps are finished.

Original: Dropout and layer dropout values of 0.05 is used in the AM.
Revised: Dropout and layer dropout values of 0.05 is used in the AM.

Original: B.1.4 Pseudolabeling
Revised: B.1.4 Pseudolabeling

Original: For training the letter-based CTC models, we use the same Transformer-based encoder consisting of 270M 460 parameters from [37] for the AM: the encoder of our acoustic models is composed of a convolutional frontend (1D convolution with kernel-width 7 and stride 3 followed by GLU activation) followed by 36 4-heads Transformer 462 blocks [38].
Revised: For training the letter-based CTC models, we use the same Transformer-based encoder consisting of 270M 464 parameters from [37] for the AM: the encoder of our acoustic models is composed of a convolutional frontend (1D convolution with kernel-width 7 and stride 3 followed by GLU activation) followed by 36 4-heads Transformer 466 blocks [38].

Original: The self-attention dimension is 768 and the feed-forward network (FFN) dimension is 3072 in each
Revised: The self-attention dimension is 768 and the feed-forward network (FFN) dimension is 3072 in each

Original: Transformer block.
Revised: Transformer block.

Original: We uses 64 × Nvidia 32GB V100 GPUs for training.
Revised: We uses 64 × Nvidia 32GB V100 GPUs for training.

Original: The output of the encoder is followed 464 by a linear layer to the output classes.
Revised: The output of the encoder is followed 468 by a linear layer to the output classes.

Original: We use dropout after the convolution layer.
Revised: We use dropout after the convolution layer.

Original: For all Transformer layers,we use dropout of 0 .
Revised: For all Transformer layers,we use dropout of 0 .

Original: 2 on the self-attention and on the FFN, and layer drop [12] value of 0 . 2 .
Revised: 2 on the self-attention and on the FFN, and layer drop [12] value of 0 . 2 .

Original: Token set for all 466 acoustic models consists of 26 English alphabet letters, augmented with the apostrophe and a word boundarytoken.
Revised: Token set for all 470 acoustic models consists of 26 English alphabet letters, augmented with the apostrophe and a word boundarytoken.

Original: To speed up training, we also use mixed-precision training.
Revised: To speed up training, we also use mixed-precision training.

Original: We use the same SpecAugment and learning 468 rate schemes as discussed above for STC models.
Revised: We use the same SpecAugment and learning 472 rate schemes as discussed above for STC models.

Original: We decay learning rate by a factor of 2 each time the WERreaches a plateau on the validation sets.
Revised: We decay learning rate by a factor of 2 each time the WERreaches a plateau on the validation sets.

Original: B.2 Handwriting Recognition
Revised: B.2 Handwriting Recognition

Original: For training the handwriting recognition models, we have used the open-source code 4 based on a prior workfrom [42] and adapted it for our use case.
Revised: For training the handwriting recognition models, we have used the open-source code 4 based on a prior workfrom [42] and adapted it for our use case.

Original: It uses depthwise separable convolutions as the main computational 473 block and the model consists of about 39 million parameters.
Revised: It uses depthwise separable convolutions as the main computational 477 block and the model consists of about 39 million parameters.

Original: We use 8 × Nvidia V100 32GB GPUs for trainingthe models.
Revised: We use 8 × Nvidia V100 32GB GPUs for trainingthe models.

Original: It should also be noted that we were unable to find an open source implementation of Yousef et al.
Revised: It should also be noted that we were unable to find an open source implementation of Yousef et al.

Original: [43] and our 476 baseline is our best attempt to reproduce their results.
Revised: [43] and our 480 baseline is our best attempt to reproduce their results.

Original: We use Adam Optimizer [22] with an initial learning rate of 0 .
Revised: We use Adam Optimizer [22] with an initial learning rate of 0 .

Original: 02 and run training using a batchsize of 8 per 478 GPU for all the experiments.
Revised: 02 and run training using a batchsize of 8 per 482 GPU for all the experiments.

Original: All image are scaled to a maximum width, height of 600, 32 pixels respectivelybefore feeding to the neural network.
Revised: All image are scaled to a maximum width, height of 600, 32 pixels respectivelybefore feeding to the neural network.

Original: We use exponential learning rate decay schedule with a gamma factor of10 − 1 / 90000 ≈ 0 . 99997 .
Revised: We use exponential learning rate decay schedule with a gamma factor of10 − 1 / 90000 ≈ 0 . 99997 .

Original: We use random projection transformations as the augmentation method while training.
Revised: We use random projection transformations as the augmentation method while training.

Original: C Appendix: A simple letter to word encoder
Revised: C Appendix: A simple letter to word encoder

Original: Since, we are working on direct-to-word models for ASR, a major hurdle would be to transcribe words not found 483 in the training vocabulary.
Revised: Since, we are working on direct-to-word models for ASR, a major hurdle would be to transcribe words not found 487 in the training vocabulary.

Original: Recently, [8] propose an end-to-end model which outputs words directly, yet is stillable to dynamically modify the lexicon with words not seen at training time.
Revised: Recently, [8] propose an end-to-end model which outputs words directly, yet is stillable to dynamically modify the lexicon with words not seen at training time.

Original: They use a convnet based letter to word embedding encoder which is jointly trained with the acoustic model and is able to accommodate new 486 words at testing time.
Revised: They use a convnet based letter to word embedding encoder which is jointly trained with the acoustic model and is able to accommodate new 490 words at testing time.

Original: In this work, we propose a simpler system for letter to word encoder which can be embedded into the acoustic 488 model and thus avoiding the needs for a separate network.
Revised: In this work, we propose a simpler system for letter to word encoder which can be embedded into the acoustic 492 model and thus avoiding the needs for a separate network.

Original: Consider A L , A W are the alphabets for letters, wordsrespectively and l max is the maximum length of a word.
Revised: Consider A L , A W are the alphabets for letters, wordsrespectively and l max is the maximum length of a word.

Original: We assume A L also contains two special letters - c blank 490 which produces blank token required for CTC/STC models and c pad which is used to pad all words, blank tokento the same maximum word length l max .
Revised: We assume A L also contains two special letters - c blank 494 which produces blank token required for CTC/STC models and c pad which is used to pad all words, blank tokento the same maximum word length l max .

Original: We let the acoustic model produce a ( |A L | × l max ) dimensional 492 vector for each time frame.
Revised: We let the acoustic model produce a ( |A L | × l max ) dimensional 496 vector for each time frame.

Original: [Figure]
Revised: [Figure]

Original: Since CTC/STC expect scores for A W and blank for each time frame, we carefully construct a matrix E of 494 1 s and 0 s which converts a vector of size ( |A L | × l max ) to ( |A W | + 1) as shown in Figure 11 .
Revised: Since CTC/STC expect scores for A W and blank for each time frame, we carefully construct a matrix E of 498 1 s and 0 s which converts a vector of size ( |A L | × l max ) to ( |A W | + 1) as shown in Figure 11 .

Original: Since weare using c pad token for padding, we can always assume every words and blank 
Revised: Since weare using c pad token for padding, we can always assume every words and blank token is a sequence of l max

Original: [Table]
Revised: [Table]

Revised: and it produces the score for a word or blank .

Original:  [Table]
Revised: We use A L { a − z, ' , c blank , c pad } and l max = 20 for all our experiments on LibriSpeech using words as the output tokens. [Table] [Table]

Revised: From Table

Original: [Table]
Revised: [Table]

Original: From Table 3, we can see that we are able to match the performance of the word based models from [8] using a 500 much smaller acoustic model.
Revised: we can see that we are able to match the performance of the word based models from [8] using a [Table] much smaller acoustic model.

Original: Also, our system is simpler as we do not have a separate network for letter-to-word
Revised: Also, our system is simpler as we do not have a separate network for letter-to-word

Original: [Table]
Revised: [Table]

Original: encoder.
Revised: encoder.

Revised: [Table]

Original: D Appendix: Hyperparameters for token insertion penalty, λ
Revised: D Appendix: Hyperparameters for token insertion penalty, λ

Original: In Table 4, we report the hyperparameters used for training STC models on LibriSpeech (Table 1) and IAM 504 (Table 2).
Revised: In Table [Table] we report the hyperparameters used for training STC models on LibriSpeech (Table 1) and IAM 508 (Table 2).

Revised: The best performing values are found by running a grid search over the hyperparameters and measuringperformance on the development set.

Original: It can be seen that with higher pDrop values, a higher token insertion  penalty values are usuallypreferred.
Revised: It can be seen that with higher p _ drop values, a higher token insertion 510 penalty values D.1 Sensitivity of the model performance when setting token insertion penalty, λ

Original: [Table]
Revised: [Table]

Revised: [Table]