Original: Star Temporal Classification: Sequence Modeling with Partially Labeled Data
Revised: Star Temporal Classification: Sequence Modeling with Partially Labeled Data

Original: Abstract
Revised: Abstract

Original: We develop an algorithm which can learn from partially labeled and unsegmented sequential data.
Revised: We develop an algorithm which can learn from partially labeled and unsegmented sequential data.

Original: Most sequential loss functions, such as Connectionist Temporal
Revised: Most sequential loss functions, such as Connectionist Temporal

Original: Classification (CTC), break down when many labels are missing.
Revised: Classification (CTC), break down when many labels are missing.

Original: We address this problem with Star Temporal Classification (STC) which uses a special star token to allow alignments which include all possible tokens whenever a token could be missing.
Revised: We address this problem with Star Temporal Classification (STC) which uses a special star token to allow alignments which include all possible tokens whenever a token could be missing.

Original: We express STC as the composition of weighted finite-state transducers (WFSTs) and use GTN (a framework for automatic differentiation with WFSTs) to compute gradients.
Revised: We express STC as the composition of weighted finite-state transducers (WFSTs) and use GTN (a framework for automatic differentiation with WFSTs) to compute gradients.

Original: We perform extensive experiments on automatic speech recognition.
Revised: We perform extensive experiments on automatic speech recognition.

Original: These experiments show that STC can close the performance gap with supervised baseline to about 1% WER when up to 70% of the labels are missing.
Revised: These experiments show that STC can close the performance gap with supervised baseline to about 1% WER when up to 70% of the labels are missing.

Original: We also perform experiments in handwriting recognition to show that our method easily applies to other sequence classification tasks
Revised: We also perform experiments in handwriting recognition to show that our method easily applies to other sequence classification tasks

Original:  Introduction
Revised:  Introduction

Original: Applications of machine learning in settings with little or no labeled data are growing rapidly.
Revised: Applications of machine learning in settings with little or no labeled data are growing rapidly.

Original: Much prior work in temporal classification focuses on semi-supervised or self-supervised learning [6, 37].
Revised: Much prior work in temporal classification focuses on semi-supervised or self-supervised learning [6, 37].

Original: Instead, we focus on weakly supervised learning, in which the learner receives incomplete labels for the examples in the data set.
Revised: Instead, we focus on weakly supervised learning, in which the learner receives incomplete labels for the examples in the data set.

Original: Examples include multi-instance learning [1, 44], and partial-label learning [18, 9, 25].
Revised: Examples include multi-instance learning [1, 44], and partial-label learning [18, 9, 25].

Original: In our case, we assume that each example in the training set is partially labeled, as in Figure 1: neither the number of missing words nor their positions in the label sequence are known in advance.
Revised: In our case, we assume that each example in the training set is partially labeled, as in Figure 1: neither the number of missing words nor their positions in the label sequence are known in advance.

Original: [Figure]
Revised: [Figure]

Original: Partial labeling problem could arise in many practical scenarios.
Revised: Partial labeling problem could arise in many practical scenarios.

Original: In semi-supervised learning or transfer learning of sequence labeling tasks, we can only consider high confidence tokens in the label sequence and discard rest of the tokens.
Revised: In semi-supervised learning or transfer learning of sequence labeling tasks, we can only consider high confidence tokens in the label sequence and discard rest of the tokens.

Original: These labels with incomplete tokens can be treated as partial labels.
Revised: These labels with incomplete tokens can be treated as partial labels.

Original: When transcribing a dataset, we can randomly label parts of the dataset and use them as partial labels, at a much lower cost than transcribing the full dataset.
Revised: When transcribing a dataset, we can randomly label parts of the dataset and use them as partial labels, at a much lower cost than transcribing the full dataset.

Original: We can also consider semi-automatic ways to label a dataset.
Revised: We can also consider semi-automatic ways to label a dataset.

Original: Restoring damaged texts [2, 3] where some of the writing is illegible could be considered as a partial labeling problem.
Revised: Restoring damaged texts [2, 3] where some of the writing is illegible could be considered as a partial labeling problem.

Original: We also often face partial alignment problem [5] in practice where beginning and ending tokens are missing.
Revised: We also often face partial alignment problem [5] in practice where beginning and ending tokens are missing.

Original: This is a sub-problem of the more general version of partial labeling problem that we solve in this work.
Revised: This is a sub-problem of the more general version of partial labeling problem that we solve in this work.

Original: To solve this weak supervision problem we develop the Star Temporal Classification (STC) loss function.
Revised: To solve this weak supervision problem we develop the Star Temporal Classification (STC) loss function.

Original: The STC algorithm explicitly accounts for the possibility of missing labels in the output sequence at any position, through a special star token.
Revised: The STC algorithm explicitly accounts for the possibility of missing labels in the output sequence at any position, through a special star token.

Original: For the example of Figure 1, the STC label sequence is * from * airplane * the * a * spider * (where * can be anything but the following label token).
Revised: For the example of Figure 1, the STC label sequence is * from * airplane * the * a * spider * (where * can be anything but the following label token).

Original: We show how to simply implement STC in the weighted finite-state transducer (WFST) framework.
Revised: We show how to simply implement STC in the weighted finite-state transducer (WFST) framework.

Original: Moreover, through the use of automatic differentiation with WFSTs, gradients of the loss function with respect to the inputs are computed automatically, allowing STC to be used with standard neural network training pipelines.
Revised: Moreover, through the use of automatic differentiation with WFSTs, gradients of the loss function with respect to the inputs are computed automatically, allowing STC to be used with standard neural network training pipelines.

Original: For practical applications, a naive implementation of STC using WFSTs is intractably slow.
Revised: For practical applications, a naive implementation of STC using WFSTs is intractably slow.

Original: We propose several optimizations to improve the efficiency of the STC loss.
Revised: We propose several optimizations to improve the efficiency of the STC loss.

Original: The star token we use results in much smaller graphs since it combines many arcs into one.
Revised: The star token we use results in much smaller graphs since it combines many arcs into one.

Original: The star token also enables us to substantially reduce the amount of memory transfers between the GPU, on which the network is computed, and the CPU, on which the STC algorithm is evaluated.
Revised: The star token also enables us to substantially reduce the amount of memory transfers between the GPU, on which the network is computed, and the CPU, on which the STC algorithm is evaluated.

Original: These optimizations result in only a minor increase in model training time when using the STC loss over a CTC baseline.
Revised: These optimizations result in only a minor increase in model training time when using the STC loss over a CTC baseline.

Original: The STC loss can be applied to most sequence transduction tasks including speech recognition, handwriting recognition, machine translation, and action recognition from videos.
Revised: The STC loss can be applied to most sequence transduction tasks including speech recognition, handwriting recognition, machine translation, and action recognition from videos.

Original: Here we demonstrate the effectiveness of STC on both automatic speech recognition and offline handwriting recognition.
Revised: Here we demonstrate the effectiveness of STC on both automatic speech recognition and offline handwriting recognition.

Original: In speech recognition, with up to 70% of the labels missing, STC is able to achieve similar (about 1%
Revised: In speech recognition, with up to 70% of the labels missing, STC is able to achieve similar (about 1%

Original: WER difference) performance as supervised baseline .
Revised: WER difference) performance as supervised baseline .

Original: Similarly in handwriting recognition STC can achieve low character error rates with up to 50% of the labels missing.
Revised: Similarly in handwriting recognition STC can achieve low character error rates with up to 50% of the labels missing.

Original: In both cases STC yields dramatic gains over a baseline system which does not explicitly handle missing labels.
Revised: In both cases STC yields dramatic gains over a baseline system which does not explicitly handle missing labels.

Original: To summarize, the main contributions of this work are:
Revised: To summarize, the main contributions of this work are:

Original: • We introduce STC, a novel temporal classification algorithm which explicitly handles an arbitrary number of missing labels with an indeterminate location.
Revised: • We introduce STC, a novel temporal classification algorithm which explicitly handles an arbitrary number of missing labels with an indeterminate location.

Original: • We simplify the implementation of STC by using WFSTs and automatic differentiation.
Revised: • We simplify the implementation of STC by using WFSTs and automatic differentiation.

Original: • We describe key optimizations for STC enabling it to be used with minimal increase in training time.
Revised: • We describe key optimizations for STC enabling it to be used with minimal increase in training time.

Original: • We show in practical speech and handwriting recognition tasks that STC yields low word and character error rates with up to 70% of the labels missing.
Revised: • We show in practical speech and handwriting recognition tasks that STC yields low word and character error rates with up to 70% of the labels missing.

Original:  Related Work
Revised:  Related Work

Original: Many generalizations and variations of CTC have been proposed.
Revised: Many generalizations and variations of CTC have been proposed.

Original: Graph-based temporal classification (GTC) [32] generalizes CTC to allow for an n -best list of possible labels.
Revised: Graph-based temporal classification (GTC) [32] generalizes CTC to allow for an n -best list of possible labels.

Original: [40] also extends CTC for using with for multiple labels.
Revised: [40] also extends CTC for using with for multiple labels.

Original: [23] propose selfless-CTC which disallows self-loops for non-blank output tokens.
Revised: [23] propose selfless-CTC which disallows self-loops for non-blank output tokens.

Original: Unlike these alternatives, STC allows for an arbitrary number of missing tokens anywhere in the label for a given example.
Revised: Unlike these alternatives, STC allows for an arbitrary number of missing tokens anywhere in the label for a given example.

Original: A recent work, concurrent with this work, proposes a wild card version of CTC [5].
Revised: A recent work, concurrent with this work, proposes a wild card version of CTC [5].

Original: However, in that work the wild cards, or missing tokens, can only be at the beginning or end of the label.
Revised: However, in that work the wild cards, or missing tokens, can only be at the beginning or end of the label.

Original: Extended Connectionist Temporal Classification (ECTC) [17] was developed for weakly supervised video action labeling.
Revised: Extended Connectionist Temporal Classification (ECTC) [17] was developed for weakly supervised video action labeling.

Original: However, the weak supervision in this case refers to the lack of a segmentation, which is the common paradigm for CTC.
Revised: However, the weak supervision in this case refers to the lack of a segmentation, which is the common paradigm for CTC.

Original: The extensions to CTC are intended to handle the much higher ratio of input video frames to output action labels.
Revised: The extensions to CTC are intended to handle the much higher ratio of input video frames to output action labels.

Original: [11] develop a sequence level loss function which can learn from noisy labels.
Revised: [11] develop a sequence level loss function which can learn from noisy labels.

Original: They extend the auto segmentation criterion [7] to include a noise model which explicitly handles up to one missing label between tokens in the output.
Revised: They extend the auto segmentation criterion [7] to include a noise model which explicitly handles up to one missing label between tokens in the output.

Original: Unlike that work, STC extends the more commonly used CTC loss and allows for any number of missing tokens.
Revised: Unlike that work, STC extends the more commonly used CTC loss and allows for any number of missing tokens.

Original: The idea of expressing CTC as the composition of WFSTs is well known [27, 23, 41].
Revised: The idea of expressing CTC as the composition of WFSTs is well known [27, 23, 41].

Original: However, recently developed WFST-based frameworks which support automatic differentiation [15, 19] make the development of variations of the CTC loss much simpler.
Revised: However, recently developed WFST-based frameworks which support automatic differentiation [15, 19] make the development of variations of the CTC loss much simpler.

Original: WFSTs have been modified by to take into account of noisy transcripts [16, 31, 4] by composing the WFST graphs of word sequences.
Revised: WFSTs have been modified by to take into account of noisy transcripts [16, 31, 4] by composing the WFST graphs of word sequences.

Original: However, in this work the WFST composition is between significantly larger WFST graphs (label sequence and neural network model output sequence).
Revised: However, in this work the WFST composition is between significantly larger WFST graphs (label sequence and neural network model output sequence).

Original: Also, the composition is performed during the training of the neural network which makes it a more challenging engineering problem.
Revised: Also, the composition is performed during the training of the neural network which makes it a more challenging engineering problem.

Original:  Method 3.1 Problem Description
Revised:  Method 3.1 Problem Description

Original: We consider the problem of temporal classification [20], which predicts an output sequence y =
Revised: We consider the problem of temporal classification [20], which predicts an output sequence y =

Original: [ y 1 , ...., y U ] ∈
Revised: [ y 1 , ...., y U ] ∈

Original: A 1 × U , where A is a fixed alphabet of possible output tokens, from an unsegmented input sequence x = [ x 1 , ..., x T ] ∈ R d × T , where each x i is a d -dimensional feature vector.
Revised: A 1 × U , where A is a fixed alphabet of possible output tokens, from an unsegmented input sequence x = [ x 1 , ..., x T ] ∈ R d × T , where each x i is a d -dimensional feature vector.

Original: We also assume that the length of the output U is less than or equal to that of the input, T .
Revised: We also assume that the length of the output U is less than or equal to that of the input, T .

Original: A partially labeled output sequence ˜ y of length ˜ U , such that ( ˜ U ≤ U ), is formed by removing zero or more elements from the true output sequence y .
Revised: A partially labeled output sequence ˜ y of length ˜ U , such that ( ˜ U ≤ U ), is formed by removing zero or more elements from the true output sequence y .

Original: Given a training set D ≜ { x i , ˜ y
Revised: Given a training set D ≜ { x i , ˜ y

Original: i : i = 1 , . . .
Revised: i : i = 1 , . . .

Original: , N } consisting of input sequences x i =
Revised: , N } consisting of input sequences x i =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x T i ] and partially labeled output sequences ˜ y
Revised: , x T i ] and partially labeled output sequences ˜ y

Original: i =
Revised: i =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y ˜ U
Revised: , y ˜ U

Original: i ] our goal is to learn temporal models which can predict true output sequences y i .
Revised: i ] our goal is to learn temporal models which can predict true output sequences y i .

Original: We consider this a weakly supervised temporal classification problem as the learning algorithm only has access to a subset of the true output labels during training.
Revised: We consider this a weakly supervised temporal classification problem as the learning algorithm only has access to a subset of the true output labels during training.

Original: 3.2 Connectionist Temporal Classification (CTC)
Revised: 3.2 Connectionist Temporal Classification (CTC)

Original: Connectionist Temporal Classification (CTC) [13] is a widely used loss function for training sequence models where the alignment of input sequence with the target labels is not known in advance.
Revised: Connectionist Temporal Classification (CTC) [13] is a widely used loss function for training sequence models where the alignment of input sequence with the target labels is not known in advance.

Original: Consider an input sequence x
Revised: Consider an input sequence x

Original: = [ x 1 , . .
Revised: = [ x 1 , . .

Original: , x T ] and target sequence y =
Revised: , x T ] and target sequence y =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y U ] ∈
Revised: , y U ] ∈

Original: A 1 × U , where
Revised: A 1 × U , where

Original: A is a finite alphabet of possible output tokens and U ≤ T .
Revised: A is a finite alphabet of possible output tokens and U ≤ T .

Original: The CTC loss uses a special blank token, ⟨ b ⟩ , to represents frames which do not correspond to an output token.
Revised: The CTC loss uses a special blank token, ⟨ b ⟩ , to represents frames which do not correspond to an output token.

Original: An alignment between the input x and output y in CTC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ] where π t ∈
Revised: An alignment between the input x and output y in CTC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ] where π t ∈

Original: A ∪ { blank } .
Revised: A ∪ { blank } .

Original: The CTC collapse function B maps an alignment to an output, B ( π ) =
Revised: The CTC collapse function B maps an alignment to an output, B ( π ) =

Original: y .
Revised: y .

Original: The function B removes all but one of any consecutively repeated tokens and then removes blank tokens e.g., (a, b, blank , blank , b, b, blank , a) (cid:55)→ (a, b, b, a).
Revised: The function B removes all but one of any consecutively repeated tokens and then removes blank tokens e.g., (a, b, blank , blank , b, b, blank , a) (cid:55)→ (a, b, b, a).

Original: Given the input x , each output is independent of all other outputs, hence the probability of an alignment P ( π | x ) is:
Revised: Given the input x , each output is independent of all other outputs, hence the probability of an alignment P ( π | x ) is:

Original: [Equation]
Revised: [Equation]

Original: There can be many possible alignments π for a given x and y pair.
Revised: There can be many possible alignments π for a given x and y pair.

Original: The CTC loss computes the negative log probability of y given x by marginalizing over all possible alignments:
Revised: The CTC loss computes the negative log probability of y given x by marginalizing over all possible alignments:

Original: [Equation]
Revised: [Equation]

Original: The sum over all alignments can be computed efficiently with dynamic programming as described by [13].
Revised: The sum over all alignments can be computed efficiently with dynamic programming as described by [13].

Original: The CTC loss can be computed purely with WFST 2 operations as described in Figure 2.
Revised: The CTC loss can be computed purely with WFST 2 operations as described in Figure 2.

Original: The composition of the emission graph (Figure 2a) which has arc weights weights corresponding to log P ( π t | x ) and the label graph (Figure 2b) is shown in Figure 2c.
Revised: The composition of the emission graph (Figure 2a) which has arc weights weights corresponding to log P ( π t | x ) and the label graph (Figure 2b) is shown in Figure 2c.

Original: Each path from a start to a final state in Figure 2c is a valid CTC alignment.
Revised: Each path from a start to a final state in Figure 2c is a valid CTC alignment.

Original: The negation of the forward score of the composed graph gives the CTC loss.
Revised: The negation of the forward score of the composed graph gives the CTC loss.

Original: These operations are differentiable and gradients can easily be propagated backward from the loss to a neural network to train the model.
Revised: These operations are differentiable and gradients can easily be propagated backward from the loss to a neural network to train the model.

Original: Constructing the CTC loss from simpler WFST graphs also simplifies the implementation of variations of CTC.
Revised: Constructing the CTC loss from simpler WFST graphs also simplifies the implementation of variations of CTC.

Original: For example, the variations of [32] only require changes to the label graph Y ctc .
Revised: For example, the variations of [32] only require changes to the label graph Y ctc .

Original: However, when efficiency is a primary concern, custom GPU kernels can be faster than a generic implementation with WFSTs.
Revised: However, when efficiency is a primary concern, custom GPU kernels can be faster than a generic implementation with WFSTs.

Original: [Figure]
Revised: [Figure]

Original: 3.3 Star Temporal Classification (STC)
Revised: 3.3 Star Temporal Classification (STC)

Original: The label graph of CTC, L ctc (Figure 2b), constructed from partial labels does not allow for the true target as a possibility.
Revised: The label graph of CTC, L ctc (Figure 2b), constructed from partial labels does not allow for the true target as a possibility.

Original: The STC algorithm addresses this problem by allowing for zero or more tokens from the alphabet between any two tokens in the partial label.
Revised: The STC algorithm addresses this problem by allowing for zero or more tokens from the alphabet between any two tokens in the partial label.

Original: Like CTC, an alignment between the input x and output y in STC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ] where π t ∈
Revised: Like CTC, an alignment between the input x and output y in STC is represented by a sequence of length T ; π = [ π 1 , π 2 , ..., π T ] where π t ∈

Original: A ∪ { blank } and y is a partial label of B ′ ( π ) .
Revised: A ∪ { blank } and y is a partial label of B ′ ( π ) .

Original: Unlike CTC, the STC collapse function B ′ ( π ) only removes blank tokens.
Revised: Unlike CTC, the STC collapse function B ′ ( π ) only removes blank tokens.

Original: In other words, we do not allow self-loops on non blank tokens in the
Revised: In other words, we do not allow self-loops on non blank tokens in the

Original: STC label graph.
Revised: STC label graph.

Original: This enables us to use a token insertion penalty as discussed in Section 3.3.1.
Revised: This enables us to use a token insertion penalty as discussed in Section 3.3.1.

Original: a/w b/w c/w d/w → ⟨ * ⟩ /w (
Revised: a/w b/w c/w d/w → ⟨ * ⟩ /w (

Original: [Figure]
Revised: [Figure]

Original: w5 =
Revised: w5 =

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ,w4) a/w b/w c/w d/w → ⟨ * ⟩ \a/w (b
Revised: ,w4) a/w b/w c/w d/w → ⟨ * ⟩ \a/w (b

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: To make the computation of the STC loss efficient, STC uses a special star token, ⟨∗⟩ , which represents every token in the alphabet.
Revised: To make the computation of the STC loss efficient, STC uses a special star token, ⟨∗⟩ , which represents every token in the alphabet.

Original: It can be used to collapse the arcs between any two nodes in the WFST graph as shown in Figure 3a.
Revised: It can be used to collapse the arcs between any two nodes in the WFST graph as shown in Figure 3a.

Original: We also define ⟨∗⟩\ t =
Revised: We also define ⟨∗⟩\ t =

Original: { y : y ∈ A ; y ̸ = t } which is the relative complement of t in A .
Revised: { y : y ∈ A ; y ̸ = t } which is the relative complement of t in A .

Original: We use the token, ⟨∗⟩\ t , in the label graph of STC, L stc , to avoid counting the same alignment multiple times as dicsussed in Section 3.3.2 The STC label graph, L stc , for the partial label ( a, b, c ) is shown in Figure 4.
Revised: We use the token, ⟨∗⟩\ t , in the label graph of STC, L stc , to avoid counting the same alignment multiple times as dicsussed in Section 3.3.2 The STC label graph, L stc , for the partial label ( a, b, c ) is shown in Figure 4.

Original: The graph allows any alignment with zero or more tokens in between any two tokens of the given partial label sequence.
Revised: The graph allows any alignment with zero or more tokens in between any two tokens of the given partial label sequence.

Original: We also manually add the ⟨∗⟩ and ⟨∗⟩\ t tokens to the emission graph E X with weights corresponding to their log-probabilities for each timestep t given by:
Revised: We also manually add the ⟨∗⟩ and ⟨∗⟩\ t tokens to the emission graph E X with weights corresponding to their log-probabilities for each timestep t given by:

Original: [Equation]
Revised: [Equation]

Original: ⟨ b ⟩ /  a/  ⟨ * ⟩ \a/ λ 2 ⟨ b ⟩ / 3 b/ ⟨ b ⟩ / a/0 ⟨ * ⟩ \a/ λ ⟨ b ⟩ / b/  ⟨ * ⟩ \b/ λ 4 ⟨ b ⟩ / 5 c/0 ⟨ b ⟩ / b/ ⟨ * ⟩ \b/ λ ⟨ b ⟩ / c/  ⟨ * ⟩ \c/ λ
Revised: ⟨ b ⟩ /  a/  ⟨ * ⟩ \a/ λ 2 ⟨ b ⟩ / 3 b/ ⟨ b ⟩ / a/0 ⟨ * ⟩ \a/ λ ⟨ b ⟩ / b/  ⟨ * ⟩ \b/ λ 4 ⟨ b ⟩ / 5 c/0 ⟨ b ⟩ / b/ ⟨ * ⟩ \b/ λ ⟨ b ⟩ / c/  ⟨ * ⟩ \c/ λ

Original: 6 ⟨ b ⟩ / 10 ⟨ * ⟩ /0 ⟨ b ⟩ / c/ ⟨ * ⟩ \c/ λ ⟨ b ⟩ / ⟨ * ⟩ / ⟨ b ⟩ / ⟨ * ⟩ /
Revised: 6 ⟨ b ⟩ / 10 ⟨ * ⟩ /0 ⟨ b ⟩ / c/ ⟨ * ⟩ \c/ λ ⟨ b ⟩ / ⟨ * ⟩ / ⟨ b ⟩ / ⟨ * ⟩ /

Original: 3.3.1 Token Insertion Penalty, λ
Revised: 3.3.1 Token Insertion Penalty, λ

Original: We noticed that model training with STC does not work if we train directly with the STC label graph, even when 10% of the labels are missing.
Revised: We noticed that model training with STC does not work if we train directly with the STC label graph, even when 10% of the labels are missing.

Original: This is because the new paths introduced in STC from
Revised: This is because the new paths introduced in STC from

Original: CTC (marked in red in Figure 4) would allow the model to produce a lot of possible output sequences for any given example and can confuse the model, especially during the early stages of training.
Revised: CTC (marked in red in Figure 4) would allow the model to produce a lot of possible output sequences for any given example and can confuse the model, especially during the early stages of training.

Original: To circumvent this problem, we use a parameter called token insertion penalty, λ to add a penalty when using lot of new tokens.
Revised: To circumvent this problem, we use a parameter called token insertion penalty, λ to add a penalty when using lot of new tokens.

Original: We define STC loss in the following way
Revised: We define STC loss in the following way

Original: [Equation]
Revised: [Equation]

Original: where P path i is the probability of an alignment path i , p ∈ (0 , 1] and k i is the number of additional star tokens for path i .
Revised: where P path i is the probability of an alignment path i , p ∈ (0 , 1] and k i is the number of additional star tokens for path i .

Original: This down weight paths which include many additional tokens.
Revised: This down weight paths which include many additional tokens.

Original: The token insertion penalty is in log-space and is defined as λ = log( p ) .
Revised: The token insertion penalty is in log-space and is defined as λ = log( p ) .

Original: We use a exponential decay scheme to gradually reduce the penalty as the model starts training.
Revised: We use a exponential decay scheme to gradually reduce the penalty as the model starts training.

Original: [Equation]
Revised: [Equation]

Original: where p 0 , p max , τ are hyperparameters and λ t denotes the token insertion penalty used in STC for training step t .
Revised: where p 0 , p max , τ are hyperparameters and λ t denotes the token insertion penalty used in STC for training step t .

Original: For choosing value of τ , it is useful think in terms of half-life t 1 / 2 =
Revised: For choosing value of τ , it is useful think in terms of half-life t 1 / 2 =

Original: τln (2) , which is the number of time steps taken for p t to reach ( p 0 + p max ) /
Revised: τln (2) , which is the number of time steps taken for p t to reach ( p 0 + p max ) /

Original: 3.3.2 On the importance of ⟨∗⟩\ t token
Revised: 3.3.2 On the importance of ⟨∗⟩\ t token

Original: The " ⟨∗⟩\ t " token which matches any token except the token, t is useful to not count the same alignment path multiple times while computing the STC loss.
Revised: The " ⟨∗⟩\ t " token which matches any token except the token, t is useful to not count the same alignment path multiple times while computing the STC loss.

Original: We illustrate it through the two possible
Revised: We illustrate it through the two possible

Original: STC label graphs from Figure 5. Consider that the partial label has just one word "the" and number of time steps, T = 3.
Revised: STC label graphs from Figure 5. Consider that the partial label has just one word "the" and number of time steps, T = 3.

Original: In Figure 5a, the alignment path {"the", "the", "cat"} is counted multiple times when we use start token ⟨∗⟩ directly 0 → 1 → 2 → 4 and 0 → 2 → 4 → 4.
Revised: In Figure 5a, the alignment path {"the", "the", "cat"} is counted multiple times when we use start token ⟨∗⟩ directly 0 → 1 → 2 → 4 and 0 → 2 → 4 → 4.

Original: However, if we use ⟨∗⟩\ the token as shown in Figure 5b , we do not have this issue of accepting the path "the the cat" multiple times.
Revised: However, if we use ⟨∗⟩\ the token as shown in Figure 5b , we do not have this issue of accepting the path "the the cat" multiple times.

Original: It should also be noted that the " ⟨∗⟩\ t " token is only present after the token t but not before the token t in the label graph of STC from Figure 4.
Revised: It should also be noted that the " ⟨∗⟩\ t " token is only present after the token t but not before the token t in the label graph of STC from Figure 4.

Original: So, the label graph would still accept all possible labels sequences for the given partial label.
Revised: So, the label graph would still accept all possible labels sequences for the given partial label.

Original: 3.3.3 Implementation Details
Revised: 3.3.3 Implementation Details

Original: With the recent advent of frameworks such as GTN[15], K2[19] which support autograd on WFSTs, we focus on the engineering details used by us to use these frameworks and train large-scale recognition systems below.
Revised: With the recent advent of frameworks such as GTN[15], K2[19] which support autograd on WFSTs, we focus on the engineering details used by us to use these frameworks and train large-scale recognition systems below.

Original: We use the CPU-based WFST algorithms from GTN [15] to implement the STC criterion.
Revised: We use the CPU-based WFST algorithms from GTN [15] to implement the STC criterion.

Original: Multiple threads are used to compute STC loss in parallel for all of the examples in a batch.
Revised: Multiple threads are used to compute STC loss in parallel for all of the examples in a batch.

Original: Since the neural network model is run on GPU, the emissions needed by STC must be copied to the CPU, and the
Revised: Since the neural network model is run on GPU, the emissions needed by STC must be copied to the CPU, and the

Original: STC gradients must be copied back to GPU.
Revised: STC gradients must be copied back to GPU.

Original: To reduce the amount of data transfer between the CPU and the GPU, we transfer values corresponding only to the tokens present in the partial labels and the star tokens.
Revised: To reduce the amount of data transfer between the CPU and the GPU, we transfer values corresponding only to the tokens present in the partial labels and the star tokens.

Original: This works since the gradients corresponding to all other tokens are zero.
Revised: This works since the gradients corresponding to all other tokens are zero.

Original: Figure 6 shows an overview of the STC training pipeline.
Revised: Figure 6 shows an overview of the STC training pipeline.

Original: The input sequence is typically passed through a neural network on GPU to produce a frame wise distribution (in log-space) over A∪{ blank } .
Revised: The input sequence is typically passed through a neural network on GPU to produce a frame wise distribution (in log-space) over A∪{ blank } .

Original: The output number of frames depend on the size of the input sequence, as well as the amount of padding and the stride of the neural network model architecture.
Revised: The output number of frames depend on the size of the input sequence, as well as the amount of padding and the stride of the neural network model architecture.

Original: We then compute the log-probability of star token, ⟨∗⟩ and ⟨∗⟩\ t tokens ∀ t ∈
Revised: We then compute the log-probability of star token, ⟨∗⟩ and ⟨∗⟩\ t tokens ∀ t ∈

Original: y using Equation 3.
Revised: y using Equation 3.

Original: We construct the emission graph E X with weights as log-probabilities and the label graph L stc on CPU.
Revised: We construct the emission graph E X with weights as log-probabilities and the label graph L stc on CPU.

Original: Using the WFST operations shown in Figure 2, we compute STC loss and gradients of loss with respect to arc weights (log probability output) of E X .
Revised: Using the WFST operations shown in Figure 2, we compute STC loss and gradients of loss with respect to arc weights (log probability output) of E X .

Original: These gradients are copied back to GPU and the gradients of parameters in neural network are computed using backpropagation.
Revised: These gradients are copied back to GPU and the gradients of parameters in neural network are computed using backpropagation.

Original: Then the model can be trained using standard gradient descent methods.
Revised: Then the model can be trained using standard gradient descent methods.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Experimental Setup
Revised:  Experimental Setup

Original: [Figure]
Revised: [Figure]

Original: .1 Automatic Speech Recognition (ASR)
Revised: .1 Automatic Speech Recognition (ASR)

Original: We use LibriSpeech [33] dataset, containing 960 hours of training audio with paired transcriptions for our speech recognition experiments.
Revised: We use LibriSpeech [33] dataset, containing 960 hours of training audio with paired transcriptions for our speech recognition experiments.

Original: The standard LibriSpeech validation sets ( dev-clean and dev-other ) are used to tune all hyperparameters, as well as to select the best models.
Revised: The standard LibriSpeech validation sets ( dev-clean and dev-other ) are used to tune all hyperparameters, as well as to select the best models.

Original: We report final word error rate (WER) performance on the test sets ( test-clean and test-other ).
Revised: We report final word error rate (WER) performance on the test sets ( test-clean and test-other ).

Original: We use the top 50K words (sorted by occurance frequency) from the official language model (LM) training data provided with LibriSpeech as the alphabet for training the models.
Revised: We use the top 50K words (sorted by occurance frequency) from the official language model (LM) training data provided with LibriSpeech as the alphabet for training the models.

Original: These 50K words cover 99.04% of all the word occurances in the LM training data.
Revised: These 50K words cover 99.04% of all the word occurances in the LM training data.

Original: The model architectures and the STC loss are implemented with the ASR application [35] of the flashlight 3 machine-learning framework and the C++ API of GTN.
Revised: The model architectures and the STC loss are implemented with the ASR application [35] of the flashlight 3 machine-learning framework and the C++ API of GTN.

Original: 4.1.1 Label Generation for Weakly Supervised Setup
Revised: 4.1.1 Label Generation for Weakly Supervised Setup

Original: As LibriSpeech dataset consists of fully labeled data, we need to drop the labels manually to simulate the weakly supervised setup.
Revised: As LibriSpeech dataset consists of fully labeled data, we need to drop the labels manually to simulate the weakly supervised setup.

Original: We quantify this using a parameter p _ drop ∈ [0 , 1] , which denotes the probability of dropping a word from the transcript label.
Revised: We quantify this using a parameter p _ drop ∈ [0 , 1] , which denotes the probability of dropping a word from the transcript label.

Original: It can be seen that higher value of p _ drop corresponds to higher number of missing words in the transcript and vice-versa.
Revised: It can be seen that higher value of p _ drop corresponds to higher number of missing words in the transcript and vice-versa.

Original: Figures 7a-b show the histogram of percentage of the labeled words present in each training sample in LibriSpeech for different values of p _ drop which closely follows a normal distribution.
Revised: Figures 7a-b show the histogram of percentage of the labeled words present in each training sample in LibriSpeech for different values of p _ drop which closely follows a normal distribution.

Original: Also, from the definition of p _ drop , it naturally follows that p _ drop = 0 corresponds to supervised training setup while p _ drop = 1 corresponds to unsupervised training.
Revised: Also, from the definition of p _ drop , it naturally follows that p _ drop = 0 corresponds to supervised training setup while p _ drop = 1 corresponds to unsupervised training.

Original: To make sure our method is robust to different types of partial labels, we also test our method on two other ways of generating the data :
Revised: To make sure our method is robust to different types of partial labels, we also test our method on two other ways of generating the data :

Original: Randomly split the training set into p parts and use a different p _ drop for each split (Figure 7c) 2.
Revised: Randomly split the training set into p parts and use a different p _ drop for each split (Figure 7c) 2.

Original: Randomly split the set of all words in training vocab into p parts, use a different p _ drop for each split (Figure 7d) 3 https://github.com/flashlight/flashlight
Revised: Randomly split the set of all words in training vocab into p parts, use a different p _ drop for each split (Figure 7d) 3 https://github.com/flashlight/flashlight

Original: [Figure]
Revised: [Figure]

Original: For all the new partially labeled datasets created, we prune samples which have empty transcriptions before using them for training.
Revised: For all the new partially labeled datasets created, we prune samples which have empty transcriptions before using them for training.

Original: 4.1.2 Pseudo-labeling
Revised: 4.1.2 Pseudo-labeling

Original: To further achieve better word error rate (WER), we generate pseudo labels (PLs) on the training set using the word-based model trained with STC and use these PLs to train a letter-based model, via a regular CTC approach.
Revised: To further achieve better word error rate (WER), we generate pseudo labels (PLs) on the training set using the word-based model trained with STC and use these PLs to train a letter-based model, via a regular CTC approach.

Original: The PLs are generated using rescoring as described in Section B.1.2 on the training set using the hyperparameters optimized on validation set.
Revised: The PLs are generated using rescoring as described in Section B.1.2 on the training set using the hyperparameters optimized on validation set.

Original: 4.2 Handwriting Recognition (HWR)
Revised: 4.2 Handwriting Recognition (HWR)

Original: [Figure]
Revised: [Figure]

Original: We test our approach on IAM Handwriting database [26], which is a widely used benchmark for handwriting recognition.
Revised: We test our approach on IAM Handwriting database [26], which is a widely used benchmark for handwriting recognition.

Original: The dataset contains 78 different characters and a white-space symbol, | .
Revised: The dataset contains 78 different characters and a white-space symbol, | .

Original: We use Aachen data splits 4 to divide the dataset into three subsets: 6,482 lines for training, 976 lines for validation and 2,915 lines for testing.
Revised: We use Aachen data splits 4 to divide the dataset into three subsets: 6,482 lines for training, 976 lines for validation and 2,915 lines for testing.

Original: To create weakly supervised labels, we use the same methodology using p _ drop as described in 4.1.1.
Revised: To create weakly supervised labels, we use the same methodology using p _ drop as described in 4.1.1.

Original: However, we drop the labels at character-level instead of word-level.
Revised: However, we drop the labels at character-level instead of word-level.

Original: This is because it is efficient to express the STC label graph with " ⟨∗⟩ " tokens when we use the same type of targets which are being dropped.
Revised: This is because it is efficient to express the STC label graph with " ⟨∗⟩ " tokens when we use the same type of targets which are being dropped.

Original: In ASR, the targets are dropped at word level, hence we choose words and in HWR we use character targets.
Revised: In ASR, the targets are dropped at word level, hence we choose words and in HWR we use character targets.

Original: Figure 8 shows a training example from IAM along with weakly supervised labels generated for different p _ drop values.
Revised: Figure 8 shows a training example from IAM along with weakly supervised labels generated for different p _ drop values.

Original: Additional implementation and training details (models, tokens, optimization, other hyperparameters and settings) for the ASR and HWR experiments are in the Appendix B.
Revised: Additional implementation and training details (models, tokens, optimization, other hyperparameters and settings) for the ASR and HWR experiments are in the Appendix B.

Original: [Table]
Revised: [Table]

Original:  Results 5.1 Automatic Speech Recognition
Revised:  Results 5.1 Automatic Speech Recognition

Original: In Table 1, we compare the WER performance of STC models for a fixed value of p _ drop = 0 .
Revised: In Table 1, we compare the WER performance of STC models for a fixed value of p _ drop = 0 .

Original: 7 and also when p _ drop is dependent on the sample or the word.
Revised: 7 and also when p _ drop is dependent on the sample or the word.

Original: We also compare these results with the models trained directly with CTC on the partial labels.
Revised: We also compare these results with the models trained directly with CTC on the partial labels.

Original: We can see that STC performs better than CTC in all the settings.
Revised: We can see that STC performs better than CTC in all the settings.

Original: We observe that performing beam search decoding
Revised: We observe that performing beam search decoding

Original: P _ DROP
Revised: P _ DROP

Original: M ETHOD
Revised: M ETHOD

Original: D EV CER
Revised: D EV CER

Original: T EST CER
Revised: T EST CER

Original: T RANSFORMER [21] 4.
Revised: T RANSFORMER [21] 4.

Original: CNN+CTC[43] 4.
Revised: CNN+CTC[43] 4.

Original: LSTM W /A TTN [28] 3.
Revised: LSTM W /A TTN [28] 3.

Original: CNN + CTC
Revised: CNN + CTC

Original: (O UR B ASELINE ) 3.
Revised: (O UR B ASELINE ) 3.

Original: CNN + CTC 5.
Revised: CNN + CTC 5.

Original: 7.7 CNN + STC 5.
Revised: 7.7 CNN + STC 5.

Original: CNN + CTC 8.
Revised: CNN + CTC 8.

Original: [Figure]
Revised: [Figure]

Original: CNN + STC 5.
Revised: CNN + STC 5.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CNN
Revised: CNN

Original: [Figure]
Revised: [Figure]

Original: CTC
Revised: CTC

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CTC
Revised: CTC

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: and an additional second-pass rescoring with a Transformer LM can reduce the WER of the models significantly.
Revised: and an additional second-pass rescoring with a Transformer LM can reduce the WER of the models significantly.

Original: We also see that pseudo labeling the training set using STC model trained on partial labels and then training a letter-based CTC model can further improve WER performance.
Revised: We also see that pseudo labeling the training set using STC model trained on partial labels and then training a letter-based CTC model can further improve WER performance.

Original: We can get competitive results compared with the fully supervised results
Revised: We can get competitive results compared with the fully supervised results

Original: 5.2 Handwriting Recognition
Revised: 5.2 Handwriting Recognition

Original: In Table we compare the CER performance of CTC and STC for various values of p _ drop with greedy decoding.
Revised: In Table we compare the CER performance of CTC and STC for various values of p _ drop with greedy decoding.

Original: We can see that using STC clearly gives better performance over CTC trained models for p _ drop = 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 .
Revised: We can see that using STC clearly gives better performance over CTC trained models for p _ drop = 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 .

Original: The performance gap between the supervised baseline and models trained with partial labels using STC is larger compared to ASR experiments.
Revised: The performance gap between the supervised baseline and models trained with partial labels using STC is larger compared to ASR experiments.

Original: This is because we did not use LM decoding or pseudo-labeling steps which should help in reducing CER.
Revised: This is because we did not use LM decoding or pseudo-labeling steps which should help in reducing CER.

Original: Hyperparameters used for token insertion penalty, λ to train STC models for ASR, HWR experiments are in Appendix D.
Revised: Hyperparameters used for token insertion penalty, λ to train STC models for ASR, HWR experiments are in Appendix D.

Original: 5.3 Runtime Performance of STC
Revised: 5.3 Runtime Performance of STC

Original: In Figure 9, we compare the epoch time of STC, CTC models on LibriSpeech and IAM datasets.
Revised: In Figure 9, we compare the epoch time of STC, CTC models on LibriSpeech and IAM datasets.

Original: For a fair comparison, we use the same model that is used to report STC results for each dataset.
Revised: For a fair comparison, we use the same model that is used to report STC results for each dataset.

Original: The experiments on LibriSpeech, IAM are run on 32, 8 Nvidia 32GB V100 GPUs and uses C++, Python
Revised: The experiments on LibriSpeech, IAM are run on 32, 8 Nvidia 32GB V100 GPUs and uses C++, Python

Original: APIs of GTN respectively.
Revised: APIs of GTN respectively.

Original: We were able to run an epoch on LibriSpeech in about 300 seconds for the STC model and in about 250 seconds for the CTC model.
Revised: We were able to run an epoch on LibriSpeech in about 300 seconds for the STC model and in about 250 seconds for the CTC model.

Original: Optimizing the data transfer between CPU and GPU by only moving the tokens which are present in the current training sample (Section 3.3.3) is a crucial step in achieving good performance.
Revised: Optimizing the data transfer between CPU and GPU by only moving the tokens which are present in the current training sample (Section 3.3.3) is a crucial step in achieving good performance.

Original: Without this optimization, it would take about 5400 seconds per epoch on LibriSpeech as the alphabet size, |A| of 50000 is very high.
Revised: Without this optimization, it would take about 5400 seconds per epoch on LibriSpeech as the alphabet size, |A| of 50000 is very high.

Original: For IAM experiments, we did not perform the data transfer optimization for STC as |A| is only 79.
Revised: For IAM experiments, we did not perform the data transfer optimization for STC as |A| is only 79.

Original: It can be see that the epoch time of STC is about 25% more compared to CTC.
Revised: It can be see that the epoch time of STC is about 25% more compared to CTC.

Original:  Conclusion
Revised:  Conclusion

Original: In a variety of sequence labeling tasks (ASR, HWR), we show that STC enables training models with partially labeled data and can give strong performance.
Revised: In a variety of sequence labeling tasks (ASR, HWR), we show that STC enables training models with partially labeled data and can give strong performance.

Original: Weakly supervised data can be collected in semi-automatic ways and can alleviate labeling of full training data, thus reducing production costs.
Revised: Weakly supervised data can be collected in semi-automatic ways and can alleviate labeling of full training data, thus reducing production costs.

Original: We also show that using WFSTs with a differentiable framework is flexible and powerful tool for researchers to solve a completely new set of problems, like the example we have shown in this paper.
Revised: We also show that using WFSTs with a differentiable framework is flexible and powerful tool for researchers to solve a completely new set of problems, like the example we have shown in this paper.

Original: STC requires additional hyperparameter tuning and these vary by p _ drop .
Revised: STC requires additional hyperparameter tuning and these vary by p _ drop .

Original: The label graph can be easily modified to include the cases where the missing label’s position or the number of missing tokens is known in advance.
Revised: The label graph can be easily modified to include the cases where the missing label’s position or the number of missing tokens is known in advance.

Original: Also, STC can be specialized to allow a particular range of missing tokens (e.g. between 2 and 4 characters).
Revised: Also, STC can be specialized to allow a particular range of missing tokens (e.g. between 2 and 4 characters).

Original: A direct potential extension is to study the application of
Revised: A direct potential extension is to study the application of

Original: STC to noisy labels
Revised: STC to noisy labels

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Appendix: Weighted Finite-State Transducers
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: A weighted finite-state transducer (WFST) is a generalization of a finite-state automaton (FSA) [29, 30, 39] where each transition has an input label from an alphabet Σ an output label from an alphabet ∆ and scalar weight w .

Original: Figure 10a shows an example WFST with nodes representing states and arcs representing transitions.

Original: A path from an initial to a final state encodes a mapping from an input sequence i ∈

Original: Σ ∗ to an output sequence o ∈ ∆ ∗ and a corresponding score.

Original: While WFST operations can be performed in any semiring, in this work we only use the log semiring.

Original: [Figure]

Original: , T

Original: [Figure]

Original: In this work we use composition and shortest distance operations on WFSTs.

Original: The composition operation combines WFSTs from different modalities.

Original: [-Given two WFSTs T 1 and T 2 , if T 1 transduces a to b with weight w 
-]
Revised: [-Given two WFSTs T 1 and T 2 , if T 1 transduces a to b with weight w 
-]

Original: and T 2 transduces b to c with weight w 2 , then their composition T 1 ◦ T 2 transduces a to c with weight w 1 + w 2 .

Original: The forward score operation is the shortest distance from a start state to a final state in the log semiring.

Original: Given a transducer T 1 , the forward score is the log-sum-exp of the scores of all paths from any start state to any final state.

Original: The example output graphs from composition and forward score operations are shown in Figure 10c and Figure 10d respectively.

Original: A.1 Autograd with WFSTs

Original: Most operations on WFST are differentiable with respect to the arc weights of the input graphs.

Original: This allows WFSTs to be used dynamically to train neural networks.

Original: Frameworks like GTN [15] and k2 [19] implement automatic differentiation with WFSTs.

Original: For the purpose of this work, we use the GTN framework.

Original: B Appendix: Additional Implementation Details B.1 Automatic Speech Recognition

Original: We keep the original 16kHz sampling rate and compute log-mel filterbanks with 80 coefficients for a 25ms sliding window, strided by 10ms.

Original: All features are normalized to have zero mean and unit variance per input sequence before feeding them into the acoustic model.

Original: We use SpecAugment [34] as the data augmentation method for all ASR experiments.

Original: B.1.1 Model architecture

Original: The acoustic model (AM) architecture is composed of a convolutional frontend (1-D convolution with kernel- width 15 and stride 8 followed by GLU activation) followed by 36 ×

Original: 4- heads Transformer blocks [38] with relative positional embedding.

Original: The self-attention dimension is 384 and the feed-forward network (FFN) dimension is 3072 in each Transformer block.

Original: The output of the final Transformer block if followed by a Linear layer with output dimension of 580 and a letter-to-word encoder (see Appendix C) to the output classes (word vocabulary + blank ).

Original: For all Transformer layers, we use dropout on the self-attention and on the FFN, and layer drop [12], dropping entire layers at the FFN level.

Original: We apply LogSoftmax operation on each output frame to produce a probability distribution (in log-space) over output classes.

Original: The model consists of 70 million parameters and we use 32 × Nvidia 32GB V100 GPUs for training.

Original: B.1.2 Decoding

Original: Beam-Search decoder : In our experiments, we use a beam-search decoder following [8] which leverages a n -gram language model to decrease the word error rate.

Original: The beam-search decoder outputs a transcription ˆ y that maximizes the following objective

Original: [Equation]

Original: where log P LM ( . ) is the log-likelihood of the language model, α is the weight of the language model, β is a word insertion weight and | ˆ y | is the transcription length in words.

Original: The hyperparameters α and β are optimized on the validation set.

Original: We use 5-gram LM trained on the official LM training data provided with LibriSpeech for beam-search decoding.

Original: Rescoring :

Original: To further decrease the word error rate, we use the top- K hypothesis from beam search decoding and perform rescoring [37] with a Transformer LM to reorder the hypotheses according to the following score:

Original: [Equation]

Original: where log P LM ′ ( . ) is the log-likelihood of the Transformer LM, α , β are the hyperparameters as described above and | ˆ y | is the transcription length in characters.

Original: We use the pre-trained Transformer LM on LibriSpeech from [24] which has a perplexity of 50 on dev-other transcripts.

Original: In this work, top 512 hypothesis from beam search decoding are used as candidates for rescoring.

Original: B.1.3 Optimizer

Original: For training the word-based STC models, we use the Adagrad optimizer [10] with a learning rate warmup scheme that increases linearly from 0 to 0 .

Original: 02 in 16000 training steps for all the experiments.

Original: We halve the learning rate initially after 400 epochs and every 200 epochs after that.

Original: All models are trained with dynamic batching with a batch size of 240 audio sec per GPU.

Original: We use SpecAugment [34] with two frequency masks, and ten time masks with maximum time mask ratio of p = 0.1, the maximum frequency bands masked by one frequency mask is 30, and the maximum frames masked by the time mask is 50; time warping is not used.

Original: SpecAugment is turned on only after 32000 training steps are finished.

Original: Dropout and layer dropout values of 0.05 is used in the AM.

Original: B.1.4 Pseudolabeling

Original: For training the letter-based CTC models, we use the same Transformer-based encoder consisting of 270M parameters from [37] for the AM: the encoder of our acoustic models is composed of a convolutional frontend (1D convolution with kernel-width 7 and stride 3 followed by GLU activation) followed by 36 4-heads Transformer blocks [38].

Original: The self-attention dimension is 768 and the feed-forward network (FFN) dimension is 3072 in each

Original: Transformer block.

Original: We uses 64 × Nvidia 32GB V100 GPUs for training.

Original: The output of the encoder is followed by a linear layer to the output classes.

Original: We use dropout after the convolution layer.

Original: For all Transformer layers, we use dropout of 0 .

Original: 2 on the self-attention and on the FFN, and layer drop [12] value of 0 . 2 .

Original: Token set for all acoustic models consists of 26 English alphabet letters, augmented with the apostrophe and a word boundary token.

Original: To speed up training, we also use mixed-precision training.

Original: We use the same SpecAugment and learning rate schemes as discussed above for STC models.

Original: We decay learning rate by a factor of 2 each time the WER reaches a plateau on the validation sets.

Original: B.2 Handwriting Recognition

Original: For training the handwriting recognition models, we have used the open-source code 5 based on a prior work from [42] and adapted it for our use case.

Original: It uses depthwise separable convolutions as the main computational block and the model consists of about 39 million parameters.

Original: We use 8 × Nvidia V100 32GB GPUs for training the models.

Original: It should also be noted that we were unable to find an open source implementation of Yousef et al.

Original: [43] and our baseline is our best attempt to reproduce their results.

Original: We use Adam Optimizer [22] with an initial learning rate of 0 .

Original: 02 and run training using a batchsize of 8 per GPU for all the experiments.

Original: All image are scaled to a maximum width, height of 600, 32 pixels respectively before feeding to the neural network.

Original: We use exponential learning rate decay schedule with a gamma factor of 10 − 1 / 90000 ≈ 0 . 99997 .

Original: We use random projection transformations as the augmentation method while training.

Original: C Appendix: A simple letter to word encoder

Original: Since, we are working on direct-to-word models for ASR, a major hurdle would be to transcribe words not found in the training vocabulary.

Original: Recently, [8] propose an end-to-end model which outputs words directly, yet is still able to dynamically modify the lexicon with words not seen at training time.

Original: They use a convnet based letter to word embedding encoder which is jointly trained with the acoustic model and is able to accommodate new words at testing time.

Original: In this work

Original: [Figure]

Original: we propose a simpler system for letter to word encoder which can be embedded into the acoustic model and thus avoiding the needs for a separate network.

Original: Consider A L , A W are the alphabets for letters, words respectively and l max is the maximum length of a word.

Original: We assume A L also contains two special letters - c blank which produces blank token required for CTC/STC models and c pad which is used to pad all words, blank token to the same maximum word length l max .

Original: We let the acoustic model produce a ( |A L | × l max ) dimensional vector for each time frame.

Original: [Figure]

Original: Since CTC/STC expect scores for A W and blank for each time frame, we carefully construct a matrix E of 1 s and 0 s which converts a vector of size ( |A L | × l max ) to ( |A W | + 1) as shown in Figure 11 .

Original: Since we are using c pad token for padding, we can always assume every words and blank token is a sequence of l max letters.

Original: Each row in E is constructed by concatenating the one-hot representation of each letter (including c pad ) and it produces the score for a word or blank .

Original: We use A L = { a − z, ' , c blank , c pad } and l max = 20 for all our experiments on LibriSpeech using words as the output tokens.

Original: [Table]

Original: From Table 3, we can see that we are able to match the performance of the word based models from [8] using a much smaller acoustic model.

Original: Also, our system is simpler as we do not have a separate network for letter-to-word encoder.

Original: D Appendix: Hyperparameters for token insertion penalty, λ

Original: In Table 4, we report the hyperparameters used for training STC models on LibriSpeech (Table 1) and IAM (Table 2).

Original: The best performing values are found by running a grid search over the hyperparameters and measuring performance on the development set.

Original: It can be seen that with higher p

Original: _ drop values, a higher token insertion penalty values are usually preferred.

Original: [Table]
Revised: [N/A]

Original: D.1 Sensitivity of the model performance when setting token insertion penalty, λ

Original: In Table 5, we report the effect of hyper parameter choices on the model performance for IAM dataset for p _ drop = 0 .

Original: 1 and p _ drop = 0 .

Original: We see that the model performance is more sensitive with respect to hyperparameter choices p _ drop =

Original: 7 compared to p _

Original: =

Original: [Table]

Original: [Table]