Original: D IFFERENTIABLE W EIGHTED F INITE -S TATE T RANSDUCERS
Revised: D IFFERENTIABLE W EIGHTED F INITE -S TATE T RANSDUCERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a framework for automatic differentiation with weighted ﬁnite-state transducers (WFSTs) allowing them to be used dynamically at training time.
Revised: We introduce a framework for automatic differentiation with weighted ﬁnite-state transducers (WFSTs) allowing them to be used dynamically at training time.

Original: Through the separation of graphs from operations on graphs, this framework enables the exploration of new structured loss functions which in turn eases the encoding of prior knowledge into learning algorithms.
Revised: Through the separation of graphs from operations on graphs, this framework enables the exploration of new structured loss functions which in turn eases the encoding of prior knowledge into learning algorithms.

Original: We show how the framework can combine pruning and back-off in transition models with various sequencelevel loss functions.
Revised: We show how the framework can combine pruning and back-off in transition models with various sequencelevel loss functions.

Original: We also show how to learn over the latent decomposition of phrases into word pieces.
Revised: We also show how to learn over the latent decomposition of phrases into word pieces.

Original: Finally, to demonstrate that WFSTs can be used in the interior of a deep neural network, we propose a convolutional WFST layer which maps lower-level representations to higher-level representations and can be used as a drop-in replacement for a traditional convolution.
Revised: Finally, to demonstrate that WFSTs can be used in the interior of a deep neural network, we propose a convolutional WFST layer which maps lower-level representations to higher-level representations and can be used as a drop-in replacement for a traditional convolution.

Original: We validate these algorithms with experiments in handwriting recognition and speech recognition.
Revised: We validate these algorithms with experiments in handwriting recognition and speech recognition.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Weighted ﬁnite-state transducers (WFSTs) are a commonly used tool in speech and language processing (Knight & May, 2009; Mohri et al., 2002).
Revised: Weighted ﬁnite-state transducers (WFSTs) are a commonly used tool in speech and language processing (Knight & May, 2009; Mohri et al., 2002).

Original: They are most frequently used to combine predictions from multiple already trained models.
Revised: They are most frequently used to combine predictions from multiple already trained models.

Original: In speech recognition, for example, WFSTs are used to combine constraints from an acoustic-to-phoneme model, a lexicon mapping words to pronunciations, and a word-level language model.
Revised: In speech recognition, for example, WFSTs are used to combine constraints from an acoustic-to-phoneme model, a lexicon mapping words to pronunciations, and a word-level language model.

Original: However, combining separately learned models using WFSTs only at inference time has several drawbacks, including the well-known problems of exposure bias (Ranzato et al., 2015) and label bias (Bottou, 1991; Lafferty et al., 2001).
Revised: However, combining separately learned models using WFSTs only at inference time has several drawbacks, including the well-known problems of exposure bias (Ranzato et al., 2015) and label bias (Bottou, 1991; Lafferty et al., 2001).

Original: Given that gradients may be computed for most WFST operations, using them only at the inference stage of a learning system is not a hard limitation.
Revised: Given that gradients may be computed for most WFST operations, using them only at the inference stage of a learning system is not a hard limitation.

Original: We speculate that this limitation is primarily due to practical considerations.
Revised: We speculate that this limitation is primarily due to practical considerations.

Original: Historically, hardware has not been sufﬁciently performant to make training with WFSTs tractable.
Revised: Historically, hardware has not been sufﬁciently performant to make training with WFSTs tractable.

Original: Also, no implementation exists with the required operations which supports automatic differentiation in a high-level yet efﬁcient manner.
Revised: Also, no implementation exists with the required operations which supports automatic differentiation in a high-level yet efﬁcient manner.

Original: We develop a framework for automatic differentiation through operations on WFSTs.
Revised: We develop a framework for automatic differentiation through operations on WFSTs.

Original: We show the utility of this framework by leveraging it to design and experiment with existing and novel learning algorithms.
Revised: We show the utility of this framework by leveraging it to design and experiment with existing and novel learning algorithms.

Original: Automata are a more convenient structure than tensors to encode prior knowledge into a learning algorithm.
Revised: Automata are a more convenient structure than tensors to encode prior knowledge into a learning algorithm.

Original: However, not training with them limits the extent to which this prior knowledge can be incorporated in a useful manner.
Revised: However, not training with them limits the extent to which this prior knowledge can be incorporated in a useful manner.

Original: A framework for differentiable WFSTs allows the model to jointly learn from training data as well as prior knowledge encoded in WFSTs.
Revised: A framework for differentiable WFSTs allows the model to jointly learn from training data as well as prior knowledge encoded in WFSTs.

Original: This enables the learning algorithm to incorporate such knowledge in the best possible way.
Revised: This enables the learning algorithm to incorporate such knowledge in the best possible way.

Original: Use of WFSTs conveniently decomposes operations from data ( i .e.
Revised: Use of WFSTs conveniently decomposes operations from data ( i .e.

Original: graphs).
Revised: graphs).

Original: For example, rather than hand-coding sequence-level loss functions such as Connectionist Temporal Classiﬁcation (CTC) (Graves et al., 2006) or the Automatic Segmentation Criterion (ASG) (Collobert et al., 2016), we may specify the core assumptions of the criteria in graphs and compute the resulting loss with graph operations.
Revised: For example, rather than hand-coding sequence-level loss functions such as Connectionist Temporal Classiﬁcation (CTC) (Graves et al., 2006) or the Automatic Segmentation Criterion (ASG) (Collobert et al., 2016), we may specify the core assumptions of the criteria in graphs and compute the resulting loss with graph operations.

Original: This facilitates exploration in the space of such structured loss functions.
Revised: This facilitates exploration in the space of such structured loss functions.

Original: We show the utility of the differentiable WFST framework by designing and testing several algorithms.
Revised: We show the utility of the differentiable WFST framework by designing and testing several algorithms.

Original: For example, bi-gram transitions may be added to CTC with a transition WFST.
Revised: For example, bi-gram transitions may be added to CTC with a transition WFST.

Original: We scale transitions to large token set sizes by encoding pruning and back-off in the transition graph.
Revised: We scale transitions to large token set sizes by encoding pruning and back-off in the transition graph.

Original: Word pieces are commonly used as the output of speech recognition and machine translation models (Chiu et al., 2018; Sennrich et al., 2016).
Revised: Word pieces are commonly used as the output of speech recognition and machine translation models (Chiu et al., 2018; Sennrich et al., 2016).

Original: The word piece decomposition for a word is learned [Figure] [Figure] ASG(emissions, transitions, target): # Compute constrained and normalization graphs:
Revised: The word piece decomposition for a word is learned from gtn import * def ASG(emissions, transitions, target): # Compute constrained and normalization graphs:

Original: A = intersect(intersect(target, transitions), emissions)
Revised: A = intersect(intersect(target, transitions), emissions)

Original: Z = intersect(transitions, emissions)
Revised: Z = intersect(transitions, emissions)

Original: # Forward both graphs: A_score = forward_score(A)
Revised: # Forward both graphs: A_score = forward_score(A)

Original: Z_score =
Revised: Z_score =

Original: forward_score(Z) # Compute loss: loss = negate(subtract(A_score, Z_score))
Revised: forward_score(Z) # Compute loss: loss = negate(subtract(A_score, Z_score))

Original: # Clear previous gradients: emissions.zero_grad()
Revised: # Clear previous gradients: emissions.zero_grad()

Original: [Figure]
Revised: [Figure]

Revised: _grad()

Original: [Figure]
Revised: [Figure]

Revised: gradients

Original: [Figure]
Revised: [Figure]

Original: graph
Revised: [Figure] [Figure] _graph [Figure]

Original: [Figure]
Revised: [Figure]

Original: with a task-independent model.
Revised: with a task-independent model.

Original: Instead, we use WFSTs to marginalize over the latent word piece decomposition at training time.
Revised: Instead, we use WFSTs to marginalize over the latent word piece decomposition at training time.

Original: This lets the model learn decompositions salient to the task at hand.
Revised: This lets the model learn decompositions salient to the task at hand.

Original: Finally, we show that WFSTs may be used as layers themselves intermixed with tensor-based layers.
Revised: Finally, we show that WFSTs may be used as layers themselves intermixed with tensor-based layers.

Original: We propose a convolutional WFST layer which maps lower-level representations to higher-level representations.
Revised: We propose a convolutional WFST layer which maps lower-level representations to higher-level representations.

Original: The WFST convolution can be trained with the rest of the model and results in improved accuracy with fewer parameters and operations as compared to a traditional convolution.
Revised: The WFST convolution can be trained with the rest of the model and results in improved accuracy with fewer parameters and operations as compared to a traditional convolution.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: • A framework for automatic differentiation with WFSTs.
Revised: • A framework for automatic differentiation with WFSTs.

Original: The framework supports both C++ and Python front-ends and is available at https://www.anonymized.com .
Revised: The framework supports both C++ and Python front-ends and is available at https://www.anonymized.com .

Original: • We show that the framework may be used to express both existing sequence-level loss functions and to design novel sequence-level loss functions.
Revised: • We show that the framework may be used to express both existing sequence-level loss functions and to design novel sequence-level loss functions.

Original: • We propose a convolutional WFST layer which can be used in the interior of a deep neural network to map lower-level representations to higher-level representations.
Revised: • We propose a convolutional WFST layer which can be used in the interior of a deep neural network to map lower-level representations to higher-level representations.

Original: • We demonstrate the effectiveness of using WFSTs in the manners described above with experiments in automatic speech and handwriting recognition.
Revised: • We demonstrate the effectiveness of using WFSTs in the manners described above with experiments in automatic speech and handwriting recognition.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A wealth of prior work exists using weighted ﬁnite-state automata in speech recognition, natural language processing, optical character recognition, and other applications (Breuel, 2008; Knight &
Revised: A wealth of prior work exists using weighted ﬁnite-state automata in speech recognition, natural language processing, optical character recognition, and other applications (Breuel, 2008; Knight &

Original: May, 2009; Mohri, 1997; Mohri et al., 2008; Pereira et al., 1994).
Revised: May, 2009; Mohri, 1997; Mohri et al., 2008; Pereira et al., 1994).

Original: However, the use of WFSTs is limited mostly to the inference stage of a predictive system.
Revised: However, the use of WFSTs is limited mostly to the inference stage of a predictive system.

Original: For example, Kaldi, a commonly used toolkit for automatic speech recognition, uses WFSTs extensively, but in most cases for inference or to estimate the parameters of shallow models (Povey et al., 2011).
Revised: For example, Kaldi, a commonly used toolkit for automatic speech recognition, uses WFSTs extensively, but in most cases for inference or to estimate the parameters of shallow models (Povey et al., 2011).

Original: In some cases, WFSTs are used statically to incorporate ﬁxed lattices in discriminative sequence criteria (Vesel`y et al., 2013).
Revised: In some cases, WFSTs are used statically to incorporate ﬁxed lattices in discriminative sequence criteria (Kingsbury, 2009; Kingsbury et al., 2012; Su et al., 2013; Vesel`y et al., 2013).

Original: Implementations of sequence criteria in end-to-end style training are typically hand-crafted with careful consideration for speed (Amodei et al., 2016; Collobert et al., 2019; Povey et al., 2016).
Revised: Implementations of sequence criteria in end-to-end style training are typically hand-crafted with careful consideration for speed (Amodei et al., 2016; Collobert et al., 2019; Povey et al., 2016).

Original: The use of hand-crafted implementations reduces ﬂexibility which limits research.
Revised: The use of hand-crafted implementations reduces ﬂexibility which limits research.

Original: In some cases, such as the fully differentiable beam search of Collobert et al.
Revised: In some cases, such as the fully differentiable beam search of Collobert et al.

Original: , achieving the necessary computational efﬁciency with a WFST-based implementation may not yet be tractable.
Revised: , achieving the necessary computational efﬁciency with a WFST-based implementation may not yet be tractable.

Original: However, as a ﬁrst step, we show that in many common cases we can have the expressiveness afforded by the differentiable
Revised: However, as a ﬁrst step, we show that in many common cases we can have the expressiveness afforded by the differentiable

Original: WFST framework without paying an unacceptable penalty in execution time.
Revised: WFST framework without paying an unacceptable penalty in execution time.

Revised: The ideas of learning with WFSTs (Eisner, 2002) and automatic differentiation through operations on graphs (Bottou et al., 1997) are not new.

Revised: However, no simple and efﬁcient frameworks exist.

Original: Related to this work, and inspiring the name of our framework, are the graph transformer networks of Bottou et al.
Revised: Especially related to this work, and inspiring the name of our framework, are the graph transformer networks of Bottou et al.

Original: Generalized graph transducers (Bottou et al., 1996) are more expressive than WFSTs, allowing arbitrary data as edge labels.
Revised: Generalized graph transducers (Bottou et al., 1996) are more expressive than WFSTs, allowing arbitrary data as edge labels.

Original: When composing these graphs, one deﬁnes an edge matching function and a “transformer” to construct the resulting structure.
Revised: When composing these graphs, one deﬁnes an edge matching function and a “transformer” to construct the resulting structure.

Original: While not this general, differentiable WFSTs nevertheless allow for a vast design space of interesting algorithms, and perhaps make a more pragmatic trade-off between ﬂexibility and efﬁciency.
Revised: While not this general, differentiable WFSTs nevertheless allow for a vast design space of interesting algorithms, and perhaps make a more pragmatic trade-off between ﬂexibility and efﬁciency.

Original: Highly efﬁcient libraries for operations on WFSTs exist, notably OpenFST and its predecessor
Revised: Highly efﬁcient libraries for operations on WFSTs exist, notably OpenFST and its predecessor

Original: FSM (Allauzen et al., 2007; Mohri et al., 2000).
Revised: FSM (Allauzen et al., 2007; Mohri et al., 2000).

Original: We take inspiration from OpenFst in the interface and implementation of many of our functions.
Revised: We take inspiration from OpenFst in the interface and implementation of many of our functions.

Original: However, the design implications of operating on WFSTs with automatic differentiation are quite different than those of the use cases OpenFST has been optimized for.
Revised: However, the design implications of operating on WFSTs with automatic differentiation are quite different than those of the use cases OpenFST has been optimized for.

Original: We also draw inspiration from libraries for automatic differentiation and deep learning (Collobert et al., 2011; Paszke et al., 2019; Pratap et al., 2019; Tokui et al., 2015).
Revised: We also draw inspiration from libraries for automatic differentiation and deep learning (Collobert et al., 2011; Paszke et al., 2019; Pratap et al., 2019; Tokui et al., 2015).

Original: Some of the algorithms we propose, with the goal of demonstrating the utility of the differentiable
Revised: Some of the algorithms we propose, with the goal of demonstrating the utility of the differentiable

Original: WFST library, are inspired by prior work.
Revised: WFST library, are inspired by prior work.

Original: Prior work has explored pruning the set of allowed alignments with CTC, and in particular limiting the spacing between output tokens (Liu et al., 2018).
Revised: Prior work has explored pruning the set of allowed alignments with CTC, and in particular limiting the spacing between output tokens (Liu et al., 2018).

Original: Learning n -gram word decompositions with both differentiable (Liu et al., 2017) and nondifferentiable (Chan et al., 2017) loss functions has also been explored.
Revised: Learning n -gram word decompositions with both differentiable (Liu et al., 2017) and nondifferentiable (Chan et al., 2017) loss functions has also been explored.

Original:  D IFFERENTIABLE W EIGHTED F INITE -S TATE T RANSDUCERS
Revised:  D IFFERENTIABLE W EIGHTED F INITE -S TATE T RANSDUCERS

Original: A weighted ﬁnite-state acceptor A is a 6 -tuple consisting of an alphabet Σ , a set of states Q , start states Q s , accepting states Q a , a transition function π ( q, p ) which maps elements of Q × Σ to elements of Q , and a weight function ω ( q, p ) which maps elements of Q × Σ to R .
Revised: A weighted ﬁnite-state acceptor A is a 6 -tuple consisting of an alphabet Σ , a set of states Q , start states Q s , accepting states Q a , a transition function π ( q, p ) which maps elements of Q × Σ to elements of Q , and a weight function ω ( q, p ) which maps elements of Q × Σ to R .

Original: A weighted ﬁnite-state transducer T is a 7 -tuple which augments an acceptor with an output alphabet ∆ .
Revised: A weighted ﬁnite-state transducer T is a 7 -tuple which augments an acceptor with an output alphabet ∆ .

Original: The transition function π ( q, p, r ) and weight function ω ( q, p, r ) map elements of Q × Σ × ∆ to elements of Q and R respectively.
Revised: The transition function π ( q, p, r ) and weight function ω ( q, p, r ) map elements of Q × Σ × ∆ to elements of Q and R respectively.

Original: In other words, each edge of a transducer connects two states and has an input label p , an output label r and a weight w ∈ R .
Revised: In other words, each edge of a transducer connects two states and has an input label p , an output label r and a weight w ∈ R .

Original: We denote an input by p =
Revised: We denote an input by p =

Original: [ p 1 , . .
Revised: [ p 1 , . .

Original: . , p T ] where each p i ∈ Σ .
Revised: . , p T ] where each p i ∈ Σ .

Original: An acceptor A accepts the input p if there exists a sequence of states q i +

Revised: An acceptor A accepts the input p if there exists a sequence of states q i +

Original: =
Revised: =

Original: π ( q i , p i )
Revised: π ( q i , p i )

Original: ∈ Q such that q 1 ∈ Q s and q T +1 ∈ Q a .
Revised: ∈ Q such that q 1 ∈ Q s and q T +1 ∈ Q a .

Original: With a slight abuse of notation, we let p ∈
Revised: With a slight abuse of notation, we let p ∈

Original: A denote that A accepts p .
Revised: A denote that A accepts p .

Original: The score of p is given by s ( p ) =
Revised: The score of p is given by s ( p ) =

Original: (cid:80)
Revised: (cid:80)

Original: Ti =1 ω ( q i , p i ) .
Revised: Ti =1 ω ( q i , p i ) .

Original: Let r =
Revised: Let r =

Original: [ r 1 , . .
Revised: [ r 1 , . .

Original: , r T ] be a path with r i ∈ ∆ .
Revised: , r T ] be a path with r i ∈ ∆ .

Original: A transducer T transduces the input p to the output r ( i.e. ( p , r ) ∈ T ) if there exists a sequence of states q i +

Revised: A transducer T transduces the input p to the output r ( i.e. ( p , r ) ∈ T ) if there exists a sequence of states q i +

Original: =
Revised: =

Original: π ( q i , p i , r i
Revised: π ( q i , p i , r i

Original: ) ∈ Q such that q 1 ∈ Q s and q T +1 ∈ Q a .
Revised: ) ∈ Q such that q 1 ∈ Q s and q T +1 ∈ Q a .

Original: The score of the pair ( p , r ) is given by s ( p , r ) =
Revised: The score of the pair ( p , r ) is given by s ( p , r ) =

Original: (cid:80) Ti =1 ω ( q i , p i , r i ) .
Revised: (cid:80) Ti =1 ω ( q i , p i , r i ) .

Original: We restrict operations to the log and tropical semirings.
Revised: We restrict operations to the log and tropical semirings.

Original: In both cases the accumulation of weights along a path is with addition.
Revised: In both cases the accumulation of weights along a path is with addition.

Original: In the log semiring the accumulation over path scores is with logsum-exp which we denote by logadd i s i = log (cid:80)
Revised: In the log semiring the accumulation over path scores is with logsum-exp which we denote by logadd i s i = log (cid:80)

Original: i e s i .
Revised: i e s i .

Original: In the tropical semiring, the accumulation over path scores is with the max, max i s i .
Revised: In the tropical semiring, the accumulation over path scores is with the max, max i s i .

Original: We allow (cid:15) transitions in both acceptors and transducers.
Revised: We allow (cid:15) transitions in both acceptors and transducers.

Original: An (cid:15) input (or output) on an edge means the edge can be traversed without consuming an input (or emitting an output).
Revised: An (cid:15) input (or output) on an edge means the edge can be traversed without consuming an input (or emitting an output).

Original: This allows inputs to map to outputs of differing length.
Revised: This allows inputs to map to outputs of differing length.

Original: 3. O PERATIONS
Revised: 3. O PERATIONS

Original: We brieﬂy describe a subset of the most important operations implemented in our differentiable
Revised: We brieﬂy describe a subset of the most important operations implemented in our differentiable

Original: WFST framework.
Revised: WFST framework.

Original: For a more detailed discussion of operations on WFSTs see e.g. Mohri (2009).
Revised: For a more detailed discussion of operations on WFSTs see e.g. Mohri (2009).

Original: Intersection We denote by B =
Revised: Intersection We denote by B =

Original: A 1 ◦ A 2 the intersection of two acceptors A 

Revised: A 1 ◦ A 2 the intersection of two acceptors A

Original: and A 2 .
Revised: and A 2 .

Original: The intersected graph B contains all paths which are accepted by both inputs.
Revised: The intersected graph B contains all paths which are accepted by both inputs.

Original: The score of any path in B is the sum of the scores of the two corresponding paths in A 1 and A 2 .
Revised: The score of any path in B is the sum of the scores of the two corresponding paths in A 1 and A 2 .

Original: Composition The same symbol denotes the composition of two transducers
Revised: Composition The same symbol denotes the composition of two transducers

Original: U =
Revised: U =

Original: T 1 ◦ T 2 .
Revised: T 1 ◦ T 2 .

Original: If Ttransduces p to u and T 2 transduces u to r then U transduces p to r .
Revised: If Ttransduces p to u and T 2 transduces u to r then U transduces p to r .

Original: As in intersection, the score of the path in the composed graph is the sum of the path scores from the input graphs.
Revised: As in intersection, the score of the path in the composed graph is the sum of the path scores from the input graphs.

Original: Forward and Viterbi Score
Revised: Forward and Viterbi Score

Original: The forward score of T is logadd ( p , r ) ∈A s ( p , r ) .
Revised: The forward score of T is logadd ( p , r ) ∈T s ( p , r ) .

Original: Similarly, the Viterbi score of T is max ( p , r ) ∈T s ( p , r ) .
Revised: Similarly, the Viterbi score of T is max ( p , r ) ∈T s ( p , r ) .

Original: Viterbi Path The Viterbi path of T is given by arg max ( p , r ) ∈T s ( p , r ) .

Original: 1 a:a/ a: ε / (a)
Revised: 1 a:a/ a: ε / (a)

Original: Token graph T a 1 a: ε / 2 b:ab/ (b) Label graph Y 1 a: ε / a: ε / 2 b:ab/ b: ε / (c) Alignment graph T ◦ Y  a/-3 b/ c/  a/

Revised: Token graph T a 1 a: ε / 2 b:ab/ (b) Label graph Y 1 a: ε / a: ε / 2 b:ab/ b: ε / (c) Alignment graph T ◦ Y  a/-3 b/ c/  a/

Original: b/- c/-  a/-1 b/ c/-  a/1 b/- c/ (d) Emissions graph E X 1 a/- 2 a/ 3 b/- 4 a/-  b/ b/  b/- b/- (e) Constrained graph A = T ◦
Revised: b/- c/-  a/-1 b/ c/-  a/1 b/- c/ (d) Emissions graph E X 1 a/- 2 a/ 3 b/- 4 a/-  b/ b/  b/- b/- (e) Constrained graph A = T ◦

Original: Y ◦
Revised: Y ◦

Original: E X
Revised: E X

Revised: Viterbi Path The Viterbi path of T is given by arg max ( p , r ) ∈T s ( p , r ) .

Revised: The forward score, Viterbi score, and the Viterbi path for an acceptor A are deﬁned in the same way using just p .

Revised: In a directed acyclic graph these operations can be computed in time linear in the size of the graph with the well-known forward and Viterbi algorithms (Jurafsky, 2000; Rabiner, 1989).

Original: We support standard rational operations including the union ( A + B ), concatenation ( AB ), and the
Revised: We support standard rational operations including the union ( A + B ), concatenation ( AB ), and the

Original: Kleene closure ( A ∗ ) of WFSTs.
Revised: Kleene closure ( A ∗ ) of WFSTs.

Original: To enable automatic differentiation through complete computations, we support slightly non-standard operations.
Revised: To enable automatic differentiation through complete computations, we support slightly non-standard operations.

Original: We allow for the negation of all the arcs weights in a graph and the addition or subtraction of the arc weights of two identically structured graphs.
Revised: We allow for the negation of all the arcs weights in a graph and the addition or subtraction of the arc weights of two identically structured graphs.

Original: 3. A UTOMATIC D IFFERENTIATION
Revised: 3. A UTOMATIC D IFFERENTIATION

Original: For the sake of automatic differentiation, every operation in section 3.1 accepts as inputs one or more graphs and returns a graph as output.
Revised: For the sake of automatic differentiation, every operation in section 3.1 accepts as inputs one or more graphs and returns a graph as output.

Original: For example, the forward score returns a “scalar” graph with a single arc between a start state and accepting state, with the score as the arc weight.
Revised: For example, the forward score returns a “scalar” graph with a single arc between a start state and accepting state, with the score as the arc weight.

Original: The Viterbi path outputs the linear graph representing the best path p ∗ with p ∗ i as the arc labels.
Revised: The Viterbi path outputs the linear graph representing the best path p ∗ with p ∗ i as the arc labels.

Original: Jacobians of the output graph weights with respect to the weights of the input graphs may be computed for all of the operations described so far.
Revised: Jacobians of the output graph weights with respect to the weights of the input graphs may be computed for all of the operations described so far.

Original: Hence, the chain rule may be used to compute Jacobians or gradients of outputs with respect to the inputs of compositions of graph operations.
Revised: Hence, the chain rule may be used to compute Jacobians or gradients of outputs with respect to the inputs of compositions of graph operations.

Original: The WFST framework implements reverse-mode automatic differentiation following existing deep learning frameworks (Pratap et al., 2019; Paszke et al., 2019).
Revised: The WFST framework implements reverse-mode automatic differentiation following existing deep learning frameworks (Pratap et al., 2019; Paszke et al., 2019).

Original: Figure 1 gives an example Python implementation which computes gradients for the ASG criterion.
Revised: Figure 1 gives an example Python implementation which computes gradients for the ASG criterion.

Original: In order to enable second-order differentiation, among other design considerations, the gradient with respect to a graph is also a graph.
Revised: In order to enable second-order differentiation, among other design considerations, the gradient with respect to a graph is also a graph.

Original: This adds little additional overhead as the two graphs share the underlying graph topology.
Revised: This adds little additional overhead as the two graphs share the underlying graph topology.

Original: The weights of the gradient graph are the gradients with respect to the weights of the graph.
Revised: The weights of the gradient graph are the gradients with respect to the weights of the graph.

Original:  L EARNING A LGORITHMS
Revised:  L EARNING A LGORITHMS

Original: Let X =
Revised: Let X =

Original: [ x 1 , . . .
Revised: [ x 1 , . . .

Original: x T ] ∈
Revised: x T ] ∈

Original: X be a sequence of observations and y =
Revised: X be a sequence of observations and y =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y U ] ∈ Y a label.
Revised: , y U ] ∈ Y a label.

Original: A sequence-level objective for the ( X , y ) pair can be speciﬁed with two weighted automata:
Revised: A sequence-level objective for the ( X , y ) pair can be speciﬁed with two weighted automata:

Original: [Equation]
Revised: [Equation]

Original: The target constrained graph, A X , y , is constructed from the observation and target pair and the normalization graph, Z X , is constructed only from the observation.
Revised: The target constrained graph, A X , y , is constructed from the observation and target pair and the normalization graph, Z X , is constructed only from the observation.

Original: We require A X , y ⊆
Revised: We require A X , y ⊆

Original: Z X .
Revised: Z X .

Original: The key ingredients to make equation 1 operational are 1) the structure of A and Z and 2) the source of the arc weights.
Revised: The key ingredients to make equation 1 operational are 1) the structure of A and Z and 2) the source of the arc weights.

Original: The graph structures can be speciﬁed through functions on graphs.
Revised: The graph structures can be speciﬁed through functions on graphs.

Original: The initial arc weights come from the data itself or arbitrary learning algorithms (deep networks, n -gram language models, etc. )
Revised: The initial arc weights come from the data itself or arbitrary learning algorithms (deep networks, n -gram language models, etc. )

Original: capable of providing useful scores on elements of X and Y .
Revised: capable of providing useful scores on elements of X and Y .

Original: The graphs A and Z used by the ASG loss function can be constructed from operations on simpler graphs.

Original: The ASG criterion assumes that emitting a new token requires consuming the token at least

Original: <b>/ (
Revised: <b>/ (

Original: [Figure]
Revised: [Figure]

Revised: graph.

Original: a/  <b>/ a/  b/  <b>/0 a/0 <b>/ b/ 5 <b>/0 b/0 <b>/ <b>/ (
Revised: a/  <b>/ a/  b/  <b>/0 a/0 <b>/ b/ 5 <b>/0 b/0 <b>/ <b>/ (

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: < s > b a c a/ w ( < s > ,a) b/ w ( < s > ,b) c/ w ( < s > ,c) a/ w (a,a) b/ w (a,b) c/ w (a,c) a/ w (b,a) b/ w (b,b) c/ w (b,c) a/ w (c,a) b/ w (c,b) c/ w (c,c)
Revised: < s > b a c a/ w ( < s > ,a) b/ w ( < s > ,b) c/ w ( < s > ,c) a/ w (a,a) b/ w (a,b) c/ w (a,c) a/ w (b,a) b/ w (b,b) c/ w (b,c) a/ w (c,a) b/ w (c,b) c/ w (c,c)

Original: [Figure]
Revised: [Figure]

Original: a, b b b, a b,b (cid:15) c
Revised: a, b b b, a b,b (cid:15) c

Original: a/ w (a, b, a) (cid:15) / β (a, b) b/ w (b, b) a/ w (b, a) (cid:15) / β (b) c/ w (c) b/ w (b)
Revised: a/ w (a, b, a) (cid:15) / β (a, b) b/ w (b, b) a/ w (b, a) (cid:15) / β (b) c/ w (c) b/ w (b)

Original: [Figure]
Revised: [Figure]

Revised: The ASG (Collobert et al., 2016) and CTC (Graves et al., 2006) sequence criteria are commonly used in speech and handwriting recognition.

Original:  once, as in the graph, T a , of ﬁgure 2a.
Revised: See e.g. Hannun (2017) for background on the mechanics and motivation of CTC. To demonstrate the utility of our framework, we show how to construct these criteria from operations on simpler WFSTs following the formulation in equation 1. The graphs A and Z used by the ASG loss function can be constructed from operations on simpler graphs. The ASG criterion assumes that emitting a new token requires consuming the token at least once, as in the graph, T a , of ﬁgure 2a.

Original: The full token graph is the closure of the union of the individual token graphs.
Revised: The full token graph is the closure of the union of the individual token graphs.

Original: Assuming an alphabet of { a, b, c } gives T =
Revised: Assuming an alphabet of { a, b, c } gives T =

Original: ( T a + T b + T c ) ∗ .
Revised: ( T a + T b + T c ) ∗ .

Original: Composing
Revised: Composing

Original: T with the label graph Y (ﬁg. 2b) of “ab” results in the alignment graph (ﬁg. 2c).
Revised: T with the label graph Y (ﬁg. 2b) of “ab” results in the alignment graph (ﬁg. 2c).

Original: The alignment graph speciﬁes the allowed correspondences between arbitrary length paths and the desired label.
Revised: The alignment graph speciﬁes the allowed correspondences between arbitrary length paths and the desired label.

Original: Intersecting the alignment graph with the emissions graph E X (ﬁg. 2d) results in the constrained graph A (ﬁg. 2e).
Revised: Intersecting the alignment graph with the emissions graph E X (ﬁg. 2d) results in the constrained graph A (ﬁg. 2e).

Original: The arc weights of E X depend on X and can be the output of a deep neural network.
Revised: The arc weights of E X depend on X and can be the output of a deep neural network.

Original: Here Z is simply the emissions graph.
Revised: Here Z is simply the emissions graph.

Original: We may add bigram transitions by including a bigram transition graph, B , as in ﬁgure 4a.
Revised: We may add bigram transitions by including a bigram transition graph, B , as in ﬁgure 4a.

Original: In summary, the graph operations are:
Revised: In summary, the graph operations are:

Original: [Equation]
Revised: [Equation]

Original: The primary difference between ASG and CTC is the inclusion of a blank  token, < b > ,  represented by the graph in ﬁgure 3a.
Revised: The primary differences between ASG and CTC are the use of bigram transitions in the former and the inclusion of a blank token in the latter. The blank token, < b > , is represented by the graph in ﬁgure 3a.

Original: Constructing CTC amounts to including the blank token graph when constructing the full token graph T .
Revised: Constructing CTC amounts to including the blank token graph when constructing the full token graph T .

Original: The intersection T ◦Y then results in the CTC alignment graph (ﬁg. 3b).
Revised: The intersection T ◦ Y then results in the CTC alignment graph (ﬁg. 3b).

Original: Note, this version of CTC does not force transitions on < b > between repeats tokens.
Revised: Note, this version of CTC does not force transitions on < b > between repeats tokens.

Original: This requires remembering the previous state and hence is more involved (see Appendix A.1 for details).
Revised: This requires remembering the previous state and hence is more involved (see Appendix A.1 for details).

Original: A beneﬁt of constructing sequence-level criteria by composing operations on simpler graphs is the access to a large design space of loss functions with which we can encode useful priors.
Revised: A beneﬁt of constructing sequence-level criteria by composing operations on simpler graphs is the access to a large design space of loss functions with which we can encode useful priors.

Original: For example we could construct a “spike” CTC, a “duration-limited” CTC, or an “equally spaced” CTC by substituting the appropriate token graphs into equation 2 (see Appendix A.2 for details).
Revised: For example we could construct a “spike” CTC, a “duration-limited” CTC, or an “equally spaced” CTC by substituting the appropriate token graphs into equation 2 (see Appendix A.2 for details).

Original: 4. T RANSITIONS
Revised: 4. T RANSITIONS

Original: The ASG criterion was constructed with a bigram transition graph B .
Revised: The ASG criterion was constructed with a bigram transition graph B .

Original: We can use B to add transitions to CTC with the same operations in equation 2 but with the addition of the blank token. 
Revised: We can use B to add transitions to CTC with the same operations in equation 2 but with the addition of the blank token. [Figure]

Original: Dense n -gram graphs require O ( C n − 1 ) states and O ( C n ) arcs for C tokens, computationally intractable as n and C grow.
Revised: Dense n -gram graphs require O ( C n − 1 ) states and O ( C n ) arcs for C tokens, computationally intractable as n and C grow.

Original: The dense graph also suffers from sample efﬁciency problems as most transitions will not be observed in the training set.
Revised: The dense graph also suffers from sample efﬁciency problems as most transitions will not be observed in the training set.

Original: We can alleviate these issues with pruning and back-off (Katz, 1987) which can be encoded in a WFST (Mohri et al., 2008).
Revised: We can alleviate these issues with pruning and back-off (Katz, 1987) which can be encoded in a WFST (Mohri et al., 2008).

Original: Figure 4b demonstrates back-off for a trigram model.
Revised: Figure 4b demonstrates back-off for a trigram model.

Original: The trigram (a, b, a) exists so no back-off is needed.
Revised: The trigram (a, b, a) exists so no back-off is needed.

Original: Neither the trigram (a, b, c) nor the bigram (b, c) exist, so the model backs off to the 0 -gram state ( (cid:15) ) and then transitions to the unigram state (c).[Figure] [Figure] [Figure]
Revised: Neither the trigram (a, b, c) nor the bigram (b, c) exist, so the model backs off to the 0 -gram state ( (cid:15) ) and then transitions to the unigram state (c).

Original: he

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 4. M ARGINALIZED W ORD P IECE D ECOMPOSITIONS
Revised: 4. M ARGINALIZED W ORD P IECE D ECOMPOSITIONS

Original: Using word pieces has several beneﬁts over grapheme outputs without the sample and computational efﬁciency issues, and lexical constraints of outputting words directly.
Revised: Using word pieces has several beneﬁts over grapheme outputs without the sample and computational efﬁciency issues, and lexical constraints of outputting words directly.

Original: However, the set of pieces and the decomposition for a word are learned in an unsupervised and task independent manner (Kudo, 2018; Sennrich et al., 2016).
Revised: However, the set of pieces and the decomposition for a word are learned in an unsupervised and task independent manner (Kudo, 2018; Sennrich et al., 2016).

Original: These are then ﬁxed and used to train the task speciﬁc model.
Revised: These are then ﬁxed and used to train the task speciﬁc model.

Original: The word piece decomposition for a given phrase is not important, serving only as a stepping stone to more accurate models.
Revised: The word piece decomposition for a given phrase is not important, serving only as a stepping stone to more accurate models.

Original: This assumption can be made explicit by marginalizing over the set of decompositions for a target label while training the task speciﬁc model.
Revised: This assumption can be made explicit by marginalizing over the set of decompositions for a target label while training the task speciﬁc model.

Original: This allows the model to select the decomposition(s) which are most salient to the task at hand.
Revised: This allows the model to select the decomposition(s) which are most salient to the task at hand.

Original: Implementing marginalization over word piece decompositions usually requires non-trivial changes to carefully optimized implementations of existing sequence criterion.
Revised: Implementing marginalization over word piece decompositions usually requires non-trivial changes to carefully optimized implementations of existing sequence criterion.

Original: In the differentiable WFST framework this can be implemented in a plug-and-play fashion by incorporating a single lexicon graph.
Revised: In the differentiable WFST framework this can be implemented in a plug-and-play fashion by incorporating a single lexicon graph.

Original: The lexicon transducer L , which maps sequences of sub-word tokens to graphemes, is the closure of the union of the individual sub-word-to-grapheme graphs.
Revised: The lexicon transducer L , which maps sequences of sub-word tokens to graphemes, is the closure of the union of the individual sub-word-to-grapheme graphs.

Original: A composition with the label graph, L ◦ Y , gives the decomposition graph for the label y .
Revised: A composition with the label graph, L ◦ Y , gives the decomposition graph for the label y .

Original: Figure 5 shows an example for the label “the”.
Revised: Figure 5 shows an example for the label “the”.

Original: The decomposition graph can be used in place of the label graph to construct any down stream sequence criterion.
Revised: The decomposition graph can be used in place of the label graph to construct any down stream sequence criterion.

Original: For example, the constrained graph of equation 2 becomes:
Revised: For example, the constrained graph of equation 2 becomes:

Original: [Equation]
Revised: [Equation]

Original: where L =
Revised: where L =

Original: ( L 1 + . . .
Revised: ( L 1 + . . .

Original: + L C ) ∗ and C is the number of word piece tokens.
Revised: + L C ) ∗ and C is the number of word piece tokens.

Original: 4. C ONVOLUTIONAL WFST S
Revised: 4. C ONVOLUTIONAL WFST S

Original: Thus far we have only discussed applications of WFSTs to combine the output of other models.
Revised: Thus far we have only discussed applications of WFSTs to combine the output of other models.

Original: Figure 6 demonstrates a convolutional WFST layer which can function as an arbitrary layer in a tensor-based architecture.
Revised: Figure 6 demonstrates a convolutional WFST layer which can function as an arbitrary layer in a tensor-based architecture.

Original: Like a standard convolution, the WFST convolution is speciﬁed by a set of kernels, a receptive ﬁeld size k , and a stride s .
Revised: Like a standard convolution, the WFST convolution is speciﬁed by a set of kernels, a receptive ﬁeld size k , and a stride s .

Original: Each kernel, K i , is a WFST.
Revised: Each kernel, K i , is a WFST.

Original: Let the hidden units in a receptive ﬁeld at position t be given by H t : t + k  .
Revised: Let the hidden units in a receptive ﬁeld at position t be given by H t : t + k ∈ R d × k .

Revised: The receptive ﬁeld graph, R H t : t + k , is a linear graph with k nodes and d edges between consecutive nodes.

Original: The hidden units are the edge weights inthe receptive ﬁeld graph, R H t : t + k .
Revised: The activations in H t : t + k yield the edge weights in the receptive ﬁeld graph.

Original: The output of the i -th kernel applied to position t is
Revised: The output of the i -th kernel applied to position t is

Original: [Equation]
Revised: [Equation]

Revised: ◦R

Revised: [Equation]

Revised: k

Revised: [Equation]

Original:  the forward score of the receptive ﬁeld graph composed with the kernel graph.
Revised: [Table] [Table] the forward score of the receptive ﬁeld graph composed with the kernel graph [Table]

Revised: The structure and labels of the kernel WFSTs are problem speciﬁc.

Original: The kernel transducers can be structured to impose a desired correspondence.
Revised: For example, the kernel transducers can be structured to impose a desired correspondence.

Original: For example, we construct kernels which map lower-level tokens such as letters to higher level tokens like word pieces.
Revised: We construct kernels which transduce lower-level tokens such as letters to higher level tokens like word pieces.

Original: Without any other constraints, the interpretation of the input as letter scores and the output as word piece scores is implicit in the structure of the graphs, and can be learned by the model up to any indistinguishable permutation.
Revised: Without any other constraints, the interpretation of the input as letter scores and the output as word piece scores is implicit in the structure of the graphs [Table] and can be learned by the model up to any indistinguishable permutation.

Original: The kernel graphs themselves have edge weights which, since equation 4 is differentiable, are easily learned along with the other parameters of the model.[Table] [Table]
Revised: The kernel graphs themselves have edge weights which, since equation 4 is differentiable, can be initialized randomly and learned along with the other parameters of the model.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We study several of the algorithms from section 4 with experiments in ofﬂine handwriting recognition (HWR) and automatic speech recognition (ASR).
Revised: We study several of the algorithms from section 4 with experiments in ofﬂine handwriting recognition (HWR) and automatic speech recognition (ASR).

Revised: For both applications, we use a variation of the time-depth separable (TDS) convolutional architecture (Hannun et al., 2019).

Original: A  detailed description of the network architectures, optimization procedures and data processingused is in Appendix B.
Revised: A more detailed description of the network architectures, optimization procedures and data processing is in Appendix B.

Original: We report character error rate (CER) without the use of an external language model or lexical constraint.
Revised: We report character error rate (CER) without the use of an external language model or lexical constraint.

Original: Code to reproduce our experiments is open-source and available at https://www.anonymized.com .
Revised: Code to reproduce our experiments is open-source and available at https://www.anonymized.com .

Revised: All of the tensor based operations are on the GPU while the

Revised: WFST operations are on the CPU.

Revised: The WFST operations are executed in parallel over the examples in the batch.

Original: Ofﬂine Handwriting Recognition To facilitate comparisons to prior academic studies, we test our approach on handwritten line recognition using the IAM database (Marti & Bunke, 2002).
Revised: Ofﬂine Handwriting Recognition To facilitate comparisons to prior academic studies, we test our approach on handwritten line recognition using the IAM database (Marti & Bunke, 2002).

Original: We use the standard training set and report results on the ﬁrst validation set available with the data set distribution.
Revised: We use the standard training set and report results on the ﬁrst validation set available with the data set distribution.

Original: The splits have changed from those used by prior work (Graves et al., 2008; Pham et al., 2014; Voigtlaender et al., 2016) which makes comparisons to such work less meaningful.
Revised: The splits have changed from those used by prior work (Graves et al., 2008; Pham et al., 2014; Voigtlaender et al., 2016) which makes comparisons to such work less meaningful.

Original: Automatic Speech Recognition Speech recognition experiments are performed on the LibriSpeech (Panayotov et al., 2015) and Wall Street Journal (WSJ) (Paul & Baker, 1992) corpora.
Revised: Automatic Speech Recognition Speech recognition experiments are performed on the LibriSpeech (Panayotov et al., 2015) and Wall Street Journal (WSJ) (Paul & Baker, 1992) corpora.

Original: For
Revised: For

Original: LibriSpeech, we train with the “train-clean-100” subset and report results on the clean validation set.
Revised: LibriSpeech, we train with the “train-clean-100” subset and report results on the clean validation set.

Original: For WSJ we train on the full corpus and report results on the “dev-93” validation set.
Revised: For WSJ we train on the full corpus and report results on the “dev-93” validation set.

Original: 5. T RANSITIONS
Revised: 5. T RANSITIONS

Original: Table 1 shows that including transitions can improve CER substantially, particularly when no blank token is used.
Revised: Table 1 shows that including transitions can improve CER substantially, particularly when no blank token is used.

Original: We speculate that bigram transitions help in this case primarily because of the duration model they provide.
Revised: We speculate that bigram transitions help in this case primarily because of the duration model they provide.

Original: As previously observed (Graves et al., 2006), with a blank token, the emissions of non-blank outputs spike at a single frame, and no duration model is needed.
Revised: As previously observed (Graves et al., 2006), with a blank token, the emissions of non-blank outputs spike at a single frame, and no duration model is needed.

Original: In Table 2, we see that pruning with back-off results in no discernible loss in accuracy over a dense transition model while yielding substantial run-time improvements.
Revised: In Table 2, we see that pruning with back-off results in no discernible loss in accuracy over a dense transition model while yielding substantial run-time improvements.

Original: With 1000 word pieces, the epoch time using a graph with pruned bigrams is nearly two orders of magnitude faster than that of the dense graph. 
Revised: With 1000 word pieces, the epoch time using a graph with pruned bigrams is nearly two orders of magnitude faster than that of the dense graph. [Figure]

Original: 5. M ARGINALIZED W ORD P IECE D ECOMPOSITIONS
Revised: 5. M ARGINALIZED W ORD P IECE D ECOMPOSITIONS

Original: We experiment with marginalization over word piece decompositions in both ASR and HWR.
Revised: We experiment with marginalization over word piece decompositions in both ASR and HWR.

Original: We use the SentencePiece toolkit (Kudo & Richardson, 2018) to compute the token set from the training set text.
Revised: We use the SentencePiece toolkit (Kudo & Richardson, 2018) to compute the token set from the training set text.

Original: For all datasets, larger token set sizes result in worse CER, though the effect is mild on
Revised: For all datasets, larger token set sizes result in worse CER, though the effect is mild on

Original: LibriSpeech and more pronounced on IAM.
Revised: LibriSpeech and more pronounced on IAM.

Original: Appendix ﬁgure 10 suggests this is primarily due to the different dataset sizes.
Revised: Appendix ﬁgure 11 suggests this is primarily due to the different dataset sizes.

Original: Figures 7a,7b, and 7c demonstrate that word piece marginalization recovers some of the accuracy lost when using larger token sets.
Revised: Figures 7a,7b and 7c demonstrate that word piece marginalization recovers some of the accuracy lost when using larger token sets.

Original: We examine the Viterbi decomposition on the LibriSpeech training set.
Revised: We examine the Viterbi decomposition on the LibriSpeech training set.

Original: Appendix table 4 shows that in some cases the most frequently selected decomposition for a word is more phonologically plausible than the SentencePiece decomposition.
Revised: Appendix table 4 shows that in some cases the most frequently selected decomposition for a word is more phonologically plausible than the SentencePiece decomposition.

Original: For IAM, we ﬁnd that the model favors decompositions using lower-level output tokens but does still rely on higher-level word pieces in many cases.
Revised: For IAM, we ﬁnd that the model favors decompositions using lower-level output tokens but does still rely on higher-level word pieces in many cases.

Original: We also explore the interaction between subsampling in the network and marginalization.
Revised: We also explore the interaction between subsampling in the network and marginalization.

Original: One beneﬁt of marginalization is the ability to dynamically choose the decomposition suitable to a given frame rate as demonstrated by 7d.
Revised: One beneﬁt of marginalization is the ability to dynamically choose the decomposition suitable to a given frame rate as demonstrated by 7d.

Original: We examine the Viterbi decomposition for a few examples from the training set in ﬁgure 8.
Revised: We examine the Viterbi decomposition for a few examples from the training set in ﬁgure 8.

Original: We see that a single model dynamically switches between different decompositions depending on the input.
Revised: We see that a single model dynamically switches between different decompositions depending on the input.

Original: If the handwriting is tightly spaced, for example, a higher-level token is preferred.
Revised: If the handwriting is tightly spaced, for example, a higher-level token is preferred.

Original: [Figure]

Original: 5. C ONVOLUTIONAL WFST S
Revised: 5. C ONVOLUTIONAL WFST S

Original: [Table] [Figure] [Table] We experiment with the convolutional WFST layer on the IAM data set.
Revised: We experiment with the convolutional WFST layer on the IAM data set.

Original: The kernels in the WFST layer transduce letters to 200 word pieces.
Revised: The kernels in the WFST layer transduce letters to 200 word pieces.

Original: We use a CTC-style graph structure(ﬁg. 3b) for the kernels and hence the input to the WFST has an additional dimension for the < b > token. 
Revised: We use a CTC-style graph structure for the kernels and hence the input to the WFST has an additional dimension for the < b > token. Figure 9 gives and example of a kernel and receptive ﬁeld WFST.

Original: We place the WFST convolution in between the third and fourth TDS groups.
Revised: We place the WFST convolution in between the third and fourth TDS groups.

Original: The model is trained to predict 1000 word pieces at the output layer.
Revised: The model is trained to predict 1000 word pieces at the output layer.

Original: We compare the WFST convolution with a traditional convolution in the same position.
Revised: We compare the WFST convolution with a traditional convolution in the same position.

Original: Both convolutional layers have a kernel width k = 5 , a stride of 4 , c i = 80 input channels, and c o = 200 output channels.

Original: Table 3 demonstrates that the WFST convolution outperforms the traditional convolution.
Revised: Table 3 demonstrates that the WFST convolution outperforms the traditional convolution.

Original: The WFST convolution is also less expensive as the number of parameters and the number of operations needed scale with the lengths of the word pieces in letters( w in table 3) rather than the number of input channels.
Revised: The WFST convolution is also less expensive as the number of parameters and the number of operations needed scale with the lengths of the word pieces in letters rather than the number of input channels.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown that combining automatic differentiation with WFSTs is not only possible, but a promising route to new learning algorithms and architectures.
Revised: We have shown that combining automatic differentiation with WFSTs is not only possible, but a promising route to new learning algorithms and architectures.

Original: The design space of structured loss functions that may be encoded and efﬁciently implemented through the use of differentiable WFSTs is vast.
Revised: The design space of structured loss <b>: ε / 1 t:th/ t: ε / 2 <b>: ε / 3 h: ε / <b>: ε / h: ε /0 h: ε / 4 <b>: ε / <b>: ε / (a) Kernel WFST  a/-3. z/0.6 <b>/3. a/-4.

Revised: z/-1.0 <b>/0.

Revised: a/3.

Revised: z/-4.7 <b>/1.

Revised: a/-3.

Revised: z/-4.1 <

Revised: b>/3.

Revised: [Figure]

Revised: [Table]

Revised: functions that may be encoded and efﬁciently implemented through the use of differentiable WFSTs is vast.

Original: We have only begun to explore it.
Revised: We have only begun to explore it

Original: We see many other related and exciting paths for future work in automatic differentiation with WFSTs.
Revised: We see many other related and exciting paths for future work in automatic differentiation with WFSTs.

Original: First, we aim to continue to bridge the gap between static computations with WFSTs while performing inference and dynamic computations with WFSTs during model training.
Revised: First, we aim to continue to bridge the gap between static computations with WFSTs while performing inference and dynamic computations with WFSTs during model training.

Original: This will require further optimizations and potentially approximations when integrating higher level lexical and language modeling constraints.
Revised: This will require further optimizations and potentially approximations when integrating higher level lexical and language modeling constraints.

Original: Second, the use of WFSTs as a substitute for tensor-based layers in a deep architecture is intriguing.
Revised: Second, the use of WFSTs as a substitute for tensor-based layers in a deep architecture is intriguing.

Original: This may be more effective than traditional layers in imposing prior knowledge on the relationship between higher-level and lower-level representations or in discovering useful representations from data.
Revised: This may be more effective than traditional layers in imposing prior knowledge on the relationship between higher-level and lower-level representations or in discovering useful representations from data.

Original: A A LGORITHMS A. R EPEAT T OKENS IN CTC
Revised: A A LGORITHMS A. R EPEAT T OKENS IN CTC

Original: The blank token in CTC ( < b > ) also serves to disambiguate consecutive repeat tokens in the output from a single token corresponding to two input frames.
Revised: The blank token in CTC ( < b > ) also serves to disambiguate consecutive repeat tokens in the output from a single token corresponding to two input frames.

Original: Enforcing this construction in the WFST framework requires keeping track of the previously generated token and only allowing transitions on a new token or on < b > .
Revised: Enforcing this construction in the WFST framework requires keeping track of the previously generated token and only allowing transitions on a new token or on < b > .

Original: Let T 1 , . .
Revised: Let T 1 , . .

Original: , T C be the individual token graphs, including the blank token graph as in ﬁgure 3a.
Revised: , T C be the individual token graphs, including the blank token graph as in ﬁgure 3a.

Original: The full token graph to disallow repeat consecutive transitions is given by
Revised: The full token graph to disallow repeat consecutive transitions is given by

Original: [Equation]
Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Original: In practice, constructing the graph with equation 5 will result in more states and arcs than are needed.
Revised: In practice, constructing the graph with equation 5 will result in more states and arcs than are needed.

Original: Instead, we construct the graph directly.
Revised: Instead, we construct the graph directly.

Original: However, we note that these operations may be used along with (cid:15) removal and state minimization to keep the resulting token graph T small.
Revised: However, we note that these operations may be used along with (cid:15) removal and state minimization to keep the resulting token graph T small.

Original: A. P RIORS WITH CTC
Revised: A. P RIORS WITH CTC

Original: The individual token graphs in CTC do not assume much about the duration or spacing of the tokens at the input level.
Revised: The individual token graphs in CTC do not assume much about the duration or spacing of the tokens at the input level.

Original: The graph in ﬁgure 2a states only that non-blank tokens must align to at least one frame.
Revised: The graph in ﬁgure 2a states only that non-blank tokens must align to at least one frame.

Original: The graph in ﬁgure 3a states that the < b > token is optional.
Revised: The graph in ﬁgure 3a states that the < b > token is optional.

Original: The differentiable WFST framework simpliﬁes the construction of variations of these token-level graphs with potentially useful alternative assumptions.
Revised: The differentiable WFST framework simpliﬁes the construction of variations of these token-level graphs with potentially useful alternative assumptions.

Original: For example, we can construct a “spike”
Revised: For example, we can construct a “spike”

Original: [Figure]
Revised: CTC which only allows a single repetition of a non-blank label by using the graph in ﬁgure 10a.

Original: Similarly, a “duration-limited” CTC could use a token graph as in ﬁgure 9b.
Revised: Similarly, a “duration-limited” CTC could use a token graph as in ﬁgure 10b.

Original: In this case, the duration of a token is limited to be between one and two input frames.
Revised: In this case, the duration of a token is limited to be between one and two input frames.

Original: We could specify a range of allowed distances between non-blank tokens using the blank token graph in ﬁgure 9c.
Revised: We could specify a range of allowed distances between non-blank tokens using the blank token graph in ﬁgure 10c.

Original: These may all be mixed and matched on a per-token basis.
Revised: These may all be mixed and matched on a per-token basis.

Original: B E XPERIMENTAL S ETUP
Revised: B E XPERIMENTAL S ETUP

Original: In both handwriting recognition and speech recognition experiments, we use a variation of the timedepth separable (TDS) convolutional architecture (Hannun et al., 2019).
Revised: In both handwriting recognition and speech recognition experiments, we use a variation of the timedepth separable (TDS) convolutional architecture (Hannun et al., 2019).

Original: The code and settings needed to reproduce our experiments are available at https://www.anonymized.com . 
Revised: The code and settings needed to reproduce our experiments are available at https://www.anonymized.com . [Table]

Original: B. H ANDWRITING R ECOGNITION
Revised: B. H ANDWRITING R ECOGNITION

Original: For experiments in HWR, we propose a variant of the TDS model which preserves the 2D structure in the image.
Revised: For experiments in HWR, we propose a variant of the TDS model which preserves the 2D structure in the image.

Original: The 2D TDS block ﬁrst applies a 3D convolution with kernel size 1 × k h × k w on a 4dimensional input of size c × d × h ×
Revised: The 2D TDS block ﬁrst applies a 3D convolution with kernel size 1 × k h × k w on a 4dimensional input of size c × d × h ×

Original: w where c and d are the separated channel and depth dimensions and w and h are the height and width of the image respectively.
Revised: w where c and d are the separated channel and depth dimensions and w and h are the height and width of the image respectively.

Original: Following the 3D convolution, is a 1 × 1 convolution with c × d input and output channels.
Revised: Following the 3D convolution, is a 1 × 1 convolution with c × d input and output channels.

Original: We apply dropout, the ReLU non-linearity and residual connections as in the original TDS architecture.
Revised: We apply dropout, the ReLU non-linearity and residual connections as in the original TDS architecture.

Original: In place of layer normalization, we use 2D instance normalization (Isola et al., 2017) with a learned afﬁne transformation.
Revised: In place of layer normalization, we use 2D instance normalization (Isola et al., 2017) with a learned afﬁne transformation.

Original: Unless otherwise noted, we use 4 groups of three TDS blocks each.
Revised: Unless otherwise noted, we use 4 groups of three TDS blocks each.

Original: In between each TDS block we apply a standard 2D convolution and optionally sub-sample the image in the height and width dimensions.
Revised: In between each TDS block we apply a standard 2D convolution and optionally sub-sample the image in the height and width dimensions.

Original: For letter-based experiments we sub-sample the height by 2 and the width by 2 only after the ﬁrst two groups.
Revised: For letter-based experiments we sub-sample the height by 2 and the width by 2 only after the ﬁrst two groups.

Original: For word piece experiments we sub-sample both the height and width by a factor of 2 after every group for an overall sub-sampling factor of 16 × in each dimension.
Revised: For word piece experiments we sub-sample both the height and width by a factor of 2 after every group for an overall sub-sampling factor of 16 × in each dimension.

Original: We use a uniform kernel size of 5 × 7 .
Revised: We use a uniform kernel size of 5 × 7 .

Original: We start the number of channels ( c × d ) at 8 and increase it to 16 ,and 64 in between each TDS group.
Revised: We start the number of channels ( c × d ) at 8 and increase it to 16 ,and 64 in between each TDS group.

Original: For optimization we use a simple stochastic gradient descent with an initial learning rate of 10 −which is reduced every 100 epochs by a factor of 2 .
Revised: For optimization we use a simple stochastic gradient descent with an initial learning rate of 10 −which is reduced every 100 epochs by a factor of 2 .

Original: All models are trained for 400 epochs and we save the model which achieves the best results on the validation set during training.
Revised: All models are trained for 400 epochs and we save the model which achieves the best results on the validation set during training.

Original: We use a mini-batch size of 32 with distributed data parallel training.
Revised: We use a mini-batch size of 32 with distributed data parallel training.

Original: Bounding boxes for the line-level segmentation for each page are provided along with the ground truth texts.
Revised: Bounding boxes for the line-level segmentation for each page are provided along with the ground truth texts.

Original: We perform no other data pre-processing other than a simple normalization, removing the global pixel mean and standard deviation from each image.
Revised: We perform no other data pre-processing other than a simple normalization, removing the global pixel mean and standard deviation from each image.

Original: We use three forms of data augmentation–a random resize and crop, a random rotation, and a random color change.
Revised: We use three forms of data augmentation–a random resize and crop, a random rotation, and a random color change.

Original: B. S PEECH R ECOGNITION
Revised: B. S PEECH R ECOGNITION

Original: For experiments in speech recognition, we use a 1D TDS architecture with 3 groups each containing 5 TDS blocks.
Revised: For experiments in speech recognition, we use a 1D TDS architecture with 3 groups each containing 5 TDS blocks.

Original: In between each group we sub-sample the input in the time-dimension by a factor of 2 for an overall sub-sampling factor of 8.
Revised: In between each group we sub-sample the input in the time-dimension by a factor of 2 for an overall sub-sampling factor of 8.

Original: The TDS blocks all use a kernel width of 5 .
Revised: The TDS blocks all use a kernel width of 5 .

Original: The ﬁrst group has 4 channels, and we double the number of channels in between each group.
Revised: The ﬁrst group has 4 channels, and we double the number of channels in between each group.

Original: For optimization we use stochastic gradient descent.
Revised: For optimization we use stochastic gradient descent.

Original: Initial learning rates are tuned for each model type in the range 10 − 2 to 1 .
Revised: Initial learning rates are tuned for each model type in the range 10 − 2 to 1 .

Original: The learning rate is reduced every 100 epochs by a factor of 2 .
Revised: The learning rate is reduced every 100 epochs by a factor of 2 .

Original: All models are trained for 400 epochs and we save the model which achieves the best results on the validation set during training.
Revised: All models are trained for 400 epochs and we save the model which achieves the best results on the validation set during training.

Original: We use a mini-batch size of 16 with distributed data parallel training.
Revised: We use a mini-batch size of 16 with distributed data parallel training.

Revised: For features, we use 80 log-mel-scale ﬁlter banks on windows of 20 milliseconds every 10 milliseconds.

Revised: We normalize individual examples by removing the mean and dividing by the standard deviation.

Revised: We apply SpecAugment for regularization following the “LD” policy but without timewarping (Park et al., 2019).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N u m b e r o f w o r d p i ece s
Revised: N u m b e r o f w o r d p i ece s

Original: WSJ
Revised: WSJ

Original: LibriSpeech
Revised: LibriSpeech

Original: Word

Original: Most common

Original: SentencePiece

Original: Decomposition

Original: Count Decomposition Count able a, ble , able wind w, in, d wi, n, d single si, ng, le , s, ing, le move mo, ve mov, e ring ri, ng r, ing For features, we use 80 log-mel-scale ﬁlter banks on windows of 20 milliseconds every 10 milliseconds.
Revised: IAM

Original: We normalize individual examples by removing the mean and dividing by the standard deviation.

Original: We apply SpecAugment for regularization following the “LD” policy but without timewarping (Park et al., 2019).