Global Normalization for Streaming Speech Recognition in a Modular Framework
Abstract
We introduce the Globally Normalized Autoregressive Transducer (GNAT) for addressing the label bias problem in streaming speech recognition.
Our solution admits a tractable exact computation of the denominator for the sequence-level normalization.
Through theoretical and empirical results, we demonstrate that by switching to a globally normalized model, the word error rate gap between streaming and non-streaming speech-recognition models can be greatly reduced (by more than 50% on the Librispeech dataset).
This model is developed in a modular framework which encompasses all the common neural speech recognition models.
The modularity of this framework enables controlled comparison of modelling choices and creation of new models.
A JAX implementation of our models has been open sourced.
Introduction
Deep neural network models have been tremendously successful in the ﬁeld of automatic speech recognition (ASR).
Several different models have been proposed over the years: cross-entropy (CE) models with a deep feed-forward architecture [19], connectionist temporal classiﬁcation (CTC) models [14] with recurrent architectures such as long short-term memory (LSTM) [20] and more recently sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) [8], recurrent neural network transducer (RNN-T) [13], and hybrid autoregressive transducer (HAT) [40].
When conﬁgured in non-streaming mode, these neural ASR models have reached state-of-the-art word error rate (WER) on many tasks.
However, the WER signiﬁcantly worsens when they are operating in streaming mode.
In this paper, we argue that one main cause of this WER gap is that all the existing models are constrained to be locally normalized which makes them susceptible to the label bias problem [36, 2, 24, 4].
To address this problem we introduce new category of globally normalized models called Globally Normalized Autoregressive Transducer (GNAT).
Our contributions are:
(1) Addressing the label bias problem in streaming ASR through global normalization that signiﬁcantly closes more than 50% of the WER gap between streaming and non-streaming ASR.
(2) Efﬁcient, accelerator-friendly algorithms for the exact computation of the global normalization under the ﬁnite context assumption.
(3) A modular framework for neural ASR which encompasses all the common models (CE, CTC, LAS, RNN-T, HAT), allowing creation of new ones, and extension to their globally normalized counterparts.
Streaming Speech Recognition
For an input feature sequence x = x 1 . . .
x T , usually represented as a sequence of real valued feature vectors (such as log mel), and a ﬁnite output alphabet Σ , we wish to predict the corresponding output label sequence y =
y 1 . . .
y U , y i ∈ Σ .
We call each element x
i in x a frame , and each element y i in y an output label .
Common ASR models do not directly predict y , but rather an alignment label sequence z
= z 1 . . .
z V , z i ∈ Σ ∪ ∆ .
∆ is a ﬁnite alphabet of control labels, such as the blank label in CTC or RNN-T, or the end-of-sequence label in LAS.
There is a deterministic mapping B ( z ) : (Σ ∪ ∆) ∗ → Σ ∗ for obtaining y from z (e.g. in RNN-T, we simply remove all the blank labels from z ).
An ASR model can then be broken down in two tasks: (1) assigning a score (cid:81) i ω
( z i | x , z <i ) to each alignment sequence z , where ω ( z i | x , z <i ) > 0 is the alignment score of predicting a single alignment label; (2) ﬁnding (usually approximately) arg max y (cid:80)
z | B ( z )=
y (cid:81) i ω
( z i | x , z <i ) .
A non-streaming ASR model’s alignment score ω
( z i | x , z <i ) has access to the entire x for any i .
In contrast, a streaming model’s alignment score takes the form ω ( z i | x <t
( i ) , z <i ) : it only has access a preﬁx x <t ( i ) of the input feature sequence, where t ( i ) is the frame to which z
i is aligned .
Streaming models can be seen as special case of non-streaming models with respect to the alignment scores.
All the common neural ASR models use a locally normalized alignment score which satisﬁes the constraint (cid:80) z ∈ Σ ∪ ∆ ω ( z i =
z | x , z <i ) = 1 .
This is achieved by applying the softmax function to the last layer activations.
The local normalization constraint makes ω ( z i | x , z <i ) easily interpretable as a conditional probability distribution P ω ( z i | x , z <i ) , and thus (cid:81) i ω
( z i | x , z <i ) easily interpretable as P ω ( z | x ) .
With samples from the true data distribution P ( x , y ) , the modeling parameters are optimized by minimizing the negative log-conditional-likelihood loss E P ( x , y )
[ − log P ω ( y | x )] =
E P ( x , y ) [ − log (cid:80) z | B ( z )=
y P ω ( z | x )] .
2.1 Label Bias in Streaming ASR
For a non-streaming, locally normalized model, the negative log-conditional-likelihood loss is minimized in the limit (assuming enough model capacity) by setting ω ( z i | x , z <i ) to the true conditional probability P ( z i | x , z <i ) , leading to (cid:81) i ω
( z i | x , z <i ) being equal to the true posterior P ( z | x ) .
For a streaming model, x is replaced by x <t ( i ) in the alignment score (e.g. by using a unidirectional encoder).
Here the negative log-conditional-likelihood loss is minimized in the limit by setting ω ( z i | x <t ( i ) , z <i ) to P ( z i | x <t ( i ) , z <i ) .
As a result, the product (cid:81) i ω
( z i | x <t ( i ) , z <i ) =
(cid:81) i P ( z i | x <t ( i ) , z <i ) is in general not equal to P ( z | x ) anymore.
In other words, using a streaming locally normalized model means that the estimated alignment sequence posterior is the product of some locally normalized alignment scores which depend only on partial input x <t ( i ) , and as a result can no longer accurately represent the true conditional distribution.
This will bias the model towards predictions with low-entropy estimated posterior probabilities at each decoding step.
This degrades the model ability to revise previous decisions, a phenomenon called label bias [24, 2].
2.2 Global Normalization
Traditionally, globally normalized models such as Conditional Random Fields (CRF) [24] are used to address the label bias problem.
This paper seeks to apply global normalization to modern neural architectures that are more similar to CTC, RNN-T, or LAS, rather than traditional linear models, with the purpose of addressing label bias problem for streaming ASR models.
A globally normalized model does not constrain the alignment score ω
( z i | x , z <i ) to be locally normalized; it only requires it to be any non-negative score, as long as the denominator Z ( x ) = (cid:80) z (cid:81) i ω
( z i | x , z <i ) is ﬁnite.
The ﬁnite denominator allows us to interpret (cid:81)
i ω
( z i | x , z <i )
Z ( x ) as a conditional probability distribution P ω ( z | x ) .
Minimum negative log-conditional-likelihood training can be more expensive for globally normalized models due to the need to compute Z ( x ) and the corresponding gradients.
However with our proposed modular framework, globally normalized model training can be made practical with careful modelling choices on modern hardware.
Any locally normalized model is by deﬁnition also globally normalized, whose Z ( x ) = 1 .
While the reverse is not true in general, Appendix E shows that under non-streaming settings with a mild condition, locally and globally normalized models actually express the same class of conditional distributions.
Based on this observation, we argue that under non-streaming settings, with adequately powerful neural architectures, maximum log-conditional-likelihood training should yield behaviorly similar locally or globally normalized models, and thus similar WERs in testing.
Results from [37, 17] and our own experiments in Section 6 validate this.
A Modular Framework for Neural ASR
In this section, we introduce a modular framework for neural ASR, using the weighted ﬁnite state automaton (WFSA) formalism to calculate the conditional probabilities via alignment scores.
The modular framework clearly expresses the modelling choices and enables practical globlally normalized model training and inference.
We use the WFSA formalism as the language for describing our framework because of its succinctness and precision.
Our algorithms are implemented from scratch to run on TPUs, without using existing toolkits such as OpenFst [1], Kaldi [33] or K2 [21].
3.1 Preliminaries
We begin with an introduction to the relevant concepts and notations.
A semiring ( K , ⊕ , ⊗ , ¯ 0 , ¯ 1) consists of a set K together with an associative and commutative operation ⊕ and an associative operation ⊗ , with respective identities ¯0 and ¯1 , such that ⊗ distributes over ⊕ , and ¯ ⊗ x = x ⊗ ¯0 = ¯0 .
The real semiring ( R + , + , × , 0 , 1) is used when the weights represent probabilities.
The log semiring ( R ∪ {−∞} , ⊕ log , + , −∞ , 0) , isomorphic to the real semiring via the log mapping, is often used in practice for numerical stability.
The tropical semiring ( R ∪ {−∞} , max , + , −∞ , 0) is often used in shortest-path applications.
A weighted ﬁnite-state automaton (WFSA)
A =
(Σ , Q, i, F, ρ, E ) over a semiring K is speciﬁed by a ﬁnite alphabet Σ , a ﬁnite set of states Q , an initial state i ∈ Q , a set of ﬁnal states F ⊆ Q , a ﬁnal state weight assignment ρ : F → K , and a ﬁnite set of transitions E ⊆ Q × (Σ ∪ { (cid:15) } )
× K × Q ( (cid:15) denotes the empty label sequence).
Given a transition e ∈ E , p [ e ] denotes its origin or previous state, n [ e ] its destination or next state, o [ e ] its label, and ω [ e ] its weight.
A path π
= e 1 . . .
e k is a sequence of consecutive transitions e
i ∈ E : n [ e i − 1 ] =
p [ e i ] , i = 2 , . . .
k .
The functions n , p , and ω on transitions can be extended to paths by setting: n [ π ] = n [ e k ] and p [ π ] =
p [ e 1 ] and by deﬁning the weight of a path as the ⊗ -product of the weights of its constituent transitions: ω
[ π ] = ω [ e 1 ] ⊗ · · · ⊗ ω [ e k ]) .
An unweighted ﬁnite-state automaton (FSA) A = (Σ , Q, i, F, E ) is simply a WFSA whose transitions and ﬁnal states are all weighted by ¯1 .
Π( Q 1 , Q 2 ) is the set of all paths from a subset Q 1 ⊆ Q to a subset Q 2 ⊆ Q .
Π( Q 1 , y , Q 2 ) is the subset of all paths of Π( Q 1 , Q 2 ) with label sequence y
=
y 1 . . .
y U , y i ∈ Σ .
A path in Π( { i } , F ) is said to be accepting or successful .
The weight associated by WFSA A to any label sequence y is given by A ( y ) = (cid:76) π ∈ Π( { i } , y ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
The weight of A is the ⊕ -sum of weights of all accepting paths W ( A ) =
(cid:76) π ∈ Π( { i } ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
For a semiring K where ⊗ is also commutative, the intersection (or Hadamard product) of two WFSA A 1 and A 2 is deﬁned as: [Equation]
y ) ⊗ A 2 ( y ) .
[28] gives an algorithm to compute the intersection.
We can view y as a WFSA that accepts only y with weight ¯1 , then A ( y )
=
W ( A ∩ y ) .
3.2 Probabilistic Modeling and Inference on Acyclic Recognition Lattices
For any feature sequence x = x 1 . . .
x T , we use a model with trainable parameters θ to induce a recognition lattice WFSA A θ, x = (Σ , Q θ, x , i θ, x , F θ, x , ρ θ, x , E θ, x ) .
For a label sequence y =
y 1 . . .
y U , the weight associated with it,
A θ, x ( y ) , is interpreted under the log semiring as the unnormalized negative log conditional probability P
[Equation]
The recognition lattice A θ, x is designed to be acyclic, and therefore the weight of the automata in both the numerator and denominator above can be efﬁciently computed by visiting the states of the corresponding WFSA in topological order [27].
See Appendix C for our accelerator friendly version of this algorithm.
We can thus train an ASR model by minimizing the negative log-conditionallikelihood on the training corpus D , and choosing θ (cid:63) =
arg min θ E ( x , y ) ∈D [ − log( P θ ( y | x ))]
= arg min θ E ( x , y ) ∈D
[ W ( A θ, x ) − W ( A θ, x ∩ y )] .
[Equation]
In general, there can be more than one path in A θ, x that accepts the same y .
During inference, ﬁnding the optimal ˆ y = arg max y P θ ( y | x ) requires running the potentially expensive WFSA disambiguation algorithm [30] on A θ, x .
As a cheaper approximation, we instead look for the shortest path ˆ π in A θ, x under the tropical semiring, and use the corresponding label sequence as the prediction, again using the standard shortest path algorithm for an acyclic WFSA [27].
3.3 Inducing the WFSA
Our framework decomposes the sequence prediction task in ASR into three components, each playing a speciﬁc role in inducing the recognition lattice
A θ, x .
• The context dependency FSA C
=
(Σ , Q C , i C , F C , E C ) is an (cid:15) -free, unweighted FSA, whose states encode the history of the label sequence produced so far.
C is ﬁxed for a given model, independent of input x . •
The alignment lattice FSA L T =
(Σ , Q T , i T , F T , E T ) is an acyclic, unweighted FSA, whose states encode the alignment between input frames x and output labels y .
L T depends on only the length T of input x . •
The weight function ω θ, x : Q T × Q C × (Σ ∪ (cid:15) )
→ K .
ω θ, x is the only component that contains trainable parameters and requires full access to x .
This function deﬁnes the transition weights in the recognition lattice A θ, x .
We will discuss how one can deﬁne these components in detail in the next section.
With ( C, L T , ω θ, x ) given, the recognition lattice A θ, x is deﬁned as follows:
[Equation]
c ) (cid:1) | y ∈ Σ , ( q a , y, q (cid:48) a ) ∈ E T , ( q c , y, q (cid:48) c ) ∈ E C (cid:111) ∪ (cid:110)(cid:0) ( q a , q c ) , (cid:15), ω θ, x ( q a , q c , (cid:15) ) , ( q (cid:48) a , q c ) (cid:1) | (
[Equation]
a
[Equation]
(cid:15), q (cid:48)
a [Equation] In other words, the topology (states and unweighted transitions) of the recognition lattice
A θ, x is the same as the FSA intersection L T ∩ C (see Figure 1c for a concrete example); and the transition weights are deﬁned using ω θ, x . The (cid:15) -freeness of C and the acyclicity of L T implies that the recognition lattice A θ, x is also acyclic.
Components of a GNAT Model
In this section, we deﬁne globally normalized autoregressive transducer (GNAT) through the framework above, by specifying each model component.
4.1 Context Dependency
GNAT uses an n -gram context-dependency deﬁned by C n = (Σ , Q n , i n , F n , E n ) , where Q n = Σ ≤ n corresponds to a label history of length up to n .
The initial state i n = (cid:15) is the empty label sequence.
The transitions E n ⊆ Q n × Σ × Q n correspond to truncated concatenation: E n = (cid:8) ( q, y, q (cid:48) )
| q ∈ Q, y ∈ Σ (cid:9) where q (cid:48) is the sufﬁx of qy with length at most n .
For example, when n = 2 , the transition ( ab, c, bc ) goes from state ab to state bc with label c .
All states are ﬁnal: F n = Q n .
See Figure 1a for the FSA C 1 when Σ = { a, b } .
Appendix C demonstrates how C n can be efﬁciently intersected during the shortest distance computation thanks to the highly regular structure of C n .
Although not studied in this paper, our modular framework makes it easy to switch to a more sophisticated context dependency, such as clustered histories often used for context-dependent phone models, or a variable context length as used in n -gram language models [29].
(cid:15) a b a b a b a b
[Figure]
[Figure]
3 Σ (cid:15) Σ (cid:15) Σ (cid:15) (
[Figure]
alignment lattice with T
[Figure]
0 ,(cid:15) 1 ,a 1 ,(cid:15) 1 ,b 2 ,(cid:15) 2 ,a 2 ,b 3 ,(cid:15) 3 ,a 3 ,b (cid:15) a b (cid:15) a b (cid:15) a b (cid:15) a b (cid:15) a b (cid:15) a b (cid:15) a b (0 , 0) (0 , 1) (0 , 2) (1 , 0) (1 , 1) (1 , 2) (2 , 0) (2 , 1) (2 , 2) (3 , 0) (cid:15)
[Figure]
(cid:15)
[Figure]
[Figure]
(cid:15)
[Figure]
4.2 Alignment Lattices
Given the input feature sequence length T , the alignment lattice FSA L T deﬁnes all the possible alignments between the feature sequence and allowed label sequences.
Since the feature sequence length T usually differs from the label sequence length U , many different alignments between the feature sequence and a label sequence can be deﬁned.
The states in an alignment lattice FSA encode how the next label or (cid:15) -transition corresponds to some position in the feature sequence.
We can choose different structures for L T by encoding one or both of the positions in the feature sequence and the label sequence.
A simple example is the frame dependent alignment similar to [14], where each frame is aligned to at most one label:
[Equation]
Here any state t < T represents a position in the feature sequence.
We start by aligning to the initial frame of the feature sequence, and repeatedly shift to the next frame for every subsequent label or (cid:15) -transition until all frames have been visited.
Figure 1b depicts a frame dependent alignment lattice.
To allow a label sequence longer than the feature sequence, we can use the k -constrained label and frame dependent alignment similar to [13]:
[Equation]
Here, up to k consecutive label transitions can align to any single frame.
An (cid:15) -transition is then taken to explicitly shift the alignment to the next frame.
The number of labels aligned to one frame is constrained by a constant k solely in order to impose acyclicity.
Figure 1d depicts a k -constrained label and frame dependent alignment lattice.
Some models may only depend on the position in the label sequence, similar to [8].
In this case we can bound the length of the label sequence by some function l ( T ) , and use the following label dependent alignment :
[Equation]
Here, each label can be seen as aligning to the entire feature sequence, and the (cid:15) -transition serves as an explicit termination of the label sequence.
Figure 1e depicts a label dependent alignment lattice.
4.3 Weight functions
The weight function ω θ, x translates trainable parameters into transition weights for states in the recognition lattice
A θ, x .
The choice of the weight function depends on the choice of the context and alignment lattice FSA, especially the alignment lattice where the meaning of a state directly affects how the weight function can access x .
The experiments discussed in this paper use frame dependent and k -constrained label and frame dependent alignment lattices.
In these two types of alignment lattices, a non-ﬁnal state q a in Q T contains a position τ ( q a ) in the feature sequence (the state itself in the case of frame dependent alignment lattices; the ﬁrst value in the state in the case of label and frame dependent alignment lattices).
Weight functions can thus be deﬁned in three steps,
Feed x into an encoder, such as unidirectional or bidirectional RNN, or a self-attention encoder to obtain the sequence of hidden units h of dimension D .
In the experiments we compared streaming vs non-streaming encoders.
Map a single frame of hidden units h
[ t ] and context state q c to a ( | Σ | + 1) -dimensional vector, corresponding to the unnormalized transition weights for y ∈ Σ ∪ { (cid:15) } .
Optionally locally normalize transition weights across y ∈ Σ ∪ { (cid:15) } given ( q a , q c ) .
For step 2, we experiment with the following concrete modelling choices with varying degree of parameter sharing,
Per-state linear projection ( unshared )
For every context state q c , we obtain a D × ( | Σ | + 1) projection matrix W q c and a ( | Σ | + 1) -dim
bias vector b q c from θ , and deﬁne for y ∈ Σ ∪ { (cid:15) } :
[Equation]
[Equation]
Shared linear projection with per-state embedding ( shared-emb )
We obtain from θ (a) for every context state q c a D -dimensional state embedding E q c , (b) independent of context states a D × ( | Σ | +1) projection matrix W and a ( | Σ | + 1) -dim bias vector b , and deﬁne for y ∈ Σ ∪ { (cid:15) } :
[Equation]
Shared linear projection with RNN state embedding ( shared-rnn )
Similar to shared linear projection with per-state embedding but E q c is obtained from running an RNN (e.g. LSTM) on the n -gram label sequence represented by q c .
Discussion A Modular Framework
All the existing locally normalized models can be explained within the modular framework presented in Section 3 with a particular choice of context size, alignment lattice and of course with constraining the weights to be locally normalized using softmax function.
Appendix B presents how CE, CTC, LAS, RNNT and HAT models can be expressed within this framework.
This allows controlled comparison of different components as well as creating new models by mixing different modeling choices.
Like locally and globally normalized models in general, when the weights are locally normalized, the denominator of models deﬁned in our framework is one.
Finally, our framework is very different from traditional uses of ﬁnite state machines via a cascade of weighted ﬁnite state transducer compositions.
The separation of the weight function from the automaton topology allows an arbitrarily complex, non-linear weight function to model the dependency among alignment states, context states, and output label, which is impossible with composition cascades.
The idea of weight function has been proposed before (e.g. [38]), although as we shall see in experiments, the exact architecture of a weight function plays a crucial role in model accuracy.
Related Globally Normalized Models
There is a rich literature on the applications of globally normalized models [5, 26, 35, 24], as well as theoretical studies on the label bias problem [24, 2, 12].
In ASR, there is a lot of research on applying globally normalized models [3, 7, 6, 16, 25, 45, 18].
Among these, MMI [3, 7] is the most relevant to our work.
In MMI, a sequence level score is factorized into an acoustic model (AM) score and a language model (LM) score.
The denominator is usually approximated over a pruned lattice.
Recently [34] introduced lattice-free MMI, which replaces the word level LM with a 4-gram phone LM during training.
Both the GNAT and MMI are globally normalized.
The GNAT model differs from MMI in several ways: (1) Unlike MMI, GNAT does not require an external LM, nor does it impose any constraint on how the sequence level scores are deﬁned.
In MMI training, the LM is kept frozen while the AM parameters are optimized.
In
GNAT all the model parameters are trained together.
(3) GNAT’s recognition lattice encodes the same set of weighted alignment paths during training and inference, therefore the denominator computation in GNAT is exactly matched between training and inference.
This does not hold for the standard
MMI, because by using different LMs between training and inference, the denominator computation in training time only serves as an approximation of the denominator during inference.
Implementation-wise, GNAT trains from scratch without any need for initialization or special regularization techniques as used in the lattice-free version of MMI [34].
In addition we were able to train
GNAT models with accelerators without any techniques discussed in [34].
The concept of global normalization has also been visited with deep neural networks [17, 10, 42, 44].
These models can be seen as special cases of MMI based on aspects such as how the sequence level scores are factorized, their use of ﬁxed LM for training, and the mismatching training and inference LMs.
Thus the comparison between MMI and GNAT applies here as well.
Their use of
WFST composition cascades results in weaker modelling power compared to our weight function formulation, and their use of non-streaming encoders combined with their model formulation makes globally normalized models equally expressive to the locally normalized counterparts (Appendix E).
Challenges
The main challenge with the GNAT model is its scalability to a larger number of label contexts.
At each training step, the model requires O ( | Q C | × | Σ | ) semiring multiplications and additions.
For n -gram context dependency, | Q c | = ( | Σ | n +1 − 1) / ( | Σ | − 1) , thus the computation scale exponentially by value of n .
However as shown in Appendix D, due to the particular structure of this space the practical computation and memory cost benchmarks do not scale exponentially with 0 ≤ n ≤ 2 .
We also note that large value of n might also not be necessary: The HAT model [40] reports that a Seq2Seq model with a label history of just the two previous phonemes performs on par with a similar model with a full history trained on very large voice-search corpus.
Similar observations are reported in studies with grapheme and wordpiece units [43, 11].
Due to the data sparsity there might not be enough training data to fully represent all n -gram context states, so increasing the value of n might not necessarily lead to a WER reduction.
One way of dealing with larger numbers of states is to use standard pruning techniques to keep only some of the most common states in the training data.
Experiments Data
We use the full 960 -hour Librispeech corpus [32] for experiments.
The input features are 80-dim.
log Mel extracted from a 25 ms window of the speech signal with a 10 ms shift.
The
SpecAugment library with baseline recipe parameters were used [15].
The ground truth transcription is used without any processing and tokenized by the 28 graphemes that appear in the training data.
Architecture
Attention-based architectures allows using the same parameterization for streaming and non-streaming models, thus for all the experiments we used 12 -layer Conformer encoders [15] with model dimension 512, followed by a linear layer with output dimension 640.
The Conformer parameters are set such that the only difference between streaming and non-streaming models is the right context: at each time frame t , the streaming models only access the left context (feature frames from 1 to t ), while the non-streaming models can see the entire acoustic feature sequence.
To enforce the consistency of the encoder architecture between streaming and non-streaming modes, we removed all the sub-architecture which behaved differently between these two modes.
Speciﬁcally, we removed the convolution sub-sampling layer, and also forced the stacking layers to only stack within the left context.
The baseline experiments use a shared-rnn weight function deﬁned in section 4.3.
A single layer LSTM is used with 640 cells.
The experiments with the unshared weight function use a linear layer of size ( | Q C | × | Σ | ) × 640 to project the encoder activation at each time frame into the transition weights of the recognition lattice.
In our experiments, | Σ | = 32 .
For the n -gram context dependency, | Q c | = ( | Σ | n +1 − 1) / ( | Σ | − 1) .
The experiments with the shared-emb weight function use an embedding table of size | Q c | × 128 .
Training
All models are trained on 8 × 8 TPUs with a batch size 2048 .
The training examples with more than 1961 feature frames or more than 384 labels are ﬁltered out.
We used Adam optimizer [22] ( β 1 = 0 .
9 , β 2 = 0 .
98 , and (cid:15) =
10 − 9 ) with the Transformer learning rate schedule [41] (10k warm-up steps and peak learning rate 0 .
05 / √ 512 ).
We applied the same regularization techniques and the training hyperparameters used in the baseline recipe of [15].
Evaluation
We report WER results on standard Librispeech test sets: test_clean and test_other.
The WER is either computed with sum-path algorithm or max-path algorithm.
The sum-path algorithm approximately merges the alignment hypothesis corresponding to the same label sequence preﬁx after removal of epsilons.
In ideal decoding, sum-path should result in the most likely output label sequence.
The max-path algorithm computes the highest scoring alignment path using algorithms in Appendix C.
Baselines
The RNN-T baselines are presented in the rows corresponding to the “ RNN-T ” context dependency in Table 1a.
Unlike the other rows in Table 1a, the decoder RNN in RNN-T in principle encodes an unlimited output history as context.
The standard RNN-T model effectively uses the label frame dependent alignment lattice with an inﬁnite k .
The rows corresponding to the frame dependent alignment lattice represent a slightly modiﬁed RNN-T model, where each frame is aligned to exactly one output (either a blank, or a lexical label).
The WER differences between non-streaming RNN-T baselines in Table 1a and [15] are mainly due to our modiﬁcations to the Conformer encoder for a controlled comparison against streaming models.
Choice of the context dependency
Table 1a compares effect of n -gram context dependency for n = 1 , 2 , 3 and baseline RNN-T models.
The general observation is that increasing n leads to better WER independent of the other choices of the modeling parameters.
However, the model with 3 -gram context dependency already performs on par with RNN-T baseline.
The 2 -gram context dependency perform almost on par as baseline on clean test set while still lagging on the other test set.
This is consistent with the earlier observations in [40, 43, 11].
Choice of the alignment lattice
The comparison of different alignment lattices in Table 1a suggests that this choice does not signiﬁcantly contribute to the model performance.
While there is a performance gap for 1 -gram context dependency, we do not think there is a principal argument in favor of frame dependent alignment lattice.
We speculate that this is more due to the choice of optimization parameters.
However, the choice of lattice type can have some side effects.
For example as k in k -constrained label frame dependent alignment lattice increases, the model has more [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
ability to delay its prediction to the end of the signal.
This implicit lookahead can translate into performance gains particularly for unidirectional models.
By limiting this quantity to 1, we observed that performance on clean and other sets degrades by 34 .
9% and 36 .
7% , respectively.
Choice of the weight function normalization
Table 1b examines the effect of weight normalization on non-streaming and streaming models.
Here we present models with 1 -gram and 2 -gram context dependency with a frame dependent alignment lattice.
Note that for 0 -gram context dependency with a frame dependent alignment lattice it is easy to show that locally normalized and globally normalized models are equivalent.
For non-streaming models, the normalization seems to not have an impact on the performance quality neither for 1 -gram nor for 2 -gram context dependency experiments.
This is validates our argument in the end of Section 2.2 that locally and globally normalized models express the same class of conditional distributions under this particular setting.
On the other hand, streaming models signiﬁcantly beneﬁt from global normalization: For the clean test set, the globally normalized model outperforms the locally normalized model by about 21% relative WER for 1 -gram context dependency and by about 20% relative gain for 2 -gram context dependency.
The globally normalized model with 2 -gram context dependency also beat the baseline streaming
RNN-T model in Table 1a and performs signiﬁcantly closer to the non-streaming RNN-T baseline.
The equivalent streaming RNN-T model performs 5 .
1% on test clean and the non-streaming model performs 2 .
5% on same test set.
The globally normalized model decoded with max-path algorithm performs 3 . 8% on same test set.
The globally normalized model effectively closed almost 50% of the performance gap between streaming and non-streaming models.
The reported performance for the globally normalized models is from max-path decoding, while the baselines beneﬁt from sum-path decoding.
Comparing the locally normalized models’ WER from max-path decoding in Table 1b and their counterparts in Table 1a, it is clear that sum-path decoding leads to an extra WER gain.
This gain is more signiﬁcant on test_other.
So we expect the globally model performs even better when decoded with sum-path.
The standard sum-path algorithms use several heuristics particularly for path merging and pruning.
While similar merging techniques can be applied to the globally normalized models, the pruning heuristics require several adjustments.
This is particularly due to the nature of the globally normalized models where the transition weights are not constrained and can take any value, unlike locally normalized models where the transition weights are constrained to be positive number between 0 and 1 and sum to 1 for all the weights leaving the same state in recognition lattice.
We plan to explore approaches to improve globally normalized model decoding in the future.
Choice of the weight function architecture
Finally Table 1c compares different choices of the architectures for a streaming model with 2 -gram context dependency and frame dependent alignment lattice.
While the unshared and shared-rnn architectures are very different in terms of parameter sharing among states, both perform well, though the shared-rnn architecture performs slightly better.
The shared-emb architecture performs signiﬁcantly worst than shared-rnn architecture.
Note that the shared-rnn model is able to learn common structures across states in the context dependency while shared-emb does not have such capability.
Conclusion
The GNAT model was proposed and evaluated with the focus on the label bias problem and its impact on the performance gap between streaming and non-streaming locally normalized ASR.
The ﬁnite context property of this model allows exact computation of the sequence level normalization which makes this model differ from existing globally normalized models.
Furthermore, the same property allows accelerator friendly training and inference.
We showed that the streaming models with globally normalized criteria can signiﬁcantly close the gap between streaming and non-streaming models by more than 50%.
Finally, the modular framework introduced in this paper to explain the GNAT model encompasses all the common neural speech recognition models.
This enables fair and accurate comparison of different models via controlled modelling choices and creation of new ASR models.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A An overview example
To illustrate different components of the GNAT model, here we present a toy example of designing a speech recognition for ﬁnite alphabet Σ
= { a, b } .
Given an input feature sequence x = x 1 . . .
x T , we wish to predict the corresponding output label sequence y =
y 1 . . .
y U , y i ∈ Σ .
Our objective is to create a conditional probabilistic model P ( y | x ) which assigns the highest probability to the correct label sequences for any given feature sequence.
We construct a GNAT model with the following components:
• Context dependency: 2 -gram • Alignment lattice: frame dependent • Weight function: per-state linear projection, streaming
Next we elaborate details of each of these modules and how they are integrated to create the ﬁnal space the recognition lattice
A θ, x and the probabilistic model P ( y | x ) as described in Section 3.
A.1 Context Dependency FSA
Figure 2 presents the 2 -gram context dependency C 2 .
The set of states for this space are the initial state, 1-gram states and 2-gram states:
[Equation]
With the lexicographic order, these states are indexed as follows: state state index (cid:15) a
[Equation]
aa
[Equation]
ba bb
For this particular FSA, the transitions space is
[Equation]
where q (cid:48) is the sufﬁx of qy with length at most 2 .
All 14 transitions of this space are listed in the following table: from state label to state (cid:15) a a (cid:15) b b a a aa a b ab b a ba b b bb aa a aa aa b ab ab a ba ab b bb ba a aa ba b bb bb a ba bb [Equation]
A.2 Alignment Lattice FSA
Figure 3 depicts a frame dependent alignment lattice L 4 for four frames feature sequence x .
The states of this space are:
[Equation]
(cid:15) a b ab aa ba bb a b a b b a a b a b a b a b
[Figure]
(cid:15)
[Figure] (cid:15) Σ (cid:15) [Figure] (cid:15) where 0 is the initial state and 4 is the ﬁnal state.
Every path starting from the initial state in this automaton corresponds to one possible alignment sequence of the input feature sequence.
The example FSA in Figure 1b encodes 3 4 = 81 possible alignment sequences.
An alignment path (cid:15)a(cid:15)b corresponds to the following sequence of transitions in L 4 : [Equation]
A.3 Weight Function
The weight function ω θ, x : Q T × Q C × (Σ ∪ (cid:15) )
→ K .
ω θ, x is the only trainable component of the
GNAT model which assigns a weight to every transition of the recognition lattice A θ, x .
We ﬁrst feed x = x 1 , ..., x 4 into an encoder to obtain hidden activations
h = h 1 , ..., h 4 of dimension D .
The encoder can be any neural architecture such as DNNs, CNNs, RNNs or Transformers.
Since we are interested in streaming weight function for this example, we need to make sure the encoder is also streaming.
This means h t can only depends on x 1: t .
Finally we deﬁne a D × 3 matrix W q c and a 3 -dim bias vector b q c for any q c ∈ Q c .
The weight function is then deﬁned as 3 :
[Equation]
where W q c [ y, :] is the row of W q c corresponding to label y ∈ Σ ∪ { (cid:15) } .
The total number of trainable parameters is 7 × D × 3 + 7 × 3 parameters plus the number of parameters of the encoder function.
A.4 The recognition lattice A θ, x
Given the context dependency FSA C 2 , the frame dependent alignment lattice L 4 and the weight function ω θ, x , we are ready to derive the recognition lattice
A θ, x .
The state space has 5 × 7 states:
[Equation]
The transitions in this space is speciﬁed by the state it is originated from, ( t, q c ) , the label y ∈ Σ ∪{ (cid:15) } , weight ω θ, x ( t, q c , y ) and the state the transition is ended to ( t + 1 , q (cid:48) c ) .
For the alignment sequence of our example, (cid:15)a(cid:15)b , the transitions are: from label weight to
[Equation]
, (cid:15) )
[Equation]
The product of the above weights is the score that the GNAT model assigns to the features sequence x and alignment sequence (cid:15)a(cid:15)b : score
[Equation]
The GNAT model formulates the posterior probability P ( y | x ) by ratio of two quantities:
• numerator: sum of all the score ( x , z ) where z is an alignment between x and y .
For example if y = ab , there are only 6 possible alignments: ab(cid:15)(cid:15) , a(cid:15)b(cid:15) , a(cid:15)(cid:15)b , (cid:15)ab(cid:15) , (cid:15)a(cid:15)b , (cid:15)(cid:15)ab • denominator: sum of all the score ( x , z ) where z can be any ( | Σ | + 1) T = 3 4 =sequences.
Since the numerator computation is a special case of the denominator, we only present the denominator calculation.
To follow the computation presented in Section C, we ﬁrst deﬁne forward variable α t which is a 7 -dim real-valued vector where α t [ j ] is the total alignment scores reaching to state index j (corresponding to the state indices in Q c ) at time t : [Equation] [Equation] [Equation]
[Equation]
[Equation]
5] α [Equation] [6] the initial state α 0 is a 1 -hot vector with α
[ j ] = 1 iff j = 0 , the initial state.
At every time frame t , the transition weight matrix Ω t, Σ is deﬁned for all the transitions in E A θ, x where label is an element of Σ .
This matrix is a structured matrix with only | Σ | = 2 non-zero elements per row: Ω t, Σ
= ( t, (cid:15) ) ( t, a ) ( t, b ) ( t, aa ) , ( t, ab ) ( t, ba )
( t, bb )     ( t − 1 , (cid:15) ) ω ( t, (cid:15), a ) ω
( t, (cid:15), b )    ( t − 1 , a )   ω ( t, a, a ) ω ( t, a, b )
( t − 1 , b )     ω ( t, b, a ) ω ( t, b, b ) ( t − 1 , aa )   ω ( t, aa, a ) ω ( t, aa, b )  ( t − 1 , ab )     ω ( t, ab, a ) ω ( t, ab, b ) ( t − 1 , ba )   ω ( t, ba, a ) ω ( t, ba, b )  ( t − 1 , bb )     ω ( t, bb, a ) ω ( t, bb, b )
Similarly we denote Ω t,(cid:15) to be the transition weight matrix for all the transitions in E A θ, x where label is (cid:15) .
This matrix is a diagonal matrix corresponding to the weights of the self loops: Ω t,(cid:15) = ( t, (cid:15) )
( t, a ) ( t, b ) ( t, aa ) , ( t, ab ) ( t, ba )
( t, bb )     ( t − 1 , (cid:15) )
ω ( t, (cid:15), (cid:15) )      ( t − 1 , a ) ω ( t, a, (cid:15) )
( t − 1 , b )  ω ( t, b, (cid:15) )
( t − 1 , aa )   ω ( t, aa, (cid:15) )
( t − 1 , ab )    ω ( t, ab, (cid:15) )  ( t − 1 , ba )     ω ( t, ba, (cid:15) )
( t − 1 , bb )      ω
( t, bb, (cid:15) )
For our model, the forward variable α t can be calculated given α t − 1 and the above weights matrices as:
[Equation]
since every transition at time t + 1 is either an (cid:15) transition or a non- (cid:15) transition.
Here α (cid:48) t − 1 is the transpose of forward variable α t − 1 .
Given the above iterative equation, the forward variable at time 4 is equal to:
[Equation]
and the denominator of the GNAT model is equal to (cid:80) 6 j =
α 4 [ j ] .
Replacing the real semiring with the tropical semiring in above calculation will allow us to ﬁnd the most likely alignment sequence.
B A Modular Framework
In this section we demonstrates how the existing and common neural speech recognition models can be expressed within our proposed framework.
B.1 Cross-entropy with Alignments
The conventional cross-entropy models with feed-forward neural architectures [31] deﬁne the conditional probability of label sequence y given feature sequence x by:
[Equation]
where probability factors P θ ( y t | x t ) are derived by some neural architecture parameterized by θ :
[Equation]
where h t is the encoder activation of dimension D at time frame t , W is a weight matrix of shape | Σ | × D and b is a | Σ | -dim bias vector.
The equivalent GNAT model is conﬁgured as follow:
• Context dependency: 0 -gram C• Alignment lattice: frame dependent without (cid:15) transitions • Weight function (here (cid:15) is the initial and only state of C 0 ): – ω θ, x ( q a = t, q c = (cid:15), y = y t ) (cid:44) P θ ( y = y t | x t ) – Locally normalized – Streaming
The more advanced cross-entropy models use recurrent architectures or transformers as encoder [39].
The only difference between the GNAT equivalent of these models and above conﬁguration is that whether the encoder is streaming or not.
B.2 Listen, Attend and Spell (LAS)
This model formulates the posterior probability by directly applying chain rule (Equation 1 in [8]):
[Equation]
the posterior factors are deﬁned as (Equations 6 to 8 of [8]):
[Equation]
where
[Equation]
here Listen is a bidirectional encoder function, AttentionContext is the attention network (Equations 9 to 11 of [8]).
The equivalent GNAT model is conﬁgured as follow:
• Context dependency:
M -gram where M is the length of longest label sequence in the training data.
Note that M -gram context dependency is equivalent of the tree space truncated at depth M .
• Alignment lattice: label dependent since the probability factorizes only on label sequence. • Weight function: – ω θ, x ( q a = l, q c = q, y = y l ) (cid:44) P
θ ( y = y l | x , y <l ) – Locally normalized – Non-streaming
B.3 Recurrent Neural Transducer
The RNNT model formulate the posterior probability as marginalization of alignment sequences (Equation 1 in the RNNT paper [13]):
[Equation]
where z = z 1 , · · · , z T + L is an alignment sequence, z i ∈ Σ ∪ { (cid:15) } , T is the number of acoustic frames and L is the number of labels.
The function B ( z ) =
y removes the epsilons from the alignment sequence.
The alignment posterior is factorized along the alignment path as:
[Equation]
and ﬁnally RNNT make the following assumption:
[Equation]
which means if the preﬁx of two alignments be equal after epsilon removal, the model assigns same expansion probability for the next alignment position.
The inner terms in the above equation is deﬁned (Equations 12 to 15 of the RNNT paper):
[Equation]
where h j − u is the encoder activation at time frame j − u (referred to as the transcription network in [13]) and g u is the output of the prediction network which is a simple stack of RNNs.
The equivalent GNAT model is conﬁgured as follow:
• Context dependency:
M -gram where M is the maximum value of T + L in the training data set. • Alignment lattice: k -constrained label and frame dependent with k being the label sequence length. • Weight function: – ω θ, x ( q a = ( t, u ) , q c = q, y = z t + u +1 ) (cid:44) P
[Equation]
( z t + u +
[Equation]
x ,
[Equation]
( z <t + u + [Equation] ) = y <u ) – locally normalized – non-streaming
While the original deﬁnition of the RNNT model is based on non-streaming encoder (transcription network), this model is widely used for streaming applications by using a streaming encoder.
This is in contradiction of the forward-backward derivations in the original paper which explicitly assumes dependency on the whole sequence for any position of alignment sequence (Equation 17 of [13])
Similar to RNNT, the hybrid autoregressive transducer (HAT) [40] model can be also conﬁgured in the GNAT framework with the same parametrization as RNNT.
The only difference is the weight function.
The HAT model deﬁnes different probabilities for label transitions and (cid:15) -transitions (duration model in [40]):
[Equation]
where b t,u is a sigmoid function deﬁned in Equation 6 of [40].
(cid:15) a b (cid:15) : (cid:15) a : a b : b (cid:15) : (cid:15) a : (cid:15) b : b (cid:15) : (cid:15) a : a b : (cid:15)
B.4 Supporting CTC Style Label Deduplication
The standard CTC model [14] is very similar to a GNAT model using a frame dependent alignment lattice and a 0 -gram context dependency.
One key difference is that CTC introduces a deduplication process when turning its model output sequence to a label sequence.
Each model output of a CTC model is either a lexical label from Σ , or the special blank ( (cid:15) ) label.
To obtain the label sequence, two steps are applied on the model output in order,
Maximal consecutive repeated non-blank labels are merged into one (e.g. turning abbc into abc , or abb(cid:15)b into ab(cid:15)b ); 2.
All the (cid:15) labels are removed.
As a comparison, paths on the alignment lattice of the GNAT models in the main paper is equivalent to the model outputs in CTC, whereas the (cid:15) -free label sequence seen by the context dependency is equivalent to the label sequence in CTC.
To support the deduplication of repeated non-blank labels, we need to introduce a ﬁnite state transducer into our series of ﬁnite state machine compositions.
Similar to a ﬁnite state automaton, a weighted ﬁnite-state transducer (WFST) T = (Σ , Q, i, F, ρ, E ) over a semiring K is speciﬁed by a ﬁnite alphabet Σ , a ﬁnite set of states Q , an initial state i ∈ Q , a set of ﬁnal states F ⊆ Q , a ﬁnal state weight assignment ρ : F → K , and a ﬁnite set of transitions E . {+
+}
The meaning of Σ , Q , i , F , and ρ are identical to those of a WFSA.
The set of transitions E is instead a subset of Q × (Σ ∪ { (cid:15) } ) × (Σ ∪ { (cid:15) } )
× K × Q , i.e. containing a pair of input/output labels instead of just one.
A WFSA can be viewed as a WFST with identical input/output labels on each arc, and similar to WFSA intersection, a series of WFST can be composed into a single WFST.
We refer the readers to [28] for a full description of WFST and the composition algorithm.
Figure 4 is an example unweighted FST, when composed with another input FSA or FST, performs the CTC style label deduplication.
More generally, the unweighted label deduplication transducer D of vocabulary Σ consists of
• States Q D = Σ ∪ { (cid:15) } , i D = (cid:15) , F D = Q D • Transitions E = { ( p, x, x, x ) | p, x ∈ Q D } ∪ { ( x, x, (cid:15), x ) | x ∈ Σ }
Given a context dependency FSA C , an alignment lattice FSA L T , and the weight function ω θ, x , as deﬁned in Section 3.3, a GNAT model with CTC style label deduplication induces a WFST T θ, x as follows,
[Equation]
In other words
[Equation]
the topology of T θ, x is the same as the following cascade of FST compositions, 1.
D · C treating output (cid:15) labels in D as empty (i
.e. standard FST composition).
L T · ( D · C ) treating (cid:15) transitions in L T and input (cid:15) labels in ( D · C ) as regular labels.
and the transition weights are deﬁned using ω θ, x just like the GNAT models in the main paper.
When implemented naively, CTC style label deduplication causes a | Q D |
= ( | Σ | + 1) blow up in | Q θ, x | .
However, by inferring about states in Q D from states in Q C , we can greatly reduce the number of states needed.
For each state x ∈ Σ in Q D , we know the last non-blank label observed when reaching state x must be label x .
Similarly, for context dependencies we care about ( n -gram and string), there is a unique label x
( q c ) for all incoming arcs of each non-start state q c (start states do not have any incoming arcs in these context dependencies).
Thus, the states in Q θ, x that are reachable from the start must match one of the following patterns,
• ( q a , (cid:15), q c ) , ∀ q a ∈ Q A , q c ∈ Q X • ( q a , x ( q c ) , q c ) , ∀ q a ∈ Q A , q c ∈ Q X \ { i X }
This means the actual number of states we shall visit in computing the shortest distance is only 2 | Q A || Q C | .
C Accelerator-Friendly Computation
The standard shortest distance/path algorithm for acyclic WFSA [27] can be used for training (computing W ( A ) for some acyclic A ) and inference of a GNAT model.
To compute W ( A ) for an acyclic WFSA A , we maintain the following forward weight α q for each state q in Q A :
[Equation]
else .
The weight of A is then W ( A )
=
(cid:76)
q ∈ F A α q ⊗ ρ A ( q ) .
The recurrence in the deﬁnition of α q can be computed by visiting states in Q A in a topological order.
To make better use of the compute power of modern accelerator hardware, we observe the following properties of the C or L T presented so far that enable us to use a more vectorized variant of the shortest distance algorithm in Figure 5:
• From any topological ordering on Q T , we can derive a topological ordering on Q θ, x . • The n -gram context dependency FSA C n is deterministic, namely leaving any state there is no more than 1 transition for any label y ∈ Σ , and there is no (cid:15) -transition.
• For all three types of alignment lattices, for any non-ﬁnal state q ∈ Q T \ F T , there is a unique next state succ( q ) for transitions leaving q consuming any label y ∈ Σ .
{Initialize the length | Q C | forward weight vectors ¯ α q a } for all q a ∈ Q T do ¯ α q a ← [¯0 , . . . , ¯0] end for ¯ α i T [ i C ] ← ¯ {Compute ¯ α q a for q a (cid:54) =
i T } for all q a ∈ Q T in topological order do { Ω is a [ | Q C | , | Σ | + 1] matrix} Ω ← ¯
[Figure]
(cid:0)
[Figure]
} (cid:1) if q a has outgoing label transitions to q (cid:48) a = succ( q a ) then
[Figure]
end if for all q (cid:48)
a such that ( q a , (cid:15), q (cid:48) a ) ∈ E T do
[Figure]
end for end for
[Figure]
{Inputs: ¯ α q a and Ω[: , Σ] } if n = 1 then { Q C n contains only i C n } return ¯ α q a ¯ ⊗ (cid:76)
y ∈ Σ Ω[ i C n , y ] end if {Initialize length | Q C n | vector
¯
α } ¯ α ← [¯0 , . .
. , ¯0] {States in Q C n are numbered from 0 to | Q C n | − 1 = (cid:80) n − 1 i =1 | Σ | i following the lexicographic order} l ←for i = 0 to n − 2 do h ← l + | Σ | i ¯ α [ l · | Σ | + 1 : h · | Σ | + 1] ← ﬂatten(Ω[ l : h, Σ]) l ← h end for for i = 0 to | Σ | − 1 do ¯ α
[ l :] ← ¯ α
[ l :]¯ ⊕ ﬂatten(Ω[ l + i · | Σ | n − 2 : l + ( i + 1) · | Σ | n − 2 , Σ])
Center to an efﬁcient implementation of the algorithm in Figure 5 is the function next C .
This function receives as input the current forward weight vector ¯ α q a for states ( q a , q c ) , ∀ q c ∈ Q C , and the transition weights for leaving these states via label transitions, and returns the forward weights going to states ( q (cid:48) a , q (cid:48) c ) by taking the ( q c , y, q (cid:48) c ) transitions for y ∈ Σ .
In other words, next C [ q (cid:48) c ] = (cid:76) ( q c ,y,q (cid:48) c ) ∈
E C ¯ α q a [ q c ] ⊗ Ω[ q c , y ] .
The n -gram context dependency C n allows a particularly simple and efﬁcient implementation of next C n , as outlined in Figure 6.
The key observation is that when we number the states in Q C following the lexicographic order, the | Σ | transitions leaving the same q c lead to states in a consecutive range [ σ ( q c y 0 ) , . . .
, σ ( q c y | Σ |− 1 )] , where [ y 0 , . . .
y | Σ |− 1 ] are the lexicographically sorted labels of Σ , and σ ( s ) is the sufﬁx of label sequence s of length up to n − 1 .
During training, we also need to compute the shortest distance D ( A θ, x ∩ y ) .
We note the algorithm in Figure 5 can also be used for this purpose since ( L T ∩ C ) ∩ y
=
L T ∩ ( C ∩ y ) , and we simply need to substitute C with C ∩ y in the algorithm.
D Memory and Computation Time Benchmarks
The memory and computation benchmark of our implementation for the GNAT model is presented in
Table 2.
We present benchmarks for training and inference for different conﬁgurations of the GNAT model:
• Context dependency: 0 -gram, 1 -gram and 2 -gram • Alignment lattice: frame dependent, 1-constrained label and frame dependent • Weight functions: Per-state linear projection (unshared), Shared linear projection with per-state embedding (shared-emb), Shared linear projection with RNN state embedding (shared-rnn)
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
global 146.
[Table]
[Table]
[Table]
For each conﬁguration, the memory usage footprint is presented in terms of MB and total computation time is presented in terms of number of seconds.
The benchmarks do not include the memory and computation footprint of the encoder activations.
The training benchmarks are corresponding to the calculation of the GNAT criterion as well as all the backward gradient calculation up to the encoder activations.
The evaluation benchmarks only contain the forward pass memory and compute footprint to ﬁnd the most likely hypothesis.
All the memory and computation benchmarks are evaluated for an input batch of 32 examples each with 1024 number of frames.
Each frame is a 512 -dim vector corresponding to the encoder activations.
Each example in the input batch are assumed to have at most 256 labels.
The alphabet size is set to 32 .
The main observations are:
• The larger context dependency lead to more memory and compute footprint.
This is expected since the computation complexity is directly related to the context dependency state size.
However, interestingly, the memory and computation values do not scale exponentially by value of n in n -gram context dependency (as a result by number of states in the context dependency). • Label frame dependent alignment lattice generally leads to higher memory usage and computation time compare to the frame dependent alignment lattice.
This is expected since the label frame dependent consist of alignment paths of length 1024 + 256 = 1270 , corresponding to the sum of number of frames and number of labels. • The per-state linear projection weight function requires more memory and has longer compute time compare to the shared weights function which is expected by design.
Both shared weight functions are performing on-par of each other in terms of memory and compute.
• The global normalization requires more memory and time and the difference is more signiﬁcant for context dependency FSAs with more number of states ( 2 -gram versus 1gram).
E Equivalence of Certain Locally and Globally Normalized Models under Non-Streaming Settings
As widely discussed in literature (e.g. Section 4.6.12 of [23], [24, 2]), under streaming settings, a locally normalized model introduces an independence assumption between an output z i and future inputs x ≥ t ( i ) , when future outputs z >i are not yet known.
On the contrary, a globally normalized model using the same score factorization does not introduce the same independence assumption.
This is the key reason to a locally normalized model’s inability to express certain conditional distributions and thus prone to the label bias problem.
However, under non-streaming settings, the entire input sequence x is available for predicting each z i .
The aforementioned independence assumption is thus no longer present for locally normalized models.
We shall show that under a mild condition regarding how an alignment score ω
( z i | z
<
i , x ) makes use of information from the past outputs
z <i , for every non-streaming globally normalized model, there is a non-streaming locally normalized model that deﬁnes the same conditional distribution for the output sequence.
For an alignment score ω ( z i | z
<
i , x ) , for each output step i , deﬁne the past output index set δ i ⊆ { 1 , 2 , . .
, i − 1 } to consist of the indices of past outputs on which ω
( z i | z <i , x ) actually depend.
For example, for a GNAT model using a n -gram context dependency with n = 2 , δ i
= { i − 1 , i − 2 } , i.e. the alignment score only depends on the previous two outputs.
On the other hand, an RNN-T model with unlimited history has δ i = { 1 , 2 , . .
, i − 1 } .
When comparing locally and globally normalized models, it is important to ensure the same past output index sets are used.
Otherwise the comparison is not fair since the parameterization of these models will differ.
Given the sequence of past output index sets [ δ 1 , δ 2 , . . . ] , the alignment score function ω ( z i | z <i , x ) can also be written as ω ( z i |
z δ i , x ) , with z δ i
=
z j | j ∈ δ i } .
The sequence of past output index sets grows monotonically if for all i , δ i ∩ { 1 , 2 , . .
, i − 2 } ⊆
δ i − 1 .
In other words, each δ i can be constructed from δ i − 1 by ﬁrst removing some elements then optionally adding i − 1 .
Importantly if some j < i is not in δ i , then j is not in any subsequent δ k for any k ≥ i .
Notably, all existing work on neural globally normalized ASR models including this paper itself use score functions where the sequence of past output index sets grows monotonically [17, 10, 42, 44].
When the sequence of past output index sets grows monotonically, globally normalized models and locally normalized ones are in fact equivalent in terms of the conditional distributions that they can express.
Since any locally normalized model is by deﬁnition globally normalized, we only need to show the other direction.
Theorem E.1.
For any non-streaming globally normalized model with alignment score ω
G ( z i | z δ i , x ) , if the sequence of past index sets { δ 1 , δ 2 , . . . } grows monotonically, then there exists a non-streaming locally normalized model with alignment score of the same sequence of index sets ω L ( z i | z
δ i , x ) , such that P ω G ( z | x ) = P ω L ( z | x ) .
This theorem can be seen a result of such a globally normalized model’s corresponding Markov network representation being chordal (see Theorem 4.13 of [23]).
In an effort to be self-contained , below we instead present a direct proof that does not rely on any results on graphical models, using essentially the same technique as [9, 36].
Proof.
For any preﬁx z ≤ i , the sum of the scores of all possible sufﬁces continuing z ≤ i is commonly known as the backward score .
More speciﬁcally, in our case, the backward score at position i is
[Equation]
It is easy to see that β i only depends x and z j for j ∈ { i } ∪ δ i .
Thus β i ( z i , z δ i , x ) is a well-deﬁned function.
β i is upper bounded by the denominator Z ( x ) and thus ﬁnite.
ω G ( z j | z δ , x ) > 0 implies β i > 0 .
We can thus deﬁne the following alignment score which is a locally normalized model,
[Equation]
Then
[Equation]
Therefore [Equation] P ω G ( z | x ) =
P ω L ( z | x ) .