Global Normalization for Streaming Speech Recognition in a Modular Framework
Abstract
We introduce the Globally Normalized Autoregressive Transducer (GNAT) for addressing the label bias problem in streaming speech recognition.
Our solution admits a tractable exact computation of the denominator for the sequence-level normalization.
Through theoretical and empirical results, we demonstrate that by switching to a globally normalized model, the word error rate gap between streaming and non-streaming speech-recognition models can be greatly reduced (by more than 50% on the Librispeech dataset).
This model is developed in a modular framework which encompasses all the common neural speech recognition models.
The modularity of this framework enables controlled comparison of modelling choices and creation of new models.
A JAX implementation of our models has been open sourced.
Introduction
Deep neural network models have been tremendously successful in the ﬁeld of automatic speech recognition (ASR).
Several different models have been proposed over the years: cross-entropy (CE) models with a deep feed-forward architecture [19], connectionist temporal classiﬁcation (CTC) models [14] with recurrent architectures such as long short-term memory (LSTM) [20] and more recently sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) [8], recurrent neural network transducer (RNN-T) [13], and hybrid autoregressive transducer (HAT) [40].
When conﬁgured in non-streaming mode, these neural ASR models have reached state-of-the-art word error rate (WER) on many tasks.
However, the WER signiﬁcantly worsens when they are operating in streaming mode.
In this paper, we argue that one main cause of this WER gap is that all the existing models are constrained to be locally normalized which makes them susceptible to the label bias problem [36, 2, 24, 4].
To address this problem we introduce new category of globally normalized models called Globally Normalized Autoregressive Transducer (GNAT).
Our contributions are:
(1) Addressing the label bias problem in streaming ASR through global normalization that signiﬁcantly closes more than 50% of the WER gap between streaming and non-streaming ASR.
(2) Efﬁcient, accelerator-friendly algorithms for the exact computation of the global normalization under the ﬁnite context assumption.
(3) A modular framework for neural ASR which encompasses all the common models (CE, CTC, LAS, RNN-T, HAT), allowing creation of new ones, and extension to their globally normalized counterparts.
Streaming Speech Recognition
For an input feature sequence x = x 1 . . .
x T , usually represented as a sequence of real valued feature vectors (such as log mel), and a ﬁnite output alphabet Σ , we wish to predict the corresponding output label sequence y =
y 1 . . .
y U , y i ∈ Σ .
We call each element x
i in x a frame , and each element y i in y an output label .
Common ASR models do not directly predict y , but rather an alignment label sequence z
=
z 1 . . .
z V , z i ∈ Σ ∪ ∆ .
∆ is a ﬁnite alphabet of control labels, such as the blank label in CTC or RNN-T, or the end-of-sequence label in LAS.
There is a deterministic mapping
B ( z ) : (Σ ∪ ∆) ∗ → Σ ∗ for obtaining y from z (e.g. in RNN-T, we simply remove all the blank labels from z ).
An ASR model can then be broken down in two tasks: (1) assigning a score (cid:81) i ω
( z i | x , z <i ) to each alignment sequence z , where ω ( z i | x , z <i ) > 0 is the alignment score of predicting a single alignment label; (2) ﬁnding (usually approximately) arg max y (cid:80)
z | B ( z )=
y (cid:81) i ω
( z i | x , z <i ) .
A non-streaming ASR model’s alignment score ω
( z i | x , z <i ) has access to the entire x for any i .
In contrast, a streaming model’s alignment score takes the form ω ( z i | x <t
( i ) , z <i ) : it only has access a preﬁx x <t ( i ) of the input feature sequence, where t ( i ) is the frame to which z
i is aligned .
Streaming models can be seen as special case of non-streaming models with respect to the alignment scores.
All the common neural ASR models use a locally normalized alignment score which satisﬁes the constraint (cid:80) z ∈ Σ ∪ ∆ ω ( z i =
z | x , z <i ) = 1 .
This is achieved by applying the softmax function to the last layer activations.
The local normalization constraint makes ω ( z i | x , z <i ) easily interpretable as a conditional probability distribution P ω ( z i | x , z <i ) , and thus (cid:81) i ω
( z i | x , z <i ) easily interpretable as P ω ( z | x ) .
With samples from the true data distribution P ( x , y ) , the modeling parameters are optimized by minimizing the negative log-conditional-likelihood loss E P ( x , y )
[ − log P ω ( y | x )] =
E P ( x , y ) [ − log (cid:80) z | B ( z )=
y P ω ( z | x )] .
2.1 Label Bias in Streaming ASR
For a non-streaming, locally normalized model, the negative log-conditional-likelihood loss is minimized in the limit (assuming enough model capacity) by setting ω ( z i | x , z <i ) to the true conditional probability P ( z i | x , z <i ) , leading to (cid:81) i ω
( z i | x , z <i ) being equal to the true posterior P ( z | x ) .
For a streaming model, x is replaced by x <t ( i ) in the alignment score (e.g. by using a unidirectional encoder).
Here the negative log-conditional-likelihood loss is minimized in the limit by setting ω ( z i | x <t ( i ) , z <i ) to P ( z i | x <t ( i ) , z <i ) .
As a result, the product (cid:81) i ω
( z i | x <t ( i ) , z <i ) =
(cid:81) i P ( z i | x <t ( i ) , z <i ) is in general not equal to P ( z | x ) anymore.
In other words, using a streaming locally normalized model means that the estimated alignment sequence posterior is the product of some locally normalized alignment scores which depend only on partial input x <t ( i ) , and as a result can no longer accurately represent the true conditional distribution.
This will bias the model towards predictions with low-entropy estimated posterior probabilities at each decoding step.
This degrades the model ability to revise previous decisions, a phenomenon called label bias [24, 2].
2.2 Global Normalization
Traditionally, globally normalized models such as Conditional Random Fields (CRF) [24] are used to address the label bias problem.
This paper seeks to apply global normalization to modern neural architectures that are more similar to CTC, RNN-T, or LAS, rather than traditional linear models, with the purpose of addressing label bias problem for streaming ASR models.
A globally normalized model does not constrain the alignment score ω
( z i | x , z <i ) to be locally normalized; it only requires it to be any non-negative score, as long as the denominator Z ( x )
= (cid:80) z (cid:81) i ω
( z i | x , z <i ) is ﬁnite.
The ﬁnite denominator allows us to interpret (cid:81)
i ω
( z i | x , z <i )
Z ( x ) as a conditional probability distribution P ω ( z | x ) .
Minimum negative log-conditional-likelihood training can be more expensive for globally normalized models due to the need to compute Z ( x ) and the corresponding gradients.
However with our proposed modular framework, globally normalized model training can be made practical with careful modelling choices on modern hardware.
Any locally normalized model is by deﬁnition also globally normalized, whose Z ( x ) = 1 .
While the reverse is not true in general, Appendix E shows that under non-streaming settings with a mild condition, locally and globally normalized models actually express the same class of conditional distributions.
Based on this observation, we argue that under non-streaming settings, with adequately powerful neural architectures, maximum log-conditional-likelihood training should yield behaviorly similar locally or globally normalized models, and thus similar WERs in testing.
Results from [37, 17] and our own experiments in Section 6 validate this.
A Modular Framework for Neural ASR
In this section, we introduce a modular framework for neural ASR, using the weighted ﬁnite state automaton (WFSA) formalism to calculate the conditional probabilities via alignment scores.
The modular framework clearly expresses the modelling choices and enables practical globlally normalized model training and inference.
We use the WFSA formalism as the language for describing our framework because of its succinctness and precision.
Our algorithms are implemented from scratch to run on TPUs, without using existing toolkits such as OpenFst [1], Kaldi [33] or K2 [21].
3.1 Preliminaries
We begin with an introduction to the relevant concepts and notations.
A semiring ( K , ⊕ , ⊗ , ¯ 0 , ¯ 1) consists of a set K together with an associative and commutative operation ⊕ and an associative operation ⊗ , with respective identities ¯0 and ¯1 , such that ⊗ distributes over ⊕ , and ¯ ⊗ x = x ⊗ ¯0 = ¯0 .
The real semiring ( R + , + , × , 0 , 1) is used when the weights represent probabilities.
The log semiring ( R ∪ {−∞} , ⊕ log , + , −∞ , 0) , isomorphic to the real semiring via the log mapping, is often used in practice for numerical stability.
The tropical semiring ( R ∪ {−∞} , max , + , −∞ , 0) is often used in shortest-path applications.
A weighted ﬁnite-state automaton (WFSA)
A =
(Σ , Q, i, F, ρ, E ) over a semiring K is speciﬁed by a ﬁnite alphabet Σ , a ﬁnite set of states Q , an initial state i ∈ Q , a set of ﬁnal states F ⊆ Q , a ﬁnal state weight assignment ρ : F → K , and a ﬁnite set of transitions E ⊆ Q × (Σ ∪ { (cid:15) } )
× K × Q ( (cid:15) denotes the empty label sequence).
Given a transition e ∈ E , p [ e ] denotes its origin or previous state, n [ e ] its destination or next state, o [ e ] its label, and ω [ e ] its weight.
A path π
= e 1 . . .
e k is a sequence of consecutive transitions e
i ∈ E : n [ e i − 1 ] =
p [ e i ] , i = 2 , . . .
k .
The functions n , p , and ω on transitions can be extended to paths by setting: n [ π ] = n [ e k ] and p [ π ] =
p [ e 1 ] and by deﬁning the weight of a path as the ⊗ -product of the weights of its constituent transitions: ω
[ π ] = ω [ e 1 ] ⊗ · · · ⊗ ω [ e k ]) .
An unweighted ﬁnite-state automaton (FSA) A = (Σ , Q, i, F, E ) is simply a WFSA whose transitions and ﬁnal states are all weighted by ¯1 .
Π( Q 1 , Q 2 ) is the set of all paths from a subset Q 1 ⊆ Q to a subset Q 2 ⊆ Q .
Π( Q 1 , y , Q 2 ) is the subset of all paths of Π( Q 1 , Q 2 ) with label sequence y
=
y 1 . . .
y U , y i ∈ Σ .
A path in Π( { i } , F ) is said to be accepting or successful .
The weight associated by WFSA A to any label sequence y is given by A ( y ) = (cid:76) π ∈ Π( { i } , y ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
The weight of A is the ⊕ -sum of weights of all accepting paths W ( A ) =
(cid:76) π ∈ Π( { i } ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
For a semiring K where ⊗ is also commutative, the intersection (or Hadamard product) of two WFSA A 1 and A 2 is deﬁned as:
[Equation]
y ) ⊗ A 2 ( y ) .
[28] gives an algorithm to compute the intersection.
We can view y as a WFSA that accepts only y with weight ¯1 , then A ( y )
=
W ( A ∩ y ) .
3.2 Probabilistic Modeling and Inference on Acyclic Recognition Lattices
For any feature sequence x = x 1 . . .
x T , we use a model with trainable parameters θ to induce a recognition lattice WFSA A θ, x = (Σ , Q θ, x , i θ, x , F θ, x , ρ θ, x , E θ, x ) .
For a label sequence y =
y 1 . . .
y U , the weight associated with it,
A θ, x ( y ) , is interpreted under the log semiring as the unnormalized negative log conditional probability P
[Equation]
The recognition lattice A θ, x is designed to be acyclic, and therefore the weight of the automata in both the numerator and denominator above can be efﬁciently computed by visiting the states of the corresponding WFSA in topological order [27].
See Appendix C for our accelerator friendly version of this algorithm.
We can thus train an ASR model by minimizing the negative log-conditionallikelihood on the training corpus D , and choosing θ (cid:63) =
arg min θ E ( x , y ) ∈D [ − log( P θ ( y | x ))]
= arg min θ E ( x , y ) ∈D
[ W ( A θ, x ) − W ( A θ, x ∩ y )] .
[Equation]
In general, there can be more than one path in A θ, x that accepts the same y .
During inference, ﬁnding the optimal ˆ y = arg max y P θ ( y | x ) requires running the potentially expensive WFSA disambiguation algorithm [30] on A θ, x .
As a cheaper approximation, we instead look for the shortest path ˆ π in A θ, x under the tropical semiring, and use the corresponding label sequence as the prediction, again using the standard shortest path algorithm for an acyclic WFSA [27].
3.3 Inducing the WFSA
Our framework decomposes the sequence prediction task in ASR into three components, each playing a speciﬁc role in inducing the recognition lattice
A θ, x .
• The context dependency FSA C
=
(Σ , Q C , i C , F C , E C ) is an (cid:15) -free, unweighted FSA, whose states encode the history of the label sequence produced so far.
C is ﬁxed for a given model, independent of input x . •
The alignment lattice FSA L T =
(Σ , Q T , i T , F T , E T ) is an acyclic, unweighted FSA, whose states encode the alignment between input frames x and output labels y .
L T depends on only the length T of input x . •
The weight function ω θ, x : Q T × Q C × (Σ ∪ (cid:15) )
→ K .
ω θ, x is the only component that contains trainable parameters and requires full access to x .
This function deﬁnes the transition weights in the recognition lattice A θ, x .
We will discuss how one can deﬁne these components in detail in the next section.
With ( C, L T , ω θ, x ) given, the recognition lattice A θ, x is deﬁned as follows:
[Equation]
c ) (cid:1) | y ∈ Σ , ( q a , y, q (cid:48) a ) ∈ E T , ( q c , y, q (cid:48) c ) ∈ E C (cid:111) ∪ (cid:110)(cid:0) ( q a , q c ) , (cid:15), ω θ, x ( q a , q c , (cid:15) ) , ( q (cid:48) a , q c ) (cid:1) | (
[Equation]
a
[Equation]
(cid:15), q (cid:48)
a
[Equation]
In other words, the topology (states and unweighted transitions) of the recognition lattice
A θ, x is the same as the FSA intersection L T ∩ C (see Figure 1c for a concrete example); and the transition weights are deﬁned using ω θ, x .
The (cid:15) -freeness of C and the acyclicity of L T implies that the recognition lattice A θ, x is also acyclic.
Components of a GNAT Model
In this section, we deﬁne globally normalized autoregressive transducer (GNAT) through the framework above, by specifying each model component.
4.1 Context Dependency
GNAT uses an n -gram context-dependency deﬁned by C n = (Σ , Q n , i n , F n , E n ) , where Q n = Σ ≤ n corresponds to a label history of length up to n .
The initial state i n = (cid:15) is the empty label sequence.
The transitions E n ⊆ Q n × Σ × Q n correspond to truncated concatenation: E n = (cid:8) ( q, y, q (cid:48) )
| q ∈
Q, y ∈ Σ (cid:9) where q (cid:48) is the sufﬁx of qy with length at most n .
For example, when n = 2 , the transition ( ab, c, bc ) goes from state ab to state bc with label c .
All states are ﬁnal: F n = Q n .
See Figure 1a for the FSA C 1 when Σ = { a, b } .
Appendix C demonstrates how C n can be efﬁciently intersected during the shortest distance computation thanks to the highly regular structure of C n .
Although not studied in this paper, our modular framework makes it easy to switch to a more sophisticated context dependency, such as clustered histories often used for context-dependent phone models, or a variable context length as used in n -gram language models [29].
(cid:15)
a
[Figure]
a b a b a
[Figure] [Figure]
[Figure] [Figure]
[Figure]
3 Σ (cid:15) Σ (cid:15) Σ (cid:15) (
[Figure]
dependent alignment lattice with T = 3 .
0 ,(cid:15) 1 ,a 1 ,(cid:15) 1 ,b 2 ,(cid:15) 2 ,a 2 ,b 3 ,(cid:15) 3 ,a 3 ,b (cid:15) a b (cid:15) a b (cid:15)
a
[Figure]
(cid:15) a b (cid:15) a b (cid:15) a b (cid:15)
a
[Figure]
(0 , 0) (0 , 1) (0 , 2) (1 , 0) (1 , 1) (1 , 2) (2 , 0) (2 , 1) (2 , 2) (3 , 0) (cid:15)
[Figure]
(cid:15) (cid:15)
[Figure]
(cid:15)
[Figure] [Figure]
4.2 Alignment Lattices
Given the input feature sequence length T , the alignment lattice FSA L T deﬁnes all the possible alignments between the feature sequence and allowed label sequences.
Since the feature sequence length T usually differs from the label sequence length U , many different alignments between the feature sequence and a label sequence can be deﬁned.
The states in an alignment lattice FSA encode how the next label or (cid:15) -transition corresponds to some position in the feature sequence.
We can choose different structures for L T by encoding one or both of the positions in the feature sequence and the label sequence.
A simple example is the frame dependent alignment similar to [14], where each frame is aligned to at most one label:
[Equation]
Here any state t < T represents a position in the feature sequence.
We start by aligning to the initial frame of the feature sequence, and repeatedly shift to the next frame for every subsequent label or (cid:15) -transition until all frames have been visited.
Figure 1b depicts a frame dependent alignment lattice.
To allow a label sequence longer than the feature sequence, we can use the k -constrained label and frame dependent alignment similar to [13]:
[Equation]
Here, up to k consecutive label transitions can align to any single frame.
An (cid:15) -transition is then taken to explicitly shift the alignment to the next frame.
The number of labels aligned to one frame is constrained by a constant k solely in order to impose acyclicity.
Figure 1d depicts a k -constrained label and frame dependent alignment lattice.
Some models may only depend on the position in the label sequence, similar to [8].
In this case we can bound the length of the label sequence by some function l ( T ) , and use the following label dependent alignment :
[Equation]
Here, each label can be seen as aligning to the entire feature sequence, and the (cid:15) -transition serves as an explicit termination of the label sequence.
Figure 1e depicts a label dependent alignment lattice.
4.3 Weight functions
The weight function ω θ, x translates trainable parameters into transition weights for states in the recognition lattice
A θ, x .
The choice of the weight function depends on the choice of the context and alignment lattice FSA, especially the alignment lattice where the meaning of a state directly affects how the weight function can access x .
The experiments discussed in this paper use frame dependent and k -constrained label and frame dependent alignment lattices.
In these two types of alignment lattices, a non-ﬁnal state q a in Q T contains a position τ ( q a ) in the feature sequence (the state itself in the case of frame dependent alignment lattices; the ﬁrst value in the state in the case of label and frame dependent alignment lattices).
Weight functions can thus be deﬁned in three steps,
Feed x into an encoder, such as unidirectional or bidirectional RNN, or a self-attention encoder to obtain the sequence of hidden units h of dimension D .
In the experiments we compared streaming vs non-streaming encoders.
Map a single frame of hidden units h
[ t ] and context state q c to a ( | Σ | + 1) -dimensional vector, corresponding to the unnormalized transition weights for y ∈ Σ ∪ { (cid:15) } .
Optionally locally normalize transition weights across y ∈ Σ ∪ { (cid:15) } given ( q a , q c ) .
For step 2, we experiment with the following concrete modelling choices with varying degree of parameter sharing,
Per-state linear projection ( unshared )
For every context state q c , we obtain a D × ( | Σ | + 1) projection matrix W q c and a ( | Σ | + 1) -dim
bias vector b q c from θ , and deﬁne for y ∈ Σ ∪ { (cid:15) } :
[Equation]
[Equation]
Shared linear projection with per-state embedding ( shared-emb )
We obtain from θ (a) for every context state q c a D -dimensional state embedding E q c , (b) independent of context states a D × ( | Σ | +1) projection matrix W and a ( | Σ | + 1) -dim bias vector b , and deﬁne for y ∈ Σ ∪ { (cid:15) } :
[Equation]
Shared linear projection with RNN state embedding ( shared-rnn )
Similar to shared linear projection with per-state embedding but E q c is obtained from running an RNN (e.g. LSTM) on the n -gram label sequence represented by q c .
Discussion A Modular Framework
All the existing locally normalized models can be explained within the modular framework presented in Section 3 with a particular choice of context size, alignment lattice and of course with constraining the weights to be locally normalized using softmax function.
Appendix B presents how CE, CTC, LAS, RNNT and HAT models can be expressed within this framework.
This allows controlled comparison of different components as well as creating new models by mixing different modeling choices.
Like locally and globally normalized models in general, when the weights are locally normalized, the denominator of models deﬁned in our framework is one.
Finally, our framework is very different from traditional uses of ﬁnite state machines via a cascade of weighted ﬁnite state transducer compositions.
The separation of the weight function from the automaton topology allows an arbitrarily complex, non-linear weight function to model the dependency among alignment states, context states, and output label, which is impossible with composition cascades.
The idea of weight function has been proposed before (e.g. [38]), although as we shall see in experiments, the exact architecture of a weight function plays a crucial role in model accuracy.
Related Globally Normalized Models
There is a rich literature on the applications of globally normalized models [5, 26, 35, 24], as well as theoretical studies on the label bias problem [24, 2, 12].
In ASR, there is a lot of research on applying globally normalized models [3, 7, 6, 16, 25, 45, 18].
Among these, MMI [3, 7] is the most relevant to our work.
In MMI, a sequence level score is factorized into an acoustic model (AM) score and a language model (LM) score.
The denominator is usually approximated over a pruned lattice.
Recently [34] introduced lattice-free MMI, which replaces the word level LM with a 4-gram phone LM during training.
Both the GNAT and MMI are globally normalized.
The GNAT model differs from MMI in several ways: (1) Unlike MMI, GNAT does not require an external LM, nor does it impose any constraint on how the sequence level scores are deﬁned.
In MMI training, the LM is kept frozen while the AM parameters are optimized.
In
GNAT all the model parameters are trained together.
(3) GNAT’s recognition lattice encodes the same set of weighted alignment paths during training and inference, therefore the denominator computation in GNAT is exactly matched between training and inference.
This does not hold for the standard
MMI, because by using different LM topologies between training and inference, the denominator computation in training time only serves as an approximation of the denominator during inference.
Implementation-wise, GNAT trains from scratch without any need for initialization or special regularization techniques as used in the lattice-free version of MMI [34].
In addition we were able to train
GNAT models with accelerators without any techniques discussed in [34].
The concept of global normalization has also been visited with deep neural networks [17, 10, 42, 44].
These models can be seen as special cases of MMI based on aspects such as how the sequence level scores are factorized, their use of ﬁxed LM for training, and the mismatching training and inference LMs.
Thus the comparison between MMI and GNAT applies here as well.
Their use of
WFST composition cascades results in weaker modelling power compared to our weight function formulation, and their use of non-streaming encoders combined with their model formulation makes globally normalized models equally expressive to the locally normalized counterparts (Appendix E).
Challenges
The main challenge with the GNAT model is its scalability to a larger number of label contexts.
At each training step, the model requires O ( | Q C | × | Σ | ) semiring multiplications and additions.
For n -gram context dependency, | Q c | = ( | Σ | n +1 − 1) / ( | Σ | − 1) , thus the computation scale exponentially by value of n .
However as shown in Appendix D, due to the particular structure of this space the practical computation and memory cost benchmarks do not scale exponentially with 0 ≤ n ≤ 2 .
We also note that large value of n might also not be necessary:
The HAT model [40] reports that a Seq2Seq model with a label history of just the two previous phonemes performs on par with a similar model with a full history trained on very large voice-search corpus.
Similar observations are reported in studies with grapheme and wordpiece units [43, 11].
Due to the data sparsity there might not be enough training data to fully represent all n -gram context states, so increasing the value of n might not necessarily lead to a WER reduction.
One way of dealing with larger numbers of states is to use standard pruning techniques to keep only some of the most common states in the training data.
Experiments Data
We use the full 960 -hour Librispeech corpus [32] for experiments.
The input features are 80-dim.
log Mel extracted from a 25 ms window of the speech signal with a 10 ms shift.
The
SpecAugment library with baseline recipe parameters were used [15].
The ground truth transcription is used without any processing and tokenized by the 28 graphemes that appear in the training data.
Architecture
Attention-based architectures allows using the same parameterization for streaming and non-streaming models, thus for all the experiments we used 12 -layer Conformer encoders [15] with model dimension 512, followed by a linear layer with output dimension 640.
The Conformer parameters are set such that the only difference between streaming and non-streaming models is the right context: at each time frame t , the streaming models only access the left context (feature frames from 1 to t ), while the non-streaming models can see the entire acoustic feature sequence.
To enforce the consistency of the encoder architecture between streaming and non-streaming modes, we removed all the sub-architecture which behaved differently between these two modes.
Speciﬁcally, we removed the convolution sub-sampling layer, and also forced the stacking layers to only stack within the left context.
The baseline experiments use a shared-rnn weight function deﬁned in section 4.3.
A single layer LSTM is used with 640 cells.
The experiments with the unshared weight function use a linear layer of size ( | Q C | × | Σ | ) × 640 to project the encoder activation at each time frame into the transition weights of the recognition lattice.
In our experiments, | Σ | = 32 .
For the n -gram context dependency, | Q c | = ( | Σ | n +1 − 1) / ( | Σ | − 1) .
The experiments with the shared-emb weight function use an embedding table of size | Q c | × 128 .
Training
All models are trained on 8 × 8 TPUs with a batch size 2048 .
The training examples with more than 1961 feature frames or more than 384 labels are ﬁltered out.
We used Adam optimizer [22] ( β 1 = 0 .
9 , β 2 = 0 .
98 , and (cid:15) =
10 − 9 ) with the Transformer learning rate schedule [41] (10k warm-up steps and peak learning rate 0 .
05 / √ 512 ).
We applied the same regularization techniques and the training hyperparameters used in the baseline recipe of [15].
Evaluation
We report WER results on standard Librispeech test sets: test_clean and test_other.
The WER is either computed with sum-path algorithm or max-path algorithm.
The sum-path algorithm approximately merges the alignment hypothesis corresponding to the same label sequence preﬁx after removal of epsilons.
In ideal decoding, sum-path should result in the most likely output label sequence.
The max-path algorithm computes the highest scoring alignment path using algorithms in
Appendix C.
Baselines
The RNN-T baselines are presented in the rows corresponding to the “ RNN-T ” context dependency in Table 1a.
Unlike the other rows in Table 1a, the decoder RNN in RNN-T in principle encodes an unlimited output history as context.
The standard RNN-T model effectively uses the label frame dependent alignment lattice with an inﬁnite k .
The rows corresponding to the frame dependent alignment lattice represent a slightly modiﬁed RNN-T model, where each frame is aligned to exactly one output (either a blank, or a lexical label).
The WER differences between non-streaming RNN-T baselines in Table 1a and [15] are mainly due to our modiﬁcations to the Conformer encoder for a controlled comparison against streaming models.
Choice of the context dependency
Table 1a compares effect of n -gram context dependency for n = 1 , 2 , 3 and baseline RNN-T models.
The general observation is that increasing n leads to better
WER independent of the other choices of the modeling parameters.
However, the model with 3 -gram context dependency already performs on par with RNN-T baseline.
The 2 -gram context dependency perform almost on par as baseline on clean test set while still lagging on the other test set.
This is consistent with the earlier observations in [40, 43, 11].
Choice of the alignment lattice
The comparison of different alignment lattices in Table 1a suggests that this choice does not signiﬁcantly contribute to the model performance.
While there is a performance gap for 1 -gram context dependency, we do not think there is a principal argument in favor of frame dependent alignment lattice.
We speculate that this is more due to the choice of optimization parameters.
However, the choice of lattice type can have some side effects.
For example as k in k -constrained label frame dependent alignment lattice increases, the model has more
[Table] [Table]
[Table]
[Table] [Table] [Table]
[Table] [Table]
ability to delay its prediction to the end of the signal.
This implicit lookahead can translate into performance gains particularly for unidirectional models.
By limiting this quantity to 1, we observed that performance on clean and other sets degrades by 34 .
9% and 36 .
7% , respectively.
Choice of the weight function normalization
Table 1b examines the effect of weight normalization on non-streaming and streaming models.
Here we present models with 1 -gram and 2 -gram context dependency with a frame dependent alignment lattice.
Note that for 0 -gram context dependency with a frame dependent alignment lattice it is easy to show that locally normalized and globally normalized models are equivalent.
For non-streaming models, the normalization seems to not have an impact on the performance quality neither for 1 -gram nor for 2 -gram context dependency experiments.
This is validates our argument in the end of Section 2.2 that locally and globally normalized models express the same class of conditional distributions under this particular setting.
On the other hand, streaming models signiﬁcantly beneﬁt from global normalization: For the clean test set, the globally normalized model outperforms the locally normalized model by about 21% relative WER for 1 -gram context dependency and by about 20% relative gain for 2 -gram context dependency.
The globally normalized model with 2 -gram context dependency also beat the baseline streaming
RNN-T model in Table 1a and performs signiﬁcantly closer to the non-streaming RNN-T baseline.
The equivalent streaming RNN-T model performs 5 .
1% on test clean and the non-streaming model performs 2 .
5% on same test set.
The globally normalized model decoded with max-path algorithm performs 3 .
8% on same test set.
The globally normalized model effectively closed almost 50% of the performance gap between streaming and non-streaming models.
The reported performance for the globally normalized models is from max-path decoding, while the baselines beneﬁt from sum-path decoding.
Comparing the locally normalized models’ WER from max-path decoding in Table 1b and their counterparts in Table 1a, it is clear that sum-path decoding leads to an extra WER gain.
This gain is more signiﬁcant on test_other.
So we expect the globally model performs even better when decoded with sum-path.
The standard sum-path algorithms use several heuristics particularly for path merging and pruning.
While similar merging techniques can be applied to the globally normalized models, the pruning heuristics require several adjustments.
This is particularly due to the nature of the globally normalized models where the transition weights are not constrained and can take any value, unlike locally normalized models where the transition weights are constrained to be positive number between 0 and 1 and sum to 1 for all the weights leaving the same state in recognition lattice.
We plan to explore approaches to improve globally normalized model decoding in the future.
Choice of the weight function architecture
Finally Table 1c compares different choices of the architectures for a streaming model with 2 -gram context dependency and frame dependent alignment lattice.
While the unshared and shared-rnn architectures are very different in terms of parameter sharing among states, both perform well, though the shared-rnn architecture performs slightly better.
The shared-emb architecture performs signiﬁcantly worst than shared-rnn architecture.
Note that the shared-rnn model is able to learn common structures across states in the context dependency while shared-emb does not have such capability.
Conclusion
The GNAT model was proposed and evaluated with the focus on the label bias problem and its impact on the performance gap between streaming and non-streaming locally normalized ASR.
The ﬁnite context property of this model allows exact computation of the sequence level normalization which makes this model differ from existing globally normalized models.
Furthermore, the same property allows accelerator friendly training and inference.
[-[6] the initial state α 0 is a 1 -hot vector with α 
-]
[-and the denominator of the GNAT model is equal to (cid:80) 6 j =
-]
We showed that the streaming models with globally normalized criteria can signiﬁcantly close the gap between streaming and non-streaming models by more than 50%.
Finally, the modular framework introduced in this paper to explain the GNAT model encompasses all the common neural speech recognition models.
This enables fair and accurate comparison of different models via controlled modelling choices and creation of new ASR models.
[-E . 
-]