Global Normalization for Streaming Speech Recognition in a Modular Framework
Anonymous Author(s)
Abstract
We introduce the Globally Normalized Autoregressive Transducer (GNAT) foraddressing the label bias problem in streaming speech recognition.
Our solutionadmits a tractable exact computation of the denominator for the sequence-levelnormalization.
Through theoretical and empirical results, we demonstrate thatby switching to a globally normalized model, the word error rate gap betweenstreaming and non-streaming speech-recognition models can be greatly reduced (bymore than 50% on the Librispeech dataset).
This model is developed in a modularframework which encompasses all the common neural speech recognition models.
The modularity of this framework enables controlled comparison of modellingchoices and creation of new models.
Introduction
Deep neural network models have been tremendously successful in the ﬁeld of automatic speechrecognition (ASR).
Several different models have been proposed over the years: cross-entropy (CE)models with a deep feed-forward architecture [19], connectionist temporal classiﬁcation (CTC)models [14] with recurrent architectures such as long short-term memory (LSTM) [20], and morerecently sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) [8], recurrentneural network transducer (RNN-T) [13], and hybrid autoregressive transducer (HAT) [40].
Whenconﬁgured in non-streaming mode, these neural ASR models have reached state-of-the-art worderror rate (WER) on many tasks.
However, the WER signiﬁcantly drops when they are operating instreaming mode.
In this paper, we argue that one main cause of such WER gap is that all the existingmodels are constrained to be locally normalized which makes them susceptible to label bias problem[36, 2, 24, 4].
To address this problem, we introduce new category of globally normalized modelscalled Globally Normalized Autoregressive Transducer (GNAT).
Our contributions are:
(1) Addressing the label bias problem in streaming ASR through global normalization thatsigniﬁcantly closes more than 50% of the WER gap between streaming and non-streaming ASR.(2)
Efﬁcient, accelerator-friendly algorithms for the exact computation of the global normalization under the ﬁnite context assumption.
A modular framework for neural ASR which encompasses all the common models (CE, CTC,LAS, RNN-T, HAT), allowing creation of new ones, and extension to their globally normalizedcounterparts.
Streaming Speech Recognition
For an input feature sequence x = x 1 . . .
x T , usually represented as a sequence of real valued featurevectors (such as log mel), and a ﬁnite output alphabet Σ , we wish to predict the corresponding output label sequence y =
y 1 . . .
y U , y i ∈ Σ .
We call each element x
i in x a frame , and each elementy i in y an output label .
Common ASR models do not directly predict y , but rather a alignmentlabel sequence z
= z 1 . . .
z V , z i ∈ Σ ∪ ∆ .
∆ is a ﬁnite alphabet of control labels, such as the blanklabel in CTC or RNN-T, or the end-of-sequence label in LAS.
There is a deterministic mappingy ( z ) : (Σ ∪ ∆) ∗ → Σ ∗ for obtaining y from z (e.g. in RNN-T, we simply remove all the blank labelsfrom z ).
An ASR model can then be broken down in two tasks: (1) assigning a score (cid:81) i ω
( z i | x , z <i )to each alignment sequence z , where ω ( z i | x , z <i ) > 0 is the alignment score of predicting a singlealignment label; (2) ﬁnding (usually approximately) arg max y (cid:80)
z | y ( z )=
y (cid:81) i ω
( z i | x , z <i ) .
A non-streaming ASR model’s alignment score ω
( z i | x , z <i ) has access to the entire x for any i .
Incontrast, a streaming model’s alignment score takes the form ω ( z i | x <t
( i ) , z <i ) : it only has access apreﬁx x <t ( i ) of the input feature sequence, where t ( i ) is the frame to which z
i is aligned .
Streamingmodels can be seen as special case of non-streaming models with respect to the alignment scores.
All the common neural ASR models use a locally normalized alignment score which satisﬁes theconstraint (cid:80) z ∈ Σ ∪ ∆ ω ( z i =
z | x , z <i ) = 1 .
This is achieved by applying the softmax function to thelast layer activations.
The local normalization constraint makes ω ( z i | x , z <i ) easily interpretable
asa conditional probability distribution P ω
( z i | x , z <i ) , and thus (cid:81)
i
ω
( z i | x , z <i ) easily interpretableas P ω ( z | x ) .
With samples from the true data distribution P ( x , y ) , the modeling parameters areoptimized by minimizing the negative log-conditional-likelihood loss E P ( x , y ) [ − log P ω ( y | x )] =
E P ( x , y ) [ − log (cid:80) z | y ( z )=
y P ω ( z | x )] .
2.1 Label Bias in Streaming ASR
For a non-streaming, locally normalized model, the negative log-conditional-likelihood loss is minimized in the limit (assuming enough model capacity) by setting ω ( z i | x , z <i ) to the true conditionalprobability P ( z i | x , z <i ) , leading to (cid:81) i ω
( z i | x , z <i ) being equal to the true posterior P ( z | x ) .
For a streaming model, x is replaced by x <t ( i ) in the alignment score (e.g. by using a unidirectional encoder).
Here the negative log-conditional-likelihood loss is minimized in the limitby setting ω ( z i | x <t ( i ) , z <i ) to P ( z i | x <t ( i ) , z <i ) .
As a result, the product (cid:81) i ω
( z i | x <t ( i ) , z <i ) =
59 (cid:81) i P ( z i | x <t ( i ) , z <i ) is in general not equal to P ( z | x ) anymore.
In other words, using a streaminglocally normalized model means that the estimated alignment sequence posterior is the product ofsome locally normalized alignment scores which depend only on partial input x <t ( i ) , and as a resultcan no longer accurately represent the true conditional distribution.
This will bias the model towardspredictions with low-entropy estimated posterior probabilities at each decoding step.
This degradesthe model ability to revise previous decisions, a phenomenon called label bias [24, 2].
2.2 Global Normalization
Traditionally, globally normalized models such as conditional random ﬁelds [24] are used to addressthe label bias problem.
This paper seeks to apply global normalization to modern neural architecturesthat are more similar to CTC, RNN-T, or LAS, rather than traditional linear models with the purposeof addressing label bias problem for streaming ASR models.
A globally normalized model does not constrain the alignment score ω
( z i | x , z <i ) to be locallynormalized; it only requires it to be any non-negative score, as long as the denominator Z ( x ) =(cid:80) z (cid:81)
i ω
( z i | x , z <i ) is ﬁnite.
The ﬁnite denominator allows us to interpret (cid:81)
i ω
( z i | x , z <i )
Z ( x ) as aconditional probability distribution P ω ( z | x ) .
Minimum negative log-conditional-likelihood trainingcan be more expensive for globally normalized models due to the need to compute Z ( x ) and thecorresponding gradients.
However with our proposed modular framework, globally normalized modeltraining can be made practical with careful modelling choices on modern hardware.
Any locally normalized model is by deﬁnition also a globally normalized, whose Z ( x ) = 1 . Whilethe reverse is not true in general, Appendix E shows that under non-streaming settings with a mildcondition, locally and globally normalized models actually express the same class of conditionaldistributions P ω ( z | x ) .
Based on this observation, we argue that under non-streaming settings, withadequately powerful neural architectures, maximum log-conditional-likelihood training should yieldbehaviorly similar locally or globally normalized models, and thus similar WERs in testing.
Resultsfrom [37, 17] and our own experiments in Section 6 validate this.
A Modular Framework for Neural ASR
In this section, we introduce a modular framework for neural ASR, using the weighted ﬁnite stateautomaton (WFSA) formalism to calculate the conditional probabilities via alignment scores.
Themodular framework clearly expresses the modelling choices enabling practical globlally normalizedmodel training and inference.
We use the WFSA formalism as the language for describing ourframework because of its succinctness and precision. Our algorithms are implemented from scratchto run on TPUs, without using existing toolkits such as OpenFst [1], Kaldi [33] or K2 [21].
3.1 Preliminaries
We begin with an introduction to the relevant concepts and notations.
A semiring ( K , ⊕ , ⊗ , ¯0 , ¯1) consists of a set K together with an associative and commutative operation⊕ and an associative operation ⊗ , with respective identities ¯0 and ¯1 , such that ⊗ distributes over⊕ , and ¯0 ⊗ x = x ⊗ ¯0 = ¯0 .
The real semiring ( R + , + , × , 0 , 1) is used when the weights representprobabilities.
The log semiring ( R ∪ {∞} , ⊕ log , + , ∞ , 0) , isomorphic to the real semiring via thenegative-log mapping, is often used in practice for numerical stability.
The tropical semiring( R ∪ {∞} , min , + , ∞ , 0) is often used in shortest-path applications.
A weighted ﬁnite-state automaton (WFSA)
A =
(Σ , Q, i, F, ρ, E ) over a semiring K is speciﬁed by aﬁnite alphabet Σ , a ﬁnite set of states Q , an initial state i ∈ Q , a set of ﬁnal states F ⊆ Q , a ﬁnal stateweight assignment ρ : F → K , and a ﬁnite set of transitions E ⊆ Q × (Σ ∪ { (cid:15) } ) × K × Q
( (cid:15) denotesthe empty label sequence).
Given a transition e ∈ E , p [ e ] denotes its origin or previous state, n [ e ]its destination or next state, o [ e ] its label, and ω [ e ] its weight.
A path π
= e 1 . . .
e k is a sequence ofconsecutive transitions e
i ∈ E : n [ e i − 1 ] =
p [ e i ] , i = 2 , . . .
k .
The functions n , p , and ω on transitionscan be extended to paths by setting: n [ π ] = n [ e k ] and p [ π ]
= p [ e 1 ] and by deﬁning the weight of apath as the ⊗ -product of the weights of its constituent transitions: ω
[ π ] = ω [ e 1 ] ⊗ · · · ⊗ ω [ e k ]) .
Anunweighted ﬁnite-state automaton (FSA)
A =
(Σ , Q, i, F, E ) is simply a WFSA whose transitionsand ﬁnal states are all weighted by ¯1 .Π
( Q 1 , Q 2 ) is the set of all paths from a subset Q 1 ⊆ Q to a subset Q 2 ⊆ Q .
Π( Q 1 , y , Q 2 )is the subset of all paths of Π( Q 1 , Q 2 ) with label sequence y
=
y 1 . . .
y U , y i ∈ Σ .
A path inΠ( { i } , F ) is said to be accepting or successful .
The weight associated by A to any label sequencey is given by A ( y ) =
(cid:76) π ∈ Π( { i } , y ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
The weight of A is the ⊕ -sum of weightsof all accepting paths W ( A ) =
(cid:76) π ∈ Π( { i } ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
For a semiring K where ⊗ is alsocommutative, the intersection (or Hadamard product) of two WFSA A 1 and A 2 is deﬁned as:( A 1 ∩ A 2 )( y ) =
A 1 ( y ) ⊗ A 2 ( y ) .
[28] gives an algorithm to compute the intersection.
We can viewy as a WFSA that accepts only y with weight ¯1 , then A ( y )
=
W ( A ∩ y ) .
3.2 Probabilistic Modeling and Inference on Acyclic Recognition Lattices
For any feature sequence x = x 1 . . .
x T , we use a model with trainable parameters θ to induce a recognition lattice WFSA A θ, x = (Σ , Q θ, x , i θ, x , F θ, x , ρ θ, x , E θ, x ) .
For a label sequence y =
y 1 . . .
y U ,the weight associated with it,
A θ, x ( y ) , is interpreted under the log semiring as the unnormalizednegative log conditional probability P θ ( y | x ) =
[Equation]
The recognition lattice A θ, x is designed to be acyclic, and therefore the weight of the automata inboth the numerator and denominator above can be efﬁciently computed by visiting the states of thecorresponding WFSA in topological order [27].
See Appendix C for our accelerator friendly versionof this algorithm.
We can thus train an ASR model by minimizing the negative log-conditionallikelihood on the training corpus D , and choosing θ (cid:63) =
arg min θ E ( x , y ) ∈D [ − log( P θ ( y | x ))]
=arg min θ E ( x , y ) ∈D
[ W ( A θ, x ∩ y ) − W
( A θ, x )] .
In general, there can be more than one path in A θ, x that accepts the same y .
During inference, ﬁndingthe optimal ˆ y = arg max y P θ
( y | x ) requires running the potentially expensive WFSA disambiguationalgorithm [30] on A θ, x .
As a cheaper approximation, we instead look for the shortest path ˆ π in A θ, xunder the tropical semiring, and use the corresponding label sequence as the prediction, again usingthe standard shortest path algorithm for an acyclic WFSA [27].
3.3 Inducing the WFSA
Our framework decomposes the sequence prediction task in ASR into three components, each playinga speciﬁc role in inducing the recognition lattice
A θ, x .
• The context dependency FSA C
=
(Σ , Q C , i C , F C , E C ) is an (cid:15) -free, unweighted FSA,whose states encode the history of the label sequence produced so far.
C is ﬁxed for a given
GNAT model, independent of input x .
• The alignment lattice FSA L T =
(Σ , Q T , i T , F T , E T ) is an acyclic, unweighted FSA, whose states encode the alignment between input frames x and output labels y .
L T dependson only the length T of input x .
• The weight function ω θ, x : Q T × Q C × (Σ ∪ (cid:15) )
→ K .
ω θ, x is the only component
that contains trainable parameters and requires full access to x .
This function deﬁnes the transition weights in the recognition lattice
A θ, x .
We will discuss how one can deﬁne these components in detail in the next section.
With ( C, L T , ω θ, x )given, the recognition lattice A θ, x is deﬁned as follows:
[Equation]
In other words, the topology (states and unweighted transitions) of the recognition lattice
A θ, x isthe same as the FSA intersection L T ∩ C ; and the transition weights are deﬁned using ω θ, x . The(cid:15) -freeness of C and the acyclicity of L T implies that the recognition lattice A θ, x is also acyclic.
Components of a GNAT Model
In this section, we deﬁne globally normalized autoregressive transducer (GNAT) through the framework above, by specifying each model component.
4.1 Context Dependency
For the experiments later in this paper, GNAT uses an n -gram context-dependency deﬁned by C n = (Σ , Q n , i n , F n , E n ) , where Q n = Σ ≤ n − 1 corresponds to a label history of length up to n − 1 .
The initial state i n = (cid:15) is the empty label sequence.
The transitions E n ⊆ Q n × Σ × Q n correspondto truncated concatenation: E n = (cid:8) ( q, y, q (cid:48) )
| q ∈ Q, y ∈ Σ) (cid:9) where q (cid:48) is the sufﬁx of qy with lengthat most n − 1 .
For example, when n = 3 , the transition ( ab, c, bc ) goes from state ab to state bcwith label c .
All states are ﬁnal: F n = Q n .
See Figure 1a for the FSA C 2 when Σ = { a, b } .
Theintersection L T ∩ C n is easy to compute thanks to the absence of (cid:15) -transitions in C n .
Appendix Cdemonstrates how C n can be efﬁciently intersected during the shortest distance computation.
Although not studied in this paper, our modular framework makes it easy to switch to a moresophisticated context dependency, such as clustered histories often used for context-dependent phonemodels, or a variable context length as used in n -gram language models [29].
Given the feature sequence length T , the alignment lattice FSA L T deﬁnes all the possible alignmentsbetween the feature sequence and allowed label sequences. Since the feature sequence length Tusually differs from the label sequence length
U , many different alignments between the feature
(cid:15) a b a b a b a b (a) n -gram context-dependency automaton C 2 for Σ = { a, b } .
3 Σ (cid:15) Σ (cid:15) Σ (cid:15) (b)
Frame dependent alignment lattice with T = 3 .
(0 , 0) (0 , 1) (0 , 2) (1 , 0) (1 , 1) (1 , 2) (2 , 0) (2 , 1) (2 , 2) (3 , 0) (cid:15) Σ (cid:15) Σ (cid:15) (cid:15) Σ (cid:15) Σ (cid:15) (cid:15) Σ (cid:15) Σ (cid:15)
(c) k -constrained label and frame dependent align- ment lattice with T = 3 , k = 2 .
3 Σ (cid:15) Σ (cid:15) Σ (cid:15) (
[Figure]
[Figure]
dependent alignment lattice with l ( T ) = 3 .
sequence and a label sequence can be deﬁned.
The states in an alignment lattice FSA encode how thenext label or (cid:15) -transition corresponds to some position in the feature sequence.
We can choose different structures for L T by encoding one or both of the positions in the featuresequence and the label sequence.
A simple example is the frame dependent alignment similar to [14],where each frame is aligned to at most one label:
[Equation]
Here any state t < T represents a position in the feature sequence.
We start by aligning to the initialframe of the feature sequence, and repeatedly shift to the next frame for every subsequent label or(cid:15) -transition until all frames have been visited.
Figure 1b depicts a frame dependent alignment lattice.
To allow a label sequence longer than the feature sequence, we can use the k -constrained label andframe dependent alignment similar to [13]:
[Equation]
Here, up to k consecutive label transitions can align to any single frame.
An (cid:15) -transition is then takento explicitly shift the alignment to the next frame.
The number of labels aligned to one frame isconstrained by a constant k solely in order to impose acyclicity.
Figure 1c depicts a k -constrainedlabel and frame dependent alignment lattice.
Some models may only depend on the position in the label sequence, similar to [8].
In this casewe can bound the length of the label sequence by some function l ( T ) , and use the following label dependent alignment :
[Equation]
Here, each label can be seen as aligning to the entire feature sequence, and the (cid:15) -transition serves asan explicit termination of the label sequence.
Figure 1d depicts a label dependent alignment lattice.
4.3 Weight functions
The weight function ω θ, x translates trainable parameters into transition weights for states in therecognition lattice
A θ, x .
The choice of the weight function depends on the choice of the context andalignment lattice FSA, especially the alignment lattice where the meaning of a state directly affectshow the weight function can access x .
The experiments discussed in this paper use frame dependent and k -constrained label and framedependent alignment lattices.
In these two types of alignment lattices, a non-ﬁnal state q a in Q Tcontains a position τ ( q a ) in the feature sequence (the state itself in the case of frame dependentalignment lattices; the ﬁrst value in the state in the case of label and frame dependent alignmentlattices).
Weight functions can thus be deﬁned in three steps,
Feed x into an encoder, such as unidirectional or bidirectional RNN, or a self-attentionencoder to obtain the sequence of hidden units h of dimension D .
In the experiments
we
compared streaming vs non-streaming encoders.
Map a single frame of hidden units h
[ t ] and context state q c to a ( | Σ | + 1) -dimensionalvector, corresponding to the unnormalized transition weights for y ∈ Σ ∪ { (cid:15) } .3.
Optionally locally normalize transition weights across y ∈ Σ ∪ { (cid:15) } given ( q a , q c ) .For
step
2, we experiment with the following concrete modelling choices with varying degree ofparameter sharing,
Per-state linear projection ( unshared )
For every context state q c , we obtain a D × ( | Σ | + 1)projection matrix W q c and a ( | Σ | + 1) -dim
bias vector b q c from θ , and deﬁne for y ∈ Σ ∪ { (cid:15) } : [Equation]
[Equation]
Shared linear projection with per-state embedding ( shared-emb )
We obtain from θ (a) for everycontext state q c a D -dimensional state embedding E q c , (b) independent of context states a D × ( | Σ |
+1)projection matrix W and a ( | Σ | + 1) -dim bias vector b , and deﬁne for y ∈ Σ ∪ { (cid:15) } :
[Equation]
Shared linear projection with RNN state embedding ( shared-rnn )
Similar to shared linear projection with per-state embedding but E q c is obtained from running an RNN (e.g. LSTM) on the n -gramlabel sequence represented by q c .
Discussion
A Modular Framework
All the existing locally normalized models can be explained withinthe modular framework presented in Section 3 with a particular choice of context size, alignmentlattice and of course with constraining the weights to be locally normalized using softmax function.
Appendix B presents how CE, CTC, LAS, RNNT and HAT models can be expressed within thisframework.
This allows controlled comparison of different components as well as creating newmodels by mixing different modeling choices.
Like locally and globally models in general, when theweights are locally normalized, the denominator of models deﬁned in our framework is one.
Finally, our framework is very different from traditional uses of ﬁnite state machines via a cascadeof weighted ﬁnite state transducer compositions.
The separation of the weight function from the automaton topology allows an arbitrarily complex, non-linear weight function to model the dependency among alignment states, context states, and output label, which is impossible with compositioncascades.
The idea of weight function has been proposed before (e.g. [38]), although as we shall seein experiments, the exact architecture of a weight function plays a crucial role in model accuracy.
Related Globally Normalized Models
There is a rich literature on the applications of globallynormalized models [5, 26, 35, 24], as well as theoretical studies on the label bias problem [24, 2, 12].
In ASR, there is a lot of research on applying globally normalized models [3, 7, 6, 16, 25, 45, 18].
Among these, MMI [3, 7] is the most relevant to our work.
In MMI, a sequence level score isfactorized into an acoustic model (AM) score and a language model (LM) score.
The denominatoris usually approximated over a pruned lattice.
Recently [34] introduced lattice-free MMI, whichreplaces the word level LM with a 4-gram phone LM during training.
Both the GNAT and MMI areglobally normalized.
The GNAT model differs from MMI in several ways: (1) Unlike MMI, GNATdoes not require an external LM, nor does it impose any constraint on how the sequence level scoresare deﬁned.
In MMI training, the LM is kept frozen while the AM parameters are optimized.
In GNAT all the model parameters are trained together.
(3) GNAT’s recognition lattice encodes the sameset of weighted alignment paths during training and inference, therefore the denominator computationin GNAT is exactly matched between training and inference. This does not hold for the standard MMI, because by using different LMs between training and inference, the denominator computationin training time only serves as an approximation of the denominator during inference.
Implementation-wise, GNAT trains from scratch without any need for initialization or special regularization techniques as used in the lattice-free version of MMI [34].
In addition we were able to train GNAT models with accelerators without any techniques discussed in [34].
The concept of global normalization has also been visited with deep neural networks [17, 10, 42, 44].
These models can be seen as special cases of MMI based on aspects such as how the sequencelevel scores are factorized, their use of ﬁxed LM for training, and the mismatching training andinference LMs. Thus the comparison between MMI and GNAT applies here as well.
Their use of WFST composition cascades results in weaker modelling power compared to our weight functionformulation, and their use of non-streaming encoders combined with their model formulation makesglobally normalized models equally expressive to the locally normalized counterparts (Appendix E).
Challenges
The main challenge with the GNAT model is its scalability to a larger number of labelcontexts.
At each training step, the model requires 2 × | Q C | × | Σ | multiplication and summation.
Forn -gram context dependency, | Q c | = ( | Σ | n +1 − 1) / ( | Σ |− 1) , thus the computation scale exponentiallyby value of n .
However as shown in Appendix D, due to the particular structure of this space thepractical computation and memory cost benchmarks do not scale exponentially with n .
We also notethat large value of n might also not be necessary: The HAT model [40] reports that a Seq2Seq modelwith a label history of just the two previous phonemes performs on par with a similar model with afull history trained on very large voice-search corpus.
Similar observations are reported in studieswith grapheme and wordpiece units [43, 11].
Due to the data sparsity there might not be enoughtraining to fully represent a n -gram space, so increasing the value of n might not necessarily lead toperformance improvement.
One way of dealing with large number of states is to use standard pruningtechniques to keep only some of the most common states in the training data.
ExperimentsData
We use the full 960 -hour Librispeech corpus [32] for experiments.
The input features are80-dim.
log Mel extracted from a 25 ms window of the speech signal with a 10 ms shift.
The
SpecAugment library with baseline recipe parameters were used [15].
The ground truth transcriptionis used without any processing and tokenized by the 28 graphemes that appear in the training data.
Architecture
Attention-based architectures allows using the same parameterization for streamingand non-streaming models, thus for all the experiments we used 12 -layer Conformer encoders [15]with model dimension 512, followed by a linear layer with output dimension 640.
The Conformerparameters are set such that the only difference between streaming and non-streaming models isthe right context: at each time frame t , the streaming models only access the left context (featureframes from 1 to t ), while the non-streaming models can see the entire acoustic feature sequence.
Toenforce the consistency of the encoder architecture between streaming and non-streaming modes, weremoved all the sub-architecture which behaved differently between these two modes. Speciﬁcally, we [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
removed the convolution sub-sampling layer, and also forced the stacking layers to only stack withinthe left context.
The baseline experiments use a shared-rnn weight function deﬁned in section 4.3.
Asingle layer LSTM is used with 640 cells.
The experiments with the unshared weight function use alinear layer of size ( | Q C | × | Σ | ) × 640 to project the encoder activation at each time frame into thetransition weights of the recognition lattice.
In our experiments, | Σ | = 32 .
For the n -gram contextdependency, | Q c | = ( | Σ | n +1 − 1) / ( | Σ | − 1) .
The experiments with the shared-emb weight functionuse an embedding table of size | Q c | × 128 .
Training
All models are trained on 8 × 8 TPUs with a batch size 2048 .
The training exampleswith more than 1961 feature frames or more than 384 labels are ﬁltered out.
We used Adam optimizer[22] ( β 1 = 0 .
9 , β 2 = 0 .
98 , and (cid:15) =
10 − 9 ) with the Transformer learning rate schedule [41] (10kwarm-up steps and peak learning rate 0 .
05 / √ 512 ).
We applied the same regularization techniquesand the training hyperparameters used in the baseline recipe of [15].
Evaluation
We report WER results on standard Librispeech test sets: test_clean and test_other.
The WER is either computed with sum-path algorithm or max-path algorithm.
The sum-pathalgorithm merges the alignment hypothesis corresponding to the same label sequence preﬁx afterremoval of epsilons.
In ideal decoding, sum-path should result in the most likely output label sequence.
The max-path algorithm computes the highest scoring path using algorithms in Appendix C.
Baselines
The RNN-T baselines are presented in the rows corresponding to the “ RNN-T ” contextdependency in Table 1a.
Unlike the other rows in Table 1a, the decoder RNN in RNN-T in principleencodes an unlimited output history as context. The standard RNN-T model effectively uses the labelframe dependent alignment lattice with an inﬁnite k . The rows corresponding to the frame dependentalignment lattice represent a slightly modiﬁed RNN-T model, where each frame is aligned to exactlyone output (either a blank, or a lexical label).
The WER difference between non-streaming RNN-Tbaselines in Table 1a and [15] are mainly due to our modiﬁcations to the Conformer encoder for acontrolled comparison against streaming models.
Choice of the context dependency
Table 1a compares effect of n -gram context dependency forn = 0 , 1 , 2 and baseline RNN-T models.
The general observation is that increasing n leads to betterperformance quality independent of the other choices of the modeling parameters.
However, themodel with 2 -gram context dependency already performs on par with RNN-T baseline.
The 1 -gramcontext dependency perform almost on par as baseline on clean test set while still lagging on theother test set.
This is consistent with the earlier observations in [40, 43, 11].
Choice of the alignment lattice
The comparison of different alignment lattices in Table 1asuggests that this choice does not signiﬁcantly contribute to the model performance.
While there is a performance gap for 0 -gram context dependency, we do not think there is a principal argumentin favor of frame dependent alignment lattice.
We speculate that this is more due to the choiceof optimization parameters.
However, the choice of lattice type can have some side effects.
Forexample as k in k -constrained label frame dependent alignment lattice increases, the model has moreability to delay its prediction to the end of the signal.
This implicit lookahead can translate intoperformance gains particularly for unidirectional models.
By limiting this quantity to 1, we observedthat performance on clean and other sets degrades by 34 .
9% and 36 .
7% , respectively.
Choice of the weight function normalization
Table 1b examines the effect of weight normalization on non-streaming and streaming models.
Here we present models with 1 -gram and 2 -gramcontext dependency with a frame dependent alignment lattice.
Note that for 0 -gram context dependency with a frame dependent alignment lattice it is easy to show that locally normalized and globallynormalized models are equivalent.
For non-streaming models, the normalization seems to not have animpact on the performance quality neither for 1 -gram nor for 2 -gram context dependency experiments.
This is validates our argument in the end of Section 2.2 that locally and globally normalized modelsexpress the same class of conditional distributions under this particular setting.
On the other hand,streaming models signiﬁcantly beneﬁt from global normalization: For clean test set, the globallynormalized model outperforms the locally normalized model by about 21% relative WER for 1 -gramcontext dependency and by about 20% relative gain for 2 -gram context dependency.
The globally normalized model with 2 -gram context dependency also beat the baseline streaming
RNN-T model in Table 1a and performs signiﬁcantly closer to the non-streaming RNN-T baseline.
The equivalent streaming RNN-T model performs 5 .
1% on test clean and the non-streaming modelperforms 2 .
5% on same test set.
The globally normalized model decoded with max-path algorithmperforms 3 . 8% on same test set.
The globally normalized model effectively closed almost 50% of theperformance gap between streaming and non-streaming models.
The reported performance for the globally normalized models is from max-path decoding, while thebaselines beneﬁt from sum-path decoding.
Comparing the locally normalized models’ WER frommax-path decoding in Table 1b and their counterparts in Table 1a, it is clear that sum-path decodingleads to an extra WER gain.
This gain is more signiﬁcant on test_other.
So we expect the globallymodel performs even better when decoded with sum-path.
The standard sum-path algorithms use several heuristics particularly for path merging and pruning.
While similar merging techniques can be applied to the globally normalized models, the pruningheuristics require several adjustments.
This is particularly due to the nature of the globally normalizedmodels where the transition weights are not constrained and can take any value, unlike locallynormalized models where the transition weights are constrained to be positive number between 0 and1 and sum to 1 for all the weights leaving the same state in recognition lattice.
We plan to exploreapproaches to improve globally normalized model decoding in the future.
Choice of the weight function architecture
Finally Table 1c compares different choices of thearchitectures for a streaming model with 2 -gram context dependency and frame dependent alignmentlattice.
While the unshared and shared-rnn architectures are very different in terms of parametersharing among states, both perform well, though the shared-rnn architecture performs slightly better.
The shared-emb architecture performs signiﬁcantly worst than shared-rnn architecture.
Note that theshared-rnn model is able to learn common structures across states in the context dependency whileshared-emb does not have such capability.
Conclusion
The GNAT model was proposed and evaluated with the focus on the label bias problem and its impacton the performance gap between streaming and non-streaming locally normalized ASR.
The ﬁnitecontext property of this model allows exact computation of the sequence level normalization whichmakes this model differ from existing globally normalized models.
Furthermore, the same propertyallows accelerator friendly training and inference.
We showed that the streaming models with globallynormalized criteria can signiﬁcantly close the gap between streaming and non-streaming modelsby more than 50%.
Finally, the modular framework introduced in this paper to explain the GNATmodel encompasses all the common neural speech recognition models.
This enables fair and accuratecomparison of different models via controlled modelling choices and creation of new ASR models.
A An overview example
To illustrate different components of the GNAT model, here we present a toy example of designing aspeech recognition for ﬁnite alphabet Σ
= { a, b } .
Given an input feature sequence x = x 1 . . .
x T , wewish to predict the corresponding output label sequence y =
y 1 . . .
y U , y i ∈ Σ .
Our objective is tocreate a conditional probabilistic model P ( y | x ) which assigns the highest probability to the correctlabel sequences for any given feature sequence.
We construct a GNAT model with the followingcomponents:
• context-dependency: 2 -gram
• alignment-lattice: frame dependent
• weight function: per-state linear projection, streaming
Next we elaborate details of each of these modules and how they are integrated to create the ﬁnalspace the recognition lattice
A θ, x and the probabilistic model P ( y | x ) as described in Section 3.
A.1 Context Dependency FSA
Figure 2 presents the 2 -gram context dependency C 3 .
The set of states for this space are the initialstate, 1-gram states and 2-gram states:
[Equation]
With the lexicographic order, these states are indexed as follow:state state index i a
[Equation]
ba
[Equation]
bb
For this particular FSA, the transitions space is
[Equation]
where q (cid:48) is the sufﬁx of qy with length at most 2 .
All 14 transitions of this space are listed in thefollowing table:from state label to state i a a i b b a a aa a b ab b a ba b b bb aa a aa aa b ab ab a ba ab b bb ba a aa ba b bb bb a ba bb b bb
A.2 Alignment Lattice FSA
Figure 1b depicts a frame dependent alignment lattice
L 4 for four frames feature sequence x .
Thestates of this space are:
[Equation]
i a b ab aa ba bb a b a b b a a b a b a b a b where 0 is the initial state and 4 is the ﬁnal state.
Every path starting from the initial state in thisautomaton corresponds to one possible alignment sequence of the input feature sequence.
Theexample FSA in Figure 1b encodes 3 4 = 81 possible alignment sequences.
An alignment path (cid:15)a(cid:15)bcorresponds to the following sequence of transitions in L 4 :(0 , (cid:15), 1) , (1 , a, 2) , (2 , (cid:15), 3) , (3 , b, 4)
A.3 Weight Function
The weight function ω θ, x : Q T × Q C × (Σ ∪ (cid:15) )
→ K .
ω θ, x is the only trainable component of the
GNAT model which assigns a weight to every transition of the recognition lattice A θ, x .
We ﬁrstfeed x = x 1 , ..., x 4 into an encoder to obtain hidden activations h
= h 1 , ..., h 4 of dimension D .
Theencoder can be any neural architecture such as DNNs, CNNs, RNNs or Transformers.
Since we areinterested in streaming weight function for this example, we need to make sure the encoder is alsostreaming.
This means h t can only depends on x 1: t − 1 .
Finally we deﬁne a D × 3 matrix W q c and a3 -dim bias vector b q c for any q c ∈ Q c .
The weight function is then deﬁned as:
[Equation]
θ,x 1: t (
[Equation]
t, q c , y ) [Equation] q c [ y, :] · h t + b q c [ y ] where W q c [ y, :] is the row of W q c corresponding to label y ∈ Σ ∪ { (cid:15) } .
The total number of trainableparameters is 7 × D × 3 + 7 × 3 parameters plus the number of parameters of the encoder function.
A.4 The recognition lattice A θ, x
Given the context dependency FSA C 3 , the frame dependent alignment lattice L 4 and the weightfunction ω θ, x , we are ready to derive the recognition lattice
A θ, x .
The state space has 5 × 7 states:
[Equation]
The transitions in this space is speciﬁed by the state it is originated from, ( t, q c ) , the label y ∈ Σ ∪{ (cid:15) } ,weight ω θ, x ( t, q c , y ) and the state the transition is ended to ( t + 1 , q (cid:48) c ) .
For the alignment sequenceof our example, (cid:15)a(cid:15)b , the transitions are:from label weight to
[Equation]
The product of the above weights is the score that the GNAT model assigns to the features sequencex and alignment sequence (cid:15)a(cid:15)b :
[Equation]
Note that for simplicity we use real semiring for all the score calculations in this example.
The GNAT model formulates the posterior probability P ( y | x ) by ratio of two quantities:
• numerator: sum of all the score ( x , ˜ y ) where ˜ y is an alignment between x and y .
For
example if y = ab , there are only 6 possible alignments: ab(cid:15)(cid:15)
, a(cid:15)b(cid:15) , a(cid:15)(cid:15)b , (cid:15)ab(cid:15) , (cid:15)a(cid:15)b , (cid:15)(cid:15)ab
• denominator: sum of all the score ( x , ˜ y ) where ˜ y can be any (Σ+1) T = 3 4 = 81 sequences.
Since the numerator computation is a special case of the denominator, we only present the denominatorcalculation.
To follow the computation presented in Section C, we ﬁrst deﬁne forward variable α twhich is a 7 -dim real-valued vector where α t [ j ] is the total alignment scores reaching to state index j(corresponding to the state indices in Q c ) at time t :
[Equation]
[Equation]
t [6] the initial state α 0 is a 1 -hot vector with α
[ j ] = 1 iff j = 0 , the initial state.
At every time frame t ,the transition weight matrix Ω t, Σ is deﬁned for all the transitions in E A θ, x where label is an elementof Σ .
This matrix is a structured matrix with only | Σ | = 2 non-zero elements per row:Ω t, Σ
= ( t, i ) ( t, a ) ( t, b ) ( t, aa ) , ( t, ab ) ( t, ba ) ( t, bb )
    ( t − 1 , i ) ω ( t, i, a ) ω ( t, i, b )    ( t − 1 , a )   ω ( t, a, a ) ω ( t, a, b )  ( t − 1 , b )     ω
( t, b, a ) ω ( t, b, b ) ( t − 1 , aa )   ω
( t, aa, a ) ω ( t, aa, b )  ( t − 1 , ab )     ω ( t, ab, a ) ω ( t, ab, b ) ( t − 1 , ba )   ω ( t, ba, a ) ω ( t, ba, b )  ( t − 1 , bb )     ω ( t, bb, a ) ω ( t, bb, b )
Similarly we denote Ω t,(cid:15) to be the transition weight matrix for all the transitions in E A θ, x where labelis (cid:15) .
This matrix is a diagonal matrix corresponding to the weights of the self loops:Ω t,(cid:15) =
( t, i ) ( t, a ) ( t, b ) ( t, aa ) , ( t, ab ) ( t, ba )
( t, bb )   
     ( t − 1 , i ) ω ( t, i, (cid:15) )      ( t − 1 , a ) ω ( t, a, (cid:15) )
( t − 1 , b )  ω ( t, b, (cid:15) )
( t − 1 , aa )   ω ( t, aa, (cid:15) )
( t − 1 , ab )    ω ( t, ab, (cid:15) )  ( t − 1 , ba )     ω ( t, ba, (cid:15) )
( t − 1 , bb )      ω
( t, bb, (cid:15) )
For our model, the forward variable α t can be calculated given α t − 1 and the above weights matricesas:
[Equation]
since every transition at time t + 1 is either an (cid:15) transition or a non- (cid:15) transition.
Here α (cid:48) t − 1 is thetranspose of forward variable α t − 1 .
Given the above iterative equation, the forward variable at time 4 is equal to:
[Equation] [Equation]
[Equation]
[Equation] [Equation]
+
[Equation]
) and the denominator of the GNAT model is equal to (cid:80) 6 j =
α 4 [ j ] .
Replacing the real semiring withtropical semiring in above calculation will allow us to ﬁnd the most likely alignment sequence.
B A Modular Framework
In this section we demonstrates how the existing and common neural speech recognition models canbe expressed within our proposed framework.
B.1 Cross-entropy with Alignments
The conventional cross-entropy models with feed-forward neural architectures [31] deﬁne the conditional probability of label sequence y given feature sequence x by:
[Equation]
where probability factors P θ ( y t | x t ) are derived by some neural architecture parameterized by θ :
[Equation]
where h t is the encoder activation of dimension D at time frame t , W is a weight matrix of shape| Σ | × D and b is a | Sigma | -dim bias vector.
The equivalent GNAT model is conﬁgured as follow:
• context dependency: 0 -gram C
• alignment lattice: frame dependent without (cid:15) transitions
• weight function:– ω θ, x ( q a = t, q c = i, y = y t ) (cid:44) P θ ( y = y t | x t )
– locally normalized
– streaming
here i is initial state of C 1 .
The more advanced cross-entropy models use recurrent architectures or transformers as encoder [39].
The only difference between the GNAT equivalent of these models and above conﬁguration is thatwhether the encoder is streaming or not.
B.2 Listen, Attend and Spell (LAS)
This model formulates the posterior probability by directly applying chain rule (Eq1 in [8]):
[Equation]
the posterior factors are deﬁned as (Eq6-Eq8 of [8]):
[Equation]
where
[Equation]
here Listen is a bidirectional encoder function, AttentionContext is the attention network (Eq9-Eq11of [8]).
The equivalent GNAT model is conﬁgured as follow:
• context dependency:
M -gram where M is the length of longest label sequence in the trainingdata.
Note that M -gram context dependency is equivalent of the tree space truncated atdepth M .
• alignment lattice: label dependent since the probability factorizes only on label sequence.• weight function:– ω θ, x ( q a = l, q c = q, y = y l ) (cid:44) P
θ ( y = y l | x , y <l )– locally normalized– non-streaming
B.3 Recurrent Neural Transducer
The RNNT model formulate the posterior probability as marginalization of alignment sequences (Eq1in the RNNT paper [13]):
[Equation]
where ˜ y =
˜
y 1 , · · · , ˜ y T + L is an alignment sequence, ˜ y i ∈ Σ ∪{ (cid:15) } , T is the number of acoustic framesand L is the number of labels.
The function B (˜ y ) =
y removes the epsilons from the alignmentsequence.
The alignment posterior is factorized along the alignment path as:
[Equation]
and ﬁnally RNNT make the following assumption:
[Equation]
which means if the preﬁx of two alignments be equal after epsilon removal, the model assigns sameexpansion probability for the next alignment position.
The inner terms in the above equation isdeﬁned (Eq12-Eq15 of the RNNT paper):
[Equation]
where h j − u is the encoder activation at time frame j − u (referred to as the transcription network in[13]) and g u is the output of the prediction network which is a simple stack of RNNs.
The equivalent GNAT model is conﬁgured as follow:
• context dependency:
M -gram where M is the maximum value of T + L in the training dataset.• alignment lattice: k -constrained label and frame dependent with k being the label sequencelength.
• weight function:
[Equation]
– locally normalized – non-streaming
While the original deﬁnition of the RNNT model is based on non-streaming encoder (transcriptionnetwork), this model is widely used for streaming applications by using a streaming encoder.
This isin contradiction of the forward-backward derivations in the original paper which explicitly assumesdependency on the whole sequence for any position of alignment sequence (Eq17 of [13])
Similar to RNNT, the hybrid autoregressive transducer (HAT) [40] model can be also conﬁgured inthe GNAT framework with the same parametrization as RNNT.
The only difference is the weightfunction.
The HAT model deﬁnes different probabilities for label transitions and epsilon transitions(duration model in [40]):
[Equation]
where b t,u is a sigmoid function deﬁned in Eq6 of [40].
(cid:15) a b (cid:15) : (cid:15) a : a b : b (cid:15) : (cid:15) a : (cid:15) b : b (cid:15) : (cid:15) a : a b : (cid:15)
B.4 Supporting CTC Style Label Deduplication
The standard CTC model [14] is very similar to a GNAT model using a frame dependent alignmentlattice and a 0 -gram context dependency.
One key difference is that CTC introduces a deduplicationprocess when turning its model output sequence to a label sequence.
Each model output of a CTCmodel is either a lexical label from Σ , or the special (cid:15) (blank) label.
To obtain the label sequence,two steps are applied on the model output in order,
Maximal consecutive repeated non- (cid:15) labels are merged into one (e
g. turning abbc into abc ,or abb(cid:15)b into ab(cid:15)b );2.
All the (cid:15) labels are removed.
As a comparison, paths on the alignment lattice of the GNAT models in the main paper is equivalentto the model outputs in CTC, whereas the (cid:15) free label sequence seen by the context dependency isequivalent to the label sequence in CTC.
To support the deduplication of repeated non- (cid:15) labels, weneed to introduce a ﬁnite state transducer into our series of ﬁnite state machine compositions.
Similarto a ﬁnite state automaton, a weighted ﬁnite-state transducer (WFST) T =
(Σ , Q, i, F, ρ, E ) over asemiring K is speciﬁed by a ﬁnite alphabet Σ , a ﬁnite set of states Q , an initial state i ∈ Q , a set ofﬁnal states F ⊆ Q , a ﬁnal state weight assignment ρ : F → K , and a ﬁnite set of transitions E .
3 Themeaning of Σ , Q , i , F , and ρ are identical to those of a WFSA.
The set of transitions E is instead asubset of Q × (Σ ∪ { (cid:15) } ) × (Σ ∪ { (cid:15) } )
× K × Q , i.e. containing a pair of input/output labels insteadof just one.
A WFSA can be viewed as a WFST with identical input/output labels on each arc, andsimilar to WFSA intersection, a series of WFST can be composed into a single WFST.
We referthe readers to [28] for a full description of WFST and the composition algorithm.
Figure 3 is anexample unweighted FST, when composed with another input FSA or FST, performs the CTC stylelabel deduplication.
More generally, the unweighted label deduplication transducer D of vocabularyΣ consists of
• States Q D = Σ ∪ { (cid:15) } , i D = (cid:15) , F D = Q D• Transitions E
= { ( p, x, x, x ) | p, x ∈ Q D } ∪ { ( x, x, (cid:15), x ) | x ∈ Σ }
Given a context dependency FSA C , an alignment lattice FSA L T , and the weight function ω θ, x , asdeﬁned in Section 3.3, a GNAT model with CTC style label deduplication induces a WFST T θ, x asfollows,
[Equation]
In other words
[Equation]
the topology of T θ, x is the same as the following cascade of FST compositions,
D · C treating output (cid:15) labels in D as empty (i
e. standard FST composition).
L T · ( D · C ) treating (cid:15) transitions in L T and input (cid:15) labels in ( D · C ) as regular labels
.and the transition weights are deﬁned using ω θ, x just like the GNAT models in the main paper.
When implemented naively, CTC style label deduplication causes a | Q D |
= ( V + 1) blow up in| Q θ, x | .
However, by inferring about states in Q D from states in Q C , we can greatly reduce thenumber of states needed.
For each state x ∈ Σ in Q D , we know the last non- (cid:15) label observed whenreaching state x must be label x .
Similarly, for context dependencies we care about ( n -gram andstring), there is a unique label x
( q c ) for all incoming arcs of each non-start state q c (start states do nothave any incoming arcs in these context dependencies).
Thus, the states in Q θ, x that are reachablefrom the start must match one of the following patterns,
• ( q a , (cid:15), q c ) , ∀ q a ∈ Q A , q c ∈ Q X• ( q a , x ( q c ) , q c ) , ∀ q a ∈ Q A , q c ∈ Q X \ { i X }
This means the actual number of states we shall visit in computing the shortest distance is only2 | Q A || Q C | .
C Accelerator-Friendly Computation
The standard shortest distance/path algorithm for acyclic WFSA [27] can be used for training(computing W ( A ) for some acyclic A ) and inference of a GNAT model.
To compute W ( A ) for anacyclic WFSA A , we maintain the following forward weight α q for each state q in Q A :
[Equation]
else .
The weight of A is then W ( A )
=
(cid:76)
q ∈ F A α q ⊗ ρ A ( q ) .
The recurrence in the deﬁnition of α q can becomputed by visiting states in Q A in a topological order.
To make better use of the compute power of modern accelerator hardware, we observe the followingproperties of the C or L T presented so far that enable us to use a more vectorized variant of theshortest distance algorithm in Figure 4:
• From any topological ordering on Q T , we can derive a topological ordering on Q θ, x .
• The n -gram context dependency FSA C n is deterministic, namely leaving any state there isno more than 1 transition for any label y ∈ Σ , and there is no (cid:15) -transition.•
For all three types of alignment lattices, for any non-ﬁnal state q ∈ Q T \ F T , there is aunique next state succ( q ) for transitions leaving q consuming any label y ∈ Σ .
{Initialize the length | Q C | forward weight vectors ¯ α q a } for all q a ∈ Q T do ¯ α q a ← [¯0 , . . . , ¯0] end for ¯ α i T [ i C ] ← ¯ {Compute ¯ α q a for q a (cid:54) =
i T } for all q a ∈ Q T in topological order do { Ω is a [ | Q C | , | Σ | + 1] matrix} Ω ← ¯
[Figure]
(cid:0)
[Figure]
} (cid:1) if q a has outgoing label transitions to q (cid:48) a = succ( q a ) then
[Figure]
end if for all q (cid:48)
a such that ( q a , (cid:15), q (cid:48) a ) ∈ E T do
[Figure]
end for end for
[Figure]
{Inputs: ¯ α q a and Ω[: , Σ] } if n = 1 then { Q C n contains only i C n } return ¯ α q a ¯ ⊗ (cid:76)
y ∈ Σ Ω[ i C n , y ] end if {Initialize length | Q C n | vector
¯
α } ¯ α ← [¯0 , . .
. , ¯0] {States in Q C n are numbered from 0 to | Q C n | − 1 = (cid:80) n − 1 i =1 | Σ | i following the lexicographic order} l ←for i = 0 to n − 2 do h ← l + | Σ | i ¯ α [ l · | Σ | + 1 : h · | Σ | + 1] ← ﬂatten(Ω[ l : h, Σ]) l ← h end for for i = 0 to | Σ | − 1 do ¯ α
[ l :] ← ¯ α
[ l :]¯ ⊕ ﬂatten(Ω[ l + i · | Σ | n − 2 : l + ( i + 1) · | Σ | n − 2 , Σ])
Center to an efﬁcient implementation of the algorithm in Figure 4 is the function next C .
Thisfunction receives as input the current forward weight vector ¯
α q a for states ( q a , q c ) , ∀ q c ∈ Q C , andthe transition weights for leaving these states via label transitions, and returns the forward weightsgoing to states ( q (cid:48) a , q (cid:48) c ) by taking the ( q c , y, q (cid:48) c ) transitions for y ∈ Σ .
In other words, next C [ q (cid:48) c ] = 680 (cid:76) ( q c ,y,q (cid:48) c ) ∈
E C ¯ α q a [ q c ] ⊗ Ω[ q c , y ] .
The n -gram context dependency C n allows a particularly simpleand efﬁcient implementation of next C n , as outlined in Figure 5.
The key observation is that whenwe number the states in Q C following the lexicographic order, the | Σ | transitions leaving the sameq c lead to states in a consecutive range [ σ ( q c y 0 ) , . . .
, σ ( q c y | Σ |− 1 )] , where [ y 0 , . . .
, y | Σ |− 1 ] are thelexicographically sorted labels of Σ , and σ ( s ) is the sufﬁx of label sequence s of length up to n − 1 .
During training, we also need to compute the shortest distance D ( A θ, x ∩ y ) .
We note the algorithmin Figure 4 can also be used for this purpose since ( L T ∩ C ) ∩ y = L T ∩ ( C ∩ y ) , and we simplyneed to substitute C with C ∩ y in the algorithm.
D Memory and Computation Time Benchmarks
The memory and computation benchmark of our implementation for the GNAT model is presented in
Table 2.
We present benchmarks for training and inference for different conﬁgurations of the GNATmodel:
• Context dependency: 0 -gram, 1 -gram and 2 -gram • Alignment lattice: frame dependent, 1-constrained label and frame dependent• Weight functions: Per-state linear projection (unshared), Shared linear projection withper-state embedding (shared-emb), Shared linear projection with RNN state embedding
[Table]
(shared-rnn)
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
global 158.
[Table]
local 157.
[Table]
[Table]
local 192.
global 192.
[Table] [Table] [Table] [Table]
global 205.
[Table] [Table] [Table] [Table]
global 181.
[Table]
global 199.
[Table] [Table] [Table] [Table]
[Table]
global 428.
[Table]
global 210.
[Table]
global 229.
include activations.
The training benchmarks are corresponding to thecalculation of the GNAT criterion as well as all the backward gradient calculation up to the encoder activations.
The evaluation benchmarks only contain the forward pass memory and compute footprintto ﬁnd the most likely hypothesis.
All the memory and computation benchmarks are evaluated for an input batch of 32 examples eachwith 1024 number of frames.
Each frame is a 512 -dim vector corresponding to the encoder activations.
Each example in the input batch are assumed to have at most 256 labels.
The alphabet size is set to32 .
The main observations are:
• The larger context dependency lead to more memory and compute footprint.
This is expectedsince the computation complexity is directly related to the context dependency state size.
However, interestingly, the memory and computation values do not scale exponentially byvalue of n in n -gram context dependency (as a result by number of states in the contextdependency).• label frame dependent alignment lattice generally leads to higher memory usage and computation time compare to the frame dependent alignment lattice.
This is expected since the labelframe dependent consist of alignment paths of length 1024 + 256 = 1270 , corresponding tothe sum of number of frames and number of labels.• The per-state linear projection weight function requires more memory and has longercompute time compare to the shared weights function which is expected by design.
Bothshared weight functions are performing on-par of each other in terms of memory andcompute.•
The global normalization requires more memory and time and the difference is moresigniﬁcant for context dependency FSAs with more number of states ( 2 -gram versus 1gram).
E Equivalence of Certain Locally and Globally Normalized Models underNon-Streaming Settings
As widely discussed in literature (e.g. Section 4.6.12 of [23], [24, 2]), under streaming settings, alocally normalized model introduces an independence assumption between an output z i and futureinputs x ≥ t ( i ) ,
when future outputs z >i are not yet known.
On the contrary, a globally normalizedmodel using the same score factorization does not introduce the same independence assumption.
Thisis the key reason to a locally normalized model’s inability to express certain conditional distributionsand thus prone to the label bias problem.
However, under non-streaming settings, the entire input sequence x is available for predicting each z i .
The aforementioned independence assumption is thus no longer present for locally normalized models.
We shall show that under a mild condition regarding how an alignment score ω
( z i | z
<
i , x ) makes useof information from the past outputs
z <i , for every non-streaming globally normalized model, thereis a non-streaming locally normalized model that deﬁnes the same conditional distribution for theoutput sequence.
For an alignment score ω ( z i | z
<
i , x ) , for each output step i , deﬁne the past output index set δ i ⊆{ 1 , 2 , . .
, i − 1 } to consist of the indices of past outputs on which ω
( z i | z <i , x ) actually depend.
Forexample, for a GNAT model using a full n-gram dependency with context size 2, δ i
= { i − 1 , i − 2 } ,i.e. the alignment score only depends on the output of the previous two outputs.
On the otherhand, an RNN-T model with unlimited history has δ i = { 1 , 2 , . .
, i − 1 } .
When comparing locallyand globally normalized models, it is important to ensure the same past output index sets are used.
Otherwise the comparison is not fair since the parameterization of these models will differ.
Given the sequence of past output index sets [ δ 1 , δ 2 , . . . ] , the alignment score function ω ( z i | z <i , x )can also be written as ω ( z i |
z δ i , x ) , with z δ i
=
z j | j ∈ δ i } .
The sequence of past output indexsets grows monotonically if for all i , δ i ∩ { 1 , 2 , . .
, i − 2 } ⊆
δ i − 1 .
In other words, each δ i can beconstructed from δ i − 1 by ﬁrst removing some elements then optionally adding i − 1 .
Importantly ifsome j < i is not in δ i , then j is not in any subsequent δ k for any k ≥ i .
Notably, all existing workon neural globally normalized ASR models including this paper itself use score functions where thesequence of past output index sets grows monotonically [17, 10, 42, 44].
When the sequence of past output index sets grows monotonically, globally normalized models andlocally normalized ones are in fact equivalent in terms of the conditional distributions that they canexpress.
Since any locally normalized model is by deﬁnition globally normalized, we only need toshow the other direction.
Theorem E.1.
For any non-streaming globally normalized model with alignment score ω
G ( z i | z δ i , x ) ,if the sequence of past index sets { δ 1 , δ 2 , . . . } grows monotonically, then there exists a non-streaminglocally normalized model with alignment score of the same sequence of index sets ω L ( z i |
z δ i , x ) , suchthat P ω G ( z | x ) = P ω L ( z | x ) .
This theorem can be seen a result of such a globally normalized model’s corresponding Markovnetwork representation being chordal (see Theorem 4.13 of [23]).
In an effort to be self-contained ,below we instead present a direct proof that does not rely on any results on graphical models, usingessentially the same technique as [9, 36].
Proof.
For any preﬁx z ≤ i , the sum of the scores of all possible sufﬁces continuing z ≤ i is commonlyknown as the backward score .
More speciﬁcally, in our case, the backward score at position i isβ i = (cid:88)
[Equation]
It is easy to see that β i only depends x and z j for j ∈ { i } ∪ δ i .
Thus β i ( z i , z δ i , x ) is a well-deﬁnedfunction.
β i is upper bounded by the denominator Z ( x ) and thus ﬁnite.
ω G ( z j | z δ , x ) > 0 impliesβ i > 0 .
We can thus deﬁne the following alignment score which is a locally normalized model,
[Equation]
Then
[Equation]
Therefore, P ω G ( z | x ) =
P ω L ( z | x ) .