Global Normalization for Streaming Speech Recognition in a Modular Framework
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We introduce the Globally Normalized Autoregressive Transducer (GNAT) foraddressing the label bias problem in streaming speech recognition.
Our solutionadmits a tractable exact computation of the denominator for the sequence-levelnormalization.
Through theoretical and empirical results, we demonstrate thatby switching to a globally normalized model, the word error rate gap betweenstreaming and non-streaming speech-recognition models can be greatly reduced (bymore than 50% on the Librispeech dataset).
This model is developed in a modularframework which encompasses all the common neural speech recognition models.
The modularity of this framework enables controlled comparison of modellingchoices and creation of new models.
Introduction
Deep neural network models have been tremendously successful in the ﬁeld of automatic speechrecognition (ASR).
Several different models have been proposed over the years: cross-entropy (CE)models with a deep feed-forward architecture [18], connectionist temporal classiﬁcation (CTC)models [13] with recurrent architectures such as long short-term memory (LSTM) [19], and morerecently sequence-to-sequence (Seq2Seq) models like listen, attend and spell (LAS) [8], recurrentneural network transducer (RNN-T) [12], and hybrid autoregressive transducer (HAT) [35].
Whenconﬁgured in non-streaming mode, these neural ASR models have reached state-of-the-art worderror rate (WER) on many tasks.
However, the WER signiﬁcantly drops when they are operating instreaming mode.
In this paper, we argue that one main cause of such WER gap is that all the existingmodels are constrained to be locally normalized which makes them susceptible to label bias problem[33, 2, 21, 4].
To address this problem, we introduce new category of globally normalized modelscalled Globally Normalized Autoregressive Transducer (GNAT).
Our contributions are:
(1) Addressing the label bias problem in streaming ASR through global normalization thatsigniﬁcantly closes more than 50% of the WER gap between streaming and non-streaming ASR.(2)
Efﬁcient, accelerator-friendly algorithms for the exact computation of the global normalization under the ﬁnite context assumption.
(3) A modular framework for neural ASR which encompasses all the common models (CE, CTC,LAS, RNN-T, HAT), allowing creation of new ones, and extension to their globally normalizedcounterparts.
Streaming Speech Recognition
For an input feature sequence x = x 1 . . .
x T , usually represented as a sequence of real valued featurevectors (such as log mel), and a ﬁnite output alphabet Σ , we wish to predict the corresponding outputlabel sequence y =
y 1 . . .
y U , y i ∈ Σ .
We call each element x
i in x a frame , and each elementy i in y an output label .
Common ASR models do not directly predict y , but rather a alignmentlabel sequence z
= z 1 . . .
z V , z i ∈ Σ ∪ ∆ .
∆ is a ﬁnite alphabet of control labels, such as the blanklabel in CTC or RNN-T, or the end-of-sequence label in LAS.
There is a deterministic mappingy ( z ) : (Σ ∪ ∆) ∗ → Σ ∗ for obtaining y from z (e.g. in RNN-T, we simply remove all the blank labelsfrom z ).
An ASR model can then be broken down in two tasks: (1) assigning a score (cid:81) i ω
( z i | x , z <i )to each alignment sequence z , where ω ( z i | x , z <i ) ≥ 0 is the alignment score of predicting a singlealignment label; (2) ﬁnding (usually approximately) arg max y (cid:80)
z | y ( z )=
y (cid:81) i ω
( z i | x , z <i ) .
A non-streaming ASR model’s alignment score ω
( z i | x , z <i ) has access to the entire x for any i .
Incontrast, a streaming model’s alignment score takes the form ω ( z i | x <t
( i ) , z <i ) : it only has access apreﬁx x <t ( i ) of the input feature sequence, where t ( i ) is the frame to which z
i is aligned .
Streamingmodels can be seen as special case of non-streaming models with respect to the alignment scores.
All the common neural ASR models use a locally normalized alignment score which satisﬁes theconstraint (cid:80) z ∈ Σ ∪ ∆ ω ( z i =
z | x , z <i ) = 1 .
This is achieved by applying the softmax function to thelast layer activations.
The local normalization constraint makes ω ( z i | x , z <i ) easily interpretable
asa conditional probability distribution P ω
( z i | x , z <i ) , and thus (cid:81)
i
ω
( z i | x , z <i ) easily interpretableas P ω ( z | x ) .
The modeling parameters are optimized by minimizing the negative log-conditionallikelihood loss E P ( x , y ) [ − log P ω ( y | x )] = E P ( x , y )
[ − log (cid:80) z | y ( z )=
y P ω ( z | x )] .
2.1 Label Bias in Streaming ASR
For a non-streaming, locally normalized model, the negative log-conditional-likelihood loss isminimized by setting ω ( z i | x , z <i ) to the true conditional probability P ( z i | x , z <i ) , leading to 54 (cid:81)
i ω ( z i | x , z <i ) being equal to the true posterior probability P ( z | x ) .
For a streaming model, x is replaced by x <t ( i ) in the alignment score (e.g. by using a unidirectionalencoder).
Here the negative log-conditional-likelihood loss is minimized by setting ω ( z i | x <t ( i ) , z <i )to P ( z i | x <t ( i ) , z <i ) .
As a result, the product (cid:81) i ω
( z i | x <t ( i ) , z <i ) =
(cid:81) i P ( z i | x <t ( i ) , z <i ) is ingeneral not equal to P ( y | x ) anymore.
In other words, using a streaming locally normalized modelmeans that the estimated alignment sequence posterior is the product of some locally normalizedalignment scores which depend only on partial input x <t ( i ) , and as a result can no longer accuratelyrepresent the true conditional distribution.
This will bias the model towards predictions with lowentropy estimated posterior probabilities at each decoding step.
This degrades the model ability torevise previous decisions, a phenomenon called label bias [21, 2].
2.2 Global Normalization and Non-Streaming ASR
Traditionally, globally normalized models such as conditional random ﬁelds [21] are used to addressthe label bias problem.
This paper seeks to apply global normalization to modern neural architecturesthat are more similar to CTC, RNN-T, or LAS, rather than traditional linear models with the purposeof addressing label bias problem for streaming ASR models.
A globally normalized model does not constrain the alignment score ω
( z i | x , z <i ) to be locallynormalized; it only requires it to be any non-negative score, as long as the denominator Z ( x ) =(cid:80) z (cid:81)
i ω
( z i | x , z <i ) is ﬁnite.
The ﬁnite denominator allows us to interpret (cid:81)
i ω
( z i | x , z <i )
Z ( x ) as aconditional probability distribution P ω ( z | x ) .
It is worth noting that any locally normalized model istrivially a globally normalized because Z ( x ) = 1 in this case.
Minimum negative log-conditionallikelihood training can be more expensive for globally normalized models due to the need to compute Z ( x ) and the corresponding gradients.
However with our proposed modular framework, globallynormalized model training can be made practical with careful modelling choices on modern hardware.
For non-streaming settings, [33] shows that locally and globally normalized models express thesame class of conditional distributions P ( z | x ) .
Based on this observation, we argue that undernon-streaming settings, with adequately powerful neural architectures, maximum log-conditional- likelihood training should yield behaviorly similar locally or globally normalized models, and thussimilar WERs in testing.
Results from [16] and our own experiments in Section 6 validate this.
A Modular Framework for Neural ASR
In this section, we introduce a modular framework for neural ASR, using the weighted ﬁnite stateautomaton (WFSA) formalism to calculate the conditional probabilities via alignment scores.
Themodular framework clearly expresses the modelling choices enabling practical globlally normalizedmodel training and inference.
We use the WFSA formalism as the language for describing ourframework because of its succinctness and precision, even though our algorithms cannot be directlyimplemented using existing toolkits such as OpenFst [1] or Kaldi [30].
3.1 Preliminaries
We begin with an introduction to the relevant concepts and notations.
A semiring ( K , ⊕ , ⊗ , ¯0 , ¯1) consists of a set K together with an associative and commutative operation⊕ and an associative operation ⊗ , with respective identities ¯ 0 and ¯ 1 , such that ⊗ distributes over⊕ , and ¯0 ⊗ x = x ⊗ ¯0 = ¯0 .
The real semiring ( R + , + , × , 0 , 1) is used when the weights representprobabilities.
The log semiring ( R ∪ {∞} , ⊕ log , + , ∞ , 0) , isomorphic to the real semiring via thenegative-log mapping, is often used in practice for numerical stability.
The tropical semiring( R ∪ {∞} , min , + , ∞ , 0) is often used in shortest-path applications.
A weighted ﬁnite-state automaton (WFSA)
A =
(Σ , Q, i, F, ρ, E ) over a semiring K is speciﬁed by aﬁnite alphabet Σ , a ﬁnite set of states Q , an initial state i ∈ Q , a set of ﬁnal states F ⊆ Q , a ﬁnal stateweight assignment ρ : F → K , and a ﬁnite set of transitions E ⊆ Q × (Σ ∪ { (cid:15) } ) × K × Q
( (cid:15) denotesthe empty label sequence).
Given a transition e ∈ E , p [ e ] denotes its origin or previous state, n [ e ]its destination or next state, o [ e ] its label, and ω [ e ] its weight.
A path π
= e 1 . . .
e k is a sequence ofconsecutive transitions e
i ∈ E : n [ e i − 1 ] =
p [ e i ] , i = 2 , . . .
k .
The functions n , p , and ω on transitionscan be extended to paths by setting: n [ π ] = n [ e k ] and p [ π ]
= p [ e 1 ] and by deﬁning the weight of apath as the ⊗ -product of the weights of its constituent transitions: ω
[ π ] = ω [ e 1 ] ⊗ · · · ⊗ ω [ e k ]) .
Anunweighted ﬁnite-state automaton (FSA)
A =
(Σ , Q, i, F, E ) is simply a WFSA whose transitions are all .Π
( Q 1 , Q 2 ) is the set of all paths from a subset Q 1 ⊆ Q to a subset Q 2 ⊆ Q .
Π( Q 1 , y , Q 2 )is the subset of all paths of Π( Q 1 , Q 2 ) with label sequence y
=
y 1 . . .
y U , y i ∈ Σ .
A path inΠ( { i } , F ) is said to be accepting or successful .
The weight associated by A to any label sequencey is given by A ( y ) =
(cid:76) π ∈ Π( { i } , y ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
The weight of A is the ⊕ -sum of weightsof all accepting paths W ( A ) =
(cid:76) π ∈ Π( { i } ,F ) ω
[ π ] ⊗ ρ ( n [ π ]) .
For a semiring K where ⊗ is alsocommutative, the intersection (or Hadamard product) of two WFSA A 1 and A 2 is deﬁned as:( A 1 ∩ A 2 )( y ) =
A 1 ( y ) ⊗ A 2 ( y ) .
[25] gives an algorithm to compute the intersection.
We can viewy as a WFSA that accepts only y with weight ¯1 , then A ( y )
=
W ( A ∩ y ) .
3.2 Probabilistic Modeling and Inference on Acyclic Recognition Lattices
For any feature sequence x = x 1 . . .
x T , a model with trainable parameters θ induces a recognitionlattice WFSA A θ, x = (Σ , Q θ, x , i θ, x , F θ, x , ρ θ, x , E θ, x ) .
For a label sequence y =
y 1 . . .
y U , therecognition lattice
A θ, x ( y ) under the log semiring can be viewed as the unnormalized negative logconditional probability
[Equation]
The recognition lattice A θ, x is designed to be acyclic, and therefore the weight of the automata inboth the numerator and denominator above can be efﬁciently computed by visiting the states of thecorresponding WFSA in topological order [24].
See Appendix C for our accelerator friendly versionof this algorithm.
We can thus train an ASR model by minimizing the negative log-conditionallikelihood on the training corpus D , and choosing θ (cid:63) =
arg min θ E ( x , y ) ∈D [ − log( P θ ( y | x ))]
=arg min θ E ( x , y ) ∈D
[ W ( A θ, x ∩ y ) − W
( A θ, x )] .
[Equation]
In general, there can be more than one path in A θ, x that accepts the same y .
During inference, ﬁndingthe optimal ˆ y = arg max y P θ
( y | x ) requires running the potentially expensive WFSA disambiguationalgorithm [27] on A θ, x .
As a cheaper approximation, we instead look for the shortest path ˆ π in A θ, xunder the tropical semiring, and use the corresponding label sequence as the prediction, again usingthe standard shortest path algorithm for an acyclic WFSA [24].
3.3 Inducing the WFSA
Our framework decomposes the sequence prediction task in ASR into three components, each playinga speciﬁc role in inducing the recognition lattice
A θ, x .
• The context dependency FSA C
=
(Σ , Q C , i C , F C , E C ) is an (cid:15) -free, unweighted FSA,whose states encode the history of the label sequence produced so far.
C is ﬁxed for a given
GNAT model, independent of input x .
• The alignment lattice FSA L T =
(Σ , Q T , i T , F T , E T ) is an acyclic, unweighted FSA,whose states encode the alignment between input frames x and output labels y .
L T dependson only the length T of input x .
• The weight function ω θ, x : Q T × Q C × (Σ ∪ (cid:15) )
→ K .
ω θ, x is the only component
that contains trainable parameters and requires full access to x .
This function deﬁnes thetransition weights in the recognition lattice
A θ, x .
We will discuss how one can deﬁne these components in detail in the next section.
With ( C, L T , ω θ, x )given, the recognition lattice A θ, x is deﬁned as follows:
[Equation]
In other words, the topology (states and unweighted transitions) of the recognition lattice
A θ, x isthe same as the FSA intersection L T ∩ C ; and the transition weights are deﬁned using ω θ, x . The(cid:15) -freeness of C and the acyclicity of L T implies that the recognition lattice A θ, x is also acyclic.
Components of a GNAT Model
In this section, we deﬁne globally normalized autoregressive transducer (GNAT) through the framework above, by specifying each model component.
4.1 Context Dependency
GNAT uses an n -gram context-dependency deﬁned by C n = (Σ , Q n , i n , F n , E n ) , where Q n =Σ ≤ n − 1 corresponds to a label history of length up to n − 1 .
The initial state i n = (cid:15) is the emptylabel sequence.
The transitions E n ⊆ Q n × Σ × Q n correspond to truncated concatenation: E n = (cid:8) ( q, y, q (cid:48) )
| q ∈ Q, y ∈ Σ) (cid:9) where q (cid:48) is the sufﬁx of qy with length at most n − 1 .
Forexample, when n = 3 , the transition ( ab, c, bc ) goes from state ab to state bc with label c .
All statesare ﬁnal: F n = Q n .
See Figure 1a for the FSA C 2 when Σ = { a, b } .
The intersection L T ∩ C n iseasy to compute thanks to the absence of (cid:15) -transitions in C n .
Appendix C demonstrates how C n canbe efﬁciently intersected during the shortest distance computation.
Although not studied in this paper, our modular framework makes it easy to switch to a moresophisticated context dependency, such as clustered histories often used for context-dependent phonemodels, or a variable context length as used in n -gram language models [26].
(cid:15) a b a b a b a b (
[Figure]
n -gram context-dependency automaton C 2 for Σ = { a, b } .
3 Σ (cid:15) Σ (cid:15) Σ (cid:15) (b)
Frame dependent alignment lattice with T = 3 .
(0 , 0) (0 , 1) (0 , 2) (1 , 0) (1 , 1) (1 , 2) (2 , 0) (2 , 1) (2 , 2) (3 , 0) (cid:15) Σ (cid:15) Σ (cid:15) (cid:15) Σ (cid:15) Σ (cid:15) (cid:15) Σ (cid:15) Σ (cid:15) (
[Figure]
and
[Figure] [Figure]
lattice with
[Figure] [Figure]
k
= 2 .
3 Σ (cid:15) Σ (cid:15) Σ (cid:15) (
[Figure]
alignment lattice with l
[Figure]
) = 3 .
4.2 Alignment Lattices
Given the feature sequence length T , the alignment lattice FSA L T deﬁnes all the possible alignmentsbetween the feature sequence and allowed label sequences.
Since the feature sequence length Tusually differs from the label sequence length
U , many different alignments between the featuresequence and a label sequence can be deﬁned.
The states in an alignment lattice FSA encode how thenext label or (cid:15) -transition corresponds to some position in the feature sequence.
We can choose different structures for L T by encoding one or both of the positions in the featuresequence and the label sequence.
A simple example is the frame dependent alignment similar to [13],where each frame is aligned to at most one label:
[Equation]
Here any state t < T represents a position in the feature sequence.
We start by aligning to the initialframe of the feature sequence, and repeatedly shift to the next frame for every subsequent label or(cid:15) -transition until all frames have been visited.
Figure 1b depicts a frame dependent alignment lattice.
To allow a label sequence longer than the feature sequence, we can use the k -constrained label andframe dependent alignment similar to [12]:
[Equation]
Here, up to k consecutive label transitions can align to any single frame.
An (cid:15) -transition is then takento explicitly shift the alignment to the next frame.
The number of labels aligned to one frame isconstrained by a constant k solely in order to impose acyclicity.
Figure 1c depicts a k -constrainedlabel and frame dependent alignment lattice.
Some models may only depend on the position in the label sequence, similar to [8].
In this casewe can bound the length of the label sequence by some function l ( T ) , and use the following label dependent alignment :
[Equation]
Here, each label can be seen as aligning to the entire feature sequence, and the (cid:15) -transition serves asan explicit termination of the label sequence.
Figure 1d depicts a label dependent alignment lattice.
4.3 Weight functions
The weight function ω θ, x translates trainable parameters into transition weights for states in therecognition lattice
A θ, x .
The choice of the weight function depends on the choice of the context andalignment lattice FSA, especially the alignment lattice where the meaning of a state directly affectshow the weight function can access x .
In the experiments discussed in this paper, we need concrete weight functions for frame dependentand k -constrained label and frame dependent alignment lattices.
In these two types of alignmentlattices, a non-ﬁnal state q a in Q T contains a position τ ( q a ) in the feature sequence (the state itselfin the case of frame dependent alignment lattices; the ﬁrst value in the state in the case of label andframe dependent alignment lattices).
Weight functions can thus be deﬁned in three steps,
Feed x into an encoder, such as unidirectional or bidirectional RNN or a self-attention encoder to obtain the sequence of hidden units h of dimension D .
In the experiments
we
compared streaming vs non-streaming encoders.
Map a single frame of hidden units h
[ t ] and context state q c to a ( | Σ | + 1) -dimensional vector, corresponding to the unnormalized transition weights for y ∈ Σ ∪ { (cid:15) } .
Optionally locally normalize transition weights across y ∈ Σ ∪ { (cid:15) } given ( q a , q
c ) .
For step 2, we experiment with the following concrete modelling choices with varying degree ofparameter sharing,
Per-state linear projection ( unshared )
For every context state q c , we obtain a D × ( | Σ | + 1)projection matrix W q c and a ( | Σ | + 1) -dim
bias vector b q c from θ , and deﬁne for y ∈ Σ ∪ { (cid:15) } :
[Equation]
Shared linear projection with per-state embedding ( shared-emb )
We obtain from θ (a) for everycontext state q c a D -dimensional state embedding E q c , (b) independent of context states a D × ( | Σ |
+1)projection matrix W and a ( | Σ | + 1) -dim bias vector b , and deﬁne for y ∈ Σ ∪ { (cid:15) } :
[Equation]
Shared linear projection with RNN state embedding ( shared-rnn )
Similar to shared linear projection with per-state embedding but E q c is obtained from running an RNN (e.g. LSTM) on the n -gramlabel sequence represented by q c .
Discussion
A
Modular Framework
All the existing locally normalized models can be explained withinthe modular framework presented in Section 3 with a particular choice of context size, alignmentlattice and of course with constraining the weights to be locally normalized using softmax function.
Appendix B presents how CE, CTC, LAS, RNNT and HAT models can be expressed within thisframework.
This allows controlled comparison of different components as well as creating newmodels by mixing different modeling choices.
Like locally and globally models in general, when theweights are locally normalized, the denominator of models deﬁned in our framework is one.
Finally, our framework is very different from traditional uses of ﬁnite state machines via a cascade ofweight ﬁnite state transducer compositions.
Our separation of the weight function from the automaton topology allows an arbitrarily complex, non-linear weight function to model the dependency amongalignment states, context states, and output label, which is impossible with composition cascades.
Related Globally Normalized Models
There is a rich literature on the applications of globallynormalized models [5, 23, 32, 21] as well as detailed studies on the importance of global normalizationin addressing the label bias problem [21, 2, 11].
In the context of ASR there is a lot of researchon applying globally normalized models [3, 7, 6, 15, 22, 40, 17].
Among these, MMI [3, 7] is themost relevant globally normalized criterion to our work.
In MMI a sequence level score is factorizedby a likelihood score which comes from an acoustic model and a prior score which is usually aword based language model (LM).
The denominator score is then approximated over a lattice ofhypotheses.
More recently a lattice free version of this criterion has been introduced [31] whichreplaces the word level LM with a 4-gram phone LM.
Both the GNAT and MMI criteria are globallynormalized.
The GNAT model differentiates itself from MMI in several ways. First unlike MMI itdoes not require an external LM and it does not apply any constraint on how the sequence level scoresare deﬁned.
Second, in MMI the language model is kept frozen while the acoustic model parametersare updated via optimization of the MMI criterion.
In GNAT all the model parameters are trainedtogether.
Third, GNAT provides the exact computation of the denominator while standard MMI onlyoffers an approximation.
Finally GNAT trains from scratch without any need for initialization orspecial regularization techniques as used in the lattice-free version of MMI [31].
In addition we wereable to train GNAT models with accelerators without any techniques discussed in [31].
Finally the concept of global normalization has also been visited with deep neural networks [16, 9,37, 39].
These models can be seen as special cases of MMI thus all the differences between MMIand GNAT model applies here as well.
Apart from their weaker modelling power as a result of using WFST composition cascades, all these models are non-streaming, where as explained multiple timesin our paper, global normalization and local normalization are equally expressive.
Challenges
The main challenge with the GNAT model is its scalability to a larger number of labelcontexts.
At each training step, the model requires 2 × | Q C | × | Σ | multiplication and summation.
For n -gram context dependency, | Q c | = | Σ | n +1 − 1 | Σ |− 1 , thus the computation scale exponentially by valueof n .
However as shown in Appendix D, due to the particular structure of this space the practicalcomputation and memory cost benchmarks do not scale exponentially with n .
We also note thatlarge value of n might also not be necessary: The HAT model [35] reports that a Seq2Seq modelwith a label history of just the two previous phonemes performs on par with a similar model with afull history trained on very large voice-search corpus.
Similar observations are reported in studieswith grapheme and wordpiece units [38, 10].
Due to the data sparsity there might not be enoughtraining to fully represent a n -gram space, so increasing the value of n might not necessarily lead toperformance improvement.
One way of dealing with large number of states is to use standard pruningtechniques to keep only some of the most common states in the training data.
ExperimentsData
We use the full 960 -hour Librispeech corpus [29] for experiments.
The input features are80-dim.
log Mel extracted from a 25 ms window of the speech signal with a 10 ms shift.
The
SpecAugment library with baseline recipe parameters were used [14].
The transcript truth is usedwithout any processing and tokenized by the 28 graphemes that appear in the training data.
Architecture
Attention-based architectures allows using the same parameterization for streamingand non-streaming models, thus for all the experiments we used 12 -layer Conformer encoders [14]with model dimension 512, followed by a linear layer with output dimension 640.
The Conformerparameters are set such that the only difference between streaming and non-streaming models isthe right context: at each time frame t , the streaming models only access the left context (featureframes from 1 to t ), while the non-streaming models can see the entire acoustic feature sequence.
Toenforce the consistency of the encoder architecture between streaming and non-streaming modes, weremoved all the sub-architecture which behaved differently between these two modes.
Speciﬁcally, weremoved the convolution sub-sampling layer, and also forced the stacking layers to only stack withinthe left context.
The baseline experiments use a shared-rnn weight function deﬁned in section 4.3.
A single layer LSTM is used with 640 cells.
The experiments with the unshared weight functionuse a linear layer of size ( | Q C | × | Σ | ) × 640 to project the encoder activation at each time frameinto the transition weights of the recognition lattice.
In our experiments, | Σ | = 32 .
For the n -gram [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Q c | = | Σ | n +1 − 1 | Σ |− 1 .
The experiments with the shared-emb weight function use anembedding table of size | Q c | × 128 .
Training
All models are trained on 8 × 8 TPUs with a batch size 2048 .
The training exampleswith more than 1961 feature frames or more than 384 labels are ﬁltered out.
We used Adam optimizer[20] ( β 1 = 0 .
9 , β 2 = 0 .
98 , and (cid:15) =
10 − 9 ) with the Transformer learning rate schedule [36] (10kwarm-up steps and peak learning rate 0 .
05 / √ 512 ).
We applied the same regularization techniquesand the training hyperparameters used in the baseline recipe of [14].
Evaluation
We report WER results on standard Librispeech test sets: test_clean and test_other.
The WER is either computed with sum-path algorithm or max-path algorithm.
The sum-pathalgorithm merges the alignment hypothesis corresponding to the same label sequence preﬁx afterremoval of epsilons.
In ideal decoding, sum-path should result in the most likely output label sequence.
The max-path algorithm computes the highest scoring path using algorithms in Appendix C.
Baselines
The RNN-T baselines are presented in the row corresponding to the M -gram contextdependency in Table 1a.
For frame dependent alignment lattice, M is equal to the length of thelongest feature sequence in the training data ( 1961 ) and for label frame dependent M is equal to1961 + 384 , sum of the maximum feature sequence length and the maximum label sequence length. The label frame dependent alignment lattice used for the baseline and all the other experiments isk -constrained with k set to the number of labels for each training example (maximum value 384).
The WER difference between non-streaming baselines in Table 1a and [14] are mainly due to ourmodiﬁcations to the Conformer encoder for a controlled comparison against streaming models.
Choice of the context dependency
Table 1a compares effect of n -gram context dependency forn = 0 , 1 , 2 and baseline RNN-T models.
The general observation is that increasing n leads to betterperformance quality independent of the other choices of the modeling parameters.
However, themodel with 2 -gram context dependency already performs on par with RNN-T baseline.
The 1 -gramcontext dependency perform almost on par as baseline on clean test set while still lagging on theother test set.
This is consistent with the earlier observations in [35, 38, 10].
Choice of the alignment lattice
The comparison of different alignment lattices in Table 1asuggests that this choice does not signiﬁcantly contribute to the model performance.
While there isa performance gap for 0 -gram context dependency, we do not think there is a principal argumentin favor of frame dependent alignment lattice.
We speculate that this is more due to the choiceof optimization parameters.
However, the choice of lattice type can have some side effects.
Forexample as k in k -constrained label frame dependent alignment lattice increases, the model has more ability to delay its prediction to the end of the signal.
This implicit lookahead can translate intoperformance gains particularly for unidirectional models.
By limiting this quantity to 1, we observedthat performance on clean and other sets degrades by 34 .
9% and 36 .
7% , respectively.
Choice of the weight function normalization
Table 1b examines the effect of weight normalization on non-streaming and streaming models.
Here we present models with 1 -gram and 2 -gramcontext dependency with a frame dependent alignment lattice.
Note that for 0 -gram context dependency with a frame dependent alignment lattice it is easy to show that locally normalized andglobally normalized models are equivalent.
For non-streaming models, the normalization seems tonot have an impact on the performance quality neither for 1 -gram nor for 2 -gram context dependencyexperiments.
This is expected since the full acoustic feature sequence context allows the model toavoid the label bias problem [2, 11] which is consistent with the equal expressiveness of globallyand locally normalized models under non-streaming setting [33].
On the other hand, streamingmodels signiﬁcantly beneﬁt from global normalization: For clean test set, the globally normalizedmodel outperforms the locally normalized model by about 21% relative WER for 1 -gram contextdependency and by about 20% relative gain for 2 -gram context dependency.
The globally normalized model with 2 -gram context dependency also beat the baseline streaming
RNN-T model in Table 1a and performs signiﬁcantly closer to the non-streaming RNN-T baseline.
The equivalent streaming RNN-T model performs 5 .
1% on test clean and the non-streaming modelperforms 2 .
5% on same test set.
The globally normalized model decoded with max-path algorithmperforms 3 . 8% on same test set.
The globally normalized model effectively closed almost 50% of theperformance gap between streaming and non-streaming models.
The reported performance for the globally normalized models is from max-path decoding, while thebaselines beneﬁt from sum-path decoding.
Comparing the locally normalized models’ WER frommax-path decoding in Table 1b and their counterparts in Table 1a, it is clear that sum-path decodingleads to an extra WER gain.
This gain is more signiﬁcant on test_other.
So we expect the globallymodel performs even better when decoded with sum-path.
The standard sum-path algorithms use several heuristics particularly for path merging and pruning.
While similar merging techniques can be applied to the globally normalized models, the pruningheuristics require several adjustments.
This is particularly due to the nature of the globally normalizedmodels where the transition weights are not constrained and can take any value, unlike locallynormalized models where the transition weights are constrained to be positive number between 0and 1 and sum to 1 for all the weights leaving the same state in recognition lattice.
We can alsoreduce the performance gap between max-path and sum-path by constraining the training criterion todistribute the whole probability mas into one one alignment path. This can be done by constrainingthe objective function with alignment path entropy. This effectively avoids the need for sum-pathinference. We will present these approaches in our future publication.
Choice of the weight function architecture
Finally Table 1c compares different choices of thearchitectures for a streaming model with 2 -gram context dependency and frame dependent alignmentlattice.
While the unshared and shared-rnn architectures are very different in terms of parametersharing among states, both perform well, though the shared-rnn architecture performs slightly better.
The shared-emb architecture performs signiﬁcantly worst than shared-rnn architecture.
Note that theshared-rnn model is able to learn common structures across states in the context dependency whileshared-emb does not have such capability.
Conclusion
The GNAT model was proposed and evaluated with the focus on the label bias problem and its impacton the performance gap between streaming and non-streaming locally normalized ASR.
The ﬁnitecontext property of this model allows exact computation of the sequence level normalization whichmakes this model differ from existing globally normalized models.
Furthermore, the same propertyallows accelerator friendly training and inference.
We showed that the streaming models with globallynormalized criteria can signiﬁcantly close the gap between streaming and non-streaming modelsby more than 50%.
Finally, the modular framework introduced in this paper to explain the GNATmodel encompasses all the common neural speech recognition models.
This enables fair and accuratecomparison of different models via controlled modelling choices and creation of new ASR models.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A An overview example
To illustrate different components of the GNAT model, here we present a toy example of designing aspeech recognition for ﬁnite alphabet Σ
= { a, b } .
Given an input feature sequence x = x 1 . . .
x T , wewish to predict the corresponding output label sequence y =
y 1 . . .
y U , y i ∈ Σ .
Our objective is tocreate a conditional probabilistic model P ( y | x ) which assigns the highest probability to the correctlabel sequences for any given feature sequence.
We construct a GNAT model with the followingcomponents:
• context-dependency: 2 -gram
• alignment-lattice: frame dependent
• weight function: per-state linear projection, streaming
Next we elaborate details of each of these modules and how they are integrated to create the ﬁnalspace the recognition lattice
A θ, x and the probabilistic model P ( y | x ) as described in Section 3.
A.1 Context Dependency FSA
Figure 2 presents the 2 -gram context dependency C 3 .
The set of states for this space are the initialstate, 1-gram states and 2-gram states:
[Equation]
With the lexicographic order, these states are indexed as follow:state state index i
[Equation]
ba
[Equation]
bb
For this particular FSA, the transitions space is
[Equation]
where q (cid:48) is the sufﬁx of qy with length at most 2 .
All 14 transitions of this space are listed in thefollowing table:from state label to state i a a i b b a a aa a b ab b a ba b b bb aa a aa aa b ab ab a ba ab b bb ba a aa ba b bb bb a ba bb b bb
Figure 1b depicts a frame dependent alignment lattice
L 4 for four frames feature sequence x .
Thestates of this space are:
[Equation]
i a b ab aa ba bb a b a b b a a b a b a b a b where 0 is the initial state and 4 is the ﬁnal state.
Every path starting from the initial state in thisautomaton corresponds to one possible alignment sequence of the input feature sequence.
Theexample FSA in Figure 1b encodes 3 4 = 81 possible alignment sequences.
An alignment path (cid:15)a(cid:15)bcorresponds to the following sequence of transitions in L 4 : [Equation]
A.3 Weight Function
The weight function ω θ, x : Q T × Q C × (Σ ∪ (cid:15) )
→ K .
ω θ, x is the only trainable component of the
GNAT model which assigns a weight to every transition of the recognition lattice A θ, x .
We ﬁrstfeed x = x 1 , ..., x 4 into an encoder to obtain hidden activations h
= h 1 , ..., h 4 of dimension D .
Theencoder can be any neural architecture such as DNNs, CNNs, RNNs or Transformers.
Since we areinterested in streaming weight function for this example, we need to make sure the encoder is alsostreaming.
This means h t can only depends on x 1: t − 1 .
Finally we deﬁne a D × 3 matrix W q c and a3 -dim bias vector b q c for any q c ∈ Q c .
The weight function is then deﬁned as:
[Equation]
[Equation]
] where W q c [ y, :] is the row of W q c corresponding to label y ∈ Σ ∪ { (cid:15) } .
The total number of trainableparameters is 7 × D × 3 + 7 × 3 parameters plus the number of parameters of the encoder function.
A.4 The recognition lattice A θ, x
Given the context dependency FSA C 3 , the frame dependent alignment lattice L 4 and the weightfunction ω θ, x , we are ready to derive the recognition lattice
A θ, x .
The state space has 5 × 7 states:
[Equation]
The transitions in this space is speciﬁed by the state it is originated from, ( t, q c ) , the label y ∈ Σ ∪{ (cid:15) } ,weight ω θ, x ( t, q c , y ) and the state the transition is ended to ( t + 1 , q (cid:48) c ) .
For the alignment sequenceof our example, (cid:15)a(cid:15)b , the transitions are:from label weight to
[Equation]
4 , ab
[Equation]
The product of the above weights is the score that the GNAT model assigns to the features sequencex and alignment sequence (cid:15)a(cid:15)b :
[Equation]
Note that for simplicity we use real semiring for all the score calculations in this example.
The GNAT model formulates the posterior probability P ( y | x ) by ratio of two quantities:
• numerator: sum of all the score ( x , ˜ y ) where ˜ y is an alignment between x and y .
For
example if y = ab , there are only 6 possible alignments: ab(cid:15)(cid:15) a(cid:15)b(cid:15) a(cid:15)(cid:15)b , (cid:15)ab(cid:15) ,
(cid:15)a(cid:15)b (cid:15)(cid:15)ab
• denominator: sum of all the score ( x , ˜ y ) where ˜ y can be any (Σ+1) T = 3 4 = 81 sequences.
Since the numerator computation is a special case of the denominator, we only present the denominatorcalculation.
To follow the computation presented in Section C, we ﬁrst deﬁne forward variable α twhich is a 7 -dim real-valued vector where α t [ j ] is the total alignment scores reaching to state index j(corresponding to the state indices in Q c ) at time t : [Equation]
[Equation]
t [6] the initial state α 0 is a 1 -hot vector with α
[ j ] = 1 iff j = 0 , the initial state.
At every time frame t ,the transition weight matrix Ω t, Σ is deﬁned for all the transitions in E A θ, x where label is an elementof Σ .
This matrix is a structured matrix with only | Σ | = 2 non-zero elements per row:Ω t, Σ
= ( t, i ) ( t, a ) ( t, b ) ( t, aa ) , ( t, ab ) ( t, ba ) ( t, bb )
    ( t − 1 , i ) ω ( t, i, a ) ω ( t, i, b )    ( t − 1 , a )   ω ( t, a, a ) ω ( t, a, b )  ( t − 1 , b )     ω
( t, b, a ) ω ( t, b, b ) ( t − 1 , aa )   ω
( t, aa, a ) ω ( t, aa, b )  ( t − 1 , ab )     ω ( t, ab, a ) ω ( t, ab, b ) ( t − 1 , ba )   ω ( t, ba, a ) ω ( t, ba, b )  ( t − 1 , bb )     ω ( t, bb, a ) ω ( t, bb, b )
Similarly we denote Ω t,(cid:15) to be the transition weight matrix for all the transitions in E A θ, x where labelis (cid:15) .
This matrix is a diagonal matrix corresponding to the weights of the self loops:Ω t,(cid:15) =
( t, i ) ( t, a ) ( t, b ) ( t, aa ) , ( t, ab ) ( t, ba )
( t, bb )   
     ( t − 1 , i ) ω ( t, i, (cid:15) )      ( t − 1 , a ) ω ( t, a, (cid:15) )
( t − 1 , b )  ω ( t, b, (cid:15) )
( t − 1 , aa )   ω ( t, aa, (cid:15) )
( t − 1 , ab )    ω ( t, ab, (cid:15) )  ( t − 1 , ba )     ω ( t, ba, (cid:15) )
( t − 1 , bb )      ω
( t, bb, (cid:15) )
For our model, the forward variable α t can be calculated given α t − 1 and the above weights matricesas:
[Equation]
since every transition at time t + 1 is either an (cid:15) transition or a non- (cid:15) transition.
Here α (cid:48) t − 1 is thetranspose of forward variable α t − 1 .
Given the above iterative equation, the forward variable at time 4 is equal to:
[Equation] [Equation]
[Equation] [Equation]
[Equation]
[Equation] [Equation]
Σ + Ω
[Equation]
) and the denominator of the GNAT model is equal to (cid:80) 6 j =
α 4 [ j ] .
Replacing the real semiring withtropical semiring in above calculation will allow us to ﬁnd the most likely alignment sequence.
B A Modular Framework
In this section we demonstrates how the existing and common neural speech recognition models canbe expressed within our proposed framework.
B.1 Cross-entropy with Alignments
The conventional cross-entropy models with feed-forward neural architectures [28] deﬁne the conditional probability of label sequence y given feature sequence x by:
[Equation]
where probability factors P θ ( y t | x t ) are derived by some neural architecture parameterized by θ :
[Equation]
where h t is the encoder activation of dimension D at time frame t , W is a weight matrix of shape| Σ | × D and b is a | Sigma | -dim bias vector.
The equivalent GNAT model is conﬁgured as follow:
• context dependency: 0 -gram C
• alignment lattice: frame dependent without (cid:15) transitions • weight function:
– ω θ, x ( q a = t, q c = i, y = y t ) (cid:44) P θ ( y = y t | x t )
– locally normalized – streaming
here i is initial state of C 1 .
The more advanced cross-entropy models use recurrent architectures or transformers as encoder [34].
The only difference between the GNAT equivalent of these models and above conﬁguration is thatwhether the encoder is streaming or not.
B.2 Listen, Attend and Spell (LAS)
This model formulates the posterior probability by directly applying chain rule (Eq1 in [8]):
[Equation]
the posterior factors are deﬁned as (Eq6-Eq8 of [8]):
[Equation]
where
[Equation]
here Listen is a bidirectional encoder function, AttentionContext is the attention network (Eq9-Eq11of [8]).
The equivalent GNAT model is conﬁgured as follow:
• context dependency:
M -gram where M is the length of longest label sequence in the trainingdata.
Note that M -gram context dependency is equivalent of the tree space truncated atdepth M .
• alignment lattice: label dependent since the probability factorizes only on label sequence.• weight function:– ω θ, x ( q a = l, q c = q, y = y l ) (cid:44) P
θ ( y = y l | x , y <l )– locally normalized– non-streaming
B.3 Recurrent Neural Transducer
The RNNT model formulate the posterior probability as marginalization of alignment sequences (Eq1in the RNNT paper [12]):
[Equation]
where ˜ y =
˜
y 1 , · · · , ˜ y T + L is an alignment sequence, ˜ y i ∈ Σ ∪{ (cid:15) } , T is the number of acoustic framesand L is the number of labels.
The function B (˜ y ) =
y removes the epsilons from the alignmentsequence.
The alignment posterior is factorized along the alignment path as:
[Equation]
and ﬁnally RNNT make the following assumption:
[Equation]
which means if the preﬁx of two alignments be equal after epsilon removal, the model assigns sameexpansion probability for the next alignment position.
The inner terms in the above equation isdeﬁned (Eq12-Eq15 of the RNNT paper):
[Equation]
where h j − u is the encoder activation at time frame j − u (referred to as the transcription network in[12]) and g u is the output of the prediction network which is a simple stack of RNNs.
The equivalent GNAT model is conﬁgured as follow:
• context dependency:
M -gram where M is the maximum value of T + L in the training dataset.• alignment lattice: k -constrained label and frame dependent with k being the label sequencelength.• weight function:– [Equation] , x ( [Equation]
( t,
[Equation]
) , q c = q, y =
[Equation]
y
[Equation]
+
[Equation]
[Equation]
)– locally normalized– non-streaming
While the original deﬁnition of the RNNT model is based on non-streaming encoder (transcriptionnetwork), this model is widely used for streaming applications by using a streaming encoder.
This isin contradiction of the forward-backward derivations in the original paper which explicitly assumesdependency on the whole sequence for any position of alignment sequence (Eq17 of [12])
Similar to RNNT, the hybrid autoregressive transducer (HAT) [35] model can be also conﬁgured inthe GNAT framework with the same parametrization as RNNT.
The only difference is the weightfunction.
The HAT model deﬁnes different probabilities for label transitions and epsilon transitions(duration model in [35]):
[Equation]
where b t,u is a sigmoid function deﬁned in Eq6 of [35].
(cid:15) a b (cid:15) : (cid:15) a : a b : b (cid:15) : (cid:15) a : (cid:15) b : b (cid:15) : (cid:15) a : a b : (cid:15)
B.4 Supporting CTC Style Label Deduplication
The standard CTC model [13] is very similar to a GNAT model using a frame dependent alignmentlattice and a 0 -gram context dependency.
One key difference is that CTC introduces a deduplicationprocess when turning its model output sequence to a label sequence.
Each model output of a CTCmodel is either a lexical label from Σ , or the special (cid:15) (blank) label.
To obtain the label sequence,two steps are applied on the model output in order,
Maximal consecutive repeated non- (cid:15) labels are merged into one (e
g. turning abbc into abc ,or abb(cid:15)b into ab(cid:15)b );2.
All the (cid:15) labels are removed.
As a comparison, paths on the alignment lattice of the GNAT models in the main paper is equivalentto the model outputs in CTC, whereas the (cid:15) free label sequence seen by the context dependency isequivalent to the label sequence in CTC.
To support the deduplication of repeated non- (cid:15) labels, weneed to introduce a ﬁnite state transducer into our series of ﬁnite state machine compositions.
Similarto a ﬁnite state automaton, a weighted ﬁnite-state transducer (WFST) T =
(Σ , Q, i, F, ρ, E ) over asemiring K is speciﬁed by a ﬁnite alphabet Σ , a ﬁnite set of states Q , an initial state i ∈ Q , a set ofﬁnal states F ⊆ Q , a ﬁnal state weight assignment ρ : F → K , and a ﬁnite set of transitions E .
3 Themeaning of Σ , Q , i , F , and ρ are identical to those of a WFSA.
The set of transitions E is instead asubset of Q × (Σ ∪ { (cid:15) } ) × (Σ ∪ { (cid:15) } )
× K × Q , i.e. containing a pair of input/output labels insteadof just one.
A WFSA can be viewed as a WFST with identical input/output labels on each arc, andsimilar to WFSA intersection, a series of WFST can be composed into a single WFST.
We referthe readers to [25] for a full description of WFST and the composition algorithm.
Figure 3 is anexample unweighted FST, when composed with another input FSA or FST, performs the CTC stylelabel deduplication.
More generally, the unweighted label deduplication transducer D of vocabularyΣ consists of
• States Q D = Σ ∪ { (cid:15) } , i D = (cid:15) , F D = Q D• Transitions E
= { ( p, x, x, x ) | p, x ∈ Q D } ∪ { ( x, x, (cid:15), x ) | x ∈ Σ }
Given a context dependency FSA C , an alignment lattice FSA L T , and the weight function ω θ, x , asdeﬁned in Section 3.3, a GNAT model with CTC style label deduplication induces a WFST T θ, x asfollows,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In other words
[Equation]
the topology of T θ, x is the same as the following cascade of FST compositions,
D · C treating output (cid:15) labels in D as empty (i
e. standard FST composition).
L T · ( D · C ) treating (cid:15) transitions in L T and input (cid:15) labels in ( D · C ) as regular labels.
and the transition weights are deﬁned using ω θ, x just like the GNAT models in the main paper.
When implemented naively, CTC style label deduplication causes a | Q D |
= ( V + 1) blow up in| Q θ, x | .
However, by inferring about states in Q D from states in Q C , we can greatly reduce thenumber of states needed.
For each state x ∈ Σ in Q D , we know the last non- (cid:15) label observed whenreaching state x must be label x .
Similarly, for context dependencies we care about ( n -gram andstring), there is a unique label x
( q c ) for all incoming arcs of each non-start state q c (start states do nothave any incoming arcs in these context dependencies).
Thus, the states in Q θ, x that are reachablefrom the start must match one of the following patterns,
• ( q a , (cid:15), q c ) , ∀ q a ∈ Q A , q c ∈ Q X• ( q a , x ( q c ) , q c ) , ∀ q a ∈ Q A , q c ∈ Q X \ { i X }
This means the actual number of states we shall visit in computing the shortest distance is only2 | Q A || Q C | .
C Accelerator-Friendly Computation
The standard shortest distance/path algorithm for acyclic WFSA [24] can be used for training(computing W ( A ) for some acyclic A ) and inference of a GNAT model.
To compute W ( A ) for anacyclic WFSA A , we maintain the following forward weight α q for each state q in Q A :
[Equation]
else .
The weight of A is then W ( A )
=
(cid:76)
q ∈ F A α q ⊗ ρ A ( q ) .
The recurrence in the deﬁnition of α q can becomputed by visiting states in Q A in a topological order.
To make better use of the compute power of modern accelerator hardware, we observe the followingproperties of the C or L T presented so far that enable us to use a more vectorized variant of theshortest distance algorithm in Figure 4:
• From any topological ordering on Q T , we can derive a topological ordering on Q θ, x .
• The n -gram context dependency FSA C n is deterministic, namely leaving any state there is no more than 1 transition for any label y ∈ Σ , and there is no (cid:15) -transition.
• For all three types of alignment lattices, for any non-ﬁnal state q ∈ Q T \ F T , there is a unique next state succ( q ) for transitions leaving q consuming any label y ∈ Σ .
{Initialize the length | Q C | forward weight vectors ¯ α q a } for all q a ∈ Q T do ¯ α q a ← [¯0 , . . . , ¯0] end for ¯ α i T [ i C ] ← ¯ {Compute ¯ α q a for q a (cid:54) =
i T } for all q a ∈ Q T in topological order do { Ω is a [ | Q C | , | Σ | + 1] matrix} Ω ← ¯
[Figure]
(cid:0)
[Figure]
} (cid:1) if q a has outgoing label transitions to q (cid:48) a = succ( q a ) then
[Figure]
end if for all q (cid:48)
a such that ( q a , (cid:15), q (cid:48) a ) ∈ E T do
[Figure]
end for end for
[Figure]
{Inputs: ¯ α q a and Ω[: , Σ] } if n = 1 then { Q C n contains only i C n } return ¯ α q a ¯ ⊗ (cid:76)
y ∈ Σ Ω[ i C n , y ] end if {Initialize length | Q C n | vector
¯
α } ¯ α ← [¯0 , . .
. , ¯0] {States in Q C n are numbered from 0 to | Q C n | − 1 = (cid:80) n − 1 i =1 | Σ | i following the lexicographic order} l ←for i = 0 to n − 2 do h ← l + | Σ | i ¯ α [ l · | Σ | + 1 : h · | Σ | + 1] ← ﬂatten(Ω[ l : h, Σ]) l ← h end for for i = 0 to | Σ | − 1 do ¯ α
[ l :] ← ¯ α
[ l :]¯ ⊕ ﬂatten(Ω[ l + i · | Σ | n − 2 : l + ( i + 1) · | Σ | n − 2 , Σ])
Center to an efﬁcient implementation of the algorithm in Figure 4 is the function next C .
Thisfunction receives as input the current forward weight vector ¯
α q a for states ( q a , q c ) , ∀ q c ∈ Q C , andthe transition weights for leaving these states via label transitions, and returns the forward weightsgoing to states ( q (cid:48) a , q (cid:48) c ) by taking the ( q c , y, q (cid:48) c ) transitions for y ∈ Σ .
In other words, next C [ q (cid:48) c ] = 712 (cid:76) ( q c ,y,q (cid:48) c ) ∈
E C ¯ α q a [ q c ] ⊗ Ω[ q c , y ] .
The n -gram context dependency C n allows a particularly simpleand efﬁcient implementation of next C n , as outlined in Figure 5.
The key observation is that whenwe number the states in Q C following the lexicographic order, the | Σ | transitions leaving the sameq c lead to states in a consecutive range [ σ ( q c y 0 ) , . . .
, σ ( q c y | Σ |− 1 )] , where [ y 0 , . . .
, y | Σ |− 1 ] are thelexicographically sorted labels of Σ , and σ ( s ) is the sufﬁx of label sequence s of length up to n − 1 .
During training, we also need to compute the shortest distance D ( A θ, x ∩ y ) .
We note the algorithm y
D Memory and Computation Time Benchmarks
The memory and computation benchmark of our implementation for the GNAT model is presented in
Table 2.
We present benchmarks for training and inference for different conﬁgurations of the GNATmodel:
• Context dependency: 0 -gram, 1 -gram and 2 -gram• Alignment lattice: frame dependent, 1-constrained label and frame dependent• Weight functions: Per-state linear projection (unshared), Shared linear projection withper-state embedding (shared-emb), Shared linear projection with RNN state embedding(shared-rnn)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
activations.
The training benchmarks are corresponding to thecalculation of the GNAT criterion as well as all the backward gradient calculation up to the encoder activations.
The evaluation benchmarks only contain the forward pass memory and compute footprintto ﬁnd the most likely hypothesis.
All the memory and computation benchmarks are evaluated for an input batch of 32 examples eachwith 1024 number of frames.
Each frame is a 512 -dim vector corresponding to the encoder activations.
Each example in the input batch are assumed to have at most 256 labels.
The alphabet size is set to32 .
The main observations are:
• The larger context dependency lead to more memory and compute footprint.
This is expectedsince the computation complexity is directly related to the context dependency state size.
However, interestingly, the memory and computation values do not scale exponentially byvalue of n in n -gram context dependency (as a result by number of states in the contextdependency).• label frame dependent alignment lattice generally leads to higher memory usage and computation time compare to the frame dependent alignment lattice.
This is expected since the labelframe dependent consist of alignment paths of length 1024 + 256 = 1270 , corresponding tothe sum of number of frames and number of labels.• The per-state linear projection weight function requires more memory and has longercompute time compare to the shared weights function which is expected by design.
Bothshared weight functions are performing on-par of each other in terms of memory andcompute.•
The global normalization requires more memory and time and the difference is moresigniﬁcant for context dependency FSAs with more number of states ( 2 -gram versus 1gram).