R EVISITING THE E NTROPY S EMIRING FOR N EURAL S PEECH R ECOGNITION
Anonymous authors
Paper under double-blind review
A BSTRACT
In streaming settings, speech recognition models have to map sub-sequences of speech to text before the full audio stream becomes available.
However, since alignment information between speech and text is rarely available during training, models need to learn it in a completely self-supervised way.
In practice, the exponential number of possible alignments makes this extremely challenging, with models often learning peaky or sub-optimal alignments.
Prima facie, the exponential nature of the alignment space makes it difﬁcult to even quantify the uncertainty of a model’s alignment distribution.
Fortunately, it has been known for decades that the entropy of a probabilistic ﬁnite state transducer can be computed in time linear to the size of the transducer via a dynamic programming reduction based on semirings.
In this work, we revisit the entropy semiring for neural speech recognition models, and show how alignment entropy can be used to supervise models through regularization or distillation.
We also contribute an open-source implementation of CTC and RNN-T in the semiring framework that includes numerically stable and highly parallel variants of the entropy semiring.
Empirically, we observe that the addition of alignment distillation improves the accuracy and latency of an already well-optimized teacher-student distillation model, achieving state-of-the-art performance on the Librispeech dataset in the streaming scenario.
I NTRODUCTION
Modern automatic speech recognition (ASR) systems deploy a single neural network trained in an end-to-end differentiable manner on a paired corpus of speech and text (Graves et al., 2006; Graves, 2012; Chan et al., 2015;
Sak et al., 2017; He et al., 2019).
For many applications like providing closed captions in online meetings or understanding natural language queries for smart assistants, it is imperative that an ASR model operates in a streaming fashion with low latency.
This means that before the full audio stream becomes available, the model has to produce partial recognition outputs that correspond to the already given speech.
Ground truth alignments that annotate sub-sequences of speech with sub-sequences of text are hard to collect, and rarely available in sufﬁcient quantities to be used as training data.
Thus, ASR models have to learn alignments from paired examples of un-annotated speech and text in a completely self-supervised way.
The two most popular alignment models used for neural speech recognition today are Connectionist Temporal Classiﬁcation (CTC) (Graves et al., 2006) and Recurrent Neural
Network Transducer (RNN-T) (Graves, 2012;
He et al., 2019).
They formulate a probabilistic model over the alignment space, and are trained with a negative log-likelihood (NLL) criterion.
Despite the widespread use of CTC and RNN-T, ASR models tend to converge to peaky or suboptimal alignment distributions in practice (Miao et al., 2015;
Liu et al., 2018; Yu et al., 2021).
Prior work outside of ASR has discovered that the standard NLL loss generally leads to over-conﬁdent predictions (Pereyra et al., 2017; Xu et al., 2020).
Even within ASR, there is strong theoretical evidence that sub-optimal alignments are an inevitable consequence of the NLL training criterion (Zeyer et al., 2021; Blondel et al., 2021).
A common remedy for mitigating over-conﬁdent predictions is to impose an entropy regularizer to encourage diversiﬁcation.
For example, label smoothing leads to more calibrated representations and better predictions (M¨uller et al., 2019; Meister et al., 2020).
Another popular technique is knowledge distillation, where we minimize the relative entropy between a teacher’s and student’s soft predictions, instead of training on hard labels (Hinton et al., 2015; Stanton et al., 2021).
However, it is not straightforward to calculate the entropy of the alignment distribution of a neural speech model.
This is because the total number of possible alignments is exponential in the length of the acoustic and text sequence, which makes naive calculation intractable.
Fortunately, classical results from Eisner (2001); Cortes et al.
(2006) show that the entropy of a probabilistic ﬁnite state transducer can be computed in time linear to the size of the transducer.
Their approach is based on the semiring framework for dynamic programming (Mohri, 1998), which generalizes classical algorithms like forward-backward, inside-outside, Viterbi, and belief propagation.
The unifying algebraic structure of all these classical algorithms is that state transitions and merges can be interpreted as generalized multiplication and addition operations on a semiring.
Eisner (2001); Cortes et al.
(2006) ingeniously constructed a semiring that corresponds to the computation of entropy.
While these results have been long established, open-source implementations like the OpenFST library (Allauzen et al., 2007) have not been designed with modern automatic differentiation and deep learning libraries in mind.
This is why thus far, supervising training with alignment entropy regularization or distillation has not been part of the standard neural speech recognition toolbox.
In fact, implementing the entropy semiring on top of modern ASR lattices like CTC or RNN-T is highly non-trivial.
During training, we need to compute not only the entropy in the forward pass, but also its gradients in the backward pass, which necessitates a highly numerically stable implementation.
Note that even for simple operations like calculating the binary cross entropy of a softmax distribution, naive implementations that do not use the LogSumExp trick are plagued by numerical inaccuracies.
When it comes to calculating the entropy of a sequence that might be thousands of tokens long, such inaccuracies accumulate quickly, leading to NaNs that make training impossible.
Moreover, it is crucial that the addition of the alignment entropy supervision does not incur additional forward passes through the ASR lattice beyond the one already done to compute NLL.
Our Contributions We contribute an open-source implementation of CTC and RNN-T in the semiring framework that is both numerically stable and highly parallel.
Regarding numerical stability, we ﬁnd that the vanilla entropy semiring from Eisner (2001); Cortes et al.
(2006) produces unstable outputs not just at the ﬁnal step, but also during intermediate steps of the dynamic program.
Thus, a naive implementation will result in instability in both the forward and backward pass, since automatic differentiation re-uses activations produced during the intermediate steps.
To address this, we propose a novel variant of the entropy semiring that is isomorphic to the original, but is designed to be numerically stable for both the forward and backward pass.
Regarding parallelism, our implementation allows for efﬁcient plug-and-play computations of arbitrary semirings in the dynamic programming graphs of CTC and
RNN-T.
Thus, when outputs from more than one semiring are desired, we can compute them in parallel using a single pass over the same data, by simply plugging in a new semiring that is formed via the concatenation of existing semirings.
Beyond our open-source contribution, we also experimentally validate the effectiveness of alignment supervision for regularization and distillation in Section 5.
Our ﬁrst experiment targets smallcapacity models that are more likely to learn sub-optimal alignments.
In Section 5.1, we show how alignment entropy regularization can reduce the word error rates (WER) of small LSTM models by up to 6 .
5% and small Conformer models by up to 2 .
9% relative to the baseline.
These results highlight that the optimization objective can signiﬁcantly impact the performance of the same model.
Next in Section 5.2, we propose a novel distillation objective, which we term semiring distillation.
Under this objective, the teacher model uses the uncertainty of both the token predictions and the alignments to supervise the student.
When applied to a well-optimized 0 .
6 B parameter RNN-T model
, the combined uncertainty produces better WER results in the student than either uncertainty term alone, achieving state-of-the-art results on the Librispeech dataset in the streaming setting.
An ablation study further reveals that while distillation in the token space has a small negative effect on latency, distillation in the alignment space reduces emission latency signiﬁcantly.
The rest of the paper is organized as such: Section 2 presents the semiring framework for dynamic programming.
Section 3 provides a brief summary of alignment models for neural speech recognition.
Section 4 discusses our implementation of the semiring framework.
Section 5 showcases applications for regularization and distillation.
Finally, Section 6 concludes the paper.
Extra discussions about related work, proofs, and derivations can be found in the Appendix.
S EMIRING F RAMEWORK FOR D YNAMIC P ROGRAMMING
We begin by introducing deﬁnitions for a semiring, and adapting Mohri (1998)’s semiring framework for dynamic programming to weighted directed acyclic graphs (DAG).
Unlike Mohri (1998), which applies to single-source directed graphs, we consider DAGs with multiple roots, since they appear more commonly in ASR, for example the CTC lattice.
Deﬁnition 2.1.
A monoid is a set M equipped with a binary associative operation (cid:12) , i.e. ∀ a, b, c ∈
M, ( a (cid:12) b ) (cid:12) c = a (cid:12) ( b (cid:12) c ) , and an identity element e ∈ M , i.e. ∀ a ∈ M, a (cid:12) e = e (cid:12) a = a .
M is called a commutative monoid if (cid:12) is commutative, i.e. ∀ a, b ∈ M, a (cid:12) b =
b (cid:12) a .
Deﬁnition 2.2.
A semiring is a set R equipped with addition ⊕ and multiplication ⊗ such that:
( R, ⊕ ) is a commutative monoid with identity element ¯0 .
( R, ⊗ ) is a monoid with identity element ¯1 .
⊗ distributes over ⊕ from both the left and right, i.e. ∀ a, b, c ∈ R , a ⊗ ( b ⊕ c )
= ( a ⊗ b ) ⊕ ( a ⊗ c ) and ( b ⊕ c ) ⊗ a = ( b ⊗ a ) ⊕ ( c ⊗ a ) .
¯0 is an annihilator for ( R, ⊗ ) , i.e. ∀ a ∈ R, a ⊗ ¯0 =
¯0 ⊗ a = ¯0 .
Deﬁnition 2.3.
Let G =
( V, E ) be a DAG where every edge e ∈ E is assigned a weight w : E → R .
A maximal path of G is a path starting from a root of G to one of G ’s leaves.
The weight w ( π ) of a path π is the product of weights of edges in the path.
A computation of G is the sum of the weights of all the maximal paths in G , i.e.
[Equation]
Even though MaxPaths ( G ) might be exponentially big, because ⊗ distributes over ⊕ in a semiring, only 1 multiplication operation is needed for each edge, resulting in linear complexity O ( | V | + | E | ) .
Theorem 2.4.
For a DAG G =
( V, E ) , if ( w ( E ) , ⊕ , ⊗ , ¯0 , ¯1) is a semiring, then C ( G, w, ⊕ , ⊗ ) can be computed in linear time
O ( | V | + | E | ) via dynamic programming.
Proof.
Sort the vertices in topological order, and enumerate them as v 1 , . .
. , v n .
Then in that order, inductively accumulate the weights from the edges into the vertices as follows.
We slightly abuse notation by over-loading w to also be a function that assigns weights to vertices. [Equation] if [Equation]
i
[Equation]
Let Π v denote the set of paths ending at vertex v that start from a root.
We now prove by induction that w ( v i ) = (cid:76) π ∈ Π vi w ( π ) .
The base case of v 1 is true by deﬁnition given that it is a root.
For the inductive step, we assume that ∀ j ≤ k , w ( v j ) =
(cid:76) π ∈ Π vj w ( π ) .
Observe that
[Equation]
The sum of all the weights in the leaves yields the desired computation in a single pass over G .
Below, we walk the reader through simple examples to demonstrate how Theorem 2.4 works.
Deﬁnition 2.5.
The probability semiring can be deﬁned as such: w ( e ) =
p ( e ) where p : E → [0 , 1] is the probability function, ⊕ = + , ⊗ = × , ¯0 = 0 , ¯1 = 1 .
Example 2.6.
Consider the following DAG with the probability semiring.
[Equation]
In practice, it is common to perform multiplication of probabilities in the log space instead for numerical stability.
Under the semiring framework, we can see that this is the same dynamic programming computation but with a different semiring.
Deﬁnition 2.7.
The log semiring can be deﬁned as such: w ( e ) = log p ( e ) , a ⊕ b = log( e a + e b ) , a ⊗ b = a + b , ¯0 =
−∞ , ¯1 = 0 .
Example 2.8.
Consider the DAG in Example 2.6 with the log semiring.
(Details in Appendix C.1)
[Equation]
2. E NTROPY S EMIRING
Notice that setting w ( e ) = p ( e ) log p ( e ) with either the probability semiring or the log semiring will not compute the entropy.
The salient insight made by Eisner (2001); Cortes et al.
(2006) is that entropy can be calculated via a semiring that derives its algebraic structure from the dual number system.
Dual numbers are hyper-complex numbers that can be expressed as a + b(cid:15) such that a ∈ R , b ∈ R , (cid:15) 2 = 0 where addition and multiplication are deﬁned as follows.
Addition:
[Equation]
[Equation]
Observe that dual numbers form a commutative semiring with the additive identity 0 + 0 (cid:15) and multiplicative identity 1 + 0 (cid:15) .
Now if we use dual-valued weights w : E → R × R with the real component representing the probability p ( e ) and the imaginary component representing the negative entropy p ( e ) log p ( e ) , the entropy of a weighted DAG can be computed as before via Theorem 2.4.
Deﬁnition 2.9.
The entropy semiring can be deﬁned as such:
[Equation]
[Equation]
0 (cid:105)
[Equation]
¯
[Equation]
(cid:104)
[Equation]
0 (cid:105) .
Example 2.10.
Consider the DAG in Example 2.6 with the entropy semiring. (Details in Appendix C.2)
[Equation]
N EURAL S PEECH R ECOGNITION
Speech recognition is the task of transcribing speech x into text y , and can be formulated probabilistically as a discriminative model p ( y | x ) .
Because x and y are represented as sequences of vectors, we can deﬁne an alignment model that maps sub-sequences of x to sub-sequences of y .
The alignments allow a speech recognition model to emit a partial sub-sequence of y given a partial subsequence of x , and thereby work in a streaming fashion.
Below, we brieﬂy recap CTC and RNN-T, which are the two most widely used alignment models for neural speech recognition.
Let the acoustic input be denoted as x = x 1: T where x t ∈ R d and the text labels be denoted as y
=
y 1: U where y u ∈ V are vocabulary tokens (we use : to denote an inclusive range).
The likelihood of a CTC model is deﬁned as:
[Equation]
[Equation]
where ˆ y =
ˆ y 1: T ∈
A CTC ⊂ {V ∪ (cid:15) } T corresponds to alignments such that removing blanks (cid:15) and repeated symbols from ˆ y results in y .
CTC makes an assumption of conditional independence, so the likelihoods of every token are independent of each other given the acoustic input.
The likelihood of an RNN-T model is deﬁned as:
[Equation]
where ˆ y =
ˆ y 1: T + U ∈
A RNN-T ⊂ {V ∪ (cid:15) } T
+ U corresponds to alignments such that removing blanks (cid:15) from ˆ y results in y .
Unlike CTC, the likelihood of each token in RNN-T depends on the history of tokens that come before it.
CTC and RNN-T lattices are DAGs where the model likelihoods can be computed efﬁciently via dynamic programming by transforming the sum-product form of Eqs.
(5) and (6) into product-sum form via Theorem 2.4.
Thus, even though the number of alignments is exponential in the length of the acoustic and text input O ( | x | + | y | ) , the likelihood and its gradient can be computed in time linear to the size of the lattice O ( | x || y | ) .
In practice, most implementations of CTC and RNN-T compute likelihoods in the log space for numerical stability, which corresponds to dynamic programming using a log semiring (cf. Deﬁnition 2.7).
More broadly, Eqs.
(5) and (6) can be recognized as speciﬁc instances of the more general Eq.
[Equation]
While we focus on CTC and RNN-T in our paper, other examples of neural speech recognition lattices that use dynamic programming to efﬁciently marginalize over an exponential number of alignments include Auto Segmentation Criterion (Collobert et al., 2016), Lattice-Free MMI (Povey et al., 2016), and the Recurrent Neural Aligner (Sak et al., 2017).
I MPLEMENTATION OF THE S EMIRING F RAMEWORK
While the mathematical details of the entropy semiring have been known for more than two decades, it is highly non-trivial to implement them in neural speech recognition models.
This is in part because modern deep learning models operate on much longer sequences and have to compute gradients, and in part because of intricate implementation details like applying loop skewing on the RNN-T lattice (Bagby et al., 2018).
One of the main contributions of our work is to make an opensource implementation of CTC and RNN-T in the semiring framework available to the research community (cf. Supplementary Material).
Below, we introduce two variants of the entropy semiring that are designed for numerical stability and parallelism.
4. N UMERICAL S TABILITY
A numerically stable implementation of the entropy semiring has to avoid the naive multiplication of two small numbers, and do it in log space instead.
As such, the mathematical formulation provided in Deﬁnition 2.9 is not actually numerically stable because it involves multiplying p 1 and p 2 log p 2 for two small numbers p 1 , p 2 .
Notice that applying a log morphism on just the ﬁrst argument of the dual number is not sufﬁcient to ensure numerical stability in the backward pass because lim p
[Equation]
We take care of numerical stability in both the forward and backward passes by applying a log morphism on both arguments of the dual number in the semiring.
For the second argument, the log morphism has to be applied on the entropy − p log p since we cannot take the logarithm of negative numbers.
This results in the following variant of the entropy semiring.
Deﬁnition 4.1.
The log entropy semiring can be deﬁned as such:
[Equation]
4. E NTROPY S EMIRING FOR D ISTILLATION
Minimizing the negative log likelihood of a parameterized model is equivalent to minimizing the
KL divergence between the empirical data distribution and the modeled distribution.
Knowledge distillation is a technique that proposes to use soft labels from the output of a teacher model in the place of hard labels in the ground truth data (Hinton et al., 2015).
It is often implemented as a weighted sum of two different KL divergences: one between the empirical data distribution and the model (hard labels), and one between the teacher distribution and the model (soft labels).
[Equation]
We can re-write the distillation loss in Eq.
(8) as a sum of three different terms.
[Equation]
Notice that the ﬁrst term can be computed with the log semiring, while the second and third term can be computed with the log entropy semiring.
But instead of doing three separate forward passes, we can compute the distillation loss using a single forward pass by concatenating them to form a new semiring weighted by four real number values w : E → R 4 .
Deﬁnition 4.2.
The log reverse-KL semiring can be deﬁned as such:
[Equation]
[-, (cid:104) a, b, c, d (cid:105) ⊗ (cid:104) f, g, h, i (cid:105) = (cid:104) a + f, b + g, log( e b + h + e c + g ) , log( e b + i + e d + g ) (cid:105) , ¯ 
-][Equation]
¯0 = (cid:104)−∞ , −∞ , −∞ , −∞(cid:105) , ¯1 = (cid:104) 0 , 0 , −∞ , −∞(cid:105) .
A PPLICATIONS
Below, we discuss two applications of the entropy semiring for neural speech recognition, and conduct experiments to study them.
5. E NTROPY R EGULARIZATION
Motivation Because there is an exponential number of alignments, it is difﬁcult in general for an optimization algorithm to settle on the optimal alignment given no explicit supervision.
Instead, once a set of feasible alignments has been found during training, it tends to dominate, and error signals concentrate around the vicinity of such alignments. It has been experimentally observed that
[Table]
both CTC and RNN-T tend to produce highly peaky and over-conﬁdent distributions
[Table]
and converge towards local optima (Miao et al., 2015; Liu et al., 2018; Yu et al., 2021).
There is also theoretical analysis that suggests that this phenomenon is to some extent inevitable
[Table]
and a direct result of the training criterion (Zeyer et al., 2021; Blondel et al., 2021).
We propose to ameliorate this problem via an entropy regularization mechanism that penalizes the negative entropy of the alignment distribution, which encourages the exploration of more alignment paths during training.
[Equation]
Experimental Setup We experimented with models using non-causal LSTM and Conformer (Gulati et al., 2020) encoders on the Librispeech dataset (Panayotov et al., 2015).
The acoustic input is processed as 80 -dimensional log-mel ﬁlter bank coefﬁcients using short-time Fourier transform windows of size 25 ms and stride 10 ms.
All models are trained with Adam using the optimization schedule speciﬁed in Vaswani et al.
(2017), with a 10 k warmup, batch size 2048 , and a peak learning rate of 0 . 002 .
The LSTM encoders have 4 bi-directional layers with cell size 512 and are trained for 100 k steps, while the Conformer encoders have 16 full-context attention layers with model dimension 144 and are trained for 400 k steps.
Decoding for all models is done with beam search, with the CTC decoders using a beam width of 16 , and the RNN-T decoders using a beam width of 8 , a 1 -layer LSTM with cell size 320 , and a WordPiece tokenizer with a vocabulary size of 1024 .
α Ent was selected via a grid search on { 0 . 01 , 0 . 001 } .
We report 95% conﬁdence interval estimates for the WER following Vilar (2008).
Results We see from Table 1 that adding entropy regularization improves WER performance over the baseline model in almost all cases.
On LSTMs, the improvements were the biggest, for example there was a big WER reduction of 6 . 5% in the Test-Clean case for the CTC LSTM from 7 . 7% to 7 . 2% .
The improvements were smaller for the Conformers, for example with the WER remaining the same on Dev-Clean and dropping by an absolute 0 .
1% on Test-Clean for the RNN-T model.
In general, we ﬁnd that the improvements are the biggest when the baseline performance is the worst, indicating that entropy regularization is the most effective when the baseline model has converged sub-optimally and has under-utilized its model capacity.
5. RNN-T D ISTILLATION
Motivation Distillation is a technique that uses a pre-trained teacher model to aid in the training of a student model.
It can be especially helpful in the semi-supervised setting where a teacher model is trained on a small set of labeled data and then used to generate pseudo-labels for a larger set of unlabeled data (Park et al., 2020).
A student model is then trained using the standard NLL loss on the combined set of labeled and pseudo-labeled data.
We refer to this process as hard distillation.
[Equation]
In addition to the hard pseudo-labels, we can use the soft logits from the teacher model to provide a better training signal to the student model.
Prior work for RNN-T models has implemented this by summing up the KL divergence between the state-wise posteriors of both models (Kurata & Saon, 2020; Panchapagesan et al., 2021).
We refer to this process as soft distillation.
[Equation]
Streaming models have to carefully balance the competing objectives of reducing emission latency and improving WER accuracy.
Alignments that delay emission can leverage more information to perform its prediction, but at the cost of latency.
Because both the hard and soft distillation objectives operate in token space, they neglect potentially helpful alignment information from the teacher model.
Prior work has found that alignment information from a good pre-trained model can help guide attention-based encoder-decoder models to achieve both good latency and accuracy (Inaguma & Kawahara, 2021).
We propose to add a KL divergence loss for the sequence-wise posteriors of both models to incorporate alignment information into the distillation objective.
Because this technique is based on the use of a semiring (cf. Section 4.2), we refer to it as semiring distillation.
[Equation]
Experimental Setup We set up a semi-supervised learning scenario by using Libri-Light as the unlabeled dataset (Kahn et al., 2020) and Librispeech as the labeled dataset.
First, we prepare pseudo-labels for Libri-Light from Zhang et al.
(2020b), where an iterative hard distillation process is performed with a non-causal 1 .
0 B Conformer model to arrive at high quality pseudo-labels.
Then, we train a teacher model by randomly sampling batches from Libri-Light and Librispeech in a 90 :mix.
This teacher model is used as our hard distillation baseline, and also used to train a student model via soft and semiring distillation for comparison.
Both teacher and student models use the same 0 .
6 B causal Conformer architecture as in Chiu et al.
(2022) with self-supervised pre-training on Libri-Light via a random projection quantizer.
Like Hwang et al.
, the student model has
FreqAug applied to it for data augmentation.
Both models were trained for 160 k steps with Adam using the same optimization schedule as in Vaswani et al.
(2017), but with a peak learning rate of 0 . 0015 , a 5 k warmup, and batch size 256 .
The RNN-T decoder uses a 2 -layer LSTM with cell size 1200 , beam width 8 , and a WordPiece tokenizer with a vocabulary size of 1024 .
We do a grid search for both α state and α seq over { 0 . 01 , 0 . 001 } .
We report 95% conﬁdence interval estimates for the
WER following Vilar (2008).
Results We report our results in Table 2.
We see that soft distillation results in substantial WER improvements over the hard distillation baseline, reducing WER on the Test-Clean set by 22% from 2 .
7% to 2 .
1% and on the Test-Other set by 17% from 6 .
4% to 5 .
Semiring distillation results in small WER improvements over soft distillation, further reducing WER by an absolute 0 .
1% on all the Dev and Test sets.
We further do an ablation study with semiring distillation using α state = 0 . 0 , and ﬁnd that alignment distillation alone, while under-performing soft distillation, still results in signiﬁcant WER improvements over the hard distillation baseline.
These results suggest that while a good measure of uncertainty in the token space helps improve accuracy more than a corresponding measure of uncertainty in the alignment space, these two measures of uncertainty are complementary to each other and should be used jointly for best results.
We do relative latency measurements between two models following Chiu et al.
Start and end times of every word in the output hypotheses are ﬁrst calculated, and then used to compute the average word timing difference between matching words from the hypotheses of both models.
Interestingly, we see that even though the soft distillation model obtained better WER accuracy, this came at a slight cost to emission latency ( +2 .
7 ms) relative to the baseline hard distillation model.
On the other hand, the α state = 0 . 0 semiring distillation ablation performed signiﬁcantly better in terms of relative latency ( − 65 .
7 ms), indicating that alignment information from the teacher helped the student learn to emit tokens faster.
By combining both state-wise and sequence-wise posteriors, the semiring distillation model improved on both emission latency and WER accuracy compared to the hard or soft distillation model. [Table]
[Table]
Zhang et al.
(2020a)
No
No - - - 4 .
[Table]
Zhang et al [Table] (2020a) No
Yes - - - 3 .10 .
Yu et al.
No
No 30 M - -
[Table]
[Table]
Cao et al.
No
No3 .8 .3 .8 .
Hwang et al.
Yes
No 0 .
2 B 4 .9 .4 .8 .
Moritz et al.
No
No2 .8 .3 .8 .
Yu et al.
No
[Table] Our work [Table] Finally, we compare the performance of our semiring-distilled model with prior work for Librispeech in the streaming setting in Table 3.
To the best of our knowledge, our work has achieved a new state-of-the-art on this benchmark, without using any future context in making its predictions.
C ONCLUSION
Across multiple machine learning applications in speech and natural language processing, computational biology, and even computer vision, there has been a paradigm shift towards sequence-tosequence modeling using Transformer-based architectures.
Most learning objectives on these models do not take advantage of the structured nature of the inputs and outputs.
This is in stark contrast to the rich literature of sequential modeling approaches based on weighted ﬁnite-state transducers from before the deep learning renaissance (Eisner, 2002; Mohri et al., 2002; Cortes et al., 2004).
Our work draws upon these pre deep learning era approaches to introduce alignment-based supervision for neural ASR in the form of regularization and distillation.
We believe that applying similar semiring based techniques to supervise the alignment between data of different modalities or domains will lead to better learning representations.
For example, recent work has found that imagetext models do not have good visio-linguistic compositional reasoning abilities (Thrush et al., 2022). This problem can potentially be addressed by learning a good alignment model between a natural language description of an object and its corresponding pixel-level image representation. Another area of future work can be learning to synchronize audio and visual input streams to solve tasks with multiple output streams like overlapping speech recognition or diarization.
Finally, we are excited about the gamut of new ASR research that will arise from plugging different semirings into our CTC and RNN-T semiring framework implementation.
For example, even though most mainstream approaches use the log semiring for training and decoding, a differentiable relaxation akin to Cuturi & Blondel (2017) might yield surprising ﬁndings.
Another promising direction for future work can involve calculating the mean or variance of hypothesis lengths for diagnostic or regularization purposes.
A PPENDIX
The Appendix is organized as follows: Appendix A discusses other related work in the literature.
Appendix B provides proofs that the semirings deﬁned in our paper are actual semirings.
Appendix C provides extended derivations for Examples 2.8 and 2.10.
A R ELATED W ORK
Cortes et al.
(2004) showed how weighted transducers can be used to form rational kernels for support vector machines, thereby expanding their use to classiﬁcation problems with variable-length inputs.
Thereafter, Nelken & Shieber (2006) demonstrated a rational kernel with the entropy semiring.
The use of semirings has been well-established in the conditional random ﬁelds literature: replacing the max-plus algebra with the sum-product algebra is a way to derive the forward-backward algorithm from Viterbi (Bishop & Nasrabadi, 2006; Wainwright et al., 2008).
Li & Eisner (2009) studied ﬁrst-order and second-order expectation semirings with applications to machine translation.
Eisner (2016) discussed how the inside-outside algorithm can be derived simply from backpropagation and automatic differentiation.
Liu et al.
(2018) proposed a different CTC-speciﬁc entropy regularization objective that is based on a similar motivation of avoiding bad alignments. Cuturi & Blondel (2017);
Mensch & Blondel (2018); Blondel et al.
(2021) explore various differentiable relaxations for dynamic programming based losses in deep learning.
Hannun et al.
(2020) proposed differentiable weighted ﬁnite state transducers, which focused on plug-and-play graphs that support the log and tropical semirings, as opposed to our work, which focuses on plug-and-play semirings that support the CTC and RNN-T graphs.
Because our graphs are ﬁxed and coded in Tensorﬂow itself, we are able to perform graph optimizations (for example, loop-skewing on the RNN-T lattice), compiler optimizations (for example, the XLA compiler for Tensorﬂow produces production-quality binaries for a variety of software and hardware architectures), and even semiring optimizations (for example, our proposed variants of the entropy semiring that are more numerically stable and highly parallel), thereby ensuring an especially efﬁcient implementation for neural speech recognition.
The use of both state-wise and sequence-wise posteriors in semiring distillation is similar to the direct sum entropy introduced in Farinhas et al.
(2021), which is a mix of both discrete entropy and differential entropy.
B P ROOF OF S EMIRING
Below, we prove that the probability, log, entropy, log entropy, log reverse-KL semirings deﬁned in our paper are actual semirings according to Deﬁnition 2.2.
Each proof is provided in four parts:
⊗ forms a commutative monoid with identity ¯0 .
⊕ forms a monoid with identity ¯1 .
⊗ distributes over ⊕ .
¯0 annihilates ⊗ .
B. P ROOF THAT THE P ROBABILITY S EMIRING IS A S EMIRING
Deﬁnition 2.5.
The probability semiring can be deﬁned as such: w ( e ) =
p ( e ) where p : E → [0 , 1] is the probability function, ⊕ = + , ⊗ = × , ¯0 = 0 , ¯1 = 1 .
Proof.
a ⊕ b = a + b =
b + a = b ⊕ a. a ⊕ ( b ⊕ c ) =
a + ( b + c )
= ( a + b ) + c = ( a ⊕ b ) ⊕ c. a ⊕ ¯ = a += a.
a ⊗ ( b ⊗ c ) = a × ( b × c ) =
( a × b ) × c = ( a ⊗ b ) ⊗ c. a ⊗ ¯ = a ×= a.
¯1 ⊗ a = 1 × a = a.
a ⊗ ( b ⊕ c )
=
a × ( b + c ) = ( a × b ) + ( a × c ) =
( a ⊗ b ) ⊕ ( a ⊗ c ) .
( b ⊕ c ) ⊗ a = ( b + c ) × a
= ( b × a ) + ( c × a ) =
( b ⊗ a ) ⊕ ( c ⊗ a ) .
a ⊗ ¯ = a ×== ¯0 .
¯0 ⊗ a = 0 × a == ¯0 .
B. P ROOF THAT THE L OG S EMIRING IS A S EMIRING
Deﬁnition 2.7.
The log semiring can be deﬁned as such: w ( e ) = log p ( e ) , a ⊕ b = log( e a + e b ) , a ⊗ b = a + b , ¯ 0 =
−∞ , ¯ 1 = 0 .
Proof.
a ⊕ b = log( e a + e b ) = log( e b
+ e a ) =
b ⊕ a. a ⊕ ( b ⊕ c ) =
a ⊕ log( e b + e c ) = log( e a + e b + e c ) =
log( e a + e b ) ⊕ c
= ( a ⊕ b ) ⊕ c. a ⊕ ¯ = log
( e a + e −∞ ) = a.
[Equation]
B. P ROOF THAT THE E NTROPY S EMIRING IS A S EMIRING
Deﬁnition 2.9.
The entropy semiring can be deﬁned as such: w ( e ) = (cid:104) p ( e ) , p ( e ) log p ( e ) (cid:105) , (cid:104) a, b (cid:105) ⊕ (cid:104) c, d (cid:105)
=
(cid:104)
a + c, b + d (cid:105) , (cid:104) a, b (cid:105) ⊗ (cid:104) c, d (cid:105) =
(cid:104) ac, ad + bc (cid:105) , ¯0 = (cid:104) 0 , 0 (cid:105) , ¯1 = (cid:104) 1 , 0 (cid:105) .
Proof.
(cid:104) a, b (cid:105) ⊕ (cid:104) c, d (cid:105) =
(cid:104)
a + c, b + d (cid:105) =
(cid:104) c
+ a, d + b (cid:105) =
(cid:104) c, d (cid:105) ⊕ (cid:104)
a, b (cid:105) .
(cid:104) a, b (cid:105) ⊕ ( (cid:104) c, d (cid:105) ⊕ (cid:104)
f, g (cid:105) )
= (cid:104) a, b (cid:105) ⊕ (cid:104) c
+ f, d + g (cid:105) =
(cid:104) a + c + f, b + d + g (cid:105) =
(cid:104)
a + c, b + d (cid:105) ⊕ (cid:104) f, g (cid:105) =
( (cid:104) a, b (cid:105) ⊕ (cid:104) c, d (cid:105) ) ⊕ (cid:104) f, g (cid:105) .
(cid:104) a, b (cid:105)
⊕
¯ =
(cid:104)
a + 0 , b + 0 (cid:105) =
(cid:104) a, b (cid:105) .
(cid:104) a, b (cid:105) ⊗ ( (cid:104) c, d (cid:105) ⊗ (cid:104) f, g (cid:105) ) = (cid:104) a, b (cid:105) ⊗ (cid:104) cf, cg + df (cid:105) =
(cid:104) acf, acg + adf
+ bcf (cid:105)
= (cid:104) ac, ad + bc (cid:105) ⊗ (cid:104) f, g (cid:105) =
( (cid:104) a, b (cid:105) ⊗ (cid:104) c, d (cid:105) ) ⊗ (cid:104) f, g (cid:105) .
(cid:104) a, b (cid:105) ⊗ ¯ =
(cid:104) a × 1 , a × 0 + b × 1 (cid:105) =
(cid:104) a, b (cid:105) .
¯1 ⊗ (cid:104) a, b (cid:105) =
(cid:104) 1 × a, 0 × a + 1 × b (cid:105) =
(cid:104) a, b (cid:105) .
(cid:104) a, b (cid:105) ⊗ ( (cid:104) c, d (cid:105) ⊕ (cid:104)
f, g (cid:105) )
= (cid:104) a, b (cid:105) ⊗ (cid:104) c
+ f, d + g (cid:105) =
(cid:104) ac
+ af, ad + ag + bc + bf (cid:105) =
(cid:104) ac, ad + bc (cid:105) ⊕ (cid:104) af, ag + bf (cid:105) =
( (cid:104) a, b (cid:105) ⊗ (cid:104) c, d (cid:105) ) ⊕ ( (cid:104) a, b (cid:105) ⊗ (cid:104) f, g (cid:105) ) .
( (cid:104) c, d (cid:105) ⊕ (cid:104) f, g (cid:105) ) ⊗ (cid:104) a, b (cid:105) =
(cid:104) c
+ f, d + g (cid:105) ⊗ (cid:104) a, b (cid:105)
=
(cid:104) ca + fa, cb + fb + da + ga (cid:105)
= (cid:104) ca, cb + da (cid:105) ⊕ (cid:104) fa, fb + ga (cid:105) =
( (cid:104) c, d (cid:105) ⊗ (cid:104) a, b (cid:105) )
⊕ ( (cid:104) f, g (cid:105) ) ⊗ (cid:104) a, b (cid:105) ) .
(cid:104) a, b (cid:105) ⊗ ¯ =
(cid:104) a × 0 , a × 0 + b × 0 (cid:105) =
(cid:104) 0 , 0 (cid:105) =
¯0 .
¯0 ⊗ (cid:104) a, b (cid:105)
=
(cid:104)
× a, 0 × a + 0 × b (cid:105) =
(cid:104) 0 , 0 (cid:105) =
¯0 .
B. P ROOF THAT THE L OG E NTROPY S EMIRING IS A S EMIRING
Deﬁnition 4.1.
The log entropy semiring can be deﬁned as such: w ( e ) = (cid:104) log p ( e ) , log( − p ( e ) log p ( e ))
(cid:105) , (cid:104) a, b (cid:105) ⊕ (cid:104) c, d (cid:105) =
(cid:104) log( e a + e c ) , log( e b + e d ) (cid:105) , (cid:104) a, b (cid:105) ⊗ (cid:104) c, d (cid:105) =
(cid:104) a + c, log( e a + d + e b + c ) (cid:105) , ¯0 = (cid:104)−∞ , −∞(cid:105) , ¯1 = (cid:104) 0 , −∞(cid:105) .
Proof.
(cid:104) a, b (cid:105) ⊕ (cid:104) c, d (cid:105) = (cid:104)
log( e a + e c ) , log( e b + e d ) (cid:105) =
(cid:104) log( e c
+ e a ) , log( e d + e b ) (cid:105) =
(cid:104) c, d (cid:105) ⊕ (cid:104)
a, b (cid:105) .
(cid:104) a, b (cid:105) ⊕ ( (cid:104) c, d (cid:105) ⊕ (cid:104)
f, g (cid:105) )
= (cid:104) a, b (cid:105) ⊕ (cid:104)
log( e c + e f ) , log( e d + e g ) (cid:105) =
(cid:104) log( e a + e c + e f ) , log( e b + e d + e g ) (cid:105) =
(cid:104) log( e a + e c ) , log( e b + e d ) (cid:105) ⊕ (cid:104) f, g (cid:105) =
( (cid:104) a, b (cid:105) ⊕ (cid:104) c, d (cid:105) ) ⊕ (cid:104) f, g (cid:105) .
(cid:104) a, b (cid:105) ⊕ ¯
= (cid:104) log( e a + e −∞ ) , log( e b + e −∞ ) (cid:105)
=
(cid:104) a, b (cid:105) .
(cid:104) a, b (cid:105) ⊗ ( (cid:104) c, d (cid:105) ⊗ (cid:104) f, g (cid:105) )
= (cid:104) a, b (cid:105) ⊗ (cid:104) c
+ f, log( e c + g + e d + f ) (cid:105) =
(cid:104) a + c + f, log
( e a + c + g + e a + d + f + e b + c + f ) (cid:105) =
(cid:104) a + c, log( e a + d + e b + c ) (cid:105) ⊗ (cid:104) f, g (cid:105) =
( (cid:104) a, b (cid:105) ⊗ (cid:104) c, d (cid:105) ) ⊗ (cid:104) f, g (cid:105) .
(cid:104) a, b (cid:105) ⊗ ¯ =
(cid:104) a + 0 , log( e a −∞ + e b +0 ) (cid:105) =
(cid:104) a, b (cid:105) .
¯1 ⊗ (cid:104) a, b (cid:105) =
(cid:104) {+
+}
+ a, log( e −∞ + a + e 0+ b ) (cid:105) =
(cid:104) a, b (cid:105) .
(cid:104) a, b (cid:105) ⊗ ( (cid:104) c, d (cid:105) ⊕ (cid:104)
f, g (cid:105) )
= (cid:104) a, b (cid:105) ⊗ (cid:104) log( e c + e f ) , log( e d + e g ) (cid:105) =
(cid:104) a + log( e c + e f ) , log( e a + d + e a
+ g + e b + c + e b + f ) (cid:105) =
(cid:104) a + c, log( e a + d + e b + c ) (cid:105) ⊕ (cid:104)
a + f, log( e a + g + e b + f ) (cid:105) =
( (cid:104) a, b (cid:105) ⊗ (cid:104) c, d (cid:105) ) ⊕ ( (cid:104) a, b (cid:105) ⊗ (cid:104) f, g (cid:105) ) .
( (cid:104) c, d (cid:105) ⊕ (cid:104) f, g (cid:105) ) ⊗ (cid:104) a, b (cid:105) =
(cid:104) log( e c + e f ) , log( e d + e g ) (cid:105) ⊗ (cid:104) a, b (cid:105)
= (cid:104) log( e c + e f )
+ a, log( e c + b + e f
+ b + e d + a + e g + a ) (cid:105) =
(cid:104) c
+ a, log( e c + b + e d + a ) (cid:105) ⊕ (cid:104) f
+ a, log( e f + b + e g + a ) (cid:105) = ( (cid:104) c, d (cid:105) ⊗ (cid:104) a, b (cid:105) )
⊕ ( (cid:104) f, g (cid:105) ) ⊗ (cid:104) a, b (cid:105) ) . (cid:104) a, b (cid:105) ⊗ ¯ =
(cid:104) a − ∞ , log( e a −∞ + e b −∞ ) (cid:105) =
(cid:104)−∞ , −∞(cid:105) = ¯0 . ¯0 ⊗ (cid:104) a, b (cid:105) = (cid:104)−∞ + a, log( e −∞ + a + e −∞ + b ) (cid:105) =
(cid:104)−∞ , −∞(cid:105) =
¯0 . B. P ROOF THAT THE L OG R EVERSE -KL S EMIRING IS A S EMIRING
Deﬁnition 4.2.
The log reverse-KL semiring can be deﬁned as such: w ( e ) = (cid:104) log p ( e ) , log q ( e ) , log( − q ( e ) log
[Equation]
( e )) , [Equation] ( −
[Equation]
( e )
[Equation]
p ( e ) [Equation] (cid:105) , (cid:104) a, b, c, d (cid:105) ⊕ (cid:104) f, g, h, i (cid:105) =
(cid:104) log( e a + e f ) , log( e b + e g ) , log( e c + e h ) , log( e d + e i ) (cid:105) , (cid:104)
a, b, c, d (cid:105) ⊗ (cid:104) f, g, h, i (cid:105)
= (cid:104) a + f, b + g, log( e b
+ h + e c + g ) , log( e b
+ i + e d + g ) (cid:105) , ¯0 = (cid:104)−∞ , −∞ , −∞ , −∞(cid:105) , ¯1 =
(cid:104) 0 , 0 , −∞ , −∞(cid:105) .
Proof.
1. (cid:104)
a
b c d (cid:105) ⊕ (cid:104) f, g, h, i (cid:105) =
(cid:104) log( e a + e f ) , log( e b + e g ) , log( e c + e h ) , log( e d + e i ) (cid:105)
= (cid:104) log( e f
+ e a ) , log( e g + e b ) , log( e h + e c ) , log( e i
+ e d ) (cid:105)
= (cid:104) f, g, h, i (cid:105) ⊕ (cid:104)
a, b, c, d (cid:105) .
(cid:104) a, b, c, d (cid:105) ⊕ ( (cid:104) f, g, h, i (cid:105) ⊕ (cid:104)
j, k, l, m (cid:105) )
= (cid:104) a, b, c, d (cid:105) ⊕ (cid:104)
log( e f + e j ) , log( e g + e k ) , log( e h + e l ) , log( e i + e m ) (cid:105) =
(cid:104) log( e a + e f + e j ) , log( e b + e g
+ e k ) , log( e c + e h + e l ) , log( e d + e i + e m ) (cid:105) =
(cid:104) log( e a + e f ) , log( e b + e g ) , log( e c + e h ) , log( e d + e i ) (cid:105) ⊕ (cid:104)
j, k, l, m (cid:105) =
( (cid:104) a, b, c, d (cid:105) ⊕ (cid:104) f, g, h, i (cid:105) )
⊕ (cid:104) j, k, l, m (cid:105) .
[Equation]
(cid:104)
a, b, c, d (cid:105) ⊗
[Equation]
f, g, h, i (cid:105) ⊕
[Equation]
j, k, l, m
[Equation]
[Equation]
[Equation]
log( e f + e j ) , log( e g + e k ) , log( e h + e l ) , log( e i + e m ) (cid:105) ⊗ (cid:104) a, b, c, d (cid:105) =
(cid:104) log( e f + e j )
+ a, log( e g
+ e k )
+ b, log( e g + c
+ e k
+ c + e h + b + e l + b ) , log( e g + d + e k
+
d
+ e i + b + e m + b ) (cid:105) =
(cid:104) f
+ a, g + b, log( e g + c
+ e h + b ) , log( e g + d + e i + b ) (cid:105) ⊕ (cid:104)
j
+ a, k + b, log( e k
+ c + e l + b ) , log( e k + d + e m + b ) (cid:105) =
( (cid:104) f, g, h, i (cid:105) ⊗ (cid:104) a, b, c, d (cid:105) ) ⊕ ( (cid:104) j, k, l, m (cid:105) ) ⊗ (cid:104) a, b, c, d (cid:105) ) . 4. (cid:104) a, b c
(cid:105) ⊗ ¯  = (cid:104) a − ∞ , b − ∞ , log( e b
−∞ + e c −∞ ) , log( e b
−∞
+ e d −∞ ) (cid:105) =
(cid:104)−∞ , −∞ , −∞ , −∞(cid:105) =
¯0 .
¯ 0 ⊗ (cid:104) a, b, c, d (cid:105) =
(cid:104)−∞ + a, −∞ + b, log( e −∞ + b + e −∞ + c ) , log( e −∞ + b + e −∞ + d ) (cid:105) =
(cid:104)−∞ , −∞ , −∞ , −∞(cid:105) =
¯0 .
C E XTENDED E XAMPLES
[Equation]
w ( v 1 ) =
w ( v 2 ) = 0 .
w ( v 3 ) =
( w ( v 1 ) ⊗ w ( e 1 ))
⊕ ( w ( v 2 ) ⊗ w ( e 2 )) = (0 + log p ( e 1 )) ⊕ (0 + log p ( e 2 )) = log p ( e 1 ) ⊕ log p ( e 2 ) =
log [ p ( e 1 ) + p ( e 2 )] . w ( v 4 ) = w ( v 3 ) ⊗ w ( e 3 ) = log [ p ( e 1 ) + p ( e 2 )]
⊗ log p ( e 3 ) = log [ p ( e 1 ) p ( e 3 ) + p ( e 2 ) p ( e 3 )] .
w ( v 5 ) = w ( v 3 ) ⊗ w ( e 4 ) = log [ p ( e 1 ) + p ( e 2 )] ⊗ log p ( e 4 ) = log [ p ( e 1 ) p ( e 4 ) + p ( e 2 ) p ( e 4 )] .
C log p ( π ) =
w ( v 4 ) ⊕ w ( v 5 ) = log [ p ( e 1 ) p ( e 3 ) + p ( e 2 ) p ( e 3 )] ⊕ log [ p ( e 1 ) p ( e 4 ) + p ( e 2 ) p ( e 4 )] .
= log [ p ( e 1 ) p ( e 3 ) + p ( e 2 ) p ( e 3 ) + p ( e 1 ) p ( e 4 ) + p ( e 2 ) p ( e 4 )] . C.
w ( v 1 ) = w ( v 2 ) = (cid:104) 1 , 0 (cid:105) .
w ( v 3 ) = ( w ( v 1 ) ⊗ w ( e 1 )) ⊕ ( w ( v 2 ) ⊗ w ( e 2 )) =
( (cid:104) 1 , 0 (cid:105) ⊗ (cid:104) p ( e 1 ) , p ( e 1 ) log p ( e 1 ) (cid:105) )
⊕ ( (cid:104) 1 , 0 (cid:105) + (cid:104) p ( e 2 ) , p ( e 2 ) log p ( e 2 ) (cid:105) )
= (cid:104) p ( e 1 ) , p ( e 1 ) log p ( e 1 ) (cid:105) ⊕ (cid:104) p ( e 2 ) , p ( e 2 ) log p ( e 2 ) (cid:105)
=
(cid:104) p ( e 1 ) + p ( e 2 ) , p ( e 1 ) log p ( e 1 )
+ p ( e 2 ) log p ( e 2 ) (cid:105) .
w ( v 4 ) =
w ( v 3 ) ⊗ w ( e 3 ) =
(cid:104) p ( e 1 ) + p ( e 2 ) , p ( e 1 ) log p ( e 1 )
+ p ( e 2 ) log p ( e 2 ) (cid:105) ⊗ (cid:104) p ( e 3 ) , p ( e 3 ) log p ( e 3 ) (cid:105) =
(cid:104) p ( e 1 ) p ( e 3 )
+ p ( e 2 ) p ( e 3 ) , p ( e 1 ) p ( e 3 ) log p ( e 1 ) p ( e 3 )
+ p ( e 2 ) p ( e 3 ) log p ( e 2 ) p ( e 3 ) (cid:105) .
w ( v 5 ) =
w ( v 3 ) ⊗ w ( e 4 ) =
(cid:104) p ( e 1 ) + p ( e 2 ) , p ( e 1 ) log p ( e 1 )
+ p ( e 2 ) log p ( e 2 ) (cid:105) ⊗ (cid:104) p ( e 4 ) , p ( e 4 ) log p ( e 4 ) (cid:105)
= (cid:104) p ( e 1 ) p ( e 4 )
+ p ( e 2 ) p ( e 4 ) , p ( e 1 ) p ( e 4 ) log p ( e 1 ) p ( e 4 ) + p ( e 2 ) p ( e 4 ) log p ( e 2 ) p ( e 4 ) (cid:105) .
C (cid:104) p ( π ) ,p ( π )log p ( π ) (cid:105) =
(cid:10) C p ( π ) , p ( e 1 ) p ( e 3 ) log [ p ( e 1 ) p ( e 3 )]
+ p ( e 2 ) p ( e 3 ) log [ p ( e 2 ) p ( e 3 )]
+ p ( e 1 ) p ( e 4 ) log [ p ( e 1 ) p ( e 4 )]
+ p ( e 2 ) p ( e 4 ) log [ p ( e 2 ) p ( e 4 )] (cid:11) .