A Deep Generative Acoustic Model for Compositional Automatic Speech Recognition
AfﬁliationAddress
Abstract
Inspired by the recent successes of deep generative models for Text-To-Speech(TTS) such as WaveNet and Tacotron [19, 22], this article proposes the use of a deepgenerative model tailored for Automatic Speech Recognition (ASR) as the primaryacoustic model (AM) for an overall recognition system with a separate languagemodel (LM).
Two dimensions of depth are considered: (1) the use of mixturedensity networks, both autoregressive and non-autoregressive, to generate densityfunctions capable of modeling acoustic input sequences with much more powerfulconditioning than the ﬁrst-generation generative models for ASR, Gaussian Mixture
Models / Hidden Markov Models (GMM/HMMs), and (2) the use of standard
LSTMs, in the spirit of the original tandem approach, to produce discriminativefeature vectors for generative modeling.
Combining mixture density networksand deep discriminative features leads to a novel dual-stack LSTM architecturedirectly related to the RNN Transducer [7], but with the explicit functional form ofa density, and combining naturally with a separate language model, using Bayesrule.
The proposed generative models are compared experimentally in terms oflog-likelihoods and frame accuracies.
Introduction1.1 The channel model & compositional ASR
For most of its history, the ﬁeld of ASR has used a collection of separate modules to represent differentstages of an overall processing chain.
Fred Jelinek formalized this approach within his concept ofthe “noisy channel model”, in which components chain together to form a consistent overall jointprobability P ( X, S ) , decomposed using Bayes’ theorem [9].
This model was particularly convenientgiven the strong independence assumptions in the model structures used.
Gaussian Mixture Models(GMMs) used jointly with 1st-order Hidden Markov Models (HMMs) were well-suited to thismodular approach, as they directly provide an acoustic likelihood p ( X | S ) (decomposed using strongindependence assumptions) for a sequence of acoustic feature vectors X = x 1: t conditioned ona given sequence of modeling units, e.g. phonemes, graphemes or words, S = s 1: m .
This thencombines naturally with a separate language model probability, P ( S ) to form the joint probability P ( X, S ) using Bayes rule.
Other probabilistic modules such as a pronunciation dictionary can beintroduced into the overall chain, again combining with the other components according to Bayes’rule.
In that sense, this approach to ASR is “compositional”.
1.2 Shoehorning discriminative models into the modular channel model
The adoption of discriminative models such as Deep Neural Networks (DNNs) and Long Short Term
Memory models (LSTMs) did not at ﬁrst alter this model.
The popular “hybrid” approach (still considered state-of-the-art by most the community today) simply converts the posteriors P ( s | x )(deﬁned at the level of a single observation x for a model state s ) obtained from DNNs or LSTMs toa scaled likelihood by normalizing the posterior by a state prior p ( s ) [3].
The scaled likelihood canthen plug back into the channel model using Bayes rule, as before, and can also plug into an overallsequence training objective function such as MMI or sMBR [11, 13].
1.3 Goodbye, modularity?
In contrast, end-to-end models such as Listen, Attend & Spell (LAS) [4] or RNN Transducer (RNNT) [7] dispense with the modular channel model altogether, and directly optimize a discriminativesequence-level model with a discriminative criterion.
By construction, these models are all-in-onemodels that represent P ( S | X ) directly, with no chaining of sub-module probabilities.
If one hasa large enough training set of supervised data pairs, this can be thought of as the perfect model.
State-of-the-art, or near state-of-the-art results have been reported for these models on challengingtasks [5, 1].
1.4 Combination of end-to-end models with other modules: Fusion
What has also been reported is the combination of end-to-end models with other models, e.g.an LM P ∗ ( S ) trained on vastly more text data than is found in the transcripts of the audio dataused to train P ( S | X ) .
Here the story is much less rosy.
One approach, sometimes referred to as“Shallow Fusion”, is simple interpolation of the joint end-to-end model score with the LM score, e.g. P ( S | X ) α ∗ P ∗ ( S ) (1 − α ) [10].
The external LM has also been brought in through other methods, e.g.“Deep Fusion” [25] and “Cold Fusion” [16].
Though giving some practical gains, these are heuristicapproaches with no clear mathematical justiﬁcation such as Bayes’ theorem.
1.5 A better way: a deep generative acoustic model
While the ﬁrst generation generative models used for ASR were an excellent ﬁt to the channel model,their strong independence assumptions severely hobbled their effectiveness, especially compared tothe discriminative models, DNNs and LSTMs, that replaced them [15, 13].
In contrast, generativemodels have made large advances in the area of speech synthesis.
State-of-the-art TTS approachessuch as WaveNet and Tacotron [19, 22] speciﬁcally model p ( X | S ) as a fully-conditioned autoregressive model, using the entire unidirectional sequence of past observations x t (where x t is either anacoustic feature vector or a single scalar waveform sample), and typically, the entire sequence ofsymbols
s i constituting the sequence S .
The “conditional WaveNet” model [19] exempliﬁes this well,deﬁning
[Equation]
where the input S represents e.g. text and speaker features, and where observed samples x t are targetsof e.g. an N -ways quantized softmax trained with CE, using e.g. a DNN with dilated convolutions.
Mixture density networks [2], based on either DNNs or RNNs, have also been effective as deepgenerative TTS models [23, 24].
Their use for ASR was proposed 20 years ago but not fullyinvestigated [14].
Mixture density networks can use the same conditioning over observations andlabels as the sample-by-sample WaveNet model, but speciﬁcally adopt a Gaussian mixture densityfunction: [Equation]
− 1 , S )) .
[Equation]
This is a reasonable choice when x t is a feature vector, as opposed to a scalar sample as in WaveNet.
As ASR models typically operate on feature vectors, and not at the sample level, mixture densitynetworks may be a good ﬁrst step to investigate deep generative models for ASR.
1.6 ASR with a deep generative acoustic model
The central insight that this study is based on is that there is nothing preventing us from doing ASRwith deep generative acoustic models, along the lines of the models used for TTS just discussed.
Fundamentally, whenever a likelihood p ( X | S ) is available, ASR is possible - as long as a separate
LM P ( S ) is available to form the joint probability P ( X, S ) , and also assuming a decoder is availablethat can generate, score and prune different symbol sequence hypotheses S .
The strong conditioningon the entire symbol sequence history characterizing the models discussed here means that ASRdecoding for deep generative acoustic models faces practical challenges, but those challenges are notsigniﬁcantly different than those faced by the end-to-end ASR models such as LAS and RNN-T. Theeasiest implementation may be N -best rescoring of a list of hypotheses from an existing conventional ASR system.
1.7 Deepening the original tandem approach
One generative approach to ASR that has shown lasting effectiveness is the tandem approach [8, 6],still yielding near state-of-the-art results [18, 12].
In contrast to the use of scaled likelihoodsformed directly from the output of a discriminative model, described earlier, in the tandem approach,features from a penultimate layer are extracted from the discriminative model, and used in a separategenerative model such as GMMs.
Typically these deep features are concatenated with acousticfeatures, and transformed using e.g. Principal Components Analysis (PCA) before being used asinput to the generative model[18].
The result is a consistent generative model, but now operating onfeatures that are far more discriminative than raw acoustic features by themselves.
Related work hasinvestigated the embedding of a gaussian layer directly into the same DNN architecture, enablingjoint discriminative training of feature extractor and gaussian density parameters [20, 17].
Thesestudies limited themselves to DNNs for the discriminative feature extraction, and to standard GMMsfor the generative model.
The approach proposed here extends this past work in both dimensions.
1.8 Overview
[Figure] This study explores two dimensions of depth: the depth of the features being modeled, and the depthof the density function itself.
More speciﬁcally, the features considered were:
[Figure]
• Shallow features: raw acoustic features, such as logmel energies or cepstral features;
• Deep features: features obtained from the last layer of an LSTM stack trained discrimina-
tively with a standard criterion such as CE, in the spirit of the original Tandem approach
The density models considered were:
• Shallow density layers: vanilla gaussian mixture models, though implemented in TensorFlowand trained with SGD (using either ML or CE);
• Deep density networks: mixture density density networks, both autoregressive and non- autoregressive, used to generate density functions capable of modeling acoustic feature vector sequences strongly conditioned on past features and labels, trained with ML.
Shallow mixture density models estimated using ML/SGD, modeling
shallow features
The starting point for the models described here is the vanilla gaussian mixture,
[Equation]
deﬁned for any label w m and any feature vector x t .
This likelihood makes strong independence assumptions; it is not conditioned on previous observations, nor on previous label symbols.
Nonetheless,it can plug into the complete likelihood of the utterance, Equation 1.
A simple radial covariance (using a single scalar standard deviation per mixture component) modelwas found to be convenient and effective:
[Equation]
Autoregressive and non-autoregressive deep mixture density networks
[Figure] modeling shallow features
[Figure] Figure 1 illustrates a mixture density network generated from an LSTM stack, predicting the nextacoustic feature frame using all label symbols w 1: t up to that point in time (input to the LSTM stackvia a class embedding), and in the autoregressive version of this model illustrated here, all previousacoustic features as well:
[Equation]
deﬁned for a speciﬁc alignment of labels w t to observations x t .
In the non-autoregressive version, the mixture density network uses only the label symbols (and noprevious acoustic features) to predict the next acoustic feature frame:
[Equation]
The power of either of these mixture density network models is that the density function changesdynamically as a function of the input it is provided.
In principle, via the input of the unidirectionalembedded symbol sequence, the model can leverage long-span symbol context, a key feature in thecurrent climate of heightened interest in non-standard acoustic modeling units such as graphemes.
3.1 Variants between autoregressive and non-autoregressive
Some variants on the full autoregressive model were considered:
• SH= n : size of the frame shift between observations input into the LSTM stack, and target
[Figure]
of the prediction
, if different from 1 .
E.g., SH=3 refers to predicting x t from x 1: t − 3 .
• BN= n : size of linear bottleneck on features before being fed into the LSTM stack, if any.
• ST= n : size of the stride over the frames input into the LSTM stack, if different from 1 .
E.g.ST=10 refers to only using every 10 -th frame, feeding in 0 s in between. [Figure]
Shallow mixture density layer modeling deep discriminative features
[Figure]
Figure 2 illustrates a shallow density model of deep features.
The deep features are obtained from an
LSTM stack, separately trained with CE as a discriminative acoustic encoder.
The deep features inquestion are simply the output of the last LSTM layer.
In contrast to previous tandem work using
DNNs [20], no bottleneck layer was found to be necessary, presumably due to the more compact
LSTM layer size (e.g. 512 for LSTMs in this study, vs 2048 for the DNNs in [20]).
The model hereis:
[Equation]
If the LSTM stack implementing the discriminative acoustic encoder d () is frozen, this is just avanilla GMM layer that can be trained with ML, whose features happen to be highly discriminative.
Joint training of the density function and the feature extractor d () using ML is feasible, but requiresproper handling of the Jacobian of the inverse of d () .
Joint training using CE, sMBR or MMI,however, has no such issue [20, 17, 12].
Deep mixture density networks modeling deep discriminative features
Finally, one can apply a deep density network to the modeling of deep features, as illustrated in
Figure 3.
The result is an architecture closely mirroring the well-known RNN Transducer [7], butexplicitly formulated as a density function generated from an LSTM stack encoding a label sequence,applied to deep features encoding the acoustic sequence.
The resulting likelihood is:
[Equation]
If the LSTM stack encoding the deep features is frozen, this is a straightforward mixture densitynetwork that can be trained with ML, but with highly discriminative features.
As with the shallowdensity model of deep features, it can be trained jointly using discriminative criteria; and joint trainingwith ML is again feasible but requires proper handling of the Jacobian of the inverse of d () .
Experiments
The goal of the experiments was to verify that the added conditioning of the mixture density networksexamined indeed improves log-likelihoods as expected, and to get some insights about likely WERfrom frame accuracies for these models.
[Table]
6.1 Computation of frame accuracy
Frame classiﬁcation for density networks can be evaluated by producing N density outputs for Nclass label hypotheses input to the LSTM stack.
In all experiments here, a simple label class prior
P ( s ) is estimated in-network [20] and used jointly with the likelihood produced by the density modelto make the frame classiﬁcation.
An issue for deep density networks is the label context to use when measuring frame accuracy.
In areal decoding scenario, multiple partial label sequences would be hypothesized by the decoder.
Theapproach adopted here was to measure frame accuracy for the density networks via conditioning onthe ground-truth label sequence up to time t − 1 , with only the label at time t being hypothesized.
This can be computed efﬁciently in TensorFlow by batching the input of the N label hypotheses at alltimes t , but splicing into the inference pass a set of separately generated LSTM states up to t − 1obtained from ground-truth only input.
6.2 Datasets
The dataset set used here is a set of 20000 hours of spontaneous speech from anonymized, humantranscribed Voice Search data for American English (en-us) .
The training examples are sequencesof unstacked (single-frame) feature vectors extracted from a 64 ms window shifted over the audiosamples in 30 ms increments.
The acoustic features vectors used in the “deep feature” models,
[Table]
{+based on discriminative acoustic encoder architectures, are 256 dimensional logmel energies; thoseused in the “shallow feature” models are 96 dimensional cepstral coefﬁcients, consisting of a stackof 
+}
+32+32 mel ﬁlterbank cepstral coefﬁcients (MFCCs) and their ﬁrst and second derivatives,extracted from the same 64 ms windowing scheme.
The overall training architecture closely follows the TensorFlow based implementation describedin [21].
Each training step has a batch size of 64 example utterances, with a ﬁxed static LSTMunrolling of 20 frames.
All LSTM stacks used are 5 layer models with 512 units each, for a totalof roughly 4.3M parameters, not counting the ﬁnal layer, whose size will depend on the number ofoutput classes and, for the density models, on the number of mixture components.
The class outputsare 42 context-independent (CI) phonemes or 8192 decision-tree clustered context-dependent (CD)phonemes.
A label class embedding of size 128 was used for all experiments.
CE training of thedensity models combines the acoustic likelihood output by the density model with an in-networkestimate of the state prior P ( s ) (as in used in the computation of frame accuracy) [20].
The statistics reported were somewhat unscientiﬁcally obtained from smoothed TensorBoard plotswhen the models were deemed to have converged.
Frame accuracies are based on a small held outdataset and are hence quite noisy; only 2 signiﬁcant digits are reported for them.
The costs (negativelog-likelihoods) are smoothed costs for the training criterion at convergence.
As roughly 27% of thedata is silence, a frame accuracy of anything not surpassing that can ofﬁcially be considered abysmal.
6.3 Results
Table 1 describes the costs (negative log-likelihoods) and frame accuracies for all models usingshallow mel-cepstral features.
Shallow density with shallow features: The frame classiﬁcation of an ML-trained radial gaussianmixture barely surpasses abysmal levels, at 35%.
In contrast, a reference 5 layer DNN with amatching number of parameters is at 55%.
However, CE-training the gaussian mixture model bringsthe frame accuracy up to 48%.
This may be a reasonable result, suggesting that the ﬂat structure ofthe GMM isn’t as effective a classiﬁer as a deeply structured DNN, but isn’t completely off either, iftrained discriminatively.
The cost of 127 provides a baseline for the deep density models in the restof Table 1.
Those models are expected to improve signiﬁcantly over that.
Deep autoregressive densities with shallow features: As expected, the deep density models domuch better in terms of cost than the shallow density model just discussed.
Also as expected,the autoregressive models have better log-likelihoods than the non-autoregressive models, andfurthermore, the full autoregressive models do better than the autoregressive models whose inputwas limited via the schemes for prediction shift ( SH ), feature bottleneck ( BN ) and frame stride ( ST ).
However, the trend for frame accuracy is the opposite.
The more past input is provided to the LSTMstack generating the mixture density, the worse the frame accuracy.
One perspective is that providingtoo much information about past observations makes the prediction problem too easy, the label classinformation is not necessary, and hence the prediction is not discriminative.
Results are also shownfor the unsupervised versions ( Unsup ), where all class input is merged into a single symbol.
Oneexpects that the label class information would help improve the prediction cost over the unsupervisedprediction, but that is not the case for the CI phone scenario, corresponding to the “CI AR” result, with a cost of 43.5, compared to the “Unsup AR” cost of 43.0.
The “CD AR” model does a bitbetter with a cost of 42.3.
Shifting the prediction target 3 frames into the future ( SH=3 ), however,produces a gain for the CI model over the unsupervised version, 116 vs 118.
Going from 10 mixturecomponents to 256 signiﬁcantly improves prediction cost, but doesn’t signiﬁcantly affect the frameaccuracies for the autoregressive models here.
Deep non-autoregressive densities with shallow features: Removing all past observation input makesthe prediction cost much worse than for the autoregressive models, but the frame accuracies are muchbetter.
They are however still not competitive with the simple DNN, and in fact a bit worse than the
CE-trained shallow density model.
Table 2 ups the ante on frame accuracy, via the use of deep features.
A standard CE-trained LSTM islisted at the top of the table for reference.
Shallow densities with deep features, joint training with CE: Given the form of the radial densitydeﬁned in Equation 4 it should not surprise us that the classiﬁcation capacity of an LSTM stack usingthe radial density as the last layer, trained discriminatively, would closely match the classiﬁcationcapacity of an LSTM stack with a standard logits layer.
This is what we see for both CI and CDversions, with a single radial component per mixture, achieving frame accuracies of 87% and 79%respectively, matching the standard CI and CD LSTMs.
Shallow densities with deep features, ML training: All the ML results here require pre-training of thestandard LSTM (either CI or CD) with CE ﬁrst.
There is a clear drop in performance compared to thepure CE result, especially for the CD case, with 8192 output classes to discriminate compared to 42 .
However, the deep features here are not mean/standard-dev normalized (nor PCA’d, etc.).
The simpleradial covariance model clearly could be improved upon for this particular conﬁguration, if desired.
Deep densities with deep features: The dual LSTM stack architecture does quite well as a predictiorof the deep features, using just the label class context.
The prediction cost (e.g. -400 for the 10component mixture version) is much better than the cost (-305) for the shallow density model ofthe same deep features, and in fact has signiﬁcantly better frame accuracy too, 85% vs 77%.
It maybe argued that the deep features here have essentially extracted phoneme information from the CEtraining of the standard LSTM stack, and that now the prediction problem is really a label predictionproblem, given the previously observed label class input.
However, there is nothing wrong with thatfrom an ASR point of view, as the deep features are a function of the acoustics.
Furthermore, this maybe exactly what is interesting about this last conﬁguration: using an encoding of a label sequence, weare trying to predict an encoding of an acoustic sequence.
Summary
Four generative models were described and evaluated, the cartesian product of {shallow, deep}features and {shallow, deep} densities.
This recapitulates but also extends the original tandemapproach to ASR.
As no WER evaluations were run, these results are rather tentative, but do givesome insights into the models proposed.
The log-likelihood results follow our intuitions about thestrength of the models.
The weak result for supervised vs unsupervised may suggest an issue with theexperimental setup, e.g. the ﬁxed alignments used are poor, or it could reﬂect the weakness of theradial covariance model.
The somewhat better result for the CD phone model suggests either that the CI label context is not being completely represented by the LSTM state, or that “future blindness”(not knowing the identity of the immediately following phoneme) is a signiﬁcant issue.
The frame accuracies for the models using shallow acoustic features seem too low to be useful for
ASR.
One could argue, however, that the autoregressive models, properly used in an ASR decodingframework, may perform much better than the frame accuracies suggest.
The use of deep features solves many of the problems generative models have in modeling environmental and speaker variability, and immediately provides strong discriminative power, but it maybe seen as cheating and no longer purely generative.
Given their state-of-the-art frame accuracy,presumably the corresponding generative models described here would be viable for ASR.
Thequestion then is, How does the generative nature of these models help us?
E.g. Does it allow for bettercompositionality with separate LMs, as claimed in the Introduction?
Does it provide the advantagesattributed to generative models regarding unsupervised training or adaptation?
Full ASR decodingexperiments with WERs are needed to address those questions.