A Deep Generative Acoustic Model for Compositional Automatic Speech Recognition
Abstract
Inspired by the recent successes of deep generative models for Text-To-Speech(TTS) such as WaveNet and Tacotron [20, 23], this article proposes the use of a deepgenerative model tailored for Automatic Speech Recognition (ASR) as the primaryacoustic model (AM) for an overall recognition system with a separate languagemodel (LM).
After ﬁrst exploring the viability of using maximum-likelihood (ML)and stochastic gradient descent (SGD) to optimize mixture densities implementedin TensorFlow, two dimensions of depth are considered: (1) the use of mixturedensity networks, both auto-regressive and non-auto-regressive, to generate densityfunctions capable of modeling acoustic input sequences with much more powerfulconditioning than the ﬁrst-generation generative models for ASR, Gaussian Mixture
Models / Hidden Markov Models (GMM/HMMs), and (2) the use of standard
LSTMs, in the spirit of the original tandem approach, to produce discriminativefeature vectors for generative modeling.
Combining mixture density networks anddeep discriminative features leads to a novel dual-stack LSTM architecture directlyrelated to the RNN Transducer (Graves, 2012), but with the explicit functionalform of a density, and combining naturally with a separate language model, using
Bayes rule.
The generative models discussed here are compared experimentally interms of log-likelihoods and frame accuracies.
Introduction1.1 The channel model
For most of its history, the ﬁeld of ASR has used a collection of separate modules to represent differentstages of an overall processing chain.
Fred Jelinek formalized this approach using the notion of a noisychannel model in which components chain together not only in terms of an input/output sequence butalso in the sense of forming a consistent overall joint probability, decomposed using Bayes’ theorem[9].
This model was particularly convenient given the strong independence assumptions in the modelstructures used.
Gaussian Mixture Models (GMMs) used jointly with 1st-order Hidden Markov
Models (HMMs) were well-suited to this modular approach, as they directly provide an acousticlikelihood p ( x 1: t | s 1: m ) (decomposed using the chain rule, and
strong independence assumptions)for a sequence of acoustic feature vectors x conditioned on a given sequence of modeling units, e.g.phonemes, s .
This then combines naturally with a separate language model probability, P ( W ) , using
Bayes rule.
Other probabilistic modules such as a pronunciation dictionary can be introduced intothe overall chain, again combining with the other components according to Bayes’ rule.
One cankeep adding modules, or substituting modules, and the overall model still holds.
In that sense, thisapproach to ASR is “compositional”.
1.2 Shoehorning discriminative models into the channel model
The adoption of discriminative models such as Deep Neural Networks (DNNs) and Long Short Term
Memory models (LSTMs) did not at ﬁrst alter this model.
The popular “hybrid” approach (stillconsidered state-of-the-art by most the community today) simply converts the posteriors P ( s | x ) for amodel state s obtained from DNNs or LSTMs to a scaled likelihood by normalizing the posterior bya state prior p ( s ) [3].
The scaled likelihood can then plug back into the channel model using Bayesrule, as before, and can also plug into an overall sequence training objective function such as MMIor sMBR [12, 14].
The overall construction of a sequence-level training objective, converting localframe-level scores from a discriminative model into “generative” scaled likelihoods, only to thenplug those into a discriminative sequence training criterion, may seem like a strange hybrid.
1.3 Bypassing the channel model
In contrast, end-to-end models such as Listen, Attend & Spell (LAS) [4] or RNN Transducer [7]dispense with the channel model altogether, and directly optimize a discriminative sequence-levelmodel with a discriminative criterion.
By construction, these models are all-in-one models thatrepresent P ( W | X ) directly, with no chaining of sub-module probabilities.
If one has a large enoughtraining set of supervised data pairs, this can be thought of as the perfect model, and impressive resultshave been reported on a variety of tasks for such all-in-one, standalone models.
State-of-the-art, ornear state-of-the-art results have been reported for these models on challenging tasks [5, 1].
1.4 Combination of end-to-end models with other modules
What has also been reported is the combination of end-to-end models with other models, e.g. an LM
P ∗ ( W ) trained on vastly more text data than is found in the transcripts of the audio data used to train
P ( W | X ) .
Here the story is much less rosy.
1.4.1 Fusion
One approach, sometimes referred to as “Shallow Fusion”, is simple interpolation of the joint end-toend model score with the LM score, e.g. P ( W | X ) α ∗ P ∗ ( W ) (1 − α )
The external LM has alsobeen brought in through other methods, e.g. “Deep Fusion” [27] and “Cold Fusion” [17].
Thoughgiving some practical gains, these are heuristic approaches with no clear mathematical justiﬁcationsuch as Bayes’ theorem.
1.5 Hybridization & Joint training
A seldom-reported approach paralleling the hybrid model for ASR described earlier is to form aseparate estimate P ( W ) based only on the limited audio + text transcript training data availableto train P ( W | X ) , and use it to form a scaled likelihood of the acoustics given W .
This can thencombine with P ∗ ( W ) as P ( W | X ) /P
( W ) ∗ P ∗ ( W ) , following the concept of Bayes rule, but withsome uncertainty whether the scaled likelihood is a meaningful representaiton of p ( X | W ) , andrequiring separate estimation of P ( W | X ) and P ( W ) .
It remains to be seen whether this approachis more effective empirically than the fusion techniques.
As for the conventional ASR approachwith discriminative acoustic models, the scaled likelihood can be plugged into an overall sequencetraining objective function such as MMI or sMBR with a separate language model, if desired.
Thoughwell-motivated, these approaches lack theoretical clarity.
1.6 A better way: a deep generative acoustic model
While the ﬁrst generation generative models used for ASR were an excellent ﬁt to the channel model,their strong independence assumptions severely hobbled their effectiveness, especially comparedto the discriminative models, DNNs and LSTMs, that replaced them [16, 14].
However, thoughabandoned for ASR, generative models have made large advances in the area of speech synthesis.
State-of-the-art TTS approaches such as WaveNet and Tacotron [20, 23] speciﬁcally model p ( X | S )as a fully-conditioned auto-regressive model, using the entire unidirectional sequence of past observations x t (where x t is either an acoustic feature vector or a single scalar waveform sample), andtypically, the entire sequence of symbols s i constituting the sequence S .
More speciﬁcally, the “conditional WaveNet” model is [20]:
[Equation]
where the input S represents e.g. speaker and text info, and where observed samples x t are targets of
N -way quantized softmax trained with CE, using e.g. a DNN with dilated convolutions.
Mixture density networks [2], based on either DNNs or RNNs, have also been effective as deepgenerative TTS models [25, 26].
Their use for ASR was proposed 20 years ago but not fullyinvestigated [15].
Mixture density networks have the same conditioning power as the sample-bysample WaveNet model, but speciﬁcally adopt a Gaussian mixture density function as the densityform.
[Equation]
This is a reasonable choice when x t is a feature vector, as opposed to a scalar sample as in WaveNet.
As ASR models typically operates on feature vectors from a signal windowing scheme, and not at thesample level, mixture density networks may be a good ﬁrst step to investigate deep generative modelsfor ASR.
This is the approach explored here.
1.7 Deepening the original tandem approach
One generative approach to ASR that has shown lasting effectiveness is the tandem approach [8, 6],still yielding near state-of-the-art results [19, 13]. In contrast to the use of scaled likelihoods formeddirectly from the output of a discriminative model, described earlier, in the tandem approach, featuresfrom a penultimate layer are extracted from the discriminative model, and used in a separate generativemodel such as GMMs.
Typically these deep features are concatenated with acoustic features, andtransformed using e.g. Principal Components Analysis (PCA) before being used as input to thegenerative model[19]. The result is a consistent, ﬂexible generative model, but now operating onfeatures that are far more discriminative than raw acoustic features by themselves. Related work hasinvestigated the embedding of a gaussian layer directly into the same DNN architecture, enablingjoint discriminative training of feature extractor and gaussian pdf parameters [21, 18].
These studieslimited themselves to DNNs for the discriminative feature extraction, and to standard GMMs for thegenerative model. The approach proposed here extends this past work in both dimensions. 1.8 Overview Here, after ﬁrst exploring the viability of using maximum-likelihood (ML) and stochastic gradientdescent (SGD) to optimize mixture densities implemented in TensorFlow, we explore two dimensionsof depth: (1) the use of mixture density networks, both auto-regressive and non-auto-regressive, togenerate density functions capable of modeling acoustic feature vector sequences with much morepowerful conditioning than the ﬁrst-generation generative models for ASR, Gaussian Mixture Models(GMMs) coupled with Hidden Markov Models (HMMs), and (2) the use of standard LSTMs inthe spirit of the original Tandem approach to produce discriminative feature vectors for generativemodeling.
Combining mixture density networks and deep discriminative features leads to a noveldual-stack LSTM architecture directly related to the RNN Transducer (Graves, 2012), but with theexplicit functional form of a density, and formulated speciﬁcally as an acoustic model intended foruse with a separate language model.
The models described here are evaluated on a real-world Voice Search in terms of log-likelihoods and frame accuracies.  Shallow mixture density models estimated using ML/SGD, modelingshallow features Figure 2 illustrates the simplest of the models described here: a vanilla gaussian mixture.
Implementedin TensorFlow, this shallow density layer is a gausian mixture deﬁned in terms of model parametersrepresented as TensorFlow Variables.
Simple TensorFlow operations (exponential transform andsoftmax) are used to enforce the constraints that the standard deviations and the mixing weights arepositive and that the mixing weights sum to SGD-based optimization of the density function can bedone with a number criteria the natural ﬁt for this simple generative model is Maximum Likelihood(ML) but discriminative criteria such as Cross Entropy (CE) or sequence-level discriminative criteria as MMI or sMBR can also be used [
2.1 Verifying the effectivess of ML/SGD training
ML-based estimation of GMMs was traditionally done with the highly effective ExpectationMaximization (EM) algorithm In contrast to the EM algorithm, SGD is not speciﬁcallysuited to ML/GMM optimization, but its generality is highly appealing in the context of joint optimization of density functions with deep features or density parameters that are themselves generatedby arbitrary, possibly recurrent neural network stacks.
The use of SGD for ML-based estimationof GMMs has not been widely reported. A ﬁrst step in this study was to verify that ML/SGD caneffectively learn the correct estimates given synthetic data drawn from known distributions, suchas that illustrated in Figure 1. For such low-dimensional scenarios easily inspected visually, it wasfound that as long as the features were mean/standard-dev normalized, standard rules of thumb of
GMM parameter initialization [24] led to surprisingly effective. For real-world data, two rough sanitychecks were used.
(1) TensorBoard histograms of mixing weights can diagnose unhealthy situationswhere a single gaussian component overwhelms all others in the mixture; (2) log-likelihoods on thetraining set should improve signiﬁcantly with increasing numbers of mixture components. Carefulcomparison of ML/SGD-estimated GMMs with EM-estimated GMMs, and schemes for mixturesplitting in TensorFLow, could yield insights and better performance. The point for now is to have aworking setup that can handle more than a single gaussian component, in an SGD-based optimization scenario that can be connected to joint optimization of deep features and mixture density networks
[Figure] with powerful conditioning.
[Figure]  Auto-regressive and non-autoregressive deep mixture density networks [Figure]
modeling shallow features
[Figure] Figure 3 illustrates a mixture density network generated from an LSTM stack, predicting the nextacoustic feature frame using all label symbols up to that point in time (via a class embedding),and, in the auto-regressive version of this model illustrated here, all previous acoustic features aswell.
Optionally, the mixture density network may use only the label symbols (and no previousacoustic features) to predict the next acoustic feature frame; that archtecture is referred to as thenon-autoregressive version. The power of either of these mixture density network models is that thedensity function is dynamically changing as a function of the input it is provided. In principle, viathe input of the embedded symbol sequence, the model can leverage unidirectional symbol context,a key feature in the current climate of heightened interest in non-standard acoustic modeling unitssuch as phonemes.
The auto-regressive version has the further feature that the “ground truth” of the observed past acoustic feature vector sequence enables a kind of adaptation of the predictive modelto the actual speech signal input to the model.
In contrast, the non-auto-regressive model can onlyleverage the given symbol context in making its predictions; it has to absorb all acoustic variabilityof the speech signal observed in the training set, across all speakers, speaking styles and acousticenvironmetns.
Shallow mixture density modeling deep discriminative features
TODO:
(1) describe the use of a separately-trained LSTM stack as a discriminative audio encoder,generating discriminative features that are then modeled by a shallow mixture density model; (2)describe the fact that this updates the classic & popular hybrid/tandem scheme for combiningdiscriminative and generative models in a consistent way; (3) describe the advantages afforded by
LSTMs for this purpose compared to DNNs used in the past; (4) describe the possibility of jointdiscriminative training of both the deep features and the density layer; (5) describe in contrast thechallenges posed by joint generative training, given the need to model the Jacobian of the inverse ofthe feature extraction function.
The experiments here consist of evaluating frame accuracy, log-likelihood, and ideally WER for:
The baseline discriminative model;
The shallow density model trained using ML on deep features from a frozen discriminative
acoustic encoder;
The shallow density model trained using CE on deep features from a frozen discriminative
acoustic encoder;
The shallow density model trained using CE, jointly optimizing deep features and the
discriminative acoustic encoder.
Deep mixture density networks modeling deep discriminative features:
the RNN Generative Transducer
TODO: describe the dual use of depth for both (1) the features and (2) the density model used onthose features.
This combines the shallow/deep conﬁguration and the deep/shallow conﬁgurationdescribed above.
The result is an architecture closely mirroring the well-known RNN Transducer, butexplicitly formulated as a density function generated from an LSTM stack encoding a label sequence,applied to deep features encoding the acoustic sequence.
The experiments here consist of:
Log-likeloods and frame-accuracies for the RNN Generative Transducer, compared to thesame for the simpler models
Experiments
6.1 Datasets
The training set used consisted of 20000 hours of spontaneous speech from anonymized, humantranscribed Voice Search data for American English (en-us) .
The test set is a separate anonymized,human-transcribed Voice Search dataset of about 25 hours.
The training examples are sequencesof unstacked (single-frame) feature vectors extracted from a 64 ms window shifted over the audiosamples in 30 ms increments.
The features vectors are either 256 dimensional logmel energies,or 96 dimensional cepstral coefﬁcients, consisting of a stack of 32+32+
mel ﬁlterbank cepstralcoefﬁcients (MFCCs) and their ﬁrst and second derivatives, extracted from the same 64 ms windowingscheme.
The training architecture closely follows the TensorFlow based implementation described in [22].
Each training step has a batch size of 64 example utterances, with a ﬁxed static LSTM unrolling of 20frames.
Given the total batch size of 1280 frames, 1 epoch of training over the 20000 hour training [Table] [Table] set corresponds to 1 .
875 M training steps.
All LSTM stacks used are 5 layer models with 512 unitseach, for a total of roughly 4.3M parameters, not counting the ﬁnal layer, whose size will depend onthe number of output classes and, for the density models, on the number of mixture components.
Theclass outputs are 42 context-independent phonemes or 8192 decision-tree clustered context-dependentphonemes.
6.2 Shallow density model & shallow features
Summary
Four generative models were described and evaluated. TODO: Expand this.
Coming soon.