Variational Autoencoders with implicit priors for short-duration text-independent speaker veriﬁcation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this work, we exploited different strategies to provide prior knowledge to commonly used generative modeling approaches aiming to obtain speaker-dependentlow dimensional representations from short-duration segments of speech data,making use of available information of speaker identities.
Namely, convolutionalvariational autoencoders are employed, and statistics of its learned posterior distribution are used as low dimensional representations of ﬁxed length short-durationutterances.
In order to enforce speaker dependency in the latent layer, we introduced a variation of the commonly used prior within the variational autoencodersframework, i.e. the model is simultaneously trained for reconstruction of inputsalong with a discriminative task performed on top of latent layers outputs.
Theeffectiveness of both triplet loss minimization and speaker recognition are evaluatedas implicit priors on the challenging cross-language NIST SRE 2016 setting andcompared against fully supervised and unsupervised baselines.
Introduction
Variational autoencoders (VAEs) (1; 2) have been introduced as an effective framework within thecontext of generative models that support tractable approximate inference (3), leveraging neuralnetworks both for generative modeling as well as for approximate inference, usually employing anon-informative prior.
However, follow-up works have shown that too simplistic of a prior will ingeneral lead to also simplistic posteriors which might not encode relevant information about theinputs.
Attempts to overcome the above mentioned issue include adversarial autoencoders, proposed originally in (4), which employ an adversarial game on top of latent variables.
The discriminator triesto distinguish samples from the posterior and prior distributions, while the encoder tries to producesamples that are indistinguishable from the prior.
Moreover, stochastic variational methods (5; 6)appeared as an alternative in which informative data-dependent priors can be used.
Sampling methodsare employed to estimate gradients of the variational gap, such that any prior from which one cansample can be used.
In both of the described cases, the only requirement for a prior is the possibilityof efﬁciently sampling from it.
Even though aforementioned adversarial autoencoders and stochastic variational methods allow theuse of non-trivial priors, designing prior distributions which yield desired properties on the ﬁnallylearned posterior is a challenging task in itself.
In this work, rather than explicitly matching posteriorand prior distributions, we evaluate the effectiveness of enforcing relevant properties on the posteriordistribution by introducing auxiliary discriminative tasks at train time, making use of available labels.
By doing so, we argue prior knowledge is introduced implicitly, since desired properties are directlyenforced into the posterior distribution.
The remainder of this paper is organized as follows: Section 2 includes a brief description of the VAEframework along with a brief deﬁnition of the speaker veriﬁcation problem, which we employ as atest-bed for the proposed approach.
Section 3 details the strategy we proposed in order to enforcedesired properties within the VAE’s learned posterior.
In Section 4 we evaluate our method, andﬁnally draw conclusions along with future directions in Section 5.
Background: Variational Autoencoders and Speaker Veriﬁcation
Consider p ( X, Z ) , where X is the observed data and Z is the latent representation.
The posteriordistribution p ( Z | X ) can be approximated within the family of distributions q ( Z | X, λ ) , parametrizedby λ .
The so-called variational gap has to be minimized in order to give the maximum likelihoodestimate of λ .
The variational gap is deﬁned as the Kullback-Leibler divergence between theapproximate q ( Z | X, λ ) and the true posterior over Z , p ( Z | X ) , written as KL ( q ( Z | X, λ )
|| p ( Z | X )) .
A common approach to minimize KL ( q λ ( Z | X ) || p
( Z | X )) with respect to λ is to deﬁne the Evidence
Lower Bound (ELBO) given by:
[Equation]
whose terms can be rearranged, and ELBO can be simpliﬁed to:
[Equation]
Two main components present in above equation are the inference model
q ( Z | X, λ ) and the generativemodel p ( X | Z ) .
VAEs parametrize both distributions using neural networks in an encoder/decodersetup.
The encoder takes samples from X and outputs the parameters λ of the latent variable model
[Equation]
X ) .
The decoder receives samples from Z as input and returns reconstructed data samples fromp φ ( X | Z ) .
Parameters θ and φ are the weights and biases of the neural networks which are selectedto minimize the negative ELBO using stochastic gradient descent.
The negative of the ELBO yieldsthe following loss function used for training the neural networks:
[Equation]
First term in above equation is equivalent to maximum likelihood estimation, thus it is in generalsubstituted by a reconstruction loss, while the second term can be seen as a regularizer, which tries toensure that the approximation follows the prior distribution as much as possible.
The posterior q θ ( Z | X ) is in general assumed to be an uncorrelated Gaussian.
In order to train the VAEusing stochastic gradient descent, the reparametrization trick (7; 8) is employed allowing gradientscomputation through the sampling process between encoder and decoder.
Hence, the outputs of theencoder network are the statistics of q θ ( Z | X ) and Z - input for the decoder - is ultimately obtainedby Z = µ ( X ) + σ ( X ) · (cid:15) , where µ ( X ) and σ ( X ) are the encoder’s outputs given X , while (cid:15) issampled from N (0 , I ) .
Speaker veriﬁcation consists of accepting or rejecting a claimed identity by comparing two spokenutterances, the ﬁrst of these utterances being used for enrollment (produced by the speaker with thetarget identity) and the second utterance is obtained from the veriﬁed speaker (9).
Under the text-independent setting, speaker veriﬁcation is performed on top of unconstrained spokenphrases of arbitrary length.
The added phonetic variability in this scenario represents an extra adversefactor when compared to the session and speaker variabilities, present in the text-dependent case(10).
Classical approaches for Automatic Speaker Veriﬁcation split the problem into two distinctphases: (
Proposed Model
Unlike the ELBO-based loss deﬁnition in Equation 3, we evaluate the use of an auxiliary task ontop of the posterior statistics µ ( X ) , with the aim at enforcing a multi-modal posterior with modesdepending on given class labels.
Our training loss is thus deﬁned by:
[Equation]
where the ﬁrst term, the mean squared error between the input X and its reconstructed pair X (cid:48) , isthe same as in the standard VAE setting, while the second term, D ( µ ( X ) , y ) , is some discriminativeloss which plays the role of the KL term in Equation 3, considering given class labels y .
β ∈ [0 , 1]is a tunable hyperparameter.
µ ( X ) is employed as a low-dimensional embedding of inputs for thediscriminative auxiliary task.
Two distinct choices of D ( µ ( X ) , y ) are evaluated here:
A soft triplet loss deﬁned by softplus ( || d + − d − || 2 ) , where d + and d − correspond to
a distance measure between pairs of embeddings.
d is chosen as d ( µ ( X 1 ) , µ ( X 2 ))
=1 − µ ( X 1 ) · µ ( X 2 )
|| µ ( X 1 )
||
2 || µ ( X 2 )
|| 2 , and the second term is the cosine of the smallest angle betweenµ ( X 1 ) and µ ( X 2 ) .2.
The sum of triplet loss with a multi-class classiﬁcation loss, i.e. µ ( X ) is linearly projectedinto an output layer and cross-entropy loss is measured using available labels.
We evaluate the described setting on the speaker veriﬁcation task
RMSProp is
employed foroptimization with α set to 0.99.
The global learning rate starts at 0.001 and is halved once tripletloss,
measured
on a validation set held out of training, plateaus for 30 epochs.
Training is carriedout in a single Titan X NVIDIA GPU, with minibatches of size 64.
Minibatches are contructed suchthat two random segments of different utterances belonging to the same speaker are sampled to formsame class pairs (positive), and a random sample from a different speaker is selected to compose thedifferent classes pair (negative).
β was at 0.8 for all experiments.
Experimental Setup and Results
Evaluation is performed on top of the cross-language NIST SRE 2016 setting (11).
Test data in
Tagalog and Cantonese are available, while train data is in English.
Embeddings obtained with astandard VAE, along with our two proposed strategies using two distinct D ( µ ( X ) , y ) previouslydescribed choices are compared with x-vectors, a fully-supervised approach shown to outperformi-vectors (12) in the full-recording setting (13).
Train data is composed of: Switchboard -2, phases1, 2, and 3, along with NIST SRE s from 2004 to 2010 combined with Mixer 6, which sums up toapproximately 7000 speakers, out of which we remove all the recordings of 50 speakers to be used asvalidation set.
Training is performed on top of 40-dimensional log-mel ﬁlter banks.
Only the SREportion is used for training probabilistic linear discriminant analysis (PLDA) (14), which was used asa backend at evaluation phase.
Since our model requires ﬁxed size inputs, speech segments of 256frames were randomly selected from each recording at train time.
We augment the described traindataset following the approach in (13), i.e. with additive background noise from the MUSAN corpusand reverberation by convolving room impulse responses (RIR) with original audio data (MUSANand RIR are available at www.openslr.org ).
We removed silence frames from data using a simpleenergy-based voice activity detector.
For enrollment, test, and unlabelled (used for PLDA adaptation) data, embeddings of each recordingare obtained from 256 frames windows without overlap, and then averaged, such that each testutterance is represented by a single ﬁxed dimensional representation, even though models only haveaccess to short-duration segments.
PLDA was employed as backend after dimensionality reduction of embeddings from 256 to 128 ,using linear discriminant analysis.
PLDA is trained on embeddings from the SRE partition of trainingdata, which are computed following the same approach as described for test data for the case ofour proposed models, while using the full-recordings in the case of x-vectors.
Results in terms of
Equal Error Rate (EER) are shown in Table 1 for embeddings obtained from VAEs trained both in astandard fashion, and our proposed approaches.
[Table]
As expected, including speaker identities relevantly increases the discriminability of learned representations when compared to a fully-unsupervised VAE, in both Tagalog and Cantonese evaluations.
We further notice that performing speaker recognition on top of statistics of the posterior is moreeffective than the metric learning approach of triplet loss minimization alone.
In order to overcome the relevant domain shift between train and test data due to different spokenlanguages, the model adaptation scheme introduced in (15) is utilized for PLDA.
To do so, embeddingsof unlabelled data in Cantonese and Tagalog are clustered, and clusters are used as speaker identities,which are then employed for training a second PLDA model.
The ﬁnal model is obtained by simplyaveraging the second order statistics of the two trained models.
Results, as reported in the right section of Table 1, correspond to the evaluation using the adapted
PLDA model.
Interestingly, one can notice that the higher the level of supervision employed onembeddings model training, the higher is the performance gain when adaptation is used.
By level ofsupervision we mean how relevant class labels (speaker identities in the studied case) are at train time.
Standard VAE makes no use of class labels, while triplet loss employs such information for tripletsconstruction only.
Even in the case in which our VAE is trained with cross-entropy minimization,semi-supervised settings can be used, leveraging available unlabelled data, which is not the case forx-vectors, for instance, whose training is performed in a fully-supervised fashion.
We thus argue thatan increasing level of supervision induces domain-dependent representations, and this is the reasonadaptation yields a huge improvement in such cases.
We further evaluate the discriminability of the representations corresponding to the statistics ofposterior distributions approximated by VAEs trained in a standard fashion and making use ofavailable speaker identities by plotting 2-dimensional t-SNE embeddings of µ ( X ) , computed for 10speakers held out of training.
Figures 1, 2, 3 are ordered in increasing level of supervision , whichonce more supports the claim that making use of class labels to perform discriminative tasks on topof statistics of the posterior is an effective strategy to enforce desired properties.
Conclusion
In this work, we proposed to exchange the divergence term within the variational autoencoderstraining loss by some discriminative cost, leveraging available class labels.
We thus argue such anapproach is equivalent to implicitly deﬁning prior distributions, directly inducing desired propertiesin the learned posterior distribution.
Evaluation is performed on the challenging cross-language NIST
SRE 2016 evaluation setting, for which we show embeddings obtained by such an approach arespeaker-dependent, as enforced by discriminative tasks performed at train time.
Future directionsinclude the evaluation of this framework on the semi-supervised setting, employing unlabelled datafor training of the generative model, along with labelled data.
Appendix A - Model architecture
Architectures employed for encoder and decoder are detailed in Tables 2 and 3.
Batch normalizationis used after all convolution layers.
Inputs present dimensionality [40 , 256] , corresponding to 40 ﬁlterbanks and 256 frames.
[Table] [Table]