A DAPTIVE S PEECH D URATION M ODIFICATION USING A D EEP -G ENERATIVE F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose the ﬁrst method to adaptively modify the duration of a given speech signal.
Our approach uses a Bayesian framework to deﬁne a latent attention map that links frames of the input and target utterances.
We train a masked convolutional encoder-decoder network to generate this attention map via a stochastic version of the mean absolute error loss function.
Our model also predicts the length of the target speech signal using the encoder embeddings, which determines the number of time steps for the decoding operation.
During testing, we generate the attention map as a proxy for the similarity matrix between the given input speech and an unknown target speech signal.
Using this similarity matrix, we compute a warping path of alignment between the two signals.
Our experiments demonstrate that this adaptive framework produces similar results to dynamic time warping, which relies on a known target signal, on both voice conversion and emotion conversion tasks.
We also show that the modiﬁed speech utterances achieve high user quality ratings, thus highlighting the practical utility of our method.
I NTRODUCTION
Human speech is a rich and varied mode of communication that encompasses both language/semantic information and the mood/intent of the speaker.
The latter is primarily conveyed by prosodic features, such as pitch, energy, speaking rate, and voice quality.
There are many applications where understanding and manipulating these prosodic features is required.
Consider voice conversion systems as an example.
Pitch and energy modiﬁcations are used to passively inject emotional cues into the neutral speech or to change the overall speaking style (A. Russell et al., 2003;
Schacter et al., 2011; Shankar et al., 2019a;b; Valle et al., 2019).
Prosodic features are also used to evaluate the quality/engagement in human machine dialog systems (Swerts & Krahmer, 2000), and they play a signiﬁcant role in speaker identiﬁcation and recognition systems (Park et al., 2016).
While there are many approaches for automated pitch and energy modiﬁcation (Toda et al., 2007; Aihara et al., 2012; Kaneko & Kameoka, 2017; Shankar et al., 2020b;a), comparatively little progress has been made in changing the duration/speaking rate of an utterance.
In fact, the speaking rate plays a crucial role in conveying emotions (Schmidt et al., 2016) and in diagnosing human speech pathologies (Bayerl et al., 2020).
The speaking rate is difﬁcult to manipulate because, unlike pitch or energy, there is no explicit coding for either the signal duration or speaking rhythm.
Rather, these features are implicitly deﬁned by the cardinality of the set of frames over a particular interval of interest.
This cardinality is a global parameter that masks subtle variations in the speaker rate over an utterance.
As a result, duration modiﬁcation algorithms are not adaptive.
Instead, they either require considerable user supervision or they are geared towards aligning to known speech signals.
Perhaps the earliest duration modiﬁcation method is the time-domain pitch synchronous overlap and add (TD-PSOLA) algorithm
(Charpentier & Stella, 1986).
TD-PSOLA modiﬁes the pitch and duration of a speech signal by replicating and interpolating between individual frames centered at the peaks of auto-correlation signal.
However, the user must manually specify both the portion of speech to modify and the exact manner in which it should be altered.
Hence, the method is neither automated nor adaptive.
An alternative approach is dynamic time warping (DTW), which ﬁnds the optimal time alignment between two parallel speech utterances (dtw, 2008).
DTW constructs a pairwise similarity matrix between all frames of the two utterances and estimates a warping path between the starting (0 , 0) and ending ( T s , T t ) points of the utterances based on a Viterbi-like de- [Figure] [Equation]
[Figure]
Finally, recent advancements in deep learning have led to a new generation of neural vocoders that disentangle the semantic content from the speaking style (Oord et al., 2016;
Shen et al., 2017;
Wang et al., 2017).
These vocoders can alter the speaking rate via the learned style embeddings.
While these models represent seminal contributions to speech synthesis, the latent representations are learned in an unsupervised manner, which makes it difﬁcult to control the output speaking style in predictable manner.
Another drawback of these methods is the large amount of data and computational resources required to train the models and generate new speech (Yasuda et al., 2020).
In this paper, we introduce the ﬁrst fully-automated adaptive speech duration modiﬁcation scheme.
Our approach combines the representation capabilities of deep neural networks with the structured simplicity of dynamic decoding.
Namely, we model the alignment between a source and target utterance via a latent attention map; these maps are used as replacement of similarity matrix for backtracking.
We train a masked convolutional encoder-decoder network to estimate these attention maps using a stochastic mean absolute error (MAE) formulation.
Unlike the conventional
DTW (dtw, 2008) algorithm, once trained our framework operates entirely on the source utterance without needing to reference the target.
We demonstrate our framework on a voice conversion task using the CMU-Arctic dataset (Kominek & W Black, 2004) and on three multi-speaker emotion conversion tasks using the VESUS dataset (Sager et al., 2019).
Our experiments conﬁrm that the proposed model can perform open-loop duration modiﬁcation and produces high-quality speech.
M ETHOD
Fig.
1 illustrates our underlying generative process.
Given an utterance X , we ﬁrst estimate the length T of the (unknown) target utterance Y and subsequently use it to estimate a mask M for the attention map.
The mask restricts the domain of the attention vectors A t at each frame t during the inference stage to mitigate distortion of the output speech.
We use paired data ( X tr , Y tr ) to train a convolutional encoder-decoder network to generate the attention vectors.
During testing, we ﬁrst generate the attention map from the input X and use it to produce the target speech Y .
2. L OSS F UNCTION
Formally, let X ∈ R D
× T s denote the input speech.
In this work, X corresponds to the Mel ﬁlterbank energies extracted from short-time moving window analysis, where D is the number of ﬁlterbanks, and T s is the number of temporal frames in the utterance.
Similarly, we denote the target speech as Y ∈ R D × T .
Notice that the target utterance length T is usually different from T s .
[Figure]
Our generative process for a single frame of the target speech is represented as follows:
[Equation]
where T is the estimated length of the target utterance, and Y t is the target Mel ﬁlter-bank energy features at time t .
The parameters { T 0 , b T , Y 0 t , b y } of the distributions are unknown and implicitly estimated via a deep neural network.
The neural network is parameterized by γ and θ (Fig. 1).
By treating the unknown parameters as functions of the input X , we obtain the following estimating equations for the target sequence length and frame-wise Mel ﬁlter-bank energies:
[Equation]
The functions f γ ( · ) and f θ ( · , · ) correspond to the length prediction and energy estimation component of the same deep neural network.
The variable A t ∈ R T s is an attention vector that combines framewise features of the source utterance X to generate the target frame ˆ Y t .
Our model differs from standard sequence-to-sequence model by treating neural net predictions as residuals added to input sequence itself.
Notice that the residuals depend on input and the history of predictions ˆ Y 0: t − 1 at previous time steps.
This autoregressive property allows the neural network to learn segmental and supra-segmental variations that can potentially distinguish between the speakers or emotions.
During training, we use paired data ( X, Y ) and maximize the likelihood of the target speech signal with respect to the neural network weights { θ, γ } .
This likelihood can be written as:
[Equation]
where, the second term in Eq.
(3) can be obtained via marginalization over A t as follows:
[Equation]
The variable M here denotes the attention mask.
We introduce M for convenience, as it is a deterministic function of the source length T s and the estimated target length ˆ T .
We encode the attention
A t as a one-hot vector across the T s frames of the source speech.
Thus, it follows a multinomial distribution.
For simplicity, we model A t as conditionally independent of the target length ˆ T given the mask M and the input X
(see Fig. 1).
Taking log( · ) of Eq.
(3) and combining with Eq.
(4) yields:
L
[Equation]
θ, [Equation] = [Equation]
log
[Equation]
[Equation]
[Equation]
A t
[Equation]
const. [Equation] The distribution q θ ( · ) above is an approximating distribution for the attention vectors implemented by a convolutional network.
The ﬁrst inequality uses the convexity of the − log function, and the second inequality comes from the fact that entropy H ( q θ ) ≥ 0 .
Notice that we have implicitly assumed P ( A t | X, ˆ Y 0: t − 1 , M ) has a uniform distribution over the masked region.
This is a reasonable assumption given that the masking process reduces the attention domain to a small region (see Section 2.3).
However, q θ is not penalized for deviating from this uniform distribution during training.
This ﬂexibility allows the network to learn realistic attention vectors during autoregressive decoding.
Eq.
(5) can be easily translated into a neural network loss function which we minimize for { θ, γ } :
[Equation]
(6) where λ 1 and λ 2 are the model hyperparameters that adjusts the trade-off between the two objectives and implicitly contain the variances of the Laplace distributions in Eq.
Notice that the loss in
Eq.
(6) computes an expectation over the attention maps.
We use the Monte-Carlo estimate by sampling from the attention map at each time-step.
The training procedure is therefore stochastic in nature due to this random sampling.
We mix this stochastic version with the maximum aposteriori estimate (MAP) of the attention vector with a probability of 0.2 during the start of training procedure.
1 function trainModelParameters ( X, Y ) ;
Input : ﬁlterbank energies ( X ∈ R D × T s , Y ∈ R D × T t ) Output: model parameters ( θ, γ ) 2 if epoch <
MaxEpochs then
Set t = 0 , predict target length
ˆ T =
f γ ( X ) and create the mask M ∈ R T s × T t ;
Estimate A ∈ R T s × T t using masked convolution and sample u ∼ U (0 , 1) ; if u < 0.2 then
Sample a ∈ R T s × T t as 1-hot vectors from A ;
Reconstruct using ˆ Y t =
X · a + f θ ( X, Y 0: t − 1 ) ; else
Reconstruct using ˆ Y t =
X · A + f θ ( X, Y 0: t − 1 ) ; end
Compute prediction errors and update parameters θ, γ ; epoch ← epoch + 1; 13 end 14 return θ and γ ;
2. T RAINING AND T ESTING A LGORITHM
Algorithm 1 describes our training strategy.
First, we extract the ﬁlterbank energies from the paired input-output utterances and deﬁne the ground truth for length prediction as the ratio of the inputto-output sequence length.
Next, we construct the attention mask (Sec. 2.3) for each input-output sample pair in the mini-batch.
The neural network then processes the input frames and generates an embedding for the decoder operation and to predict the target sequence length.
It further estimates an attention vector as a multinomial distribution at each decoder step inside the speciﬁed masked region.
Finally, to compute the loss, we either sample from this multinomial distribution or directly apply the maximum a posteriori attention estimate to the input sequence to get the target frames.
We use a Bernoulli sampling procedure with probability of 0 . 2 in line 5 (i.e., low contribution of the stochastic loss) to prevent the model from diverging in sub-optimal directions.
The MAP estimate helps in this regard.
Empirically, we found this to be extremely helpful in generating monotonic attention that is also group sparse in nature.
We ﬁxed the slope of attention mask in
1 function modifyDuration ( X ) ;
Input : ﬁlter-bank energy ( X ∈ R D
× T s and Y 0 )
Output: alignments ( ( x 1 , y 1 ) , ( x 2 , y 2 ) , ... )
2 Predict length of target sequence ˆ T t = f γ ( X ) ; 3 Create attention mask M ∈ R T s × ˆ T t and Set t = 0 ; 4 if t < ˆ T t then
Using mask M t , X , and Y 0: t − 1 estimate A t ;
Using X , Y 0: t − 1 , and A t , predict Y t ; t ← t + 1 ; 8 end 9 Run DTW backtracking on the attention matrix A ; 10 return (alignments ( x 1 , y 1 ) , ( x 2 , y 2 ) , ...
( x n , y n ) );
[Figure]
line 4 to 1 .
based on the relative difference in length observed from the training datasets.
The distribution q θ is the variational distribution which is approximated by a fully convolutional neural network shown in Fig.
The testing procedure is shown in Alg.
Here, we do not sample from the attention distribution at each time step t .
Instead, the encoder-decoder model is allowed to run autoregressively for the predicted number of time steps ( ˆ T t ).
We use this attention map as a proxy for DTW similarity matrix to run Viterbi decoding strategy and obtain a sequence of co-ordinates representing the frame-wise correspondence between the input and estimated target speech.
The generative/testing procedure (Alg. 2) is similar to the training strategy except for some minor differences.
In the generative mode, we use the predicted target sequence’s length instead of the ground truth.
Second, we do not sample from the attention vector as the estimated target sequence is no longer important.
We use the attention map as a proxy for the DTW similarity/cost matrix for
Viterbi alignment.
This alignment allows us to rearrange the input frames to give modiﬁed speech.
2. M ASKING
The mask M is used to constrain the scope of the attention mechanism to be similar in time-scale to the input.
This procedure is important for two reasons.
From a speech quality perspective, large local swings in speaking rate may generate unintelligible speech.
From an estimation perspective, the speech utterances contains hundreds (sometimes thousands) of frames.
It is difﬁcult to robustly train a deep network to generate such long attention vectors using smaller datasets.
[Figure]
i
[Figure]
t i o n
[Figure]
rr o
[Figure]
m s
[Figure]
s
[Figure]
[Figure]
We use the masks derived from Itakura parallelogram (Itakura, 1975), as illustrated in Fig. 2.
The
Itakura parallelogram is generally used to speedup the DTW algorithm when the speaking rates in the source and target utterances are expected to lie in a certain interval.
The slope of the Itakura parallelogram speciﬁes the minimum and maximum speaking rates that the reconstructed utterances are allowed to possess in comparison to the input speech.
In this paper, we ﬁxed the minimum and maximum variation in speaking rate to 0 .
8 and 1 . 25 , respectively after observing the training data.
2. N EURAL N ETWORK A RCHITECTURE
We adapt the neural network architecture from (Gehring et al., 2017) by adding residual/skip connections to the last layer and reconﬁguring the entire attention module.
Fig.
3 shows the encoder, decoder and the modiﬁed attention module of the convolutional neural network used for experiments.
The encoder is responsible for generating feature embeddings for the decoder and for predicting the relative length of target speech.
The sample operation in Fig.
3 is responsible for generating a sample from the attention distribution required for reconstruction and backpropagation.
We train our model using mini-batch gradient descent and Adam optimizer (Kingma & Ba, 2015) with a ﬁxed learning rate of 10 − 4 .
We ﬁxed the batch size to 16 .
The input X is 80-dimensional Melﬁlterbank energies spanning 0-8kHz.
The projection layer expands this input to 256 dimensions.
Both the encoder and decoder consist of 10 convolutional layers, each followed by gated linear unit.
Given the small size of training data, we use data augmentation to properly estimate the network parameters.
Speciﬁcally, we reverse the input-output sequences and randomly extract intervals of variable size (with probability 0 . 5 ) from the full speech utterance.
2. DTW B ACK -T RACKING
Our ﬁnal step uses the attention map produced by the encoder-decoder as a proxy for the DTW similarity matrix between the source and target speech frames.
This strategy allows us to train the convolutional model on a relatively small dataset (e.g., 2-3 hours) and still generate intelligible speech during open-loop modiﬁcation of new utterances.
Effectively, we apply a dynamic programming operation to the attention maps produced by the neural network to get a path of alignment from source to target, rather than rely on the noisy spectral reconstruction for resynthesis (see Algorithm 2).
To avoid skipping phonemes, we constrain the dynamic programming path to take at most one horizontal or vertical at a time while backtracking.
Once estimated, the path informs a reorganization of the source utterance frames via local contraction dilation operations.
Following this reorganization, the target speech is synthesized via the WORLD vocoder (Morise et al., 2016).
E XPERIMENTAL R ESULTS
We evaluate our model on two multi-speaker datasets: CMU-ARCTIC (Kominek & W Black, 2004) and VESUS (Sager et al., 2019).
We query various properties of the model on tasks described below.
E d i t
D
i s t a n c e ( m a t
c h r a t i o )
CMU-VC
VESUS-NA
VESUS-NH
VESUS-NS
[Figure]
3. D ATA AND V OICE M ORPHING T ASKS
CMU-ARCTIC database has 4 American English speakers (two male, two female), who we paired according to their gender, for voice conversion.
This gives us a total of 2264 sentence pairs.
We train our duration modiﬁcation framework using 2164 utterances from the database and use the remaining 100 utterances (random 50-50 split) for validation and testing the open-loop modiﬁcation properties.
VESUS is an emotional speech corpus containing 250 phrases read by 10 speakers in 4 emotion classes: neutral, angry, happy, and sad.
Each utterance in VESUS corpus contains 10 crowd-sourced emotional annotation representing the saliency of intended emotion category.
Given the variation in quality, we use only utterances that are correctly annotated by at least half of the listeners.
Our task on VESUS is to inject emotion into a neutral/monotone utterance.
Thus, we train three duration models corresponding to the three neutral-emotional pairs, resulting in the following splits:
• Neutral to Angry : 2385 utterances for training, 72 for validation and, 61 for testing.
• Neutral to Happy : 2431 utterances for training, 43 for validation and, 43 for testing.
• Neutral to Sad : 2371 utterances for training, 75 for validation and, 63 for testing.
The utterances in VESUS corpus are short with an average signal duration ranging from 1 . 5 −seconds.
Given the small sample size due to shorter length sequences, we train the convolutional neural network on CMU-ARCTIC from scratch and ﬁne-tune it for each emotion conversion task.
3. L ENGTH P REDICTION
As a sanity check, we compare the predicted utterance length by our framework with that of the ground truth parallel utterance.
As seen in Fig.
3, the utterance length is predicted by summing the encoder embeddings across time and feeding the sum into a linear layer.
At a high level, the summation aggregates information across the entire utterance without the instabilities of a recurrent architecture.
Our baselines are two most commonly used sequence-to-sequence frameworks: (i)
Gated Recurrent Unit or GRU model (Cho et al., 2014), and (ii) Transformer model (Vaswani et al., 2017).
The architecture and training strategy of baseline models are described in more detail in the appendix section.
Fig.
4 shows the error in predicting the length ratio in a ms/sec format.
Notice that, our framework mispredicts the utterance lengths by only 40 ms/sec and 65 ms/sec (on average) on CMU-ARCTIC and VESUS, respectively.
Duration prediction is particularly challenging on
VESUS due to marked differences between neutral and emotional utterances.
The median prediction error for GRU model is in the range of 400 − 600 ms per second of the input utterance.
The
Transformer model fares relatively well in comparison to GRU because of their ability to establish long-range dependency.
However, our framework performs best in this challenging scenario, likely due to our multi-task setup and the fusion of deep representation with Bayesian regularization.
3. A TTENTION A LIGNMENT
Next, we compare the open-loop alignment between the source and target speech frames estimated via our attention map with the original DWT algorithm (where both utterances are known).
To
[Figure]
[Figure]
compare the warping paths, we code the horizontal, diagonal, and vertical moves of the backtracking procedure into three classes.
We then compute the edit distance between the DTW alignment and the attention map based alignment.
Once again, we compare our method against two popular sequenceto-sequence benchmarks.
Fig.
5 illustrates the match ratio normalized by the average length of sequences.
As seen, the match ratio varies between 0 .
70 and 0 . 85 , which suggests that our approach is good at learning the general characteristics of duration modiﬁcation.
The GRU model performs poorly in this task due to its inability to learn sequence transformations in the order of 100 s of frames.
The Transformer model does relatively well on this task as they can handle long sequences.
The convolutional model however, performs best because of the Itakura masking constraint and its ability to exploit the continuity of short-time Fourier representation of speech.
Furthermore, our method can be trained on limited data resources.
In contrast, sequence-to-sequence models for voice conversion require hundreds of hours of training data along with sophisticated noise removal models to generate actual speech.
Thus, our method can be used as a tool for manipulation of speaking rate at both, local and global scale.
To our knowledge, this is the ﬁrst demonstration of its kind.
Fig.
6 shows the effect of modifying the slope of Itakura parallelogram and the horizontal/vertical movement constraint imposed during the dynamic decoding stage.
As expected, relaxing the slope constraint and increasing the number of consecutive horizontal/vertical moves provide more ﬂexibility in adjusting to the speaking rate of generated speech.
However, this ﬂexibility can lead to
[Figure]
missing or distorted phonemes, suggesting a trade-off between changing the speaking rhythm and preserving naturalness.
We allow the users to tune these knobs for their own end application.
3. C OMPONENT - WISE D URATION A NALYSIS
To further probe into the behavior of our method, Fig.
7 compares the difference in duration between our converted utterances and the ground truth target utterances for vowels, consonants and short pauses (e.g., silence and unvoiced phonemes).
We use Penn phonetic forced alignment tool (Yuan & Liberman, 2008) to get the location and duration of each phoneme in a given utterance and group them into the above three categories.
The results indicate that our proposed method faithfully modiﬁes the duration of vowels and consonants.
However, it is less effective with short pauses, as seen across all four tasks.
This trend is intuitive, as we cannot create pauses if these frames do not exist in the source utterances.
One direction of future work is to add a branch to our deep network that estimates the duration of short pauses intermittently within the utterance.
Nonetheless, our model consistently estimates the difference between vowel and consonant duration across multiple tasks, which corroborates our claim of developing a general purpose speech rate manipulation framework.
3. S PEECH R ECONSTRUCTION Q UALITY
Finally, we use crowd sourcing to obtain a mean opinion score (MOS) for the re-synthesized speech quality of the testing utterances.
The crowd sourcing was performed using Amazon mechanical turk (AMT).
We collect 5 listener ratings for each converted utterance in the test set to leverage the sample averaging effect for a reliable estimate of MOS.
Further, we also add some noisy utterances to the converted samples set to ﬂag non-invested listeners and bots on AMT.
As seen in Fig.
8, our method achieves an average MOS between 3 . 7 − 4 .
across the four tasks.
This performance is on par with the speech quality produced by state-of-the-art neural vocoders such as Oord et al.
Shen et al.
(2017); Kalchbrenner et al.
We note that CMU-ARCTIC task has the lowest
MOS, perhaps due to the longer and more complex utterances.
Interestingly, the MOS is unaffected by errors in length prediction, as evidenced by the VESUS neutral-angry emotion conversion task.
This suggests that our approach of combining the neural network attention weights with a structured
DTW algorithm provides robustness to both the speech characteristics and estimation errors.
C ONCLUSIONS
We have presented a novel deep-generative framework for adaptive speech duration modiﬁcation.
Our model used a convolutional encoder-decoder architecture to estimate attention maps to associate frames of the input speech with frames of the target speech.
The attention maps are modeled as latent variables, which lead to a stochastic formulation of the mean absolute error (MAE) loss for model training.
During testing, the attention map is directly used to approximate the similarity matrix for a DTW-style backtracking procedure.
We evaluated our framework on a voice conversion and three separate emotion conversion tasks.
Overall, our framework produced similar duration modiﬁcation as the vanilla DTW, but without requiring access to the target utterance .
Further, we showed that the re-synthesized speech had similar naturalness to most state-of-the-art neural vocoders.
D ESCRIPTION OF T RANSFORMER MODEL
We also train a vanilla Transformer model from scratch for the voice conversion task and then ﬁnetune it for emotion conversion.
The Transformer model has 6 layers of encoder and decoder and each layer has 4 attention heads.
We use the same data augmentation procedure as our proposed technique and train it via Adam optimizer with default PyTorch settings.
Similar to the GRU model, there is no masking for the encoder-decoder attention module.
We use the sum of encoder embeddings across time-axis to predict the target length sequence.
Transformer models can handle longer inputs due to their non-sequential nature and ability to form long-range dependencies between input tokens.
Therefore, they naturally perform well on estimating the length of target sequence by learning appropriate encoder representation.
The Transformer model however, fails to leverage the continuity in the short-time representation of speech signal.
An additional constraint we implicitly imposed on the baselines and proposed neural network architecture is their ability to run on a single
NVIDIA K80 GPU.
We are able to ﬁt multiple layers of convolutional and Transformer model but only single layer of GRU on this GPU.
It allowed us to be more innovative with our approach.
A. A BLATION A NALYSIS : R EMOVING I TAKURA MASKING
There are multiple components in the proposed model which work in synchronization to produce naturally sounding speech.
In addition to the generative modeling part, the two most important augmentations we have made to the standard deep neural network pipeline are: (i) using Itakura
[Figure]
L e n g t h P r e d i c t i o n
E rr o r
( m s / s e c )
E d i t
D i s t a n c e ( m a t
c h r a t i o )
[Figure]
masking for attention scope and (ii) using an attention weighted residual connection in the ﬁnal layer.
Therefore, we perform ablation experiments to understand the relative signiﬁcance of each of these augmentations.
Our ﬁrst experiment removes masking from the attention layers.
Fig.shows the model’s performance on target length prediction and approximating the DTW similarity matrix.
The results in Fig.
11(a) indicate that the length prediction performance is roughly similar to the proposed model.
This is expected because, we hypothesized that it is the multi-task setup that allows the convolutional network to estimate length with a relatively small error (in ms/sec).
The match ratio metric however, (shown in Fig.
11(b)) is considerably worse.
Itakura masking procedure acts as a good inductive bias/prior on the attention map because the speech rate do not ﬂuctuate drastically in human conversations.
Therefore, the attention map is localized in our model.
L e n g t h P r e d i c t i o n
E rr o r
( m s / s e c )
E d i t
D i s t a n c e ( m a t
c h r a t i o ) (a) (b)
A. A BLATION A NALYSIS : R EMOVING R ESIDUAL CONNECTION
Our second ablation experiment involves removing the attention weighted residual connection from the ﬁnal layer of decoder.
Fig.
12(a) shows that the model is able to estimate the target sequence lengths with a relatively low error rate.
This is again due to multi-task training setup.
The match ratio (Fig. 12(b)) in this experiment is better than the no-masking results but, worse than the proposed model.
Therefore, we can conﬁdently say that Itakura masking helps in approximating DTW similarity matrix.
Further, the presence of residual connection is extremely important as it provides a good starting point for the convolutional network to start predicting the target frames.
Since the linguistic content of input and target utterances are same, the residual connection allows the neural network to inherit input speech properties which is helpful in auto-regressive generation mode.