Original: D EEP V OICE 3: 2000-S PEAKER N EURAL T EXT - TO -S PEECH
Revised: D EEP V OICE 3: 2000-S PEAKER N EURAL T EXT - TO -S PEECH

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present Deep Voice 3, a fully-convolutional attention-based neural textto-speech (TTS) system.
Revised: We present Deep Voice 3, a fully-convolutional attention-based neural textto-speech (TTS) system.

Original: Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training an order of magnitude faster.
Revised: Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training an order of magnitude faster.

Original: We scale Deep Voice 3 to dataset sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers.
Revised: We scale Deep Voice 3 to dataset sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers.

Original: In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods.
Revised: In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods.

Original: We also describe how to scale inference to ten million queries per day on a single GPU server
Revised: We also describe how to scale inference to ten million queries per day on a single GPU server.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text-to-speech (TTS) systems convert written language into human speech.
Revised: Text-to-speech (TTS) systems convert written language into human speech.

Original: TTS systems are used in a variety of applications, such as human-technology interfaces, accessibility for the visuallyimpaired, media and entertainment.
Revised: TTS systems are used in a variety of applications, such as human-technology interfaces, accessibility for the visuallyimpaired, media and entertainment.

Original: Traditional TTS systems are based on complex multi-stage hand-engineered pipelines (Taylor, 2009).
Revised: Traditional TTS systems are based on complex multi-stage hand-engineered pipelines (Taylor, 2009).

Original: Typically, these systems ﬁrst transform text into a compact audio representation, and then convert this representation into audio using an audio waveform synthesis model called a vocoder.
Revised: Typically, these systems ﬁrst transform text into a compact audio representation, and then convert this representation into audio using an audio waveform synthesis method called a vocoder.

Original: Recent work on neural TTS has demonstrated impressive results, yielding pipelines with simpler features, fewer components, and higher quality synthesized speech.
Revised: Recent work on neural TTS has demonstrated impressive results, yielding pipelines with simpler features, fewer components, and higher quality synthesized speech.

Original: There is not yet a consensus on the optimal neural network architecture for TTS.
Revised: There is not yet a consensus on the optimal neural network architecture for TTS.

Original: However, sequence-to-sequence models (Wang et al., 2017; Sotelo et al., 2017; Arık et al., 2017) have shown promising results.
Revised: However, sequence-to-sequence models (Wang et al., 2017; Sotelo et al., 2017; Arık et al., 2017) have shown promising results.

Original: In this paper, we propose a novel, fully-convolutional architecture for speech synthesis, scale it to very large audio data sets, and address several real-world issues that arise when attempting to deploy an attention-based TTS system.
Revised: In this paper, we propose a novel, fully-convolutional architecture for speech synthesis, scale it to very large audio data sets, and address several real-world issues that arise when attempting to deploy an attention-based TTS system.

Original: Speciﬁcally, we make the following contributions:
Revised: Speciﬁcally, we make the following contributions:

Original: We propose a fully-convolutional character-to-spectrogram architecture, which enables fully parallel computation and trains an order of magnitude faster than analogous architectures using recurrent cells (e.g., Wang et al., 2017).
Revised: We propose a fully-convolutional character-to-spectrogram architecture, which enables fully parallel computation and trains an order of magnitude faster than analogous architectures using recurrent cells (e.g., Wang et al., 2017).

Original: We show that our architecture trains quickly and scales to the LibriSpeech ASR dataset (Panayotov et al., 2015), which consists of 820 hours of audio data from 2484 speakers.
Revised: We show that our architecture trains quickly and scales to the LibriSpeech ASR dataset (Panayotov et al., 2015), which consists of 820 hours of audio data from 2484 speakers.

Original: We demonstrate that we can generate monotonic attention behavior, avoiding error modes commonly affecting sequence-to-sequence models.
Revised: We demonstrate that we can generate monotonic attention behavior, avoiding error modes commonly affecting sequence-to-sequence models.

Original: We compare the quality of several waveform synthesis methods, including WORLD (Morise et al., 2016),
Revised: We compare the quality of several waveform synthesis methods, including WORLD (Morise et al., 2016),

Original: Grifﬁn-Lim (Grifﬁn & Lim, 1984), and WaveNet (Oord et al., 2016).
Revised: Grifﬁn-Lim (Grifﬁn & Lim, 1984), and WaveNet (Oord et al., 2016).

Original: We describe the implementation of an inference kernel for Deep Voice 3, which can serve up to ten million queries per day on one single-GPU server.
Revised: We describe the implementation of an inference kernel for Deep Voice 3, which can serve up to ten million queries per day on one single-GPU server.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work builds upon the state-of-the-art in neural speech synthesis and attention-based sequenceto-sequence learning.
Revised: Our work builds upon the state-of-the-art in neural speech synthesis and attention-based sequenceto-sequence learning.

Original: Several recent works tackle the problem of synthesizing speech with neural networks, including
Revised: Several recent works tackle the problem of synthesizing speech with neural networks, including

Original: Deep Voice 1 (Arık et al., 2017), Deep Voice 2 (Arık et al., 2017), Tacotron (Wang et al., 2017),
Revised: Deep Voice 1 (Arık et al., 2017), Deep Voice 2 (Arık et al., 2017), Tacotron (Wang et al., 2017),

Original: Char2Wav (Sotelo et al., 2017), VoiceLoop (Taigman et al., 2017), SampleRNN (Mehri et al., 2017), and WaveNet (Oord et al., 2016).
Revised: Char2Wav (Sotelo et al., 2017), VoiceLoop (Taigman et al., 2017), SampleRNN (Mehri et al., 2017), and WaveNet (Oord et al., 2016).

Original: Deep Voice 1 & 2 retain the traditional structure of
Revised: Deep Voice 1 & 2 retain the traditional structure of

Original: TTS pipelines, separating grapheme-to-phoneme conversion, duration and frequency prediction, and waveform synthesis.
Revised: TTS pipelines, separating grapheme-to-phoneme conversion, duration and frequency prediction, and waveform synthesis.

Original: In contrast to Deep Voice 1 & 2, Deep Voice 3 employs an attentionbased sequence-to-sequence model, yielding a more compact architecture.
Revised: In contrast to Deep Voice 1 & 2, Deep Voice 3 employs an attentionbased sequence-to-sequence model, yielding a more compact architecture.

Original: Similar to Deep Voice 3, Tacotron and Char2Wav propose sequence-to-sequence models for neural TTS.
Revised: Similar to Deep Voice 3, Tacotron and Char2Wav propose sequence-to-sequence models for neural TTS.

Original: Tacotron is a neural text-to-spectrogram conversion model, used with Grifﬁn-Lim for spectrogram-to-waveform synthesis.
Revised: Tacotron is a neural text-to-spectrogram conversion model, used with Grifﬁn-Lim for spectrogram-to-waveform synthesis.

Original: Char2Wav predicts the parameters of the WORLD vocoder (Morise et al., 2016) and uses a SampleRNN conditioned upon WORLD parameters for waveform generation.
Revised: Char2Wav predicts the parameters of the WORLD vocoder (Morise et al., 2016) and uses a SampleRNN conditioned upon WORLD parameters for waveform generation.

Original: In contrast to
Revised: In contrast to

Original: Char2Wav and Tacotron, Deep Voice 3 avoids Recurrent Neural Networks (RNNs)1 to speed up training.
Revised: Char2Wav and Tacotron, Deep Voice 3 avoids Recurrent Neural Networks (RNNs) to speed up training.

Original:  Deep Voice 3 makes attention-based TTS feasible for a production TTS system with no compromise on accuracy.
Revised: 1 Deep Voice 3 makes attention-based TTS feasible for a production TTS system with no compromise on accuracy by avoiding common attention errors.

Original: Finally, WaveNet and SampleRNN are neural vocoder models for waveform synthesis.
Revised: Finally, WaveNet and SampleRNN are neural vocoder models for waveform synthesis.

Original: There are also numerous alternatives for high-quality hand-engineered vocoders in the literature, such as STRAIGHT (Kawahara et al., 1999), Vocaine (Agiomyrgiannakis, 2015), and WORLD (Morise et al., 2016).
Revised: There are also numerous alternatives for highquality hand-engineered vocoders in the literature, such as STRAIGHT (Kawahara et al., 1999), Vocaine (Agiomyrgiannakis, 2015), and WORLD (Morise et al., 2016).

Original: Deep Voice 3 adds no novel vocoder, but has the potential to be integrated with different waveform synthesis methods with slight modiﬁcations of its architecture.
Revised: Deep Voice 3 adds no novel vocoder, but has the potential to be integrated with different waveform synthesis methods with slight modiﬁcations of its architecture.

Original: Automatic speech recognition (ASR) datasets are often much larger than traditional TTS corpora but tend to be less clean, as they typically involve multiple microphones and background noise.
Revised: Automatic speech recognition (ASR) datasets are often much larger than traditional TTS corpora but tend to be less clean, as they typically involve multiple microphones and background noise.

Original: Although prior work has applied TTS methods to ASR datasets (Yamagishi et al., 2010), Deep
Revised: Although prior work has applied TTS methods to ASR datasets (Yamagishi et al., 2010), Deep

Original: Voice 3 is, to the best of our knowledge, the ﬁrst TTS system to scale to thousands of speakers with a single model.
Revised: Voice 3 is, to the best of our knowledge, the ﬁrst TTS system to scale to thousands of speakers with a single model.

Original: Sequence-to-sequence models (Sutskever et al., 2014; Cho et al., 2014) encode a variable-length input into hidden states, which are then processed by a decoder to produce a target sequence.
Revised: Sequence-to-sequence models (Sutskever et al., 2014; Cho et al., 2014) encode a variable-length input into hidden states, which are then processed by a decoder to produce a target sequence.

Original: An attention mechanism allows a decoder to adaptively select encoder hidden states to focus on while generating the target sequence (Bahdanau et al., 2015).
Revised: An attention mechanism allows a decoder to adaptively select encoder hidden states to focus on while generating the target sequence (Bahdanau et al., 2015).

Original: Attention-based sequence-to-sequence models are widely applied in machine translation (Bahdanau et al., 2015), speech recognition (Chorowski et al., 2015), and text summarization (Rush et al., 2015).
Revised: Attention-based sequence-to-sequence models are widely applied in machine translation (Bahdanau et al., 2015), speech recognition (Chorowski et al., 2015), and text summarization (Rush et al., 2015).

Original: Recent improvements in attention mechanisms relevant to Deep Voice 3 include enforced-monotonic attention during training (Raffel et al., 2017), fully-attentional non-recurrent architectures (Vaswani et al., 2017), and convolutional sequenceto-sequence models (Gehring et al., 2017).
Revised: Recent improvements in attention mechanisms relevant to Deep Voice 3 include enforced-monotonic attention during training (Raffel et al., 2017), fully-attentional non-recurrent architectures (Vaswani et al., 2017), and convolutional sequenceto-sequence models (Gehring et al., 2017).

Original: Deep Voice 3 demonstrates the utility of monotonic attention during training in TTS, a new domain where monotonicity is expected.
Revised: Deep Voice 3 demonstrates the utility of monotonic attention during training in TTS, a new domain where monotonicity is expected.

Original: Alternatively, we show that with a simple heuristic to only enforce monotonicity during inference, a standard attention mechanism can work just as well or even better.
Revised: Alternatively, we show that with a simple heuristic to only enforce monotonicity during inference, a standard attention mechanism can work just as well or even better.

Original: Deep Voice 3 also builds upon the convolutional sequence-to-sequence architecture from Gehring et al.
Revised: Deep Voice 3 also builds upon the convolutional sequence-to-sequence architecture from Gehring et al.

Original: by introducing a positional encoding similar to that used in Vaswani et al.
Revised: by introducing a positional encoding similar to that used in Vaswani et al.

Original: (2017), augmented with a rate adjustment to account for the mismatch between input and output domain lengths.
Revised: (2017), augmented with a rate adjustment to account for the mismatch between input and output domain lengths.

Original:  M ODEL A RCHITECTURE
Revised:  M ODEL A RCHITECTURE

Original: In this section, we present our fully-convolutional sequence-to-sequence architecture for TTS (see
Revised: In this section, we present our fully-convolutional sequence-to-sequence architecture for TTS (see

Original: Fig. 1).
Revised: Fig. 1).

Original: Our architecture is capable of converting a variety of textual features (e.g. characters, phonemes, stresses) into a variety of vocoder parameters, e.g. mel-band spectrograms, linear-scale log magnitude spectrograms, fundamental frequency, spectral envelope, and aperiodicity parameters.
Revised: Our architecture is capable of converting a variety of textual features (e.g. characters, phonemes, stresses) into a variety of vocoder parameters, e.g. mel-band spectrograms, linear-scale log magnitude spectrograms, fundamental frequency, spectral envelope, and aperiodicity parameters.

Original: These vocoder parameters can be used as inputs for audio waveform synthesis models.
Revised: These vocoder parameters can be used as inputs for audio waveform synthesis models.

Original: The Deep Voice 3 architecture consists of three components:
Revised: The Deep Voice 3 architecture consists of three components:

Original: • Encoder :
Revised: • Encoder :

Original: A fully-convolutional encoder, which converts textual features to an internal learned representation.
Revised: A fully-convolutional encoder, which converts textual features to an internal learned representation.

Original: Decoder : A fully-convolutional causal decoder, which decodes the learned representation with a multi-hop convolutional attention mechanism into a low-dimensional audio representation (mel-band spectrograms) in an autoregressive manner.

Original: • Converter : A fully-convolutional post-processing network, which predicts ﬁnal output features (depending on the vocoder choice) from the decoder hidden states.

Original: Unlike the decoder, the converter is non-causal and can thus depend on future context information.

Original: The overall objective function to be optimized is a linear combination of the losses from the decoder (Section 3.5) and the converter (Section 3.7).

Original: Done + MelInput Attention Block x N Griffin-Lim
Revised: [Figure]

Original: Wave

Original: Converter

Original: WORLD
Revised: WORLD

Original: WaveNet

Original: Speaker-Embedding x N Convolution Block(Causal) TextEmbedding Convolution Block
Revised: [Figure]

Original: Decoder PreNet Encoder PreNet Encoder PostNet Encoder ( key, value ) query
Revised: query

Original: Converter x N
Revised: Converter x N

Original: FC
Revised: FC

Original: Wave Wave
Revised: Wave Wave

Original: MelOutput
Revised: MelOutput

Original: FC
Revised: FC

Original: Decoder "
Revised: Decoder "

Revised: • Decoder : A fully-convolutional causal decoder, which decodes the learned representation with a multi-hop convolutional attention mechanism into a low-dimensional audio representation (mel-scale spectrograms) in an autoregressive manner.

Revised: • Converter : A fully-convolutional post-processing network, which predicts ﬁnal vocoder parameters (depending on the vocoder choice) from the decoder hidden states.

Revised: Unlike the decoder, the converter is non-causal and can thus depend on future context information.

Original:  In multi-speaker scenario, trainable speaker embeddings as in Arık et al.
Revised: The overall objective function to be optimized is a linear combination of the losses from the decoder (Section 3.5) and the converter (Section 3.7). We separate decoder and converter and apply multi-task training, because it makes attention learning easier in practice. To be speciﬁc, the loss for mel-spectrogram prediction guides training of the attention mechanism, because the attention is trained with the gradients from mel-spectrogram prediction besides vocoder parameter prediction. In multi-speaker scenario, trainable speaker embeddings as in Arık et al.

Original: (2017) are used across encoder, decoder and converter.
Revised: (2017) are used across encoder, decoder and converter.

Original: Next, we describe each of these components and the data preprocessing in detail.
Revised: Next, we describe each of these components and the data preprocessing in detail.

Original: Model hyperparameters are available in Table 4 within Appendix C.
Revised: Model hyperparameters are available in Table 4 within Appendix C.

Original: 3. T EXT P REPROCESSING
Revised: 3. T EXT P REPROCESSING

Original: Text preprocessing is crucial for good performance.
Revised: Text preprocessing is crucial for good performance.

Original: Feeding raw text (characters with spacing and punctuation) yields acceptable performance on many utterances.
Revised: Feeding raw text (characters with spacing and punctuation) yields acceptable performance on many utterances.

Original: However, some utterances may have mispronunciations of rare words, or may yield skipped words and repeated words.
Revised: However, some utterances may have mispronunciations of rare words, or may yield skipped words and repeated words.

Original: We alleviate these issues by normalizing the input text as follows:
Revised: We alleviate these issues by normalizing the input text as follows:

Original: We uppercase all characters in the input text.
Revised: We uppercase all characters in the input text.

Original: We remove all intermediate punctuation marks.
Revised: We remove all intermediate punctuation marks.

Original: We end every utterance with a period or question mark.
Revised: We end every utterance with a period or question mark.

Original: We replace spaces between words with special separator characters which indicate the duration of pauses inserted by the speaker between words.
Revised: We replace spaces between words with special separator characters which indicate the duration of pauses inserted by the speaker between words.

Original: We use four different word separators, indicating (i) slurred-together words, (ii) standard pronunciation and space characters, (iii) a short pause between words, and (iv) a long pause between words.
Revised: We use four different word separators, indicating (i) slurred-together words, (ii) standard pronunciation and space characters, (iii) a short pause between words, and (iv) a long pause between words.

Original: For example, the sentence “Either way, you should shoot very slowly,” with a long pause after “way” and a short pause after “shoot”, would be written as “Either way%you should shoot/very slowly%.”
Revised: For example, the sentence “Either way, you should shoot very slowly,” with a long pause after “way” and a short pause after “shoot”, would be written as “Either way%you should shoot/very slowly%.”

Original: with % representing a long pause and / representing a short pause for encoding convenience.
Revised: with % representing a long pause and / representing a short pause for encoding convenience.

Original: 3. J OINT R EPRESENTATION OF C HARACTERS AND P HONEMES
Revised: 3. J OINT R EPRESENTATION OF C HARACTERS AND P HONEMES

Original: Deployed TTS systems (e.g., Capes et al., 2017; Gonzalvo et al., 2016) should include a way to modify pronunciations to correct common mistakes (which typically involve proper nouns, foreign words, and domain-speciﬁc jargon).
Revised: Deployed TTS systems (e.g., Capes et al., 2017; Gonzalvo et al., 2016) should include a way to modify pronunciations to correct common mistakes (which typically involve proper nouns, foreign words, and domain-speciﬁc jargon).

Original: A conventional way to do this is to maintain a dictionary to map words to their phonetic representations.
Revised: A conventional way to do this is to maintain a dictionary to map words to their phonetic representations.

Original: Our model can directly convert characters (including punctuation and spacing) to acoustic features, and hence learns an implicit grapheme-to-phoneme model.
Revised: Our model can directly convert characters (including punctuation and spacing) to acoustic features, and hence learns an implicit grapheme-to-phoneme model.

Original: This implicit conversion is difﬁcult to correct when the model makes mistakes.
Revised: This implicit conversion is difﬁcult to correct when the model makes mistakes.

Original: Thus, in addition to character models, we also train phoneme-only models and mixed character-and-phoneme models by allowing phoneme input option explicitly.
Revised: Thus, in addition to character models, we also train phoneme-only models and mixed character-and-phoneme models by allowing phoneme input option explicitly.

Original: These models are identical to character-only models, except that the input layer of the encoder sometimes receives  embeddingsfor the phonemes instead of characters.
Revised: These models are identical to character-only models, except that the input layer of the encoder sometimes receives phoneme and phoneme stress embeddings instead of character embeddings.

Original: A phoneme-only model requires a preprocessing step to convert words to their phoneme representations (by using an external phoneme dictionary or a separately trained grapheme-to-phoneme model) 3 .
Revised: A phoneme-only model requires a preprocessing step to convert words to their phoneme representations (by using an external phoneme dictionary or a separately trained grapheme-to-phoneme model) 3 .

Original: A mixed character-and-phoneme model requires a similar preprocessing step, except for words not in the phoneme dictionary.
Revised: A mixed character-and-phoneme model requires a similar preprocessing step, except for words not in the phoneme dictionary.

Original: These out-of-vocabulary words are input as characters, allowing the model to use its implicitly learned grapheme-to-phoneme model.
Revised: These out-of-vocabulary words are input as characters, allowing the model to use its implicitly learned grapheme-to-phoneme model.

Original: While training a mixed character-and-phoneme model, every word is replaced with its phoneme representation with some ﬁxed probability at each training iteration.
Revised: While training a mixed character-and-phoneme model, every word is replaced with its phoneme representation with some ﬁxed probability at each training iteration.

Original: We ﬁnd that this improves pronunciation accuracy and minimizes attention errors, especially when generalizing to utterances longer than those seen during training.
Revised: We ﬁnd that this improves pronunciation accuracy and minimizes attention errors, especially when generalizing to utterances longer than those seen during training.

Original: More importantly, models that support phoneme representation allow correcting mispronunciations using a phoneme dictionary, a desirable feature of deployed systems.
Revised: More importantly, models that support phoneme representation allow correcting mispronunciations using a phoneme dictionary, a desirable feature of deployed systems.

Original: 3. C ONVOLUTION B LOCKS FOR S EQUENTIAL P ROCESSING
Revised: 3. C ONVOLUTION B LOCKS FOR S EQUENTIAL P ROCESSING

Original: Decoder
Revised: Decoder

Original: Attention Block
Revised: Attention Block

Original: Converter …
Revised: Converter …

Original: Conv Block
Revised: Conv [Table]

Original: Encoder
Revised: Encoder

Revised: Text Embeddings

Original: FC Conv Block Conv Block + values keys ⨉ p 0 .
Revised: FC Conv Block Conv Block + values keys ⨉ p 0 .

Original: Mel input
Revised: Mel input

Original: Attention Block
Revised: Attention Block

Original: Conv Block …
Revised: Conv Block …

Original: Linear Spectrogram
Revised: Linear Spectrogram

Original: WORLD features

Original: Input conv split " ⨉ + ⨉
Revised: Input conv split " ⨉ + ⨉

Original: Output c 2c c c p 0 .
Revised: Output c 2c c c p 0 .

Revised: FC

Revised: ReLu dropout ⨉ N prenet ・・・

Revised: FC

Original: ReLu dropout
Revised: ReLu dropout

Original: ReLu

Original: FC … …
Revised: FC … …

Original: Conv Block FC
Revised: Conv Block FC

Original: FC Done"
Revised: FC Done

Original: Mel Output

Original: WORLD Block
Revised: WORLD Block

Original: FC "
Revised: FC "

Original: F
Revised: F

Original: FC
Revised: FC

Original: Voiced
Revised: Voiced

Original: FC
Revised: FC

Original: Spectral Envelope
Revised: Spectral Envelope

Original: Aperiodicity
Revised: Aperiodicity

Original: FC
Revised: FC

Original: Input
Revised: Input

Original: Conv Block upsample by repetition WORLD Block values keys query
Revised: Conv Block upsample by repetition WORLD Block values keys query

Original: Positional Encoding
Revised: Positional Encoding

Original: Positional Encoding + +
Revised: Positional Encoding + +

Original: FC FC
Revised: FC FC

Original: FC • inference mask (optional) •
Revised: FC • inference mask (optional) •

Original: Output softmax OR monotonic attention dropout
Revised: Output softmax OR monotonic attention dropout

Original: FC / # of timesteps sqrt
Revised: FC / # of timesteps sqrt

Original: FC dropout
Revised: FC dropout

Original: FC
Revised: FC

Original: Speaker Embedding
Revised: Speaker Embedding

Original: FC softsign + + FC softsign 
Revised: FC softsign + + FC softsign + + FC softsign

Original: Causal Conv Block ⨉ N decoder + p 0 .⨉ + ⨉ ・・・
Revised: Causal Conv Block ⨉ N decoder + p 0 .⨉ + ⨉ ・・・

Original: Speaker Embedding
Revised: Speaker Embedding

Original: FC softsign +
Revised: FC softsign +

Original: Speaker Embedding
Revised: Speaker Embedding

Original: FC sigmoid ⨉ ω query (can be a constant)raise topower grif?in lim
Revised: FC sigmoid ⨉ ω query (can be a constant) to

Original: AudioWORLD Synthesis
Revised: Audio

Original: FC sigmoid ⨉ ω initialω key (can be a constant)
Revised: FC sigmoid ⨉ ω initialω key (can be a constant)

Original: By providing a sufﬁciently large receptive ﬁeld, stacked convolutional layers can utilize long-term context information in sequences without introducing any sequential dependency in computation.
Revised: By providing a sufﬁciently large receptive ﬁeld, stacked convolutional layers can utilize long-term context information in sequences without introducing any sequential dependency in computation.

Original: We use the convolution block depicted in Fig.
Revised: We use the convolution block depicted in Fig.

Original: 2 as the main sequential processing unit to encode hidden representations of text and audio.
Revised: 2 as the main sequential processing unit to encode hidden representations of text and audio.

Original: The convolution block consists of a 1-D convolution ﬁlter, a gated-linear unit as a learnable nonlinearity (Dauphin et al., 2017), a residual connection to the input, and a scaling factor of √ 0 .
Revised: The convolution block consists of a 1-D convolution ﬁlter, a gated-linear unit as a learnable nonlinearity (Dauphin et al., 2017), a residual connection to the input, and a scaling factor of √ 0 .

Original: The gated linear unit provides a linear path for the gradient ﬂow, which alleviates the vanishing gradient issue for stacked convolution blocks while retaining non-linearity.
Revised: The gated linear unit provides a linear path for the gradient ﬂow, which alleviates the vanishing gradient issue for stacked convolution blocks while retaining non-linearity.

Original: To introduce speaker-dependent control, a speaker-dependent embedding is added as a bias to the convolution ﬁlter output, after a softsign function.
Revised: To introduce speaker-dependent control, a speaker-dependent embedding is added as a bias to the convolution ﬁlter output, after a softsign function.

Original: As in Arık et al.

Original: (2017), we use the softsign nonlinearity because it limits the range of the output while also avoiding the saturation problem that exponential-based nonlinearities sometimes exhibit.
Revised: We use the softsign nonlinearity because it limits the range of the output while also avoiding the saturation problem that exponentialbased nonlinearities sometimes exhibit.

Original: As in Gehring et al.

Original: (2017), we initialize the convolution ﬁlter weights with zero-mean and unit-variance activations throughout the entire network.
Revised: We initialize the convolution ﬁlter weights with zero-mean and unit-variance activations throughout the entire network.

Original: The convolutions in the architecture can be either non-causal (e.g. in text encoder) or causal (e.g. inaudio decoder).
Revised: The convolutions in the architecture can be either non-causal (e.g. in encoder and converter) or causal (e.g. in decoder).

Original: To preserve the sequence length, inputs are padded with k − 1 timesteps of zeros on the left for causal convolutions and ( 
Revised: To preserve the sequence length, inputs are padded with k − 1 timesteps of zeros on the left for causal convolutions and ( k − 1) / 2 timesteps of zeros on the left and on the right for non-causal convolutions, where k is an odd convolution ﬁlter width. 5 Dropout is applied to the inputs prior to the convolution for regularization.

Original: 3. E NCODER
Revised: 3. E NCODER

Original: The encoder network (depicted in Fig. 1) begins with an embedding layer, which converts characters or phonemes into trainable vector representations, h e .
Revised: The encoder network (depicted in Fig. 1) begins with an embedding layer, which converts characters or phonemes into trainable vector representations, h e .

Original: These embeddings h e are ﬁrst projected via a fully-connected layer from the embedding dimension to a target dimensionality.
Revised: These embeddings h e are ﬁrst projected via a fully-connected layer from the embedding dimension to a target dimensionality.

Original: Then, they are processed through a series of convolution blocks described in Section 3.3 to extract time-dependent text information.
Revised: Then, they are processed through a series of convolution blocks described in Section 3.3 to extract time-dependent text information.

Original: Lastly, they are projected back to the embedding dimension to create the attention key vectors h k .
Revised: Lastly, they are projected back to the embedding dimension to create the attention key vectors h k .

Original: The attention value vectors are computed from attention key vectors and text embeddings, h v = √ 0 .
Revised: The attention value vectors are computed from attention key vectors and text embeddings, h v = √ 0 .

Original: 5( h k + h e ) ,as in Gehring et al. (2017), to jointly consider the local information in h e and the long-term context information in h k .
Revised: 5( h k + h e ) , to jointly consider the local information in h e and the long-term context information in h k .

Original: The key vectors h k are used by each attention block to compute attention weights, whereas the ﬁnal context vector is computed as a weighted average over the value vectors h v (see Section 3.6).
Revised: The key vectors h k are used by each attention block to compute attention weights, whereas the ﬁnal context vector is computed as a weighted average over the value vectors h v (see Section 3.6).

Original: 3. D ECODER
Revised: 3. D ECODER

Original: The decoder (depicted in Fig. 1) generates audio in an autoregressive manner by predicting a group of r future audio frames conditioned on the past audio frames.
Revised: The decoder (depicted in Fig. 1) generates audio in an autoregressive manner by predicting a group of r future audio frames conditioned on the past audio frames.

Original: Since the decoder is autoregressive, it must use causal convolution blocks (described in Section 3.3).
Revised: Since the decoder is autoregressive, it must use causal convolution blocks.

Original: We choose mel-band log-magnitude spectrogram as the compact low-dimensional audio frame representation.
Revised: We choose mel-band log-magnitude spectrogram as the compact low-dimensional audio frame representation.

Original: We empirically observed that decoding multiple frames together (i.e. having r > 1 ) yields better audio quality, similar to Wang et al.
Revised: Similar to Wang et al. , we empirically observed that decoding multiple frames together (i.e. having r > 1 ) yields better audio quality.

Original: The decoder network starts with multiple fully-connected layers with rectiﬁed linear unit (ReLU) nonlinearities to preprocess input mel-spectrograms (denoted as “PreNet” in Fig. 1).
Revised: The decoder network starts with multiple fully-connected layers with rectiﬁed linear unit (ReLU) nonlinearities to preprocess input mel-spectrograms (denoted as “PreNet” in Fig. 1).

Original: Then, it is followed by a series of causal convolution and attention blocks (described in Section 3.6).
Revised: Then, it is followed by a series of causal convolution and attention blocks.

Original: These convolution blocks generate the queries used to attend over the encoder’s text-based outputs.
Revised: These convolution blocks generate the queries used to attend over the encoder’s hidden states (see Section 3.6).

Original: Lastly,  fully-connected layers output the next group of r audio frames and also a binary “ﬁnal frame” predictionparameter (indicating whether the last frame of the utterance has been synthesized).
Revised: Lastly, a fully-connected layer output the next group of r audio frames and also a binary “ﬁnal frame” prediction (indicating whether the last frame of the utterance has been synthesized).

Original: Dropout is applied before each fully-connected layer prior to the attention blocks, except for the ﬁrst one.
Revised: Dropout is applied before each fully-connected layer prior to the attention blocks, except for the ﬁrst one.

Original: An L1 loss 6 is computed using the output mel-spectrograms and a binary cross-entropy loss is computed using the ﬁnal-frame prediction.
Revised: An L1 loss 6 is computed using the output mel-spectrograms and a binary cross-entropy loss is computed using the ﬁnal-frame prediction.

Original: 3. A TTENTION B LOCK
Revised: 3. A TTENTION B LOCK

Original: Decoder
Revised: We use a dot-product attention mechanism (depicted in Fig. 3) similar to Vaswani et al. The attention mechanism uses a query vector (the hidden states of the decoder) and the per-timestep key vectors from the encoder to compute attention weights, and then outputs a context vector computed as the weighted average of the value vectors. We observe empirical beneﬁts from introducing a inductive bias where the attention follows a monotonic progression in time. Thus, we add a positional encoding to both the key and the query vectors.

Original: Attention Block
Revised: These positional encodings h p are chosen as h p ( i ) = sin ( ω s i / 10000 k/d ) (for even i ) or cos ( ω s i / 10000 k/d ) (for odd i ), where i is the timestep index, k is the channel index in the positional encoding, d is the total number of channels in the positional encoding, and ω s is the position rate of the encoding. The position rate dictates the average slope of the line in the attention distribution, roughly corresponding to speed of speech.

Original: Converter …
Revised: For a single speaker, ω s is set to one for the query, and odd convolution widths the

Original: Conv Block
Revised: We choose L1 loss since it yields the best result empirically. Other loss such as L2 may suffer from outlier spectral features, which may correspond to non-speech noise.

Original: Encoder
Revised: Decoder

Original: Text Embeddings

Original: FC Conv Block Conv Block +
Revised: Attention Block Converter …

Original: [Figure]
Revised: [Figure]

Original:  keys [Figure]
Revised: Encoder Text Embeddings FC Conv Block Conv Block + values keys ⨉ p 0 .

Original: Mel input Attention Block
Revised: Mel input Attention Block

Original: Conv Block …
Revised: Conv Block …

Original: Linear Spectrogram
Revised: Linear Spectrogram

Original: WORLD features
Revised: WORLD features

Original: Input conv split " ⨉ + ⨉
Revised: Input conv split " ⨉ + ⨉

Original: Output c 2c c cp 0 .
Revised: Output c 2c c c

Revised: [Figure]

Original: FC ReLu dropout ⨉ N prenet ・・・
Revised: FC ReLu dropout ⨉ N prenet ・・・

Original: FC ReLu dropout
Revised: FC ReLu dropout

Original: FC … …
Revised: FC … …

Original: Conv Block FC
Revised: Conv Block FC

Original: FC Done "
Revised: FC Done "

Original: Mel Output
Revised: Mel Output

Original: WORLD Block
Revised: WORLD Block

Original: FC " F
Revised: FC " F

Original: FC
Revised: FC

Original: Voiced
Revised: Voiced

Original: FC
Revised: FC

Original: Spectral Envelope
Revised: Spectral Envelope

Original: Aperiodicity
Revised: Aperiodicity

Original: FC
Revised: FC

Original: Input
Revised: Input

Original: Conv Block upsample by repetition WORLD Blockvalues keys query
Revised: Conv Block upsample by repetition WORLD Block

Original: Positional Encoding Positional Encoding + +
Revised: [Figure]

Original: FCFC
Revised: FC

Original: FC • inference mask (optional) • Output softmax OR monotonic attention dropout FC / # of timesteps sqrt
Revised: [Figure]

Original: FCdropout FC Speaker Embedding 
Revised: FC Speaker Embedding FC softsign

Original: FC softsign + +
Revised: + +

Original: FC softsign + + FC softsign
Revised: FC softsign + + FC softsign

Original: Causal Conv Block ⨉ N decoder + p 0 .⨉ + ⨉ ・・・
Revised: Causal Conv Block ⨉ N decoder + p 0 .⨉ + ⨉ ・・・

Original: Speaker Embedding
Revised: [Figure]

Original: FC softsign + 
Revised: + [Figure]

Original: Speaker Embedding FC " ⨉ ω query (can be a constant) raise to power grif?in lim
Revised: raise to power grif?in lim

Original: WORLD Synthesis
Revised: WORLD Synthesis

Original: FC " ⨉ ω initial ω key (can be a constant)

Original: [Figure]
Revised: [Figure]

Original: applied at inference by adding a mask of large negative values to the logits. One of two possible attention schemes is used: softmax or monotonic attention from Raffel et al. During training, attention weights are dropped out.
Revised: WaveNet Audio Audio ⨉ ⨉ FC " ・・・

Original: [Figure]
Revised: [Figure]

Original: The functionality of the attention mechanism is to relate speciﬁc portions of the input text to speciﬁc portions of the output audio.

Original: We focus on a dot-product attention mechanism (depicted in Fig. 3), similar to Vaswani et al.

Original: The attention mechanism uses a query vector (the hidden state of the decoder) and the per-timestep key vectors from the encoder to compute attention weights, and then outputs a context vector computed from the weighted average of the value vectors.

Original: The dot product is maximized when key and query vectors are aligned, which should occur when the pronounced audio frame corresponds to the underlying textual feature. Ideally, encoder and decoder architectures should learn to generate the key and query vectors from scratch. Yet, in many cases, we observe beneﬁts from introducing a favorable inductive bias where the attention distribution follows a monotonic function in time. Thus, we add a positional encoding to both the key and the query vectors.
Revised: of one or two dominant layers is sufﬁcient for high-quality output.)

Original: These positional encodings h p are chosen as h p ( i ) = sin ( ω s i / 10000 k/d ) (for even i ) or cos ( ω s i / 10000 k/d ) (for odd i ), where i is the timestep index, k is the channel index in the positional encoding, d is the total number of channels in the positional encoding, and ω s is the position rate of the encoding. The position rate dictates the average slope of the line in the attention distribution, roughly corresponding to speed of speech. For a single speaker, ω s is set to one for the decoder and ﬁxed for the encoder to the ratio of output timesteps to input timesteps (computed across the entire dataset).
Revised: be ﬁxed for the key to the ratio of output timesteps to input timesteps (computed across the entire dataset).

Original: For multi-speaker datasets, ω s is computed for both the encoder and decoder from the speaker embedding for each speaker (depicted in Fig. 3).
Revised: For multi-speaker datasets, ω s is computed for both the key and query from the speaker embedding for each speaker (depicted in Fig. 3).

Original: As sine and cosine functions form an orthonormal basis, this initialization yields an attention distribution in the form of a diagonal line (see Fig.
Revised: As sine and cosine functions form an orthonormal basis, this initialization yields an attention distribution in the form of a diagonal line (see Fig.

Original: 4 (a)).
Revised: 4 (a)).

Original: We initialize the fully-connected layer weights used to compute hidden attention vectors to the same values for the query projection and the key projection.
Revised: We initialize the fully-connected layer weights used to compute hidden attention vectors to the same values for the query projection and the key projection.

Original: Positional encodings are used in all attention blocks.
Revised: Positional encodings are used in all attention blocks.

Original: We use context normalization as in Gehring et al.
Revised: We use context normalization as in Gehring et al.

Original: A fully-connected layer is applied to the context vector to generate the output of the attention block.
Revised: A fully-connected layer is applied to the context vector to generate the output of the attention block.

Original: Overall, positional encodings improvequality of the convolutional attention mechanism.
Revised: Overall, positional encodings improve the convolutional attention mechanism.

Original: Production-quality TTS systems have very low tolerance for attention errors.
Revised: Production-quality TTS systems have very low tolerance for attention errors.

Original: Hence, besides positional encodings, we consider additional strategies to minimize the cases of repeating or skipping words.
Revised: Hence, besides positional encodings, we consider additional strategies to eliminate the cases of repeating or skipping words.

Original: One approach is to substitute the canonical attention mechanism with the monotonic attention mechanism introduced in Raffel et al.
Revised: One approach is to substitute the canonical attention mechanism with the monotonic attention mechanism introduced in Raffel et al.

Original: (2017), which approximates hard-monotonic stochastic decoding with soft-monotonic attention by training in expectation.
Revised: (2017), which approximates hard-monotonic stochastic decoding with soft-monotonic attention by training in expectation.

Original: Despite the improved monotonicity, this strategy may yield a more diffused attention distribution.
Revised: Despite the improved monotonicity, this strategy may yield a more diffused attention distribution.

Original: In some cases, several characters are attended at the same time and high quality speech couldn’t be obtained.
Revised: In some cases, several char- acters are attended at the same time and high quality speech couldn’t be obtained.

Original: We attribute this to the unnormalized attention coefﬁcients of the soft alignment, potentially resulting in weak signal from the encoder.
Revised: We attribute this to the unnormalized attention coefﬁcients of the soft alignment, potentially resulting in weak signal from the encoder.

Original: Thus, we propose an alternative strategy of constraining attention weights only at inference to be monotonic, preserving the training procedure without any constraints.
Revised: Thus, we propose an alternative strategy of constraining attention weights only at inference to be monotonic, preserving the training procedure without any constraints.

Original: Instead of computing the softmax over the entire input, we instead compute the softmax only over a ﬁxed window starting at the last attended-to position and going forward several timesteps 8 .
Revised: Instead of computing the softmax over the entire input, we instead compute the softmax only over a ﬁxed window starting at the last attended-to position and going forward several timesteps 8 .

Original: The initial position is set to zero and is later computed as the index of the highest attention weight within the current window.
Revised: The initial position is set to zero and is later computed as the index of the highest attention weight within the current window.

Original: This strategy also enforces monotonic attentiondistribution at inference, as shown in Fig. 4 and yields superior speech quality.
Revised: This strategy also enforces monotonic attention at inference as shown in Fig. 4, and yields superior speech quality.

Original: 3. C ONVERTER
Revised: 3. C ONVERTER

Original: The converter network predicts the vocoder parameters for audio waveform generation from the last hidden states of the decoder. To process these time-varying hidden states efﬁciently, we use stacked convolution blocks (described in Section 3.3).
Revised: The converter network takes as inputs the activations from the last hidden layer of the decoder, applies several non-causal convolution blocks, and then predicts parameters for downstream vocoders.

Original: Unlike the decoder, the converter is non-causal and non-autoregressive, so it can use future context information to predict its outputs.
Revised: Unlike the decoder, the converter is non-causal and non-autoregressive, so it can use future context from the decoder to predict its outputs.

Original: The loss function of the converter network depends on the type of the vocoder used:
Revised: The loss function of the converter network depends on the type of the vocoder used:

Original: Grifﬁn-Lim vocoder : Grifﬁn-Lim algorithm converts spectrograms to time domain audio 
Revised: Grifﬁn-Lim vocoder : Grifﬁn-Lim algorithm converts spectrograms to time-domain audio waveforms by iteratively

Original: waveforms by iteratively estimating the unknown phases.
Revised: estimating the unknown phases.

Original: We ﬁnd raising the spectrogram to a power parametrized by a sharpening factor before waveform synthesis is helpful for improved audio quality, as suggested in Wang et al.
Revised: We ﬁnd raising the spectrogram to a power parametrized by a sharpening factor before waveform synthesis is helpful for improved audio quality, as suggested in Wang et al.

Original: L1 loss is used for prediction of linear-scale log-magnitude spectrograms.
Revised: L1 loss is used for prediction of linear-scale log-magnitude spectrograms.

Original: Decoder Attention Block
Revised: [Table]

Original: Converter ……
Revised: Converter …

Original: Conv Block
Revised: [Table]

Original: Encoder
Revised: Encoder

Original: Text/Phonemes
Revised: Text/Phonemes

Original: FC
Revised: FC

Original: Conv Block
Revised: Conv Block

Original: Conv Block + values keys ⨉ p 0 .
Revised: Conv Block + values keys ⨉ p 0 .

Original: Mel input
Revised: Mel input

Original: Attention Block
Revised: [Table]

Original: Attention Block
Revised: Attention Block

Original: Conv Block … …
Revised: Conv Block … …

Original: Linear Spectrogram WORLD features
Revised: Linear Spectrogram WORLD features

Original: Input conv split " ⨉ + ⨉

Original: Output
Revised: [Figure]

Original: Bi-GRU
Revised: Bi-GRU

Original: Bi-GRU c 2c c c p 0 .c
Revised: Bi-GRU c 2c [Figure] p 0 .c

Original: FC
Revised: FC

Original: ReLu dropout ⨉ N ・・・
Revised: ReLu dropout ⨉ N ・・・

Original: FC
Revised: FC

Original: ReLu dropout …
Revised: ReLu dropout …

Original: FC … …
Revised: FC … …

Original: Conv Block FC
Revised: Conv Block FC

Original: FC
Revised: FC

Original: Done "
Revised: Done "

Original: Mel Output
Revised: Mel Output

Original: WORLD [Table]
Revised: WORLD Block

Original: [Figure]
Revised: FC " F FC

Original: [Table]
Revised: Voiced

Original: [Figure]
Revised: FC

Original: [Table]
Revised: Spectral Envelope Aperiodicity FC

Original: [Figure]
Revised: Input

Original: Conv Block upsample by repetition
Revised: Conv Block upsample by repetition

Original: WORLD Block [Figure] Positional Encoding (rate ω)
Revised: WORLD Block values keys query Positional Encoding (rate ω)

Original: Positional Encoding (rate 1) +
Revised: Positional Encoding (rate 1) +

Original: +
Revised: +

Original: FC FC
Revised: FC FC

Original: FC • 
Revised: FC • inference mask (optional) • Output

Original: [Figure]
Revised: [Figure]

Original: optional)
Revised: OR

Original: [Figure]
Revised: [Figure]

Revised: dropout

Revised: FC / timesteps sqrt

Original: FC
Revised: FC

Original: WORLD vocoder :
Revised: WORLD vocoder :

Original: The WORLD vocoder is based on (Morise et al., 2016).
Revised: The WORLD vocoder is based on (Morise et al., 2016).

Original: As vocoder parameters, we predict a boolean value (whether the current frame is voiced or unvoiced), an F0 value (if the frame is voiced), the spectral envelope, and the aperiodicity parameters.
Revised: As vocoder parameters, we predict a boolean value (whether the current frame is voiced or unvoiced), an F0 value (if the frame is voiced), the spectral envelope, and the aperiodicity parameters.

Original: We use a cross-entropy loss for the voiced-unvoiced prediction, and L1 losses for all other predictions (see Fig. 5), 3.
Revised: We use a cross-entropy loss for the voiced-unvoiced prediction, and L1 losses for all other predictions (see Fig. 5), 3.

Original: WaveNet vocoder : We separately train a WaveNet to be used as a vocoder treating melscale log-magnitude spectrograms as vocoder parameters.
Revised: WaveNet vocoder : We separately train a WaveNet to be used as a vocoder treating melscale log-magnitude spectrograms as vocoder parameters.

Original: These vocoder parameters are input as external conditioners to the network.
Revised: These vocoder parameters are input as external conditioners to the network.

Original: The training procedure and the architecture besides the conditioner are similar to the WaveNet described in Arık et al.
Revised: The WaveNet is trained using ground-truth mel-spectragrams and audio waveforms. The architecture besides the conditioner is similar to the WaveNet described in Arık et al.

Original: While the WaveNet in Arık et al.
Revised: While the WaveNet in Arık et al.

Original: (2017) is conditioned with linear-scale log-magnitude spectrograms, we observed better performance with mel-scale spectrograms, which corresponds to a more compact representation of audio.
Revised: (2017) is conditioned with linear-scale log-magnitude spectrograms, we observed better performance with mel-scale spectrograms, which corresponds to a more compact representation of audio.

Original: To predict mel-scale spectrograms, L1 loss on linear-scale spectrograms are also used besides L1 loss on mel-scale spectrograms at decoder.
Revised: In addition to L1 loss on mel-scale spectrograms at decode, L1 loss on linear-scale spectrogram is also applied as Grifﬁn-Lim vocoder. [Table]

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we present several different experiments and metrics to evaluate our speech synthesis system.
Revised: In this section, we present several different experiments and metrics to evaluate our speech synthesis system.

Revised: We quantify the performance of our system and compare it to other recently published neural

Original:  Data: For single-speaker synthesis, we use an internal English speech dataset containing approximately 20 hours of audio with a sample rate of 48 kHz.
Revised: TTS systems. Data: For single-speaker synthesis, we use an internal English speech dataset containing approximately 20 hours of audio with a sample rate of 48 kHz.

Original: For multi-speaker synthesis, we use the VCTK (Yamagishi et al., 2009) and LibriSpeech (Panayotov et al., 2015) datasets.
Revised: For multi-speaker synthesis, we use the VCTK (Yamagishi et al., 2009) and LibriSpeech (Panayotov et al., 2015) datasets.

Original: The VCTK dataset consists of audio for 108 speakers, with a total duration of ∼ 44 hours.
Revised: The VCTK dataset consists of audios for 108 speakers, with a total duration of ∼ 44 hours.

Original: The LibriSpeech [Table] set consist of audio for 2484 speakers, with a total duration of ∼ 820 hours.
Revised: The LibriSpeech dataset consists of audios for 2484 speakers, with a total duration of ∼ 820 hours.

Original: The sample rate iskHz for VCTK and 16 kHz for LibriSpeech.
Revised: The sample rate is 48 kHz for VCTK and 16 kHz for LibriSpeech.

Original: Fast Training: We compare Deep Voice 3 to Tacotron, a recently published attention-based TTS system.
Revised: Fast Training: We compare Deep Voice 3 to Tacotron, a recently published attention-based TTS system.

Original: For our system on single-speaker data, the average training iteration time (for batch size 4) is 0.06 seconds using one GPU as opposed to 0.59 seconds for Tacotron, indicating a ten-fold increase in training speed.
Revised: For our system on single-speaker data, the average training iteration time (for batch size 4) is 0.06 seconds using one GPU as opposed to 0.59 seconds for Tacotron, indicating a ten-fold increase in training speed.

Original: In addition, Deep Voice 3 converges after ∼ 500 K iterations for all three datasets in our experiment, while Tacotron requires ∼ 2 M iterations as suggested in Wang et al.
Revised: In addition, Deep Voice 3 converges after ∼ 500 K iterations for all three datasets in our experiment, while Tacotron requires ∼ 2 M iterations as suggested in Wang et al.

Original: This signiﬁcant speedup is due to the fully-convolutional architecture of Deep Voice 3, which exploits the parallelism of a GPU during training.
Revised: This signiﬁcant speedup is due to the fully-convolutional architecture of Deep Voice 3, which exploits the parallelism of a GPU during training.

Original: Attention Error Modes: Attention-based neural TTS systems may run into several error modes that can reduce synthesis quality – including (i) repeated words, (ii) mispronunciations, and (iii) skipped words.
Revised: Attention Error Modes: Attention-based neural TTS systems may run into several error modes that can reduce synthesis quality – including (i) repeated words, (ii) mispronunciations, and (iii) skipped words.

Revised: 9 One reason for (i) and (iii) is that the attention-based model does not impose a monotonically progressing mechanism.

Original: In order to track the occurrence of attention errors, we construct a custom 100-sentence test set (see Appendix E) that includes particularly-challenging cases from deployed TTS systems (e.g. dates, acronyms, URLs, repeated words, proper nouns, foreign words etc.)
Revised: In order to track the occurrence of attention errors, we construct a custom 100-sentence test set (see Appendix E) that includes particularly-challenging cases from deployed TTS systems (e.g. dates, acronyms, URLs, repeated words, proper nouns, foreign words etc.)

Original: Attention error counts (listed in Table 1) indicate that the model trained using a mix of characters and phonemes, using standard attention during training and forced monotonic attention during inference, largely outperforms other approaches.
Revised: Attention error counts are listed in Table 1 and indicate that the model with joint representation of characters and phonemes, trained with standard attention mechanism but enforced the monotonic constraint at inference, largely outperforms other approaches.

Original: Naturalness: We demonstrate that choice of waveform synthesis matters for naturalness ratings and compare it to other published neural TTS systems.
Revised: Naturalness: We demonstrate that choice of waveform synthesis matters for naturalness ratings and compare it to other published neural TTS systems.

Original: Results in Table 2 indicate that WaveNet, a neural vocoder, achieves the highest MOS of 3 . 78 , followed by WORLD and Grifﬁn-Lim at 3 .and 3 . 62 , respectively.
Revised: Results in Table 2 indicate that WaveNet, a neural vocoder, achieves the highest MOS of 3 . 78 , followed by WORLD and Grifﬁn-Lim at 3 . 63 and 3 . 62 , respectively.

Original: Thus, we show that the most natural waveform synthesis can be done with a neural vocoder and that basic spectrogram inversion techniques can match advanced vocoders.
Revised: Thus, we show that the most natural waveform synthesis can be done with a neural vocoder, and that basic spectrogram inversion techniques can match advanced vocoders with high quality single speaker data.

Original: The WaveNet vocoder sounds more natural as the WORLD vocoder introduces various noticeable artifacts.
Revised: The WaveNet vocoder sounds more natural as the WORLD vocoder introduces various noticeable artifacts.

Original: Yet, lower inference latency may render the WORLD vocoder preferable: the heavily engineered WaveNet implementation runs at 3X realtime per CPU core (Arık et al., 2017), while 
Revised: Yet, lower inference latency may render the WORLD vocoder preferable: the heavily engineered WaveNet implementation runs at 3X realtime per CPU core (Arık et al., 2017), while WORLD runs up to 40X realtime per CPU core (see the subsection below). (i) “D AA M AH N AH N T . V EH JH AH T EH T EH R IY AH N .”

Revised: , (ii) “D AE M AH N AE N T .

Revised: V EH JH AH T EH R IY AH N .”

Original: WORLD runs up to 40X realtime per CPU core (see the subsection below).
Revised: , (iii) “D AH N T . V EH JH AH T EH R IY AH N .” [Table] [Table]

Original: Multi-Speaker Synthesis:
Revised: Multi-Speaker Synthesis:

Original: To demonstrate that our model is capable of handling multi-speaker speech synthesis effectively, we train our models on the VCTK and LibriSpeech data sets.
Revised: To demonstrate that our model is capable of handling multi-speaker speech synthesis effectively, we train our models on the VCTK and LibriSpeech data sets.

Original: For LibriSpeech (an ASR dataset), we apply a preprocessing step of standard denoising (using SoX (Bagwell, 2017)) and splitting long utterances into multiple at pause locations (which are determined by
Revised: For LibriSpeech (an ASR dataset), we apply a preprocessing step of standard denoising (using SoX (Bagwell, 2017)) and splitting long utterances into multiple at pause locations (which are determined by

Original: Gentle (Ochshorn & Hawkins, 2017)).
Revised: Gentle (Ochshorn & Hawkins, 2017)).

Original: Results are presented in Table 3.
Revised: Results are presented in Table 3.

Original: We purposefully include ground-truth samples in the set being evaluated, because the accents in datasets are likely to be unfamiliar to our North American crowdsourced raters.
Revised: We purposefully include ground-truth samples in the set being evaluated, because the accents in datasets are likely to be unfamiliar to our North American crowdsourced raters.

Original: Including these ground truth samples encourages judgment based on accent rather than solely on model quality.

Original: Our model with the WORLD vocoder achieves a comparable MOS of 3.44 on VCTK in contrast to 3.69 from Deep Voice 2, which is the state-of-the-art multi-speaker neural TTS  using WaveNet as vocoder and seperately optimized neural duration and  frequency prediction models.
Revised: Our model with the WORLD vocoder achieves a comparable MOS of 3.44 on VCTK in contrast to 3.69 from Deep Voice 2, which is the state-of-theart multi-speaker neural TTS system using WaveNet as vocoder and seperately optimized phoneme duration and fundamental frequency prediction models.

Original: We expect further improvement by using WaveNet for multi-speaker synthesis, although it may substantially slow down inference.
Revised: We expect further improvement by using WaveNet for multi-speaker synthesis, although it may substantially slow down inference.

Original: The MOS on Lib- (i) “D AA M AH N AH N T . V EH JH AH T EH T EH R IY AH N .”
Revised: The MOS on LibriSpeech is lower compared to VCTK, which we mainly attribute to the lower quality of the training dataset due to the various recording conditions and noticeable background noise.

Original: , (ii) “D AE M AH N AE N T . V EH JH AH T EH R IY AH N .” , (iii) “D AH N T . V EH JH AH T EH R IY AH N .”
Revised: In the literature, Yamagishi et al.

Original: [Table] [Table] riSpeech is lower, which we mainly attribute to the lower quality of the training dataset due to the recording conditions and background noise.
Revised: (2010) also observes worse performance, when apply parametric TTS method to ASR dataset instead of high quality TTS dataset.

Original: Lastly, we observe that the learned speaker embeddings lie in a meaningful latent space (see Fig. 7 in Appendix D), as in Arık et al.
Revised: Lastly, we ﬁnd that the learned speaker embeddings lie in a meaningful latent space (see Fig. 7 in Appendix D).

Original: Optimizing Inference for Deployment:
Revised: Optimizing Inference for Deployment:

Original: In order to deploy a neural TTS system in a cost-effective manner, the system must be able to handle high trafﬁc similar to alternative systems on a comparable amount of hardware.
Revised: In order to deploy a neural TTS system in a cost-effective manner, the system must be able to handle as much trafﬁc as alternative systems on a comparable amount of hardware.

Original: We target a throughput of ten million queries per day (116 queries per second (QPS) 11) on a single-GPU server with twenty CPU cores, which we ﬁnd is comparable in cost to already-deployed TTS systems.
Revised: To do so, we target a throughput of ten million queries per day or 116 queries per second (QPS) 11 on a single-GPU server with twenty CPU cores, which we ﬁnd is comparable in cost to commercially deployed TTS systems.

Original: By implementing custom GPU kernels for the Deep Voicearchitecture and parallelizing WORLD synthesis across CPUs, we demonstrate that our model can handle ten million queries per day.
Revised: By implementing custom GPU kernels for the Deep Voice 3 architecture and parallelizing WORLD synthesis across CPUs, we demonstrate that our model can handle ten million queries per day.

Original: We provide more details on the implementation in Appendix B.
Revised: We provide more details on the implementation in Appendix B.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce Deep Voice 3, a neural text-to-speech system based on a novel fully-convolutional sequence-to-sequence acoustic model with a position-augmented attention mechanism.
Revised: We introduce Deep Voice 3, a neural text-to-speech system based on a novel fully-convolutional sequence-to-sequence acoustic model with a position-augmented attention mechanism.

Original: We describe common error modes in sequence-to-sequence speech synthesis models and show that we successfully avoid these common error modes with Deep Voice 3.
Revised: We describe common error modes in sequence-to-sequence speech synthesis models and show that we successfully avoid these common error modes with Deep Voice 3.

Original: We show that our model is agnostic of the waveform synthesis method, and adapt it for Grifﬁn-Lim spectrogram inversion, WaveNet, and
Revised: We show that our model is agnostic of the waveform synthesis method, and adapt it for Grifﬁn-Lim spectrogram inversion, WaveNet, and

Original: WORLD vocoder synthesis.
Revised: WORLD vocoder synthesis.

Original: We demonstrate also that our architecture is capable of multispeaker speech synthesis by augmenting it with trainable speaker embeddings, a technique described in
Revised: We demonstrate also that our architecture is capable of multispeaker speech synthesis by augmenting it with trainable speaker embeddings, a technique described in

Original: Deep Voice 2.
Revised: Deep Voice 2.

Original: Finally, we describe the production-ready Deep Voice 3 system in full including text normalization and performance characteristics, and demonstrate state-of-the-art quality through extensive MOS evaluations.
Revised: Finally, we describe the production-ready Deep Voice 3 system in full including text normalization and performance characteristics, and demonstrate state-of-the-art quality through extensive MOS evaluations.

Original: Future work will involve improving the implicitly learned grapheme-tophoneme model, jointly training with a neural vocoder, and training on cleaner and larger datasets to scale to model the full variability of human voices and accents from hundreds of thousands of speakers.
Revised: Future work will involve improving the implicitly learned grapheme-tophoneme model, jointly training with a neural vocoder, and training on cleaner and larger datasets to scale to model the full variability of human voices and accents from hundreds of thousands of speakers.

Original: Appendices A D ETAILED M ODEL A RCHITECTURE OF D EEP V OICE
Revised: Appendices A D ETAILED M ODEL A RCHITECTURE OF D EEP V OICE

Original: The detailed model architecture in depicted in Fig.
Revised: The detailed model architecture in depicted in Fig.

Original: Decoder
Revised: Decoder

Original: Converter …
Revised: Converter …

Original: Encoder
Revised: Encoder

Original: Text Embeddings
Revised: Text Embeddings

Original: FC
Revised: FC

Original: Conv Block
Revised: Conv Block

Original: Conv Block + values keys ⨉ p 0 .
Revised: Conv Block + values keys ⨉ p 0 .

Original: Mel input
Revised: Mel input

Original: Attention Block
Revised: Attention Block

Original: Conv Block …
Revised: Conv Block …

Original: Linear Spectrogram
Revised: Linear Spectrogram

Original: WORLD features
Revised: WORLD features

Original: FC
Revised: FC

Original: ReLu dropout ⨉ N prenet (cid:29420)(cid:29420)(cid:29420)
Revised: ReLu dropout ⨉ N prenet (cid:29420)(cid:29420)(cid:29420)

Original: FC
Revised: FC

Original: ReLu dropout
Revised: ReLu dropout

Original: FC … …
Revised: FC … …

Original: Conv Block
Revised: Conv Block

Original: FC FC
Revised: FC FC

Original: Done 𝜎
Revised: Done 𝜎

Original: Mel Output
Revised: Mel Output

Original: Conv Block upsample by repetition
Revised: Conv Block upsample by repetition

Original: WORLD Block
Revised: WORLD Block

Original: FC FC
Revised: FC FC

Original: Speaker Embedding
Revised: Speaker Embedding

Original: FC softsign + + FC softsign + +
Revised: FC softsign + + FC softsign + +

Original: FC softsign
Revised: FC softsign

Original: Causal Conv Block ⨉ N decoder + p 0 .⨉
Revised: Causal Conv Block ⨉ N decoder + p 0 .⨉

Original: + ⨉ (cid:29420)(cid:29420)(cid:29420) raise to power
Revised: + ⨉ (cid:29420)(cid:29420)(cid:29420) raise to power

Original: Grifﬁn-Lim
Revised: Grifﬁn-Lim

Original: WORLD Synthesis
Revised: WORLD Synthesis

Original: Wave ⨉ ⨉
Revised: Wave ⨉ ⨉

Original: FC 𝜎 (cid:29420)(cid:29420)(cid:29420)
Revised: FC 𝜎 (cid:29420)(cid:29420)(cid:29420)

Original: WaveNet
Revised: WaveNet

Original: Wave Wave
Revised: Wave Wave

Original: B O PTIMIZING D EEP V OICE 3 FOR DEPLOYMENT
Revised: B O PTIMIZING D EEP V OICE 3 FOR DEPLOYMENT

Original: Running inference with a TensorFlow graph turns out to be prohibitively expensive, averaging approximately 1 QPS 12 .
Revised: Running inference with a TensorFlow graph turns out to be prohibitively expensive, averaging approximately 1 QPS 12 .

Original: Instead, we implement custom GPU kernels for Deep Voice 3 inference.
Revised: Instead, we implement custom GPU kernels for Deep Voice 3 inference.

Original: Due to the complexity of the model and the large number of output timesteps, launching individual kernels for different operations in the graph (convolutions, matrix multiplications, unary and binary operations etc.) is impractical: the overhead of launch a CUDA kernel is approximately 50 µ s, which, when aggregated across all operations in the model and all output timesteps, limits throughput to approximately 10 QPS.
Revised: Due to the complexity of the model and the large number of output timesteps, launching individual kernels for different operations in the graph (convolutions, matrix multiplications, unary and binary operations etc.) is impractical: the overhead of launch a CUDA kernel is approximately 50 µ s, which, when aggregated across all operations in the model and all output timesteps, limits throughput to approximately 10 QPS.

Original: Thus, we implement a single kernel for the entire model, which avoids the overhead of launching many CUDA kernels.
Revised: Thus, we implement a single kernel for the entire model, which avoids the overhead of launching many CUDA kernels.

Original: Finally, instead of batching computation in the kernel, our kernel operates on a single utterance and we launch as many concurrent streams as there are Streaming Multiprocessors (SMs) on the GPU.
Revised: Finally, instead of batching computation in the kernel, our kernel operates on a single utterance and we launch as many concurrent streams as there are Streaming Multiprocessors (SMs) on the GPU.

Original: Every kernel is launched with one block, so we expect the GPU to schedule one block per SM, allowing us to scale inference speed linearly with the number of SMs.
Revised: Every kernel is launched with one block, so we expect the GPU to schedule one block per SM, allowing us to scale inference speed linearly with the number of SMs.

Original: On a single Nvidia Tesla P100 GPU with 56 SMs, we achieve an inference speed of 115 QPS, which corresponds to our target ten million queries per day.
Revised: On a single Nvidia Tesla P100 GPU with 56 SMs, we achieve an inference speed of 115 QPS, which corresponds to our target ten million queries per day.

Original: We parallelize WORLD synthesis across all
Revised: We parallelize WORLD synthesis across all

Original: CPUs on the server, permanently pinning threads to CPUs in order to maximize cache performance.
Revised: CPUs on the server, permanently pinning threads to CPUs in order to maximize cache performance.

Original: In this setup, GPU inference is the bottleneck, as WORLD synthesis on 20 cores is faster than
Revised: In this setup, GPU inference is the bottleneck, as WORLD synthesis on 20 cores is faster than

Original: QPS.
Revised: QPS.

Original: We believe that inference can be made signiﬁcantly faster through more optimized kernels, smaller models, and ﬁxed-precision arithmetic; we leave these aspects to future work.
Revised: We believe that inference can be made signiﬁcantly faster through more optimized kernels, smaller models, and ﬁxed-precision arithmetic; we leave these aspects to future work.

Original: C M ODEL H YPERPARAMETERS
Revised: C M ODEL H YPERPARAMETERS

Original: All hyperparameters of the models used in this paper are shown in Table 4.[Table]
Revised: All hyperparameters of the models used in this paper are shown in Table 4.

Original: Decoder

Original: [Table]
Revised: [Table]

Original: Converter Layers / Conv.
Revised: Converter Layers / Conv.

Original: Width / Channels
Revised: Width / Channels

Original: [Table]
Revised: [Table]

Original: D L ATENT S PACE OF THE L EARNED E MBEDDINGS
Revised: D L ATENT S PACE OF THE L EARNED E MBEDDINGS

Original: Similar to Arık et al.
Revised: Similar to Arık et al.

Original: , we apply principal component analysis to the learned speaker embeddings and analyze the speakers based on their ground truth genders.
Revised: , we apply principal component analysis to the learned speaker embeddings and analyze the speakers based on their ground truth genders.

Original: Fig.
Revised: Fig.

Original: 7 shows the genders of the speakers in the space spanned by the ﬁrst two principal components.
Revised: 7 shows the genders of the speakers in the space spanned by the ﬁrst two principal components.

Original: We observe a very clear separation between male and female genders, suggesting the low-dimensional speaker embeddings constitute a meaningful latent space.
Revised: We observe a very clear separation between male and female genders, suggesting the low-dimensional speaker embeddings constitute a meaningful latent space.

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: E 100- SENTENCE TEST SET
Revised: E 100- SENTENCE TEST SET

Original: The 100 sentences used to quantify the results in Table 1 are listed below (note that % symbol corresponds to pause): 
Revised: The 100 sentences used to quantify the results in Table 1 are listed below (note that % symbol corresponds to pause): [Figure]