Original: F AST S PEECH 2: F AST AND H IGH -Q UALITY E ND - TOE ND T EXT TO S PEECH
Revised: F AST S PEECH 2: F AST AND H IGH -Q UALITY E ND - TOE ND T EXT TO S PEECH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al., 2019) can synthesize speech signiﬁcantly faster than previous autoregressive models with comparable quality.
Revised: Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al., 2019) can synthesize speech signiﬁcantly faster than previous autoregressive models with comparable quality.

Original: The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS.
Revised: The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS.

Original: However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simpliﬁcation, both of which limit the voice quality.
Revised: However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simpliﬁcation, both of which limit the voice quality.

Original: In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simpliﬁed output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs.
Revised: In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simpliﬁed output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs.

Original: Speciﬁcally, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference.
Revised: Speciﬁcally, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference.

Original: We further design FastSpeech 2s, which is the ﬁrst attempt to directly generate speech waveform from text in parallel, enjoying the beneﬁt of fully end-to-end inference.
Revised: We further design FastSpeech 2s, which is the ﬁrst attempt to directly generate speech waveform from text in parallel, enjoying the beneﬁt of fully end-to-end inference.

Original: Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even
Revised: Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even

Original: faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models.
Revised: faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models.

Original: Audio samples are available at https://fastspeech2.github.io/fastspeech2/ .
Revised: Audio samples are available at https://fastspeech2.github.io/fastspeech2/ .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural network based text to speech (TTS) has made rapid progress and attracted a lot of attention in the machine learning and speech community in recent years (Wang et al., 2017;
Revised: Neural network based text to speech (TTS) has made rapid progress and attracted a lot of attention in the machine learning and speech community in recent years (Wang et al., 2017;

Original: Shen et al., 2018;
Revised: Shen et al., 2018;

Original: Ming et al., 2016;
Revised: Ming et al., 2016;

Original: Arik et al., 2017; Ping et al., 2018; Ren et al., 2019; Li et al., 2019).
Revised: Arik et al., 2017; Ping et al., 2018; Ren et al., 2019; Li et al., 2019).

Original: Previous neural TTS models (Wang et al., 2017; Shen et al., 2018; Ping et al., 2018; Li et al., 2019)
Revised: Previous neural TTS models (Wang et al., 2017; Shen et al., 2018; Ping et al., 2018; Li et al., 2019)

Original: ﬁrst generate mel-spectrograms autoregressively from text and then synthesize speech from the generated mel-spectrograms using a separately trained vocoder (Van Den Oord et al., 2016; Oord et al., 2017;
Revised: ﬁrst generate mel-spectrograms autoregressively from text and then synthesize speech from the generated mel-spectrograms using a separately trained vocoder (Van Den Oord et al., 2016; Oord et al., 2017;

Original: Prenger et al., 2019; Kim et al., 2018; Yamamoto et al., 2020; Kumar et al., 2019).
Revised: Prenger et al., 2019; Kim et al., 2018; Yamamoto et al., 2020; Kumar et al., 2019).

Original: They usually suffer from slow inference speed and robustness (word skipping and repeating) issues (Ren et al., 2019).
Revised: They usually suffer from slow inference speed and robustness (word skipping and repeating) issues (Ren et al., 2019).

Original: In recent years, non-autoregressive TTS models (Ren et al., 2019; Ła´ncucki, 2020; Kim et al., 2020; Lim et al., 2020; Miao et al., 2020; Peng et al., 2019) are designed to address these issues, which generate mel-spectrograms with extremely fast speed and avoid robustness issues, while achieving comparable voice quality with previous autoregressive models.
Revised: In recent years, non-autoregressive TTS models (Ren et al., 2019; Ła´ncucki, 2020; Kim et al., 2020; Lim et al., 2020; Miao et al., 2020; Peng et al., 2019) are designed to address these issues, which generate mel-spectrograms with extremely fast speed and avoid robustness issues, while achieving comparable voice quality with previous autoregressive models.

Original: Among those non-autoregressive TTS methods, FastSpeech (Ren et al., 2019) is one of the most successful models.
Revised: Among those non-autoregressive TTS methods, FastSpeech (Ren et al., 2019) is one of the most successful models.

Original: The training of FastSpeech relies on an autoregressive teacher model to provide 1)  the duration of each phoneme to train a duration predictor, and 2) the generated mel-spectrograms for knowledge distillation.
Revised: FastSpeech designs two ways to alleviate the one-to-many mapping problem: 1) Reducing data variance in the target side by using the generated mel-spectrogram from an autoregressive teacher model as the training target (i .e.

Original:  While these designs in FastSpeech ease the learning of the one-to-many mapping problem (see Section 2.1) in TTS, they also bring several disadvantages: 1) the two-stage teacher-student distillation pipeline is time-consuming; 2) the duration extracted from the attention map of the teacher model is not accurate enough, and the  target mel-spectrograms distilled from the teacher model suffer from information loss 1 due to data simpliﬁcation, both of which limit thevoice quality and prosody.
Revised: , knowledge distillation). Introducing the duration information (extracted from the attention map of the teacher model) to expand the text sequence to match the length of the mel-spectrogram sequence. While these designs in FastSpeech ease the learning of the one-to-many mapping problem (see Section 2.1) in TTS, they also bring several disadvantages: 1) The two-stage teacher-student training pipeline makes the training process complicated. The target mel-spectrograms generated from the teacher model have some information loss 1 compared with the ground-truth ones, since the quality of the audio synthesized from the generated mel-spectrograms is usually worse than that from the ground-truth ones.

Revised: The duration extracted from the attention map of teacher model is not accurate enough.

Original: In this work, we propose FastSpeech 2 to address the issues in FastSpeech and better handle the one-to-many mapping problem in non-autoregressive TTS.
Revised: In this work, we propose FastSpeech 2 to address the issues in FastSpeech and better handle the one-to-many mapping problem in non-autoregressive TTS.

Original: To simplify the training pipeline and avoid the information loss due to data simpliﬁcation in teacher-student distillation, we directly train the FastSpeech 2 model with ground-truth target instead of the simpliﬁed output from a teacher.
Revised: To simplify the training pipeline and avoid the information loss due to data simpliﬁcation in teacher-student distillation, we directly train the FastSpeech 2 model with ground-truth target instead of the simpliﬁed output from a teacher.

Original: To reduce the information gap (input does not contain all the information to predict the target) between the input (text sequence) and target output (mel-spectrograms) and alleviate the one-tomapping problem for non-autoregressive TTS model training, we introduce some variation information of speech including pitch, energy and more accurate duration into FastSpeech: in training, we extract duration, pitch and energy from the target speech waveform and directly take them as conditional inputs; in inference, we use values predicted by the predictors that are jointly trained with the FastSpeech 2 model.
Revised: To reduce the information gap (input does not contain all the information to predict the target) between the input (text sequence) and target output (mel-spectrograms) and alleviate the one-to-many mapping problem for non-autoregressive TTS model training, we introduce some variation information of speech including pitch, energy and more accurate duration into FastSpeech: in training, we extract duration, pitch and energy from the target speech waveform and directly take them as conditional inputs; in inference, we use values predicted by the predictors that are jointly trained with the FastSpeech 2 model.

Original: Considering the pitch is important for the prosody of speech and is also difﬁcult to predict due to the large ﬂuctuations along time, we convert the pitch contour into pitch spectrogram using continuous wavelet transform (Tuteur, 1988; Grossmann & Morlet, 1984) and predict the pitch in the frequency domain, which can improve the accuracy of predicted pitch.
Revised: Considering the pitch is important for the prosody of speech and is also difﬁcult to predict due to the large ﬂuctuations along time, we convert the pitch contour into pitch spectrogram using continuous wavelet transform (Tuteur, 1988; Grossmann & Morlet, 1984) and predict the pitch in the frequency domain, which can improve the accuracy of predicted pitch.

Original: To further simplify the speech synthesis pipeline, we introduce FastSpeech 2s, which does not use mel-spectrograms as intermediate output and directly generates speech waveform from text in inference, enjoying low latency in inference.
Revised: To further simplify the speech synthesis pipeline, we introduce FastSpeech 2s, which does not use mel-spectrograms as intermediate output and directly generates speech waveform from text in inference, enjoying low latency in inference.

Original: Experiments on the LJSpeech (Ito, 2017) dataset show that 1) FastSpeech 2 enjoys much simpler training pipeline (3x training time reduction) than FastSpeech while inherits its advantages of fast, robust and controllable (even more controllable in pitch and energy) speech synthesis, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models.
Revised: Experiments on the LJSpeech (Ito, 2017) dataset show that 1) FastSpeech 2 enjoys much simpler training pipeline (3x training time reduction) than FastSpeech while inherits its advantages of fast, robust and controllable (even more controllable in pitch and energy) speech synthesis, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models.

Original: We attach audio samples generated by FastSpeech 2 and 2s at https://fastspeech2.github.io/fastspeech2/ .
Revised: We attach audio samples generated by FastSpeech 2 and 2s at https://fastspeech2.github.io/fastspeech2/ .

Original: The main contributions of this work are summarized as follows:
Revised: The main contributions of this work are summarized as follows:

Original: • FastSpeech 2 achieves a 3x training speed-up over FastSpeech by simplifying the training pipeline.
Revised: • FastSpeech 2 achieves a 3x training speed-up over FastSpeech by simplifying the training pipeline.

Original: • FastSpeech 2 alleviates the one-to-many mapping problem in TTS and achieves better voice quality.
Revised: • FastSpeech 2 alleviates the one-to-many mapping problem in TTS and achieves better voice quality.

Original: FastSpeech 2s further simpliﬁes the inference pipeline for speech synthesis while maintaining high voice quality, by directly generating speech waveform from text.
Revised: FastSpeech 2s further simpliﬁes the inference pipeline for speech synthesis while maintaining high voice quality, by directly generating speech waveform from text.

Original:  F AST S PEECH 2 AND 2 S
Revised:  F AST S PEECH 2 AND 2 S

Original: In this section, we ﬁrst describe the motivation of the design in FastSpeech 2, and then introduce the architecture of FastSpeech 2, which aims to improve FastSpeech to better handle the one-tomany mapping problem, with simpler training pipeline and higher voice quality.
Revised: In this section, we ﬁrst describe the motivation of the design in FastSpeech 2, and then introduce the architecture of FastSpeech 2, which aims to improve FastSpeech to better handle the one-tomany mapping problem, with simpler training pipeline and higher voice quality.

Original: At last, we extend
Revised: At last, we extend

Original: FastSpeech 2 to FastSpeech 2s for fully end-to-end text-to-waveform synthesis.
Revised: FastSpeech 2 to FastSpeech 2s for fully end-to-end text-to-waveform synthesis

Original:  2. M OTIVATION
Revised: . 2. M OTIVATION

Original: TTS is a typical one-to-many mapping problem (Wang et al., 2017; Zhu et al., 2017; Jayne et al., 2012; Gadermayr et al., 2020), since multiple possible speech sequences can correspond to a text sequence due to variations in speech, such as pitch, duration, sound volume and prosody.
Revised: TTS is a typical one-to-many mapping problem (Wang et al., 2017; Zhu et al., 2017; Jayne et al., 2012; Gadermayr et al., 2020), since multiple possible speech sequences can correspond to a text sequence due to variations in speech, such as pitch, duration, sound volume and prosody.

Original: In nonautoregressive TTS, the only input information is text which is not enough to fully predict the variance in speech.
Revised: In nonautoregressive TTS, the only input information is text which is not enough to fully predict the variance in speech.

Original: In this case, the model is prone to overﬁt to the variations of the target speech in thetraining set, resulting in poor generalization ability.
Revised: In this case, the model is prone to overﬁt to the variations of the target speech in the

Original: FastSpeech designs two ways to alleviate the one-to-many mapping problem: 1) Reducing data variance in the target side by using the generated mel-spectrogram from an autoregressive teacher model as the training target (i.e., knowledge distillation), which can ease the one-to-many mapping problem by simplifying the target.

Original: Introducing the duration information (extracted from the attention map of the teacher model) to expand the text sequence to match the length of the melspectrogram sequence, which can ease the one-to-many mapping problem by providing more input

Original: Mel-spectrogramDecoder
Revised: Mel-spectrogramDecoder

Original: Encoder
Revised: Encoder

Original: Phoneme Embedding
Revised: Phoneme Embedding

Original: Phoneme
Revised: Phoneme

Original: Variance Adaptor
Revised: Variance Adaptor

Original: Waveform Decoder
Revised: Waveform Decoder

Original: FastSpeech 2s
Revised: FastSpeech 2s

Original: Conv1D + ReLU
Revised: Conv1D + ReLU

Original: Linear Layer
Revised: Linear Layer

Original: PositionalEncoding PositionalEncoding
Revised: PositionalEncoding PositionalEncoding

Original: Pitch Predictor
Revised: Pitch Predictor

Original: Energy Predictor … Predictor
Revised: Energy Predictor … Predictor

Original: Pitch
Revised: Pitch

Original: Energy
Revised: Energy

Original: Duration Predictor
Revised: Duration Predictor

Original: LN + Dropout
Revised: LN + Dropout

Original: Conv1D + ReLU
Revised: Conv1D + ReLU

Original: LN + Dropout
Revised: LN + Dropout

Original: Duration
Revised: Duration

Original: Transposed Conv1D
Revised: Transposed Conv1D

Original: Dilated Conv1D
Revised: Dilated Conv1D

Original: Gated Activation
Revised: Gated Activation

Original: Conv 1x
Revised: Conv 1x

Original: Conv1D x N
Revised: Conv1D x N

Original: LR (a) FastSpeech
Revised: LR (a) FastSpeech

Original: Mel-spectrogramDecoder
Revised: Mel-spectrogramDecoder

Original: Encoder
Revised: Encoder

Original: Phoneme Embedding
Revised: Phoneme Embedding

Original: Phoneme
Revised: Phoneme

Original: Variance Adaptor
Revised: Variance Adaptor

Original: Waveform Decoder
Revised: Waveform Decoder

Original: FastSpeech 2s
Revised: FastSpeech 2s

Original: Conv1D + ReLU
Revised: Conv1D + ReLU

Original: Linear Layer
Revised: Linear Layer

Original: PositionalEncoding PositionalEncoding
Revised: PositionalEncoding PositionalEncoding

Original: Pitch Predictor
Revised: Pitch Predictor

Original: Energy Predictor … Predictor
Revised: Energy Predictor … Predictor

Original: Pitch
Revised: Pitch

Original: Energy
Revised: Energy

Original: Duration Predictor
Revised: Duration Predictor

Original: LN + Dropout
Revised: LN + [Table]

Original: Conv1D + ReLU
Revised: Conv1D + ReLU

Original: LN + Dropout
Revised: LN + Dropout

Original: Duration
Revised: Duration

Original: Transposed Conv1D
Revised: Transposed Conv1D

Original: Dilated Conv1D
Revised: Dilated Conv1D

Original: Gated Activation
Revised: Gated Activation

Original: Conv 1x
Revised: Conv 1x

Original: Conv1D x N
Revised: Conv1D x N

Original: LR (b) Variance adaptor
Revised: LR (b) Variance adaptor

Original: Mel-spectrogram Decoder
Revised: Mel-spectrogramDecoder

Original: Encoder
Revised: Encoder

Original: Phoneme Embedding
Revised: Phoneme Embedding

Original: Phoneme
Revised: Phoneme

Original: Variance Adaptor
Revised: Variance Adaptor

Original: Waveform Decoder
Revised: Waveform Decoder

Original: FastSpeech 2s
Revised: FastSpeech 2s

Original: Conv1D + ReLU
Revised: Conv1D + ReLU

Original: Linear Layer
Revised: Linear Layer

Original: PositionalEncoding PositionalEncoding
Revised: PositionalEncoding PositionalEncoding

Original: Pitch Predictor
Revised: Pitch Predictor

Original: Energy Predictor … Predictor
Revised: Energy Predictor … Predictor

Original: Pitch
Revised: Pitch

Original: Energy
Revised: Energy

Original: Duration Predictor
Revised: Duration Predictor

Original: LN + Dropout
Revised: LN + Dropout

Original: Conv1D + ReLU
Revised: Conv1D + ReLU

Original: LN + Dropout
Revised: LN + Dropout

Original: Duration
Revised: Duration

Original: Transposed Conv1D
Revised: Transposed Conv1D

Original: Dilated Conv1D
Revised: Dilated Conv1D

Original: Gated Activation
Revised: Gated Activation

Original: Conv 1x
Revised: Conv 1x

Original: Conv1D x N
Revised: Conv1D x N

Original: LR (c) Duration/pitch/energy predictor
Revised: LR (c) Duration/pitch/energy predictor

Original: Mel-spectrogramDecoder
Revised: Mel-spectrogramDecoder

Original: Encoder
Revised: Encoder

Original: Phoneme Embedding
Revised: Phoneme Embedding

Original: Phoneme
Revised: Phoneme

Original: Variance Adaptor
Revised: Variance Adaptor

Original: Waveform Decoder
Revised: Waveform Decoder

Original: FastSpeech 2s
Revised: FastSpeech 2s

Original: Conv1D + ReLU
Revised: Conv1D + ReLU

Original: Linear Layer
Revised: Linear Layer

Original: PositionalEncoding PositionalEncoding
Revised: PositionalEncoding PositionalEncoding

Original: Pitch Predictor
Revised: Pitch Predictor

Original: Energy

Original: [Figure]
Revised: [Figure]

Original: information. Although knowledge distillation and the duration information extracted from teacher model can improve the training of FastSpeech, they also bring about several issues: 1) The twostage teacher-student training pipeline makes the training process complicated. The target melspectrograms generated from the teacher model have some information loss compared with the ground-truth ones, since the quality of the audio synthesized from the generated mel-spectrograms is usually worse than that from the ground-truth ones, as shown in Table 1. 3)
Revised: training set, resulting in poor generalization ability.

Original: The duration extracted from the attention map of teacher model is not accurate enough, as analyzed in Table 5a. In FastSpeech 2, we address these issues by 1)removing the teacher-student distillation to simplify the  training pipeline; 2)using ground-truth speech as the training target to avoid information loss; and 3) improving the duration accuracy and introducing more variance information to ease the one-to-many mapping problem in predicting ground-truth speech.
Revised: As mentioned in Section 1, although FastSpeech designs two ways to alleviate the one-to-many mapping problem, they also bring about several issues including 1) the complicated training pipeline; 2) information loss of target mel-spectrogram as analyzed in Table 1; and 3) not accurate enough ground-truth duration as shown in Table 5a.

Original: In the following subsection, we introduce the detailed design of FastSpeech 2.
Revised: In the following subsection, we introduce the detailed design of FastSpeech 2 which aims to address these issues.

Original: 2. M ODEL O VERVIEW
Revised: 2. M ODEL O VERVIEW

Original: The overall model architecture of FastSpeech 2 is shown in Figure 1a.
Revised: The overall model architecture of FastSpeech 2 is shown in Figure 1a.

Original: The encoder converts the phoneme embedding sequence into the phoneme hidden sequence, and then the variance adaptor adds different variance information such as duration, pitch and energy into the hidden sequence, ﬁnally the mel-spectrogram decoder converts the adapted hidden sequence into mel-spectrogram sequence in parallel.
Revised: The encoder converts the phoneme embedding sequence into the phoneme hidden sequence, and then the variance adaptor adds different variance information such as duration, pitch and energy into the hidden sequence, ﬁnally the mel-spectrogram decoder converts the adapted hidden sequence into mel-spectrogram sequence in parallel.

Original: We use the feed-forward Transformer block, which is a stack of selfattention (Vaswani et al., 2017) layer and 1D-convolution as in FastSpeech (Ren et al., 2019), as the basic structure for the encoder and mel-spectrogram decoder.
Revised: We use the feed-forward Transformer block, which is a stack of selfattention (Vaswani et al., 2017) layer and 1D-convolution as in FastSpeech (Ren et al., 2019), as the basic structure for the encoder and mel-spectrogram decoder.

Original: Different from FastSpeech that relies on a teacher-student distillation pipeline and the phoneme duration from a teacher model, FastSpeech 2 makes several improvements.
Revised: Different from FastSpeech that relies on a teacher-student distillation pipeline and the phoneme duration from a teacher model, FastSpeech 2 makes several improvements.

Original: First, we remove the teacher-student distillation pipeline, and directly use ground-truth mel-spectrograms as target for model training, which can avoid the information loss in distilled mel-spectrograms and increase the upper bound of the voice quality.
Revised: First, we remove the teacher-student distillation pipeline, and directly use ground-truth mel-spectrograms as target for model training, which can avoid the information loss in distilled mel-spectrograms and increase the upper bound of the voice quality.

Original: Second, our variance adaptor consists of not only duration predictor but also pitch and energy predictors, where 1) the duration predictor uses the phoneme duration obtained by forced alignment (McAuliffe et al., 2017) as training target, which is more accurate than that extracted from the attention map of autoregressive teacher model as veriﬁed experimentally in Section 3.2.2; and 2) the additional pitch and energy predictors can provide more variance information, which is important to ease the one-to-many mapping problem in TTS.
Revised: Second, our variance adaptor consists of not only duration predictor but also pitch and energy predictors, where 1) the duration predictor uses the phoneme duration obtained by forced alignment (McAuliffe et al., 2017) as training target, which is more accurate than that extracted from the attention map of autoregressive teacher model as veriﬁed experimentally in Section 3.2.2; and 2) the additional pitch and energy predictors can provide more variance information, which is important to ease the one-to-many mapping problem in TTS.

Original: Third, to further simplify the training pipeline and push it towards a fully end-to-end system, we propose FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder).
Revised: Third, to further simplify the training pipeline and push it towards a fully end-to-end system, we propose FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder).

Original: In the following subsections, we describe detailed designs of the variance adaptor and direct waveform generation in our method.
Revised: In the following subsections, we describe detailed designs of the variance adaptor and direct waveform generation in our method.

Original: 2. V ARIANCE A DAPTOR
Revised: 2. V ARIANCE A DAPTOR

Original: The variance adaptor aims to add variance information (e.g., duration, pitch, energy, etc.) to the phoneme hidden sequence, which can provide enough information to predict variant speech for the one-to-many mapping problem in TTS.
Revised: The variance adaptor aims to add variance information (e.g., duration, pitch, energy, etc.) to the phoneme hidden sequence, which can provide enough information to predict variant speech for the one-to-many mapping problem in TTS.

Original: We brieﬂy introduce the variance information as follows: 1) phoneme duration, which represents how long the speech voice sounds; 2) pitch, which is a key feature to convey emotions and greatly affects the speech prosody; 3) energy, which indicates framelevel magnitude of mel-spectrograms and directly affects the volume  of speech.
Revised: We brieﬂy introduce the variance information as follows: 1) phoneme duration, which represents how long the speech voice sounds; 2) pitch, which is a key feature to convey emotions and greatly affects the speech prosody; 3) energy, which indicates framelevel magnitude of mel-spectrograms and directly affects the volume and prosody of speech.

Original: More variance information can be added in the variance adaptor, such as emotion, style and speaker, and we leave it for future work.
Revised: More variance information can be added in the variance adaptor, such as emotion, style and speaker, and we leave it for future work.

Original: Correspondingly, the variance adaptor consists of 1) a duration predictor (i.e., the length regulator, as used in FastSpeech), 2) a pitch predictor, and 3) an energy predictor, as shown in Figure 1b.
Revised: Correspondingly, the variance adaptor consists of 1) a duration predictor (i.e., the length regulator, as used in FastSpeech), 2) a pitch predictor, and 3) an energy predictor, as shown in Figure 1b.

Original: In training, we take the ground-truth value of duration, pitch and energy extracted from the recordings as input into the hidden sequence to predict the target speech.
Revised: In training, we take the ground-truth value of duration, pitch and energy extracted from the recordings as input into the hidden sequence to predict the target speech.

Original: At the same time, we use the ground-truth duration, pitch and energy as targets to train the duration, pitch and energy predictors, which are used in inference to synthesize target speech.
Revised: At the same time, we use the ground-truth duration, pitch and energy as targets to train the duration, pitch and energy predictors, which are used in inference to synthesize target speech.

Original: As shown in Figure 1c, the duration, pitch and energy predictors share similar model structure (but different model parameters), which consists of a 2-layer 1D-convolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to project the hidden states into the output sequence.
Revised: As shown in Figure 1c, the duration, pitch and energy predictors share similar model structure (but different model parameters), which consists of a 2-layer 1D-convolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to project the hidden states into the output sequence.

Original: In the following paragraphs, we describe the details of the three predictors respectively.
Revised: In the following paragraphs, we describe the details of the three predictors respectively.

Original: Duration Predictor
Revised: Duration Predictor

Original: The duration predictor takes the phoneme hidden sequence as input and predicts the duration of each phoneme, which represents how many mel frames correspond to this phoneme, and is converted into logarithmic domain for ease of prediction.
Revised: The duration predictor takes the phoneme hidden sequence as input and predicts the duration of each phoneme, which represents how many mel frames correspond to this phoneme, and is converted into logarithmic domain for ease of prediction.

Original: The duration predictor is optimized with mean square error (MSE) loss, taking the extracted duration as training target.
Revised: The duration predictor is optimized with mean square error (MSE) loss, taking the extracted duration as training target.

Original: Instead of extracting the phoneme duration using a pre-trained autoregressive TTS model in
Revised: Instead of extracting the phoneme duration using a pre-trained autoregressive TTS model in

Original: FastSpeech, we use Montreal forced alignment (MFA) (McAuliffe et al., 2017) tool 2 to extract the phoneme duration, in order to improve the alignment accuracy and thus reduce the information gap between the model input and output.
Revised: FastSpeech, we use Montreal forced alignment (MFA) (McAuliffe et al., 2017) tool 3 to extract the phoneme duration, in order to improve the alignment accuracy and thus reduce the information gap between the model input and output.

Original: Pitch Predictor Previous neural network based TTS systems with pitch prediction (Arik et al., 2017; Gibiansky et al., 2017) often predict pitch contour directly.
Revised: Pitch Predictor Previous neural network based TTS systems with pitch prediction (Arik et al., 2017; Gibiansky et al., 2017) often predict pitch contour directly.

Original: However, due to high variations of ground-truth pitch, the distribution of predicted pitch values is very different from ground-truth distribution, as analyzed in Section 3.2.2.
Revised: However, due to high variations of ground-truth pitch, the distribution of predicted pitch values is very different from ground-truth distribution, as analyzed in Section 3.2.2.

Original: To better predict the variations in pitch contour, we use continuous wavelet transform (CWT) to decompose the continuous pitch series into pitch spectrogram (Suni et al., 2013; Hirose & Tao, 2015) and take the pitch spectrogram as the training target for the pitch predictor which is optimized with MSE loss.
Revised: To better predict the variations in pitch contour, we use continuous wavelet transform (CWT) to decompose the continuous pitch series into pitch spectrogram (Suni et al., 2013; Hirose & Tao, 2015) and take the pitch spectrogram as the training target for the pitch predictor which is optimized with MSE loss.

Original: In inference, the pitch predictor predicts the pitch spectrogram, which is further converted back into pitch contour using inverse continuous wavelet transform (iCWT).
Revised: In inference, the pitch predictor predicts the pitch spectrogram, which is further converted back into pitch contour using inverse continuous wavelet transform (iCWT).

Original: We describe the details of pitch extraction, CWT, iCWT and pitch predictor architecture in Appendix D. To take the pitch contour as input in both training and inference, we quantize pitch F 0 (ground-truth/predicted value for train/inference respectively) of each frame to 256 possible values in log-scale and further convert it into pitch embedding vector p and add it to the expanded hidden sequence.
Revised: We describe the details of pitch extraction, CWT, iCWT and pitch predictor architecture in Appendix D. To take the pitch contour as input in both training and inference, we quantize pitch F 0 (ground-truth/predicted value for train/inference respectively) of each frame to 256 possible values in log-scale and further convert it into pitch embedding vector p and add it to the expanded hidden sequence.

Original: Energy Predictor We compute L2-norm of the amplitude of each short-time Fourier transform (STFT) frame as the energy.
Revised: Energy Predictor We compute L2-norm of the amplitude of each short-time Fourier transform (STFT) frame as the energy.

Original: Then we quantize energy of each frame to 256 possible values uniformly, encoded it into energy embedding e and add it to the expanded hidden sequence similarly to pitch.
Revised: Then we quantize energy of each frame to 256 possible values uniformly, encoded it into energy embedding e and add it to the expanded hidden sequence similarly to pitch.

Original: We use an energy predictor to predict the original values of energy instead of the quantized values and optimize the energy predictor with MSE loss 3 .
Revised: We use an energy predictor to predict the original values of energy instead of the quantized values and optimize the energy predictor with MSE loss 4 .

Original: 2. F AST S PEECH 2 S
Revised: 2. F AST S PEECH 2 S

Original: To enable fully end-to-end text-to-waveform generation, in this subsection, we extend FastSpeechto FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder).
Revised: To enable fully end-to-end text-to-waveform generation, in this subsection, we extend FastSpeechto FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder).

Original: As shown in Figure 1a, FastSpeech 2s generates waveform conditioning on intermediate hidden, which makes it more compact in inference by discarding mel-spectrogram decoder and achieve comparable performance with a cascaded system.
Revised: As shown in Figure 1a, FastSpeech 2s generates waveform conditioning on intermediate hidden, which makes it more compact in inference by discarding mel-spectrogram decoder and achieve comparable performance with a cascaded system.

Original: We ﬁrst discuss the challengeslie in non-autoregressive text-to-waveform generation, then describe details of FastSpeech 2s, including model structure and training and inference processes.
Revised: We ﬁrst discuss the challenges in non-autoregressive text-to-waveform generation, then describe details of FastSpeech 2s, including model structure and training and inference processes.

Original: Challenges in Text-to-Waveform Generation When pushing TTS pipeline towards fully end-toend framework, there are several challenges: 1) Since  waveform contains more variance information (e.g., phase) than mel-spectrograms, the information gap between the input and output is larger than that in text-to-spectrogram generation.
Revised: Challenges in Text-to-Waveform Generation When pushing TTS pipeline towards fully endto-end framework, there are several challenges: 1) Since the waveform contains more variance information (e.g., phase) than mel-spectrograms, the information gap between the input and output is larger than that in text-to-spectrogram generation.

Original: It is difﬁcult to train on the audio clip that corresponds to the full text sequence due to the extremely long waveform samples and limited GPU memory.
Revised: It is difﬁcult to train on the audio clip that corresponds to the full text sequence due to the extremely long waveform samples and limited GPU memory.

Original: As a result, we can only train on a short audio clip that corresponds to a partial text sequence which makes it hard for the model to capture the relationship among phonemes in different partial text sequences and thus harms the text feature extraction.
Revised: As a result, we can only train on a short audio clip that corresponds to a partial text sequence which makes it hard for the model to capture the relationship among phonemes in different partial text sequences and thus harms the text feature extraction.

Original: Our Method To tackle the challenges above, we make several designs in the waveform decoder: 1)
Revised: Our Method To tackle the challenges above, we make several designs in the waveform decoder: 1)

Original: Considering that the phase information is difﬁcult to predict using a variance predictor (Engel et al., 2020), we introduce adversarial training in the waveform decoder to force it to implicitly recover the phase information by itself (Yamamoto et al., 2020).
Revised: Considering that the phase information is difﬁcult to predict using a variance predictor (Engel et al., 2020), we introduce adversarial training in the waveform decoder to force it to implicitly recover the phase information by itself (Yamamoto et al., 2020).

Original: We leverage the mel-spectrogram decoder of FastSpeech 2, which is trained on the full text sequence to help on the text feature extraction.
Revised: We leverage the mel-spectrogram decoder of FastSpeech 2, which is trained on the full text sequence to help on the text feature extraction.

Original: As shown in Figure 1d, the waveform decoder is based on the structure of WaveNet (Van Den Oord et al., 2016) including non-causal convolutions and gated activation (Van den Oord et al., 2016).
Revised: As shown in Figure 1d, the waveform decoder is based on the structure of WaveNet (Van Den Oord et al., 2016) including non-causal convolutions and gated activation (Van den Oord et al., 2016).

Original: The waveform decoder takes a sliced hidden sequence corresponding to a short audio clip as input and upsamples it with transposed 1D-convolution to match the length of audio clip.
Revised: The waveform decoder takes a sliced hidden sequence corresponding to a short audio clip as input and upsamples it with transposed 1D-convolution to match the length of audio clip.

Original: The discriminator in the adversarial training adopts the same structure in Parallel WaveGAN (Yamamoto et al., 2020).
Revised: The discriminator in the adversarial training adopts the same structure in Parallel WaveGAN (Yamamoto et al., 2020) which consists of ten layers of non-causal dilated 1-D convolutions with leaky ReLU activation function.

Original: The waveform decoder is optimized by the multi-resolution STFT loss and the  discriminator loss following Parallel WaveGAN.
Revised: The waveform decoder is optimized by the multi-resolution STFT loss and the LSGAN discriminator loss following Parallel WaveGAN.

Original: In inference, we discard the mel-spectrogram decoder and only use the waveform decoder to synthesize speech audio.
Revised: In inference, we discard the mel-spectrogram decoder and only use the waveform decoder to synthesize speech audio.

Original: 2. D ISCUSSIONS
Revised: 2. D ISCUSSIONS

Original: In this subsection, we discuss how FastSpeech 2 and 2s differentiate from previous and concurrent works.
Revised: In this subsection, we discuss how FastSpeech 2 and 2s differentiate from previous and concurrent works.

Original: Compared with Deep Voice (Arik et al., 2017) and Deep Voice 2 (Gibiansky et al., 2017)which generate waveform autoregressively and also predict variance information such as duration and pitch,
Revised: Compared with Deep Voice (Arik et al., 2017), Deep Voice 2 (Gibiansky et al., 2017) and other methods Fan et al.

Original:  Fastspeech 2 and 2s adopt self-attention based feed-forward network to generate mel-spectrograms or waveform in parallel.
Revised: Ze et al. (2013) which generate waveform autoregressively and also predict variance information such as duration and pitch, Fastspeech 2 and 2s adopt self-attention based feed-forward network to generate mel-spectrograms or waveform in parallel.

Original: While some existing non-autoregressive acoustic models (Zeng et al., 2020; Lim et al., 2020; Kim et al., 2020) mostly focus on improving the duration accuracy, FastSpeechand 2s provide more variation information (duration, pitch and energy) as inputs to reduce the information gap between the input and output.
Revised: While some existing non-autoregressive acoustic models (Zeng et al., 2020; Lim et al., 2020; Kim et al., 2020) mostly focus on improving the duration accuracy, FastSpeech 2 and 2s provide more variation information (duration, pitch and energy) as inputs to reduce the information gap between the input and output.

Original: A concurrent work (Ła ´ ncucki, 2020) employs pitch prediction in phoneme level, while FastSpeech 2 and 2s predict more ﬁne-grained pitch contour in frame level.
Revised: A concurrent work (Ła ´ ncucki, 2020) employs pitch prediction in phoneme level, while FastSpeechand 2s predict more ﬁne-grained pitch contour in frame level.

Original: In addition,  FastSpeech 2 and 2s can better predict the variations in pitch using continuous wavelet transform, which results in better prosody in synthesized speech.
Revised: In addition, to improve the prosody in synthesized speech, FastSpeech 2 and 2s further introduce continuous wavelet transform to model the variations in pitch.

Original: While some text-to-waveform models such as ClariNet (Ping et al., 2019) jointly train an autoregressive acoustic model and a non-autoregressive vocoder, FastSpeech 2s embraces the fully nonautoregressive architecture for fast inference.
Revised: While some text-to-waveform models such as ClariNet (Ping et al., 2019) jointly train an autoregressive acoustic model and a non-autoregressive vocoder, FastSpeech 2s embraces the fully nonautoregressive architecture for fast inference.

Original: A concurrent work called EATS (Donahue et al., 2020) also employs non-autoregressive architecture and adversarial training to convert text to waveform directly and mainly focuses on predicting the duration of each phoneme end-to-end using a differentiable monotonic interpolation scheme.
Revised: A concurrent work called EATS (Donahue et al., 2020) also employs non-autoregressive architecture and adversarial training to convert text to waveform directly and mainly focuses on predicting the duration of each phoneme end-to-end using a differentiable monotonic interpolation scheme.

Original: Compared with EATS, FastSpeech 2s additionally provides more variation information to ease the one-to-many mapping problem in TTS.
Revised: Compared with EATS, FastSpeech 2s additionally provides more variation information to ease the one-to-many mapping problem in TTS.

Original: Previous non-autoregressive vocoders (Oord et al., 2017; Prenger et al., 2019; Yamamoto et al., 2020; Kumar et al., 2019) are not complete text-to-speech systems, since they convert time aligned linguistic features to waveforms, and require a separate linguistic model to convert input text to linguistic features or an acoustic model to convert input text to acoustic features (e.g., melspectrograms).
Revised: Previous non-autoregressive vocoders (Oord et al., 2017; Prenger et al., 2019; Yamamoto et al., 2020; Kumar et al., 2019) are not complete text-to-speech systems, since they convert time aligned linguistic features to waveforms, and require a separate linguistic model to convert input text to linguistic features or an acoustic model to convert input text to acoustic features (e.g., melspectrograms).

Original: FastSpeech 2s is the ﬁrst attempt to directly generate waveform from phoneme sequence, instead of linguistic features or mel-spectrograms.
Revised: FastSpeech 2s is the ﬁrst attempt to directly generate waveform from phoneme sequence fully in parallel, instead of linguistic features or mel-spectrograms.

Original:  E XPERIMENTS AND R ESULTS 3. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS AND R ESULTS 3. E XPERIMENTAL S ETUP

Original: Datasets We evaluate FastSpeech 2 and 2s on LJSpeech dataset (Ito, 2017).
Revised: Datasets We evaluate FastSpeech 2 and 2s on LJSpeech dataset (Ito, 2017).

Original: LJSpeech contains 13,100 English audio clips (about 24 hours) and corresponding text transcripts.
Revised: LJSpeech contains 13,100 English audio clips (about 24 hours) and corresponding text transcripts.

Original: We split the dataset into three sets: 12,228 samples for training, 349 samples (with document title LJ003) for validation and 523 samples (with document title LJ001 and LJ002) for testing.
Revised: We split the dataset into three sets: 12,228 samples for training, 349 samples (with document title LJ003) for validation and 523 samples (with document title LJ001 and LJ002) for testing.

Original: For subjective evaluation, we randomly choose 100 samples in test set.
Revised: For subjective evaluation, we randomly choose 100 samples in test set.

Original: To alleviate the mispronunciation problem, we convert the text sequence into the phoneme sequence (Arik et al., 2017;
Revised: To alleviate the mispronunciation problem, we convert the text sequence into the phoneme sequence (Arik et al., 2017;

Original: Wang et al., 2017; Shen et al., 2018) with an open-source grapheme-to-phoneme tool 4 .
Revised: Wang et al., 2017; Shen et al., 2018) with an open-source grapheme-to-phoneme tool 5 .

Original: We transform the raw waveform into melspectrograms following Shen et al.
Revised: We transform the raw waveform into melspectrograms following Shen et al.

Original: and set frame size and hop size to 1024 and 256 with respect to the sample rate 22050.
Revised: and set frame size and hop size to 1024 and 256 with respect to the sample rate 22050.

Original: Model Conﬁguration Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al., 2019) in the encoder and the mel-spectrogram decoder.
Revised: Model Conﬁguration Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al., 2019) in the encoder and the mel-spectrogram decoder.

Original: The output linear layer in the decoder converts the hidden states into 80-dimensional mel-spectrograms and our model is optimized with mean absolute error (MAE).
Revised: The output linear layer in the decoder converts the hidden states into 80-dimensional mel-spectrograms and our model is optimized with mean absolute error (MAE).

Original: We add more detailed conﬁgurations of FastSpeechand 2s used in our experiments in Appendix A.
Revised: We add more detailed conﬁgurations of FastSpeechand 2s used in our experiments in Appendix A.

Original: The details of training and inference are added in
Revised: The details of training and inference are added in

Original: Appendix B.
Revised: Appendix B.

Original: 3. R ESULTS
Revised: 3. R ESULTS

Original: [Table]
Revised: [Table]

Original: In this section, we ﬁrst evaluate the audio quality, training and inference speedup of FastSpeech 2 and 2s.
Revised: In this section, we ﬁrst evaluate the audio quality, training and inference speedup of FastSpeechand 2s.

Original: Then we conduct analyses and ablation studies of our method 5 .
Revised: Then we conduct analyses and ablation studies of our method 6 .

Original: 3.2. M ODEL P ERFORMANCE
Revised: 3.2. M ODEL P ERFORMANCE

Original: Audio Quality To evaluate the perceptual quality, we perform mean opinion score (MOS)
Revised: Audio Quality To evaluate the perceptual quality, we perform mean opinion score (MOS)

Original: (Chu & Peng, 2006) evaluation on the test set.
Revised: (Chu & Peng, 2006) evaluation on the test set.

Original: Twenty native English speakers are asked to make quality judgments about the synthesized speech samples.
Revised: Twenty native English speakers are asked to make quality judgments about the synthesized speech samples.

Original: The text content keeps consistent among different systems so that all testers only examine the audio quality without other interference factors.
Revised: The text content keeps consistent among different systems so that all testers only examine the audio quality without other interference factors.

Original: We compare the MOS of the audio samples generated by FastSpeech 2 and FastSpeech 2s with other systems, including 1) GT , the ground-truth recordings; 2) GT (Mel + PWG) , where we ﬁrst convert the ground-truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using Parallel WaveGAN (Yamamoto et al., 2020)
Revised: We compare the MOS of the audio samples generated by FastSpeech 2 and FastSpeech 2s with other systems, including 1) GT , the ground-truth recordings; 2) GT (Mel + PWG) , where we ﬁrst convert the ground-truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using Parallel WaveGAN (Yamamoto et al., 2020)

Original: (PWG); 3) Tacotron 2 (Shen et al., 2018)
Revised: (PWG); 3) Tacotron 2 (Shen et al., 2018)

Original: (Mel + PWG) ; 4) Transformer TTS (Li et al., 2019) (Mel + PWG) ; 5) FastSpeech (Ren et al., 2019)
Revised: (Mel + PWG) ; 4) Transformer TTS (Li et al., 2019) (Mel + PWG) ; 5) FastSpeech (Ren et al., 2019)

Original: (Mel + PWG) .
Revised: (Mel + PWG) .

Original: All the systems in 3), 4) and 5) use Parallel WaveGAN as the vocoder for a fair comparison.
Revised: All the systems in 3), 4) and 5) use Parallel WaveGAN as the vocoder for a fair comparison.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: It can be seen that FastSpeech 2 can surpass and
Revised: It can be seen that FastSpeech 2 can surpass and

Original: FastSpeech 2s can match the voice quality of autoregressive models Transformer TTS and Tacotron2 .
Revised: FastSpeech 2s can match the voice quality of autoregressive models Transformer TTS and Tacotron

Original:  Importantly, FastSpeech 2 outperforms FastSpeech, which demonstrates the effectiveness of providing variance information such as pitch, energy and more accurate duration and directly takingground-truth speech as training target without using teacher-student distillation pipeline.
Revised: [Table] Importantly, FastSpeech 2 outperforms FastSpeech, which demonstrates the effectiveness of providing variance information such as pitch, energy and more accurate duration and directly taking

Original: Training and Inference Speedup FastSpeech 2 simpliﬁes the training pipeline of FastSpeech by removing the teacher-student distillation process, and thus reduces the training time.
Revised: Training and Inference Speedup FastSpeech 2 simpliﬁes the training pipeline of FastSpeech by removing the teacher-student distillation process, and thus reduces the training time.

Original: We list the total training time of Transformer TTS (the autoregressive teacher model), FastSpeech (including the training of Transformer TTS teacher model and FastSpeech student model) and FastSpeech 2 in
Revised: We list the total training time of Transformer TTS (the autoregressive teacher model), FastSpeech (including the training of Transformer TTS teacher model and FastSpeech student model) and FastSpeech 2 in

Original: Table 2.
Revised: Table 2.

Original: It can be seen that FastSpeech 2 reduces the total training time by 3 .
Revised: It can be seen that FastSpeech 2 reduces the total training time by 3 .

Original: 12 × compared with
Revised: 12 × compared with

Original: FastSpeech.
Revised: FastSpeech.

Original: Note that training time here only includes acoustic model training, without considering the vocoder training.
Revised: Note that training time here only includes acoustic model training, without considering the vocoder training.

Original: Therefore, we do not compare the training time of FastSpeech 2s here.
Revised: Therefore, we do not compare the training time of FastSpeech 2s here.

Original: We then evaluate the inference latency of FastSpeech 2 and 2s compared with the autoregressive Transformer
Revised: We then evaluate the inference latency of FastSpeech 2 and 2s compared with the autoregressive Transformer

Original: TTS model, which has the similar number of model parameters with FastSpeech 2 and 2s.
Revised: TTS model, which has the similar number of model parameters with FastSpeech 2 and 2s.

Original: We show the inference speedup for waveform generation in Table 2.
Revised: We show the inference speedup for waveform generation in Table 2.

Original: It can be seen that compared with the
Revised: It can be seen that compared with the

Original: Transformer TTS model, FastSpeech 2 and 2s speeds up the audio generation by 147 × and 167 ×[Table] respectively in waveform synthesis.
Revised: Transformer TTS model, FastSpeech 2 and 2s speeds up the audio generation by 147 × and 167 × respectively in waveform synthesis.

Original:  2s  faster than due to 
Revised: We can also see that FastSpeech 2s is faster than FastSpeechdue to fully end-to-end generation

Original: 3.2. A NALYSES ON V ARIANCE I NFORMATION
Revised: 3.2. A NALYSES ON V ARIANCE I NFORMATION

Original: [Table]
Revised: [Table]

Original: More Accurate Variance Information in Synthesized Speech In the paragraph, we measure if providing more variance information (e.g., pitch and energy) as input in FastSpeech 2 and 2s can indeed synthesize speech with more accurate pitch and energy.
Revised: More Accurate Variance Information in Synthesized Speech In the paragraph, we measure if providing more variance information (e.g., pitch and energy) as input in FastSpeech 2 and 2s can indeed synthesize speech with more accurate pitch and energy.

Original: For pitch, we compute the moments (standard deviation ( σ ), skewness ( γ ) and kurtosis ( K ))
Revised: For pitch, we compute the moments (standard deviation ( σ ), skewness ( γ ) and kurtosis ( K ))

Original: (Andreeva et al., 2014; Niebuhr & Skarnitzl, 2019)of the pitch distribution for the ground-truth speech and synthesized speech. The results are shown in Table 3.
Revised: (Andreeva et al., 2014; Niebuhr & Skarnitzl, 2019) and average dynamic time warping (DTW results are shown in Table 3.

Original: It can be seen that compared with FastSpeech, the moments ( σ , γ and K ) of generated audio of FastSpeech 2/2s are more close to the ground-truth audio, demonstrating that FastSpeech 2/2s can generate speech with more natural pitch contour (which can result in better prosody) than FastSpeech.
Revised: It can be seen that compared with FastSpeech, the moments ( σ , γ and K ) of generated audio of FastSpeech 2/2s are more close to the ground-truth audio and the average DTW distances to the ground-truth pitch are smaller than other methods, demonstrating that FastSpeech 2/2s can generate speech with more natural pitch contour (which can result in better prosody) than FastSpeech.

Original: We also conduct a case study on generated pitch contours in Appendix D.
Revised: We also conduct a case study on generated pitch contours in Appendix D.

Original: [Table]
Revised: [Table]

Original: For energy, we compute the mean absolute error (MAE) between the frame-wise energyextracted from the generated waveform and the groundtruth speech.
Revised: For energy, we compute the mean absolute error (MAE) between the frame-wise energy from the generated waveform and the ground-truth speech.

Original: To ensure that the numbers of frames in the synthesized and ground-truth speech are the same, we use the ground-truth duration extracted by MFA in both FastSpeech and FastSpeech 2.
Revised: To ensure that the numbers of frames in the synthesized and ground-truth speech are the same, we use the ground-truth duration extracted by MFA in both FastSpeech and FastSpeech 2.

Original: The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: We can see that the MAE of the energy for FastSpeech 2/2s are smaller than that for FastSpeech, indicating that they both synthesize speech audio with more similar energy to the ground-truth audio.
Revised: We can see that the MAE of the energy for FastSpeech 2/2s are smaller than that for FastSpeech, indicating that they both synthesize speech audio with more similar energy to the ground-truth audio.

Original: More Accurate Duration for Model Training We then analyze the accuracy of the provided duration information to train the duration predictor and the effectiveness of more accurate duration for better voice quality based on FastSpeech.
Revised: More Accurate Duration for Model Training We then analyze the accuracy of the provided duration information to train the duration predictor and the effectiveness of more accurate duration for better voice quality based on FastSpeech.

Original: We manually align 50 audio generated by the teacher model and the corresponding text in phoneme level and get the ground-truth phoneme-level duration.
Revised: We manually align 50 audio generated by the teacher model and the corresponding text in phoneme level and get the ground-truth phoneme-level duration.

Original: We compute the average of absolute phoneme boundary differences (McAuliffe et al., 2017) using the duration from the teacher model of FastSpeech and from MFA as  in this paper respectively.
Revised: We compute the average of absolute phoneme boundary differences (McAuliffe et al., 2017) using the duration from the teacher model of FastSpeech and from MFA as used in this paper respectively.

Original: The results are shown in Table 5a.
Revised: The results are shown in Table 5a.

Original: We can see that MFA can generate more accurate duration than the teacher model of FastSpeech.
Revised: We can see that MFA can generate more accurate duration than the teacher model of FastSpeech.

Original: Next, we replace the duration used in FastSpeech (from teacher model) with that extracted by MFA, and conduct the CMOS (Loizou, 2011) test to compare the voice quality between the two FastSpeech models trained with different durations 6 .
Revised: Next, we replace the duration used in FastSpeech (from teacher model) with that extracted by MFA, and conduct the CMOS (Loizou, 2011) test to compare the voice quality between the two FastSpeech models trained with different durations 8 .

Original: The results are listed in Table 5b and it can be seen that more accurate duration information improves the voice quality of
Revised: The results are listed in Table 5b and it can be seen that more accurate duration information improves the voice quality of

Original: FastSpeech, which veriﬁes the effectiveness of our improved duration from MFA.
Revised: FastSpeech, which veriﬁes the effectiveness of our improved duration from MFA.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3.2. A BLATION S TUDY
Revised: 3.2. A BLATION S TUDY

Original: Pitch and Energy Input We conduct ablation studies to demonstrate the effectiveness of several variance information of FastSpeech 2 and 2s, including pitch and energy 7 .
Revised: Pitch and Energy Input We conduct ablation studies to demonstrate the effectiveness of several variance information of FastSpeech 2 and 2s, including pitch and energy 9 .

Original: We conduct CMOS evaluation for these ablation studies.
Revised: We conduct CMOS evaluation for these ablation studies.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: We ﬁnd that removing the energy (Row 3 in both subtables) in FastSpeech 2 and 2s results in performance drop in terms of voice quality (-0.040 and -0.160 CMOS respectively), indicating that energy is effective for FastSpeechin improving the voice quality [Table]
Revised: We ﬁnd that removing the energy (Row 3 in both subtables) in FastSpeech 2 and 2s results in performance drop in terms of voice quality (-0.040 and -0.160 CMOS respectively), indicating that energy is effective for FastSpeechin improving the voice quality, and more effective for FastSpeech 2s.

Original: and more effective for FastSpeech 2s. We also ﬁnd that removing the pitch (Row 4 in both subtables) in FastSpeech 2 and 2s results in -0.245 and -1.130 CMOS respectively, which demonstrates the effectiveness of pitch.
Revised: We also ﬁnd that removing the pitch (Row 4 in both subtables) in FastSpeech 2 and 2s results in -0.245 and -1.130 CMOS respectively, which demonstrates the effectiveness of pitch.

Original: When we remove both pitch and energy (the last row in both subtables), the voice quality further drops, indicating that both pitch and energy can help improve the performance of FastSpeech 2 and 2s.
Revised: When we remove both pitch and energy (the last row in both subtables), the voice quality further drops, indicating that both pitch and energy can help improve the performance of FastSpeech 2 and 2s.

Original: Predicting Pitch in Frequency Domain To study the effectiveness of predicting pitch in frequency domain using continuous wavelet transform (CWT) as described in Section 2.3, we directly ﬁt the pitch contour with mean square error like energy in FastSpeech 2 and 2s.
Revised: Predicting Pitch in Frequency Domain To study the effectiveness of predicting pitch in frequency domain using continuous wavelet transform (CWT) as described in Section 2.3, we directly ﬁt the pitch contour with mean square error like energy in FastSpeech 2 and 2s.

Original: We conduct CMOS evaluation and get CMOS drops of 0.185 and 0.201 for FastSpeech 2 and 2s respectively.
Revised: We conduct CMOS evaluation and get CMOS drops of 0.185 and 0.201 for FastSpeech 2 and 2s respectively.

Revised: We also compute the moments of pitch and average DTW distance to the ground-truth pitch as shown in row 6 (denoeted as FastSpeech 2 - CWT ) in Table 3.

Original: The results demonstrate that CWT can help model the pitch better and improve the prosody of synthesized speech, and thus obtaining better CMOS score.
Revised: The results demonstrate that CWT can help model the pitch better and improve the prosody of synthesized speech, and thus obtaining better CMOS score.

Original: Mel-Spectrogram Decoder in FastSpeech 2s
Revised: Mel-Spectrogram Decoder in FastSpeech 2s

Original: To verify the effectiveness of the mel-spectrogram decoder in FastSpeech 2s on text feature extraction as described in Section 2.4, we remove the mel-spectrogram decoder and conduct CMOS evaluation.
Revised: To verify the effectiveness of the mel-spectrogram decoder in FastSpeech 2s on text feature extraction as described in Section 2.4, we remove the mel-spectrogram decoder and conduct CMOS evaluation.

Original: It causes a 0.285 CMOS drop, which demonstrates that the mel-spectrogram decoder is essential to high-quality waveform generation.
Revised: It causes a 0.285 CMOS drop, which demonstrates that the mel-spectrogram decoder is essential to high-quality waveform generation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we proposed FastSpeech 2, a fast and high-quality end-to-end TTS system, to address the issues in FastSpeech and ease the one-to-many mapping problem: 1) we directly train the model with ground-truth mel-spectrograms to simplify the training pipeline and also avoid information loss compared with FastSpeech; and 2) we improve the duration accuracy and introduce more variance information including pitch and energy to ease the one-to-many mapping problem, and improve pitch prediction by introducing continuous wavelet transform.
Revised: In this work, we proposed FastSpeech 2, a fast and high-quality end-to-end TTS system, to address the issues in FastSpeech and ease the one-to-many mapping problem: 1) we directly train the model with ground-truth mel-spectrograms to simplify the training pipeline and also avoid information loss compared with FastSpeech; and 2) we improve the duration accuracy and introduce more variance information including pitch and energy to ease the one-to-many mapping problem, and improve pitch prediction by introducing continuous wavelet transform.

Original: Moreover, based on FastSpeech 2, we further developed FastSpeech 2s, a non-autoregressive text-to-waveform generation model, which enjoys the beneﬁt of fully end-to-end inference and achieves faster inference speed.
Revised: Moreover, based on FastSpeech 2, we further developed FastSpeech 2s, a non-autoregressive text-to-waveform generation model, which enjoys the beneﬁt of fully end-to-end inference and achieves faster inference speed.

Original: Our experimental results show that FastSpeech 2 and 2s outperform FastSpeech, and FastSpeech 2 can even surpass autoregressive models in terms of voice quality, with much simpler training pipeline while inheriting the advantages of fast, robust and controllable speech synthesis of FastSpeech.
Revised: Our experimental results show that FastSpeech 2 and 2s outperform FastSpeech, and FastSpeech 2 can even surpass autoregressive models in terms of voice quality, with much simpler training pipeline while inheriting the advantages of fast, robust and controllable speech synthesis of FastSpeech.

Revised: High quality, fast and fully end-to-end training without any external libraries is deﬁnitely the ultimate goal of neural TTS and also a very challenging problem.

Revised: To ensure high quality of FastSpeech 2, we use an external high-performance alignment tool and pitch extraction tools, which may seem a little complicated, but are very helpful for high-quality and fast speech synthesis.

Original: In the future, we will  consider more variance information to further improve the voice quality andwill further speed up the inference with more light-weight model.We will also work on fully end-to-end non-autoregressive TTS without external alignment models and tools.
Revised: We believe there will be more simpler solutions to achieve this goal in the future and we will certainly work on fully end-to-end TTS without external alignment models and tools. We will also consider more variance information to further improve the voice quality and speed up the inference with more light-weight model.

Original: A M ODEL C ONFIGURATION
Revised: A M ODEL C ONFIGURATION

Original: Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al., 2019) in the encoder and the mel-spectrogram decoder.
Revised: Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al., 2019) in the encoder and the mel-spectrogram decoder.

Original: In each FFT block, the dimension of phoneme embeddings and the hidden size of the self-attention are set to 256.
Revised: In each FFT block, the dimension of phoneme embeddings and the hidden size of the self-attention are set to 256.

Original: The number of attention heads is set to 2 and the kernel sizes of the 1D-convolution in the 2-layer convolutional network after the self-attention layer are set to 9 and 1, with input/output size of 256/1024 for the ﬁrst layer and 1024/256 in the second layer.
Revised: The number of attention heads is set to 2 and the kernel sizes of the 1D-convolution in the 2-layer convolutional network after the self-attention layer are set to 9 and 1, with input/output size of 256/1024 for the ﬁrst layer and 1024/256 in the second layer.

Original: The size of the phoneme vocabulary is 76, including punctuations.
Revised: The size of the phoneme vocabulary is 76, including punctuations.

Original: In the variance predictor, the kernel sizes of the 1D-convolution are set to 3, with input/output sizes of 256/256 for both layers and the dropout rate is set to 0.5.
Revised: In the variance predictor, the kernel sizes of the 1D-convolution are set to 3, with input/output sizes of 256/256 for both layers and the dropout rate is set to 0.5.

Original: Our waveform decoder consists of 1-layer transposed 1D-convolution with ﬁlter size 64 and 30 dilated residual convolution blocks, whose skip channel size and kernel size of 1D-convolution are set to 64 and 3.
Revised: Our waveform decoder consists of 1-layer transposed 1D-convolution with ﬁlter size 64 and 30 dilated residual convolution blocks, whose skip channel size and kernel size of 1D-convolution are set to 64 and 3.

Original: The conﬁgurations of the discriminator in
Revised: The conﬁgurations of the discriminator in

Original: FastSpeech 2s are the same as Parallel WaveGAN (Yamamoto et al., 2020).
Revised: FastSpeech 2s are the same as Parallel WaveGAN (Yamamoto et al., 2020).

Original: We list hyperparameters and conﬁgurations of all models used in our experiments in Table 7.
Revised: We list hyperparameters and conﬁgurations of all models used in our experiments in Table 7.

Original: Hyperparameter
Revised: Hyperparameter

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Kernel size /
Revised: Kernel size /

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B T RAINING AND I NFERENCE
Revised: B T RAINING AND I NFERENCE

Original: We train FastSpeech 2 on 1 NVIDIA V100 GPU, with batchsize of 48 sentences.
Revised: We train FastSpeech 2 on 1 NVIDIA V100 GPU, with batchsize of 48 sentences.

Original: We use the Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 .
Revised: We use the Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 , ε = 10 − 9 and follow the same learning rate schedule in Vaswani et al.
Revised: 98 , ε = 10 − 9 and follow the same learning rate schedule in Vaswani et al.

Original: It takes 160k steps for training until convergence.
Revised: It takes 160k steps for training until convergence.

Original: In the inference process, the output mel-spectrograms of our FastSpeech 2 are transformed into audio samples using pre-trained Parallel WaveGAN (Yamamoto et al., 2020) 8 .
Revised: In the inference process, the output mel-spectrograms of our FastSpeech 2 are transformed into audio samples using pre-trained Parallel WaveGAN (Yamamoto et al., 2020) 10 .

Original: For FastSpeech 2s, we train the model on 2 NVIDIA V100 GPUs, with batchsize of 6 sentences on each GPU.
Revised: For FastSpeech 2s, we train the model on 2 NVIDIA V100 GPUs, with batchsize of 6 sentences on each GPU.

Original: The waveform decoder takes the sliced hidden states corresponding to 20,480 waveform sample clips as input.
Revised: The waveform decoder takes the sliced hidden states corresponding to 20,480 waveform sample clips as input.

Original: The optimizer and learning rate schedule for FastSpeech 2s are the same as FastSpeech 2.
Revised: The optimizer and learning rate schedule for FastSpeech 2s are the same as FastSpeech 2.

Original: The details of the adversarial training follow Parallel WaveGAN (Yamamoto et al., 2020).
Revised: The details of the adversarial training follow Parallel WaveGAN (Yamamoto et al., 2020).

Original: It takes 600k steps for training until convergence for FastSpeech 2s.
Revised: It takes 600k steps for training until convergence for FastSpeech 2s.

Original: C M ODELING P ITCH WITH C ONTINUOUS W AVELET T RANSFORM C. C ONTINUOUS W AVELET T RANSFORM
Revised: C M ODELING P ITCH WITH C ONTINUOUS W AVELET T RANSFORM C. C ONTINUOUS W AVELET T RANSFORM

Original: Given a continous pitch contour function F 0 , we can convert it to pitch spectrogram W ( τ, t ) using continuous wavelet transform (Tuteur, 1988; Grossmann & Morlet, 1984):
Revised: Given a continous pitch contour function F 0 , we can convert it to pitch spectrogram W ( τ, t ) using continuous wavelet transform (Tuteur, 1988; Grossmann & Morlet, 1984):

Original: [Equation]
Revised: [Equation]

Original: where ψ is the Mexican hat mother wavelet (Ryan, 1994), F 0 ( x ) is the pitch value in position x , τ and t are scale and position of wavelet respectively.
Revised: where ψ is the Mexican hat mother wavelet (Ryan, 1994), F 0 ( x ) is the pitch value in position x , τ and t are scale and position of wavelet respectively.

Original: The original pitch contour F 0 can be recovered from the wavelet representation W ( τ, t ) by inverse continuous wavelet transform (iCWT) using the following formula:
Revised: The original pitch contour F 0 can be recovered from the wavelet representation W ( τ, t ) by inverse continuous wavelet transform (iCWT) using the following formula:

Original: [Equation]
Revised: [Equation]

Original: Suppose that we decompose the pitch contour F 0 into 10 scales (Ming et al., 2016), F 0 can be represented by 10 separate components given by:
Revised: Suppose that we decompose the pitch contour F 0 into 10 scales (Ming et al., 2016), F 0 can be represented by 10 separate components given by:

Original: [Equation]
Revised: [Equation]

Original: where i = 1 , ..., 10 and τ 0 = 5 ms , which is originally proposed in Suni et al. (2013).
Revised: where i = 1 , ..., 10 and τ 0 = 5 ms , which is originally proposed in Suni et al. (2013).

Original: Givenwavelet components ˆ W i ( t ) , we can recompose pitch contour ˆ F 0 by the following formula (Ming et al., 2016):
Revised: Givenwavelet components ˆ W i ( t ) , we can recompose pitch contour ˆ F 0 by the following formula (Ming et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: C. 
Revised: C. I MPLEMENTATION D ETAILS

Original: I MPLEMENTATION D ETAILS

Original: [Figure]
Revised: [Figure]

Original: [Figure] and iCWT denote continuous wavelet transform and inverse continuous wavelet transform respectively.
Revised: CWT and iCWT denote continuous wavelet transform and inverse continuous wavelet transform respectively.

Original: First we extract the pitch contour using PyWorldVocoder 9 .
Revised: First we extract the pitch contour using PyWorldVocoder 11 .

Original: Since CWT is very sensitive to discontinuous signals, we preprocess the pitch contour as follows: 1) we use linear interpolation to ﬁll the unvoiced frame in pitch contour; 2) we transform the resulting pitch contour to logarithmic scale; 3) we normalize it to zero mean and unit variance for each utterance, and we have to save the original utterance-level mean and variance for pitch contour reconstruction; and 4) we convert the normalized pitch contour to pitch spectrogram using continuous wavelet transform following Equation 1.
Revised: Since CWT is very sensitive to discontinuous signals, we preprocess the pitch contour as follows: 1) we use linear interpolation to ﬁll the unvoiced frame in pitch contour; 2) we transform the resulting pitch contour to logarithmic scale; 3) we normalize it to zero mean and unit variance for each utterance, and we have to save the original utterance-level mean and variance for pitch contour reconstruction; and 4) we convert the normalized pitch contour to pitch spectrogram using continuous wavelet transform following Equation 1.

Original: As shown in Figure 2, pitch predictor consists of a 2-layer 1Dconvolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to project the hidden states into the pitch spectrogram.
Revised: As shown in Figure 2, pitch predictor consists of a 2-layer 1Dconvolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to project the hidden states into the pitch spectrogram.

Original: We also need another linear layer to predict the mean/variance of recovered pitch contour for each utterance.
Revised: To predict the mean/variance of recovered pitch contour for each utterance, we average the hidden states output by the 1D-convolutional network on the time dimension to a global vector and project it to mean and variance using a linear layer.

Original: We train the pitch predictor with ground-truth pitch spectrogram and the mean/variance of pitch contour and optimize it with mean square error.
Revised: We train the pitch predictor with ground-truth pitch spectrogram and the mean/variance of pitch contour and optimize it with mean square error.

Original: During inference, we predict the pitch spectrogram and the mean/variance of recovered pitch contour using pitch predictor, inverse the pitch spectrogram to pitch contour with inverse continuous wavelet transform (iCWT) following Equation 2, and ﬁnally denormalize it with the predicted mean/variance.
Revised: During inference, we predict the pitch spectrogram and the mean/variance of recovered pitch contour using pitch predictor, inverse the pitch spectrogram to pitch contour with inverse continuous wavelet transform (iCWT) following Equation 2, and ﬁnally denormalize it with the predicted mean/variance.

Original: D C ASE S TUDY ON P ITCH C ONTOUR
Revised: D C ASE S TUDY ON P ITCH C ONTOUR

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we conduct the case study on pitch contours of the audios generated by different methods.
Revised: we conduct the case study on pitch contours of the audios generated by different methods.

Original: We randomly choose 1 utterance from the test set and plot the pitch countor of groundtruth audio samples and that generated by FastSpeech , FastSpeech 2 , FastSpeech 2s in Figure 3.
Revised: We randomly choose 1 utterance from the test set and plot the pitch countor of groundtruth audio samples and that generated by FastSpeech , FastSpeech 2 , FastSpeech 2s in Figure 3.

Original: We can see that FastSpeech 2 and 2s can capture the variations in pitch better than FastSpeech thanks to taking pitch information as input
Revised: We can see that FastSpeech 2 and 2s can capture the variations in pitch better than FastSpeech thanks to taking pitch information as input

Original: [Figure]
Revised: [Figure]

Original: E V ARIANCE C ONTROL
Revised: E V ARIANCE C ONTROL

Original: FastSpeech 2 and 2s introduce several variance information to ease the one-to-many mapping problem in TTS.
Revised: FastSpeech 2 and 2s introduce several variance information to ease the one-to-many mapping problem in TTS.

Original: As a byproduct, they also make the synthesized speech more controllable and can be used to manually control pitch, duration and energy (volume) of synthesized audio.
Revised: As a byproduct, they also make the synthesized speech more controllable and can be used to manually control pitch, duration and energy (volume) of synthesized audio.

Original: As a demonstration, we manipulate pitch input to control the pitch of synthesized speech in this subsubsection.
Revised: As a demonstration, we manipulate pitch input to control the pitch of synthesized speech in this subsubsection.

Original: We show the mel-spectrograms before and after the pitch manipulation in Figure 4.
Revised: We show the mel-spectrograms before and after the pitch manipulation in Figure 4.

Original: From the samples, we can see that FastSpeech 2 generates high-quality mel-spectrograms after adjusting the ˆ Ffrom 0.75 to 1.50 times.
Revised: From the samples, we can see that FastSpeech 2 generates high-quality mel-spectrograms after adjusting the ˆ Ffrom 0.75 to 1.50 times.

Original: Such manipulation can also be applied to FastSpeech 2s and the results are put in the supplementary materials.
Revised: Such manipulation can also be applied to FastSpeech 2s and the results are put in the supplementary materials.

Original: We also put the audio samples controlled by other variance information in supplementary materials.
Revised: We also put the audio samples controlled by other variance information in supplementary materials.

Original: [Figure]
Revised: [Figure]