Original: DPP-
Revised: DPP-

Original: TTS: D IVERSIFYING PROSODIC FEATURES OF SPEECH VIA DETERMINANTAL POINT PROCESSES
Revised: TTS: D IVERSIFYING PROSODIC FEATURES OF SPEECH VIA DETERMINANTAL POINT PROCESSES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: With the rapid advancement in deep generative models, recent neural text-tospeech models have succeeded in synthesizing human-like speech, even in an endto-end manner.
Revised: With the rapid advancement in deep generative models, recent neural text-tospeech models have succeeded in synthesizing human-like speech, even in an endto-end manner.

Original: However, many synthesized samples often have a monotonous speaking style or simply follow the speaking style of their ground-truth samples.
Revised: However, many synthesized samples often have a monotonous speaking style or simply follow the speaking style of their ground-truth samples.

Original: Although there have been many proposed methods to increase the diversity of prosody in speech, increasing prosody variance in speech often hurts the naturalness of speech.
Revised: Although there have been many proposed methods to increase the diversity of prosody in speech, increasing prosody variance in speech often hurts the naturalness of speech.

Original: Recently, Determinantal point processes (DPPs) have shown remarkable results for modeling diversity in a wide range of machine learning tasks.
Revised: Recently, Determinantal point processes (DPPs) have shown remarkable results for modeling diversity in a wide range of machine learning tasks.

Original: However, their application in speech synthesis has not been explored.
Revised: However, their application in speech synthesis has not been explored.

Original: To enhance the expressiveness of speech, we propose DPP-TTS: a text-to-speech model based on a determinantal point process to diversify prosody in speech.
Revised: To enhance the expressiveness of speech, we propose DPP-TTS: a text-to-speech model based on a determinantal point process to diversify prosody in speech.

Original: The extent of prosody diversity can be easily controlled by adjusting parameters in our model.
Revised: The extent of prosody diversity can be easily controlled by adjusting parameters in our model.

Original: We demonstrate that DPP-TTS generates more expressive samples than baselines in the side-by-side comparison test while not harming the naturalness of the speech.
Revised: We demonstrate that DPP-TTS generates more expressive samples than baselines in the side-by-side comparison test while not harming the naturalness of the speech.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the past few years, text-to-speech models have made a lot of progress in synthesizing human-like speech (Shen et al., 2018;
Revised: In the past few years, text-to-speech models have made a lot of progress in synthesizing human-like speech (Shen et al., 2018;

Original: Ping et al., 2018; Li et al., 2019; Ren et al., 2019).
Revised: Ping et al., 2018; Li et al., 2019; Ren et al., 2019).

Original: Furthermore, in the latest studies, several text-to-speech models made high-quality speech samples even in the end-toend setting without a two-stage synthesis process (Donahue et al., 2021;
Revised: Furthermore, in the latest studies, several text-to-speech models made high-quality speech samples even in the end-toend setting without a two-stage synthesis process (Donahue et al., 2021;

Original: Kim et al., 2021).
Revised: Kim et al., 2021).

Original: Based on these technical developments, text-to-speech models are now able to generate high-ﬁdelity speech.
Revised: Based on these technical developments, text-to-speech models are now able to generate high-ﬁdelity speech.

Original: However, human speech contains many prosodic features like intonation, stress, and rhythm beyond textual features.
Revised: However, human speech contains many prosodic features like intonation, stress, and rhythm beyond textual features.

Original: Therefore, it is crucial to generate speech samples with rich prosody.
Revised: Therefore, it is crucial to generate speech samples with rich prosody.

Original: There have been many attempts to synthesize speech with rich prosodic features.
Revised: There have been many attempts to synthesize speech with rich prosodic features.

Original: To incorporate rich prosodic features into speech, reference acoustic samples like mel-spectrograms are processed through reference encoder (Skerry-Ryan et al., 2018; Wang et al., 2018) or text-to-speech models are conditioned on prosodic features like duration, pitch and energy then these features are predicted or manually controlled at inference (Ren et al., 2021; La’ncucki, 2021).
Revised: To incorporate rich prosodic features into speech, reference acoustic samples like mel-spectrograms are processed through reference encoder (Skerry-Ryan et al., 2018; Wang et al., 2018) or text-to-speech models are conditioned on prosodic features like duration, pitch and energy then these features are predicted or manually controlled at inference (Ren et al., 2021; La´ncucki, 2021).

Original: However, these methods have drawbacks that the bottleneck dimension should be carefully tuned for desirable results, or the prosody predictor just learns averaged prosodic features of training sets.
Revised: However, these methods have drawbacks that the bottleneck dimension should be carefully tuned for desirable results, or the prosody predictor just learns averaged prosodic features of training sets.

Original: Meanwhile, generative models like VAEs and ﬂow models (Hsu et al., 2019; Sun et al., 2020; Vall’es-P’erez et al., 2021; Lee et al., 2021; Valle et al., 2021) have been recently used for speech synthesis and their latent spaces are manipulated for generating more expressive speech.
Revised: Meanwhile, generative models like VAEs and ﬂow models (Hsu et al., 2019; Sun et al., 2020; Vall´esP´erez et al., 2021; Lee et al., 2021; Valle et al., 2021) have been recently used for speech synthesis and their latent spaces are manipulated for generating more expressive speech.

Original: Controlling the amount of variation in the speech is achieved by generating samples in Gaussian prior with an adequate temperature, however, it has two major drawbacks.
Revised: Controlling the amount of variation in the speech is achieved by generating samples in Gaussian prior with an adequate temperature, however, it has two major drawbacks.

Original: First, generating latent samples with high variance often make generated samples to be unstable in terms of the naturalness and intelligibility of speech.
Revised: First, generating latent samples with high variance often make generated samples to be unstable in terms of the naturalness and intelligibility of speech.

Original: Second, sampling in the latent space might only cover major modes learned during the training stage and only monotonous prosodic patterns can occur in some segments of speech.
Revised: Second, sampling in the latent space might only cover major modes learned during the training stage and only monotonous prosodic patterns can occur in some segments of speech.

Original: Determinantal point processes (DPPs) have shown great results for modeling diversity in various machine learning tasks and their applications.
Revised: Determinantal point processes (DPPs) have shown great results for modeling diversity in various machine learning tasks and their applications.

Original: Its uses include text summarization (Cho et al., 2019), recommendation systems (Gartrell et al., 2021), multi-modal output generation (Elfeki et al., 2019), diverse trajectory forecasting (Yuan & Kitani, 2020), and machine translation (Meister et al., 2021).
Revised: Its uses include text summarization (Cho et al., 2019), recommendation systems (Gartrell et al., 2021), multi-modal output generation (Elfeki et al., 2019), diverse trajectory forecasting (Yuan & Kitani, 2020), and machine translation (Meister et al., 2021).

Original: Determinantal point process (DPPs) offers us an efﬁcient subset selection method by considering both the quality and diversity of items within a set.
Revised: Determinantal point process (DPPs) offers us an efﬁcient subset selection method by considering both the quality and diversity of items within a set.

Original: Speciﬁcally, items within a set are sampled according to DPPs kernel which reﬂects the quality and diversity of items in the ground set.
Revised: Speciﬁcally, items within a set are sampled according to DPPs kernel which reﬂects the quality and diversity of items in the ground set.

Original: In some cases, we may need to incorporate the conditional information into the sampling process.
Revised: In some cases, we may need to incorporate the conditional information into the sampling process.

Original: To this end, conditional DPP (Kulesza & Taskar, 2012) has been widely used for sampling subset by setting the kernel conditioned on the speciﬁc information.
Revised: To this end, conditional DPP (Kulesza & Taskar, 2012) has been widely used for sampling subset by setting the kernel conditioned on the speciﬁc information.

Original: To use conditional DPP for sampling prosodic features of speech, two main issues should be considered.
Revised: To use conditional DPP for sampling prosodic features of speech, two main issues should be considered.

Original: First, prosodic features are usually varying in length.
Revised: First, prosodic features are usually varying in length.

Original: Second, it is ambiguous what the ground set is in this problem.
Revised: Second, it is ambiguous what the ground set is in this problem.

Original: In this work, we resolve these issues by using a new similarity metric between prosodic features and adding a prosodic diversifying module (PDM) into the framework.
Revised: In this work, we resolve these issues by using a new similarity metric between prosodic features and adding a prosodic diversifying module (PDM) into the framework.

Original: Speciﬁcally, PDM generates candidates of prosodic features of speech by mapping latent codes from normal distribution to a new latent space and these candidates are used as the ground set in the sampling process.
Revised: Speciﬁcally, PDM generates candidates of prosodic features of speech by mapping latent codes from normal distribution to a new latent space and these candidates are used as the ground set in the sampling process.

Original: In addition, the similarity between two prosodic features is evaluated using soft dynamic time warping discrepancy (Soft-DTW) (Cuturi & Blondel, 2017) which enables to evaluate the similarity between features in different lengths.
Revised: In addition, the similarity between two prosodic features is evaluated using soft dynamic time warping discrepancy (Soft-DTW) (Cuturi & Blondel, 2017) which enables to evaluate the similarity between features in different lengths.

Original: The kernel matrix of conditional DPP is learned during the training by parameters of PDM getting updated.
Revised: The kernel matrix of conditional DPP is learned during the training by parameters of PDM getting updated.

Original: Speciﬁcally, parameters of the prosody diversifying module (PDM) are updated in the training stage with conditional maximum induced cardinality (MIC) objective which is adapted from
Revised: Speciﬁcally, parameters of the prosody diversifying module (PDM) are updated in the training stage with conditional maximum induced cardinality (MIC) objective which is adapted from

Original: MIC objective introduced in Gillenwater et al.
Revised: MIC objective introduced in Gillenwater et al.

Original: We formally introduce the conditional MIC objective and its derivative for clarity in Section 4.
Revised: We formally introduce the conditional MIC objective and its derivative for clarity in Section 4.

Original: To implement the aforementioned DPP, a stochastic duration predictor and pitch predictor are introduced in this work.
Revised: To implement the aforementioned DPP, a stochastic duration predictor and pitch predictor are introduced in this work.

Original: Both the duration and pitch predictor are built upon normalizing ﬂows trained with the maximum likelihood (MLE) objective.
Revised: Both the duration and pitch predictor are built upon normalizing ﬂows trained with the maximum likelihood (MLE) objective.

Original: At inference, the prosody predictor maps latent codes from a latent space to feature spaces with the inverse ﬂow.
Revised: At inference, the prosody predictor maps latent codes from a latent space to feature spaces with the inverse ﬂow.

Original: In experiments, we compare DPP-TTS with the state-of-the-art models including VITS (Kim et al., 2021) and Flowtron (Valle et al., 2021) in terms of prosody diversity and speech quality.
Revised: In experiments, we compare DPP-TTS with the state-of-the-art models including VITS (Kim et al., 2021) and Flowtron (Valle et al., 2021) in terms of prosody diversity and speech quality.

Original: The results demonstrate that our model generates speech with richer prosody than baselines while maintaining speech naturalness.
Revised: The results demonstrate that our model generates speech with richer prosody than baselines while maintaining speech naturalness.

Original: We also demonstrate that our DPP-TTS can be used in real-time applications by evaluating the inference speed of our model.
Revised: We also demonstrate that our DPP-TTS can be used in real-time applications by evaluating the inference speed of our model.

Original: In summary, our contributions follow:
Revised: In summary, our contributions follow:

Original: • We propose a novel method for diversifying prosodic features of speech based on conditional DPPs by considering prosodic features of context words as conditions.
Revised: • We propose a novel method for diversifying prosodic features of speech based on conditional DPPs by considering prosodic features of context words as conditions.

Original: • To learn the kernel matrix of conditional DPPs, we propose to train prosody diversifying module (PDM) with the conditional maximum induced cardinality (MIC) objective.
Revised: • To learn the kernel matrix of conditional DPPs, we propose to train prosody diversifying module (PDM) with the conditional maximum induced cardinality (MIC) objective.

Original: • Experiments on the side-by-side comparison and the MOS test verify that our model outperforms the two baseline models in terms of prosodic diversity while maintaining the naturalness of speech.
Revised: • Experiments on the side-by-side comparison and the MOS test verify that our model outperforms the two baseline models in terms of prosodic diversity while maintaining the naturalness of speech.

Original:  B ACKGROUND 2. D ETERMINANTAL POINT PROCESSES
Revised:  B ACKGROUND 2. D ETERMINANTAL POINT PROCESSES

Original: The core method in this work is that prosodic features of target words are sampled according to conditional DPPs kernel which reﬂects the similarity between the target and contexts.

Original: First, we will introduce DPPs before conditional DPPs.

Original: DPPs encourage diversity within a set by discouraging sampling similar items within the ground set.
Revised: DPPs encourage diversity within a set by discouraging sampling similar items within the ground set.

Original: Formally, point process P is called a determinantal point process when Y is a random subset drawn according to P , we have, for every A ⊆ Y ,
Revised: Formally, point process P is called a determinantal point process when Y is a random subset drawn according to P , we have, for every A ⊆ Y ,

Original: [Equation]
Revised: [Equation]

Original: where K is a positive deﬁnite matrix whose eigenvalues are all between 0 and 1 and K A is a positive a deﬁnite matrix indexed by elements in A .
Revised: where K is a positive deﬁnite matrix whose eigenvalues are all between 0 and 1 and K A is a positive a deﬁnite matrix indexed by elements in A .

Original: The marginal probability of including two elements e i and e j is K ii K jj − K 2 ij = p
Revised: The marginal probability of including two elements e i and e j is K ii K jj − K 2 ij = p

Original: ( e i ∈ Y ) p ( e j ∈ Y ) − K 2 ij .
Revised: ( e i ∈ Y ) p ( e j ∈ Y ) − K 2 ij .

Original: Therefore, the value of K 2 ij models the extent of negative correlation between item i and j .
Revised: Therefore, the value of K 2 ij models the extent of negative correlation between item i and j .

Original: More frequently, DPPs are deﬁned by
Revised: More frequently, DPPs are deﬁned by

Original: L-ensemble through real and symmetric matrix L instead of the marginal kernel K :
Revised: L-ensemble through real and symmetric matrix L instead of the marginal kernel K :

Original: [Equation]
Revised: [Equation]

Original: where det ( L + I ) in the denominator acts as a normalization constant.
Revised: where det ( L + I ) in the denominator acts as a normalization constant.

Original: The marginal kernel K and kernel L has the following relation:
Revised: The marginal kernel K and kernel L has the following relation:

Original: [Equation]
Revised: [Equation]

Original: To model diversity between items, the DPP kernel L is usually constructed as a symmetric similarity matrix, where S ij represents the similarity between two items x i and x j .
Revised: To model diversity between items, the DPP kernel L is usually constructed as a symmetric similarity matrix, where S ij represents the similarity between two items x i and x j .

Original: In Kulesza & Taskar (2010), authors proposed decomposing the kernel L as a a Gram matrix incorporating a quality vector to weigh each item according to its quality :
Revised: In Kulesza & Taskar (2010), authors proposed decomposing the kernel L as a a Gram matrix incorporating a quality vector to weigh each item according to its quality :

Original: [Equation]
Revised: [Equation]

Original: where φ i ∈ R D ; D ≤ N and || φ i || 2 = 1 .
Revised: where φ i ∈ R D ; D ≤ N and || φ i || 2 = 1 .

Original: In this manner, the similarity matrix S is guaranteed to be real positive semideﬁnite matrix.
Revised: In this manner, the similarity matrix S is guaranteed to be real positive semideﬁnite matrix.

Original: 2. C ONDITIONAL DETERMINANTAL POINT PROCESSES
Revised: 2. C ONDITIONAL DETERMINANTAL POINT PROCESSES

Revised: If DPPs are used for diversifying prosodic features by setting targets of DPPs as prosodic features of sentences, it would result in diversity among generated samples.

Revised: However, there still can be monotonous patterns in each generated speech.

Revised: To resolve this issue, it is required to diversify prosodic features accounting into their neighbor prosodic features.

Revised: Therefore, we need conditional

Revised: DPPs to take into account neighboring prosodic features (contexts).

Original: By setting conditions of  P , DPPs can be extended to conditional DPPs, For  B ⊆ Y not intersecting with A we have
Revised: By setting conditions of point process P , DPPs can be extended to conditional DPPs, For a subset B ⊆ Y not intersecting with A we have

Original: [Equation]
Revised: [Equation]

Original: where I ¯ A is the matrix with ones in the diagonal entries indexed by elements of Y − A and zeros elsewhere.
Revised: where I ¯ A is the matrix with ones in the diagonal entries indexed by elements of Y − A and zeros elsewhere.

Original: In Borodin & Rains (2004), authors showed that this conditional distribution is again a
Revised: In Borodin & Rains (2004), authors showed that this conditional distribution is again a

Original: DPP, with a kernel given by
Revised: DPP, with a kernel given by

Original: [Equation]
Revised: [Equation]

Original: In conditional DPPs, items in the ground set Y are sampled according to kernel given contexts.
Revised: In conditional DPPs, items in the ground set Y are sampled according to kernel given contexts.

Original: In this work, contexts of target words are used as conditions for conditional DPPs.
Revised: In this work, contexts of target words are used as conditions for conditional DPPs.

Original: 2. S OFT D YNAMIC TIME WARPING
Revised: 2. S OFT D YNAMIC TIME WARPING

Original: To build the kernel of conditional DPPs, we need a measure of similarity between two temporal signals.
Revised: To build the kernel of conditional DPPs, we need a measure of similarity between two temporal signals.

Original: Simple Euclidean distance is not applicable because two time signals often vary in their lengths.
Revised: Simple Euclidean distance is not applicable because two time signals often vary in their lengths.

Original: In this work, to calculate the similarity between two temporal features, soft dynamic time warping (Soft DTW) discrepancy is adapted.
Revised: In this work, to calculate the similarity between two temporal features, soft dynamic time warping (Soft DTW) discrepancy is adapted.

Original: DTW computes the best possible alignment between two temporal signals.
Revised: DTW computes the best possible alignment between two temporal signals.

Original: First, for the length of n and m signals, DTW computes n by m pairwise distance matrix between these points with a speciﬁc metric (e.g., Euclidean distance, L1 distance).
Revised: First, for the length of n and m signals, DTW computes n by m pairwise distance matrix between these points with a speciﬁc metric (e.g., Euclidean distance, L1 distance).

Original: After that, this matrix is used to solve the dynamic program using Bellman’s recursion with a quadratic
Revised: After that, this matrix is used to solve the dynamic program using Bellman’s recursion with a quadratic

Original: O ( mn ) cost.
Revised: O ( mn ) cost.

Original: Unfortunately, vanilla DTW cost is not easy to optimize because it only considers a single alignment between two temporal signals.
Revised: Unfortunately, vanilla DTW cost is not easy to optimize because it only considers a single alignment between two temporal signals.

Original: Cuturi & Blondel (2017) proposed differentiable Soft-DTW which considers all possible alignments between two temporal series.
Revised: Cuturi & Blondel (2017) proposed differentiable Soft-DTW which considers all possible alignments between two temporal series.

Original: Soft-DTW is differentiable in all of its arguments with quadratic cost.
Revised: Soft-DTW is differentiable in all of its arguments with quadratic cost.

Original: Formally, Soft-DTW is deﬁned as follows:
Revised: Formally, Soft-DTW is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where A n,m denotes all possible alignments between x and y and min γ is deﬁned as
Revised: where A n,m denotes all possible alignments between x and y and min γ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: A small magnitude of γ reﬂects the true discrepancy of two temporal signals, however, optimization becomes unstable.
Revised: A small magnitude of γ reﬂects the true discrepancy of two temporal signals, however, optimization becomes unstable.

Original: T EXT - TO - SPEECH MODEL
Revised: [Figure]  DPP-TTS

Original: In this section, we ﬁrst describe the architecture of  the text-to-speech model before adding prosody diversifying module (PDM). Our text-to-speech model is adapted from FastSpeech2 (Ren et al., 2021) and comprised the main seq2seq module and stochastic prosody predictormodules. The overall architecture is depicted in Figure 1a.
Revised: Our model DPP-TTS is composed of a Seq2Seq module for generating the mel-spectrogram, a prosody predictor for predicting duration and pitch sequences, and a prosody diversifying module (PDM). At the ﬁrst stage, the base TTS model which consists of the Seq2Seq module and the prosody predictor is trained as shown in Figure 1a.

Original: [Figure]
Revised: Once the base TTS is trained, PDM is inserted in front of the prosody predictor and trained with the method which will be described in detail in Section 4. We describe the main modules of DPP-TTS and their roles in the following subsection.

Original: 3. S EQ 2S EQ MODULE
Revised: 3. M AIN MODULES OF DPP-TTS

Original:  Seq2seq model for generating mel-spectrograms from phoneme sequences is adapted from FastSpeech2  with some modiﬁcations.
Revised: Seq2Seq module The role of the Seq2seq module is generating mel-spectrograms from phoneme sequences. The module is adapted from FastSpeech2 (Ren et al., 2021) with some modiﬁcations.

Original: The model consists of four main parts: a phoneme prenet, a phoneme encoder, a variance adaptor, and a mel-spectrogram decoder.
Revised: The model consists of four main parts: a phoneme prenet, a phoneme encoder, a variance adaptor, and a mel-spectrogram decoder.

Original: In the phoneme encoder, phoneme sequences are processed through a stack of feed-forward transformer blocks with a relative positional representation (Shaw et al., 2018).
Revised: In the phoneme encoder, phoneme sequences are processed through a stack of feed-forward transformer blocks with a relative positional representation (Shaw et al., 2018).

Original: Pitch embeddings and energy embeddings are added to encoded hidden representations and then hidden representations are expanded according to ground-truth duration labels which were obtained via monotonic alignment search (Kim et al., 2020) between phonemes and mel-spectrograms.
Revised: In variance adaptor at training, pitch embeddings and energy embeddings are added to encoded hidden representations and then hidden representations are expanded according to groundtruth duration labels 1 . At inference, these prosodic features are provided from predictions of the prosody predictor.

Original: Finally, expanded representations are processed through a stack of feed-forward transformer blocks and mel-spectrograms with 80 channels are generated after the linear projection.
Revised: Finally, expanded representations are processed through a stack of feed-forward transformer blocks and mel-spectrograms with 80 channels are generated after the linear projection.

Original: The seq2seq module is trained with L 1 distance between the predicted and target mel-spectrogram.
Revised: The Seq2Seq module is trained to minimize L 1 distance between the predicted and target mel-spectrogram

Original:  In FastSpeech2, the variance adaptor consists of deterministic predictors for predicting prosodic features.
Revised: Prosody predictor In FastSpeech2, the variance adaptor consists of deterministic predictors for predicting prosodic features.

Original: However, a deterministic prosodic predictor is not expressive enough to learn the speaking style of a person.
Revised: However, a deterministic prosodic predictor is not expressive enough to learn the speaking style of a person.

Original: For diverse rhythm and pitch, a stochastic duration predictor and pitch predictor are built upon normalizing ﬂows.
Revised: For diverse rhythm and pitch, a stochastic duration predictor and pitch predictor are built upon normalizing ﬂows.

Original: Speciﬁcally, the stochastic duration predictor estimates the distribution of phoneme duration and the stochastic pitch predictor estimates the distribution of phoneme pitch from the hidden sequence.
Revised: Speciﬁcally, the stochastic duration predictor estimates the distribution of phoneme duration and the stochastic pitch predictor estimates the distribution of phoneme pitch from the hidden sequence.

Original: At the training stage, the prosody predictor learns the mapping from the distribution of prosodic features to normal distribution.
Revised: At the training stage, the prosody predictor learns the mapping from the distribution of prosodic features to normal distribution.

Original: At inference, it predicts the phoneme duration or pitch by reversing the learned ﬂows.
Revised: At inference, it predicts the phoneme-level duration or pitch by reversing the learned ﬂows.

Original: As the prosody pattern of human speech is correlated with the local context and global context of the text, we design the architecture of the model to encode both the local and global features. Speciﬁcally, dilated convolution blocks are stacked to encode local features and a transformer block is used to encode global features in the coupling layers of normalizing ﬂows.
Revised: In addition, it also serves as the density estimator for prosodic features during the training of PDM which will be described in detail in Section 4. The prosody predictor is trained to maximize a variational lower bound of the likelihood of the phoneme duration or pitch.

Original: However, using the vanilla transformer to encode the global features in the coupling layer requires too large computational complexity.
Revised: More details regarding the prosody predictor are in Appendix A.

Original: Therefore, Nystr¨omformer (Xiong et al., 2021) which is a Nystr¨om-Based algorithm for approximating self-attention is used to encode global context for more efﬁcient memory usage in the coupling layer of normalizing ﬂow. Encoded local features and global features are concatenated and processed through LayerNorm (Ba et al., 2016), Mish activation (Misra, 2019) and a 1x convolution module. Following the duration predictor model in Kim et al. (2021), variational dequantization (Ho et al., 2019) is used for the duration predictor since the phoneme duration is a discrete integer.
Revised: PDM

Original: In addition, variational augmentation (Chen et al., 2020) is used to expand channel dimensions for expressive ﬂows in both theduration and pitch predictor. The stochastic  and pitch predictor are trained by maximizing their variational lower bounds (ELBO).
Revised: Although the stochastic duration and pitch predictor are trained to generate a speech with diverse rhythm and pitch, the prosody predictor may favors major modes and it leads to the monotonous prosodic pattern in the speech.

Original: Details of the duration and pitch predictor are described in Appendix A.
Revised: For more expressive speech modeling, PDM is added in front of the prosody predictor as shown in Figure 1b.

Original: P ROSODY DIVERSIFYING MODULE FOR DIVERSE PROSODIC FEATURES
Revised: Its role is to guide latent codes from a standard normal distribution to another distribution for diverse prosodic features of speech.

Original: In this section, we explain the prosody diversifying module(PDM) which is trained with the loss based on maximum induced cardinality (MIC), the process of building DPP kernel, andﬁnally the conditional (MIC) objective for learning the kernel.
Revised: This module is trained with an objective based on conditional DPPs which is described in Section 2.2. At inference of DPP-TTS, multiple prosodic candidates are generated by PDM and the prosodic feature of speech is selected via MAP inference.

Original: Although the stochastic duration and pitch predictor are trained for generating a speech with diverse rhythm and pitch, it is still not guaranteed that the synthesized sample with features from learned duration and pitch distribution gives us human-level expressivity.
Revised: [Figure]

Original: To address this issue, conditional DPPs are introduced with the aim to learn the kernel producing diverse prosodic features.
Revised: In this section, we explain the methodology for training the prosody diversifying module (PDM). The training process mainly consists of three steps: segmentation of an input text, generation of prosodic feature candidates, building DPP kernel.

Original: A prosody diversifying module (PDM) is inserted in front of the prosody predictor to parameterize thekernel of conditional DPPs.
Revised: We also explain the conditional maximum induced cardinality (MIC) objective for training PDM. The overall training procedure is depicted in

Original: What parts of texts should be processed by PDM?
Revised: Figure 2b. 4. S EGMENTATION OF AN INPUT TEXT

Original: Although there exists theoretical phonological hierarchy (Buuren, 1978) which starts from an utterance in the top level to syllables and phonemes in the low level, it is still challenging to generalize the basic segment of prosody events across people.
Revised: In this stage, targets in input text for diversiﬁcation of prosody are chosen. The scale of each target is important for the desired goal. Phoneme-level targets would result in too dynamic prosody of generated speech. In contrast, sentence-level targets may result in a monotonous pattern in each speech corresponding to each sentence.

Original: In this work, as an alternative, noun phrases extracted from the sentence using
Revised: Therefore, we choose a few words as each target in the input text. However, naively choosing random words in the input text is not desirable because prosodic boundaries and syntax of the sentence are closely related (Cole et al., 2010).

Original:  Spacy 1 libraryare processed by PDM with an expectation to get the prosodic diversity in  the phrase level.
Revised: We choose the noun phrases in the input text as each target using Spacy 2 library to take account into the syntactic structure of the input text. Neighboring words with the same number of words as the target are selected as the left of right context. For example, as shown in Figure 2a, if the target is an Atreides , then way for and to take are selected as the left and right contexts respectively.

Original: Overview of PDM process First, context hidden sequences are processed by the trained prosody predictor and context prosodic features like duration and pitch are predicted.
Revised: Indices of targets and contexts are stored after the segmentation. 4. G ENERATION OF PROSODIC FEATURE CANDIDATES In this stage, multiple prosodic candidates are generated for the ground set of DPP as shown in

Original: In Figure 2, these predicted features are denoted as d L and d R , each represents the left context and right context.
Revised: Figure 2b. First, an input text is fed into the pre-trained phoneme prenet and encoder and a hidden sequence is generated. Second, the pre-trained prosody predictor conditioned on the hidden sequence predicts the whole prosodic features of the input text from the latent codes samples from a normal distribution.

Original: Simultaneously, PDM takes hidden target and context sequences from the output of text encoder as inputs and outputs latent code z ∈ R N × T , where N denotes the number of candidates, and T denotes the length of the target sequence. The latent codes z are processed through the prosody predictor  and N prosodic features are predicted. The overall process is shown in Figure 2.
Revised: Third, new n c latent codes from a normal distribution are fed into PDM, then the prosody predictor conditioned on hidden sequence corresponds to the target and its neighboring context predicts new n c prosodic features. Finally, newly generated target prosodic sequences and previously predicted prosodic sequences are mixed, then n c prosodic candidates are generated with the target and context entangled.

Original: 4.  DPP KERNELBUILD
Revised: 4. C ONSTRUCTION OF DPP KERNEL

Original: Based on predicted prosodic features of the context d L , d R and target sequences d 1 ,..., d N ,  the  kernel of conditional DPP is built incorporating both diversity and quality of the  set.
Revised: Generated candidates are split into left and right context d L , d R and n targets d 1 , d 2 , ... d n , then the ground set for DPP is constructed merging the targets and contexts as shown in Figure 3. The kernel of conditional DPP is built incorporating both diversity and quality of the ground set.

Original: As mentioned, the kernel of DPP can be decomposed as L =
Revised: As mentioned in Section 2.1, the kernel of DPP can be decomposed as L =

Original: Diag ( q ) · S · Diag ( q ) , where q denotes the quality vectors of predicted features and S denotes the similarity matrix.
Revised: Diag ( q ) · S · Diag ( q ) , where q denotes the quality vectors of predicted features and S denotes the similarity matrix.

Original: Similarity metric Target sequences and context sequences often vary in lengths, therefore the Euclidean distance is not applicable to calculate the similarity between two sequences.
Revised: Similarity metric Target sequences and context sequences often vary in lengths, therefore the Euclidean distance is not applicable to calculate the similarity between two sequences.

Original: The similarity metric is calculated as follows based on Soft DTW:
Revised: The similarity metric is calculated as follows based on Soft DTW:

Original: [Equation]
Revised: [Equation]

Original:  dtw D γ denotes soft-DTW discrepancy with a metric D and smoothing parameter γ .
Revised: [Equation] dtw Dγ denotes soft-DTW discrepancy with a metric D and smoothing parameter γ .

Original: When the metric D is chosen as the L 1 distance D ( x , y ) = (cid:80) i | x i − y
Revised: When the metric D is chosen as the L 1 distance D ( x , y ) = (cid:80) i | x i − y

Original: i | or half gaussian D ( x , y ) =
Revised: i | or half gaussian D ( x , y ) =

Original: || x − y ||
Revised: || x − y ||

Original: + log(2 − exp( −|| x − y || 22 ))
Revised: + log(2 − exp( −|| x − y || 22 ))

Original: , the similarity matrix is positive semi-deﬁnite (Cuturi et al., 2007; Blondel et al., 2021).
Revised: , the similarity matrix is positive semi-deﬁnite (Cuturi et al., 2007; Blondel et al., 2021).

Original: In this work, L 1 distance is used as the metric of Soft-DTW so that S to be positive semi-deﬁnite.
Revised: In this work, L 1 distance is used as the metric of Soft-DTW so that S to be positive semi-deﬁnite.

Original: Quality metric Based on  estimated density of predicted features, quality scores are calculated.
Revised: Quality metric Based on the estimated density of predicted features, quality scores are calculated.

Original: The density values of predicted features are calculated with importance sampling: p ( x ; θ ) ≈ 
Revised: The density values of predicted features are calculated with importance sampling using the prosody predictor: p ( x ; θ ) ≈ (cid:80) N i =1 p ( x , z

Original: [Equation] Ni =1 p ( x , z ; θ  q ( z  | x ; φ ) .
Revised: i ; θ ) q ( z i | x ; φ ) .

Original: In the experiment, using raw density value resulted in not enough diverse sample, therefore quality values are not penalized if the density value is greater than a speciﬁc threshold.
Revised: In the experiment, using raw density value resulted in not enough diverse sample, therefore quality values are not penalized if the density value is greater than a speciﬁc threshold.

Original: With log likelihood p ( x ) , the quality score is deﬁned as
Revised: With log likelihood p ( x ) , the quality score is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The threshold value k was set based on the average density of the dataset.
Revised: The threshold value k was set as the average density of the dataset in the experiment.

Original: Larger w value gives more weight on the quality term.

Original: In other words, samples with high density are more likely to be generated than diverse samples.

Original: Finally, the kernel of conditional DPPs is deﬁnes as L = diag ( q ) · S · diag ( q ) .
Revised: Finally, the kernel of conditional DPPs is deﬁnes as L = diag ( q ) · S · diag ( q ) .

Original: A schematic diagram of building the conditional DPPs kernel is in Figure 3.

Original: 4. O BJECTIVE FUNCTION
Revised: 4. O BJECTIVE FUNCTION

Original: We need a measure of diversity with respect to kernel L to train the PDM.
Revised: We need a measure of diversity with respect to kernel L to train the PDM.

Original: One straightforward choice is maximum likelihood (MLE) objective, log P L ( Y ) =  det( L ) / det( L  + I ) .
Revised: One straightforward choice is maximum likelihood (MLE) objective, log P L ( Y ) = log det( L ) − log det( L Y + I ) .

Original: However, there are some cases where two identical prosodic features(especially duration sequences) are sampled.
Revised: However, there are some cases where almost identical prosodic features are predicted.

Original: This leads the objective value to be close to zero which makes the training process unstable.
Revised: This leads the objective value to be close to zero which makes the training process unstable.

Original: Instead, maximum induced cardinality (MIC) (Gillenwater et al., 2018) objective which is deﬁned as E Y ∼P
Revised: Instead, maximum induced cardinality (MIC) (Gillenwater et al., 2018) objective which is deﬁned as E Y ∼P

Original: L [ | Y | ] can be an alternative.
Revised: L [ | Y | ] can be an alternative.

Original: It does not suffer from training instability since identical items in the ground set Y are allowed.
Revised: It does not suffer from training instability since identical items in the ground set Y are allowed.

Original: In this work, conditional MIC objective is used since the expected cardinality of the set without considering context is meaningless.
Revised: In this work, conditional MIC objective is used as setting contexts d L , d R as the condition.

Original: The objective function with respect to the ground set [ d L , d R , d 1 , d 2 , ..., d N ] and its derivative are  follows:
Revised: The objective function with respect to the ground set [ d L , d R , d 1 , d 2 , ..., d N ] and its derivative are as follows:

Original: Proposition 1 ( MIC objective of conditional DPPs).
Revised: Proposition 1 ( MIC objective of conditional DPPs).

Original: With respect to the ground set [ d L , d R , d 1 , d 2 , ..., d N ] , the MIC objective of conditional DPPs and its derivative are  follows:
Revised: With respect to the ground set [ d L , d R , d 1 , d 2 , ..., d N ] , the MIC objective of conditional DPPs and its derivative are as follows:

Original: [Equation]
Revised: [Equation]

Original: where A denotes the set of contexts ( d L , d R ) , ¯ A denotes the complement of the set A and θ is the parameter of PDM.
Revised: where A denotes the set of contexts ( d L , d R ) , ¯ A denotes the complement of the set A and θ is the parameter of PDM.

Original: Detailed proof is presented in Appendix B. At inference,  MAP inference is performed across sets with just a single item: [Equation] The inference of DPP-TTS is described in Figure 1b.
Revised: Detailed proof is presented in Appendix B. At inference, for predicting a single prosodic feature, MAP inference is performed across sets with just a single item as follows: x ∗ = arg max x ∈ ¯ A log det( L { x }∪ A ) .

Original: The detailed procedure of training PDM and the inference of DPP-TTS are in Appendix C.
Revised: The detailed procedure of training PDM and the inference of DPPTTS are in Appendix C.

Original: The loss function for training the PDM is the opposite of the conditional MIC objective.

Original: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrst explain the experimental setup and then human evaluation results and further evaluation results that are conducted to evaluate the proposed DPP-TTS.
Revised: In this section, we ﬁrst explain the experimental setup and then human evaluation results and further evaluation results that are conducted to evaluate the proposed DPP-TTS.

Original: Audio samples used for the evaluation are in the supplementary material.
Revised: Audio samples used for the evaluation are in the supplementary material.

Original: 5. E XPERIMENTAL SETUP
Revised: 5. E XPERIMENTAL SETUP

Original: Dataset and preprocessing We conducted experiments on the LJ Speech dataset which consists of audio clips with approximately 24 hours lengths.
Revised: Dataset and preprocessing We conducted experiments on the LJ Speech dataset which consists of audio clips with approximately 24 hours lengths.

Original: We split audio samples into 12500/100/ samples for the training, validation, and test set.
Revised: We split audio samples into 12500/100/ samples for the training, validation, and test set.

Original: Audio samples with 22kHz sampling rate are transformed into 80 bands mel-spectrograms through the Short-time Fourier transform (STFT) with 1024 window size and 256 hop length.
Revised: Audio samples with 22kHz sampling rate are transformed into 80 bands mel-spectrograms through the Short-time Fourier transform (STFT) with 1024 window size and 256 hop length.

Original: International Phonetic Alphabet (IPA) sequences are used as input for phoneme encoder.
Revised: International Phonetic Alphabet (IPA) sequences are used as input for phoneme encoder.

Original: Text sequences are converted to IPA phoneme sequences using
Revised: Text sequences are converted to IPA phoneme sequences using

Original: Phonemizer 2 software.
Revised: Phonemizer 3 software.

Original: Following Kim et al.
Revised: Following Kim et al.

Original: (2020), the converted sequences are interspersed with blank tokens, which represent the transition from one phoneme to another phoneme.
Revised: (2020), the converted sequences are interspersed with blank tokens, which represent the transition from one phoneme to another phoneme.

Original: Training  Both the basic TTS and PDM are trained using the AdamW optimizer (Loshchilov & Hutter, 2019) with β 1 = 0 . 8 , β 2 = 0 .
Revised: Training Setup Both the basic TTS and PDM are trained using the AdamW optimizer (Loshchilov & Hutter, 2019) with β 1 = 0 . 8 , β 2 = 0 .

Original: 99 and λ = 0 .
Revised: 99 and λ = 0 .

Original: The initial learning rate is 2 × 10 − 4 for the basic TTS and 1 × 10 − 5 for the PDM and the exponential learning decay is used for both of them.
Revised: The initial learning rate is 2 × 10 − 4 for the basic TTS and 1 × 10 − 5 for the PDM and the exponential learning decay is used for both of them.

Original: Basic TTS is trained with 16 batch size for 200k steps on 2 NVIDIA RTX 3080 GPUs.
Revised: Basic TTS is trained with 16 batch size for 200k steps on 2 NVIDIA RTX 3080 GPUs.

Original: PDM is trained with 8 batch size for 5k steps on same devices.
Revised: PDM is trained with 8 batch size for 5k steps on same devices.

Revised: In both the duration and pitch diversiﬁcation modeling, the quality weight of the model is set as w = 10 .

Original: Baseline We compare our model with two state-of-the-art models in terms of naturalness and prosody diversity.
Revised: Baseline We compare our model with two state-of-the-art models in terms of naturalness and prosody diversity.

Original: The ﬁrst one is VITS (Kim et al., 2021) which is an end-to-end TTS model based on conditional VAE and normalizing ﬂows.
Revised: The ﬁrst one is VITS (Kim et al., 2021) which is an end-to-end TTS model based on conditional VAE and normalizing ﬂows.

Original: In all experiments, the temperature σ of the duration predictor in VITS was set 0.8.
Revised: In all experiments, the temperature σ of the duration predictor in VITS was set 0.8.

Original: The second one is Flowtron (Valle et al., 2021) which is an autoregressive ﬂow-based TTS model.
Revised: The second one is Flowtron (Valle et al., 2021) which is an autoregressive ﬂow-based TTS model.

Original: In experiments, the temperature σ of Flowtron was set to 0.6.
Revised: In experiments, the temperature σ of Flowtron was set to 0.6.

Original: For our DPP-TTS and Flowtron, HiFi-GAN (Kong et al., 2020) is used as the vocoder for synthesizing waveforms from the mel-spectrograms.
Revised: For our DPP-TTS and Flowtron, HiFi-GAN (Kong et al., 2020) is used as the vocoder for synthesizing waveforms from the mel-spectrograms.

Original: 5. H UMAN EVALUATION RESULTS
Revised: 5. H UMAN EVALUATION RESULTS

Original: Side by side evaluation First, we conduct a side-by-side evaluation of the prosody of a sentence.
Revised: Side by side evaluation First, we conduct a side-by-side evaluation of the prosody of a sentence.

Original: In this experiment, texts from test samples from LJSpeech are synthesized.
Revised: In this experiment, texts from test samples from LJSpeech are synthesized.

Original: Target phrases extracted from the sentence are processed by PDM and pitch diversity modeling is used in DPP-TTS.
Revised: Target phrases extracted from the sentence are processed by PDM and pitch diversity modeling is used in DPP-TTS.

Original: Via
Revised: Via

Original: Amazon Mechanical Turk (AMT), we assign ﬁve testers living in the United States to a pair of audio samples (i.e., DPP-TTS and a baseline), and ask them to listen to audio samples and choose among three options: (1): sample A has more varied prosody than sample B, (2): sample A and B are equally varied in prosody, (3): the opposite of option (1).
Revised: Amazon Mechanical Turk (AMT), we assign ﬁve testers living in the United States to a pair of audio samples (i.e., DPP-TTS and a baseline), and ask them to listen to audio samples and choose among three options: A : sample A has more varied prosody than sample B, Same : sample A and B are equally varied in prosody, B : the opposite of option (1).

Original: [Table] In addition, we also conduct a side-by-side evaluation of prosody in a paragraph  since a monotonous pattern is prominent in long texts rather than a single sentence.
Revised: In addition, we also conduct a side-by-side evaluation of prosody in a paragraph that is not included in the LJSpeech dataset since a monotonous pattern is prominent in long texts rather than a single sentence.

Original: We prepared two DPP-TTS versions: (1): DPP-TTS1: a model that has only duration diversifying module and DPP-TTS2: a model that has both duration and pitch diversifying module.
Revised: We prepare two DPP-TTS versions: (1): DPP-TTS1: a model that has only duration diversifying module and DPP-TTS2: a model that has both duration and pitch diversifying modules.

Original: In this experiment, only VITS is compared as the baseline.
Revised: In this experiment, only VITS is compared as the baseline.

Original: Likewise the evaluation of a sentence, listeners are asked to choose among  three options.
Revised: Likewise the evaluation of a sentence, listeners are asked to choose among the three options.

Revised: [Table]

Original:  Results are shown in Figure 4.
Revised: [Table] [Table] [Table] Results are shown in Table 1, 2.

Original: In the side-by-side evaluation of prosody in a sentence, DPP-TTS outperforms VITSby a margin of 10 percent and Flowtron by a margin of 20 percent.
Revised: In the sideby-side evaluation of prosody in a sentence, DPP-TTS outperforms VITS and Flowtron.

Original: In the side-byside evaluation of prosody in a paragraph,  DPP-TTS1 outperforms the baseline but DPP-TTS2 does not.
Revised: In the side-by-side evaluation of prosody in a paragraph, both the DPP-TTS1 and DPPTTS2 outperform the baseline.

Original: We speculate that diversifying the duration and pitch simultaneously rather hurts the naturalness of speech.

Original: Naturalness test In this experiment, we measure the Mean-Opinion-Score (MOS) for audio samples generated by TTS models.
Revised: Naturalness test In this experiment, we measure the Mean-Opinion-Score (MOS) for audio samples evaluated in the side-byside evaluation.

Original: Likewise in the previous test, ﬁve testers are assigned to each audio sample.
Revised: Likewise in the previous test, ﬁve testers are assigned to each audio sample.

Original: Testers are asked to give a score between 1 to 5 on a 9-scale based on the sample’s naturalness.
Revised: Testers are asked to give a score between 1 to 5 on a 9-scale based on the sample’s naturalness.

Original: We used the same quality weight value w = 10 for DPP-TTS as in the side-by-side comparison test.
Revised: We used the same quality weight value w = 10 for DPP-TTS as in the side-by-side comparison test.

Original: [Table]
Revised: MOSS denotes the quality of audio from the LJSpeech dataset and MOS-P denotes the quality of audio from the paragraph. The

Original: The MOS result is described in Table 1.
Revised: MOS result is shown in Table 3.

Original: Although DPPTTS gets lower MOS than baselines, it yet still maintain areasonable MOS of 3.6.
Revised: In MOSS, our model gets lower MOS than VITS, it yet still maintain a MOS of 3.92 and outperforms Flowtron.

Original: It is noticeable there exists a large variance of MOS results in DPP-TTS compared to other
Revised: In MOS-P, our model outperforms

Original: TTS models.
Revised: VITS.

Original: we speculate that subtle differences resulted from the control of prosodic features might sound unnatural for some people. Achieving more stable samples in terms of naturalness will be our future work.
Revised: A signiﬁcant reduction of MOS in MOS-P compared to MOS-S indicates that it’s challenging to maintain the naturalness in long texts.

Original: 5. A DDITIONAL EVALUATION
Revised: 5. A DDITIONAL EVALUATION

Original: In this section, we  use the following metrics for the evaluation of our DPP-TTS and VITS: (1) σ frame : frame-level standard deviation of the pitch in a speech.
Revised: In this section, we conduct additional evaluations for our DPP-TTS and VITS 4 . We use the following metrics for the evaluation : (1) σ frame : frame-level standard deviation of the pitch in a speech.

Original: All pitch values are normalized since speech samples have different absolute pitch values.
Revised: All pitch values are normalized since speech samples have different absolute pitch values.

Original: Standard deviations are calculated overgenerated samples.
Revised: Standard deviations are averaged over 50 generated samples.

Original: DIO algorithm
Revised: DIO algorithm

Original: (Morise et al., 2009) is used for pitch extraction.
Revised: (Morise et al., 2009) is used for pitch extraction.

Original: (2): σ phoneme : phoneme-level standard deviation of duration or pitch in a speech.
Revised: (2): σ phoneme : phoneme-level standard deviation of duration or pitch in a speech.

Original: Likewise, the standard deviation values are calculated over 50 generated samples.
Revised: Likewise, the standard deviation values are averaged over 50 generated samples.

Original: (3): Determinant :  determinant of the matrix with respect to a feature set J generated by multiple samples is used to evaluate the diversity of phoneme-level prosodic features of multiple generated samples.
Revised: (3): Determinant : a determinant of the matrix with respect to a feature set J generated by multiple samples is used to evaluate the diversity of phoneme-level prosodic features of multiple generated samples.

Original: Cosine similarity is used as a metric between two sequences.
Revised: Cosine similarity is used as a metric between two sequences.

Original: Each set is constructed from 10 generated samples.
Revised: Each set is constructed from 10 generated samples.

Original: (4): Inference time : the inference time for synthesizing one-second length waveform is calculated in the text-to-speech model.
Revised: (4): Inference time : the inference time for synthesizing one-second length waveform is calculated in the text-to-speech model.

Original: The inference speed is evaluated on Intel(R) Core(TM) i7-7800X CPU and a single NVIDIA RTX 3080 GPU.
Revised: The inference speed is evaluated on Intel(R) Core(TM) i77800X CPU and a single NVIDIA RTX 3080 GPU.

Original: Computation time is averaged over 100 forward passes.
Revised: Computation time is averaged over 100 forward passes.

Original:  Table 2 shows the result of σ frame , σ phoneme , determinant and inference time.
Revised: [Table] Table 4 shows the result of σ frame , σ phoneme , determinant and inference time.

Original: At both frame and phoneme levels, the standard deviation of the pitch in DPP-TTS results in higher values than the baseline.
Revised: At both frame and phoneme levels, the standard deviation of the pitch in DPP-TTS results in higher values than the baseline.

Original: In addition, the standard deviation of duration at phoneme-level in DPP-TTS also results in a higher value than the baseline.
Revised: In addition, the standard deviation of duration at phoneme-level in DPP-TTS also results in a higher value than the baseline.

Original: It demonstrates that DPP-TTS generates a speech with more dynamic pitch and rhythm than the baseline.
Revised: It demonstrates that DPP-TTS generates a speech with more dynamic pitch and rhythm than the baseline.

Original: The determinants of duration and pitch sets of DPPTTS outperform the baseline.
Revised: The determinants of duration and pitch sets of DPPTTS outperform the baseline.

Original: It shows that DPP-TTS generates more diverse samples in terms of prosody than baseline.
Revised: It shows that DPP-TTS generates more diverse samples in terms of prosody than baseline.

Original: Finally, DPP-TTS results in 0.044 seconds of inference speed.
Revised: Finally, DPP-TTS results in 0.044 seconds of inference speed.

Original: Although its inference speed is slower than the baseline, our model is applicable in practice since the inference speed of our model is 22.7x faster than real-time.[Figure]
Revised: Although its inference speed is slower than the baseline, our model is applicable in practice since the inference speed of our model is 22.7x faster than real-time.

Original: 5. M ODEL ANALYSIS
Revised: 5. M ODEL ANALYSIS

Revised: Ablation study We conduct ablation studies to verify the effectiveness of PDM, the results are shown in Table 5.

Revised: In the side-by-side comparison, 34% more listeners choose the option that DPPTTS has more varied prosody compared to DPP-TTS without PDM.

Revised: It demonstrates that PDM contributes to prosody diversiﬁcation signiﬁcantly.

Revised: DPP-TTS also achieves a higher phoneme-level standard deviation of pitch and duration than the baseline.

Revised: Although there is a quality degradation (-0.26 MOS) compared to the baseline, DPP-TTS still achieve a MOS of 3.92.

Original:  Adjusting the extent of variation To study the impact of quality weight w , we plot (normalized) pitch and log duration values at phoneme-level with different values of quality weight .
Revised: [Table] Adjusting the extent of variation To study the impact of quality weight w , we plot (normalized) pitch and log duration values at phoneme-level with different values of quality weight.

Original: Plots are shown in Figure 5.

Original: As expected, for a large magnitude of quality weight , the model learns diversifying prosodic features with small deviations from contexts.
Revised: As expected, for a large magnitude of quality weight , the model learns diversifying prosodic features with small deviations from contexts.

Original: In contrast, for a small magnitude of quality weight , the model diversiﬁes prosodic features with large deviations from contexts.
Revised: In contrast, for a small magnitude of quality weight , the model diversiﬁes prosodic features with large deviations from contexts.

Original: We also found that a small magnitude of quality weight leads to faster convergence of the model but it easily hurts the naturalness of generated samples.
Revised: We also found that a small magnitude of quality weight leads to faster convergence of the model but it easily hurts the naturalness of generated samples.

Original: Unconditional MIC objective Instead of the conditional MIC objective, we also train our model with the vanilla MIC objective while the inference method is unchanged. We observe that generated samples lack rich prosody or result in unnatural prosody.
Revised: Figures are shows in Appendix E. [Table]

Original: It shows that it is important to sample prosodic features of target words according to processes conditioned on contexts.
Revised: determinant

Original: [Table]
Revised: [Table]

Original: Effect of number of candidates on diversity In this experiment, we study the effect of the number of candidates on the prosody diversity in generated samples.
Revised: Effect of number of candidates on diversity In this experiment, we study the effect of the number of candidates on the prosody diversity in generated samples.

Original: For a different number of candidates n c = 4 , 8 , 12 , DPP-TTS is trained up to 4k steps and all other hyperparameters remain unchanged.
Revised: For a different number of candidates n c = 4 , 8 , 12 , DPP-TTS is trained up to 4k steps and all other hyperparameters remain unchanged.

Original: After training, we calculate pitch determinants over generated samples for each setup.
Revised: After training, we calculate pitch determinants over generated samples for each setup.

Original: The result is reported in Table 3.
Revised: The result is reported in Table 6.

Original: We observe that using a large number of candidates results in more diverse samples although only a single prosody item is selected at MAP inference.
Revised: We observe that using a large number of candidates results in more diverse samples although only a single prosody item is selected at MAP inference.

Original: We also ﬁnd that using a large number of candidates has a regularization effect for training.
Revised: We also ﬁnd that using a large number of candidates has a regularization effect for training.

Original: Using a small number of candidates generates unstable speech samples in terms of naturalness, in contrast, using a large number of candidates results in more stable speech samples.
Revised: Using a small number of candidates generates unstable speech samples in terms of naturalness, in contrast, using a large number of candidates results in more stable speech samples.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have proposed a novel prosody diversifying method based on conditional determinantal point processes (DPPs).
Revised: In this paper, we have proposed a novel prosody diversifying method based on conditional determinantal point processes (DPPs).

Original: To learn the kernel matrix of conditional DPPs, conditional determinantal point process (DPPs) is parameterized additional prosody diversifying module (PDM).
Revised: To learn the kernel matrix of conditional DPPs, conditional determinantal point process (DPPs) is parameterized additional prosody diversifying module (PDM).

Original: To build the kernel, Soft dynamic time warping is adopted to measure the extent of similarity between two prosodic features, and the kernel of conditional DPPs is learned with conditional maximum induced cardinality (MIC) objective.
Revised: To build the kernel, Soft dynamic time warping is adopted to measure the extent of similarity between two prosodic features, and the kernel of conditional DPPs is learned with conditional maximum induced cardinality (MIC) objective.

Original: In the experiment, we demonstrated that our new prosody diversifying method generates more dynamic samples than the baseline while maintaining the naturalness.
Revised: In the experiment, we demonstrated that our new prosody diversifying method generates more dynamic samples than the baseline while maintaining the naturalness.

Original: In addition, we demonstrate that our model can be used in real-time applications by showing the inference speed of our model.
Revised: In addition, we demonstrate that our model can be used in real-time applications by showing the inference speed of our model.

Original: A D ETAILS OF PROSODY PREDICTOR AND PDM
Revised: A D ETAILS OF PROSODY PREDICTOR AND PDM

Original: [Figure]
Revised: [Figure]

Original: A. P ROSODY PREDICTOR
Revised: A. P ROSODY PREDICTOR

Original: Training It is hard to use the maximum likelihood objective directly to train duration predictor because the duration of each phoneme is 1) a discrete integer and a scalar, which hinders expressive transformation because invertibility should remain in normalizing ﬂow.
Revised: Training It is hard to use the maximum likelihood objective directly to train duration predictor because the duration of each phoneme is 1) a discrete integer and a scalar, which hinders expressive transformation because invertibility should remain in normalizing ﬂow.

Original: To train duration predictor, duration values are extended to continuous values using variational dequantization (Ho et al., 2019) and are augmented with extra dimensions using variational data augmentation (Chen et al., 2020).
Revised: To train duration predictor, duration values are extended to continuous values using variational dequantization (Ho et al., 2019) and are augmented with extra dimensions using variational data augmentation (Chen et al., 2020).

Original: Speciﬁcally, duration sequence d becomes continuous as d − u where u ’s value is restricted to [0 , 1) and augmented as [ d − u, v ] with a extra random variable v .
Revised: Speciﬁcally, duration sequence d becomes continuous as d − u where u ’s value is restricted to [0 , 1) and augmented as [ d − u, v ] with a extra random variable v .

Original: Two random variables u and v are sampled through approximate posterior q φ ( u, v | d, h text ) .
Revised: Two random variables u and v are sampled through approximate posterior q φ ( u, v | d, h text ) .

Original: The ELBO can be calculated as follows:
Revised: The ELBO can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: Like the duration predictor, the ELBO for pitch predictor can be calculated as follows:
Revised: Like the duration predictor, the ELBO for pitch predictor can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: (14) where p denotes the pitch sequences.
Revised: (13) where p denotes the pitch sequences.

Original: Both the duration and pitch predictor are trained on negative lower bound of likelihood.
Revised: Both the duration and pitch predictor are trained on negative lower bound of likelihood.

Original: Architecture The pitch predictor and duration predictor share identical architecture except for the additional random variable u .
Revised: Architecture The pitch predictor and duration predictor share identical architecture except for the additional random variable u .

Original: We will introduce the architecture of the duration predictor whose diagram is shown in Figure 6a.
Revised: We will introduce the architecture of the duration predictor whose diagram is shown in Figure 4a.

Original: Duration predictor consists of condition encoder for hidden sequence, posterior encoder for the duration sequence, and the main ﬂow blocks g ( · ) .
Revised: Duration predictor consists of condition encoder for hidden sequence, posterior encoder for the duration sequence, and the main ﬂow blocks g ( · ) .

Original: Speciﬁcally, posterior encoder maps latent codes from normal distribution to random variable [ u, v ] and ﬂow g ( · ) maps [ d − u, v ] to normal distribution.
Revised: Speciﬁcally, posterior encoder maps latent codes from normal distribution to random variable [ u, v ] and ﬂow g ( · ) maps [ d − u, v ] to normal distribution.

Original: Figure 6b shows the coupling of normalizing ﬂows.
Revised: Figure 4b shows the coupling of normalizing ﬂows.

Original: First, input x is split into [ x 0 , x 1 ] and then x 0 is processed by a 1x1 convolution block.
Revised: First, input x is split into [ x 0 , x 1 ] and then x 0 is processed by a 1x1 convolution block.

Original: The output of the convolution block is processed by dilated convolution and Nystromer block and their outputs are concatenated.
Revised: The output of the convolution block is processed by dilated convolution and Nystromer block and their outputs are concatenated.

Original: The concatenated output is processed by LayerNorm, Mish activation, and Convolution block.
Revised: The concatenated output is processed by LayerNorm, Mish activation, and Convolution block.

Original: Finally, spline ﬂows (Durkan et al., 2019) are parameterized by the output and then the output of spline ﬂows y and x 0 are concatenated.
Revised: Finally, spline ﬂows (Durkan et al., 2019) are parameterized by the output and then the output of spline ﬂows y and x 0 are concatenated.

Original: A. PDM
Revised: A. PDM

Original: The architecture of PDM is shown in Figure 6c.
Revised: The architecture of PDM is shown in Figure 4c.

Original: First, the hidden sequence is processed by convolution block followed by the dilated convolution block conditioned on hidden sequences, Layernorm, and 1x1 convolution block.
Revised: First, the hidden sequence is processed by convolution block followed by the dilated convolution block conditioned on hidden sequences, Layernorm, and 1x1 convolution block.

Original: After that, noise from a normal distribution is processed by 1x1 convolution followed by dilated convolution block, LayerNorm, and Mish activation.
Revised: After that, noise from a normal distribution is processed by 1x1 convolution followed by dilated convolution block, LayerNorm, and Mish activation.

Original: B P ROOF OF Proposition
Revised: As the prosody pattern of human speech is correlated with the local context and global context of the text, we design the architecture of the model to encode both the local and global features. Speciﬁcally, dilated convolution blocks are stacked to encode local features and a transformer block is used to encode global features in the coupling layers of normalizing ﬂows. However, using the vanilla transformer to encode the global features in the coupling layer requires too large computational complexity. Therefore, Nystr¨omformer (Xiong et al., 2021) which is a Nystr¨om-Based algorithm for approximating self-attention is used to encode global context for more efﬁcient memory usage in the coupling layer of normalizing ﬂow.

Revised: Encoded local features and global features are concatenated and processed through LayerNorm (Ba et al., 2016), Mish activation (Misra, 2019) and a 1x convolution module.

Revised: Following the duration predictor model in Kim et al.

Revised: (2021), variational dequantization (Ho et al., 2019) is used for the duration predictor since the phoneme duration is a discrete integer.

Revised: In addition, variational augmentation (Chen et al., 2020) is used to expand channel dimensions for expressive ﬂows in both the duration and pitch predictor.

Revised: The stochastic and pitch predictor are trained by maximizing their variational lower bounds (ELBO).

Revised: Details of the duration and pitch predictor are described in Appendix A.

Revised: B P ROOF OF Proposition

Original: Objective From equation [45] in (Kulesza & Taskar, 2012), the marginal kernel of conditional
Revised: Objective From equation [45] in (Kulesza & Taskar, 2012), the marginal kernel of conditional

Original: DPPs given the appearance of set A has a following form:
Revised: DPPs given the appearance of set A has a following form:

Original: [Equation]
Revised: [Equation]

Original: In addition, from equation [34] in Kulesza & Taskar (2012) the expected cardinality of Y given the marginal kernel K is:
Revised: In addition, from equation [34] in Kulesza & Taskar (2012) the expected cardinality of Y given the marginal kernel K is:

Original: [Equation]
Revised: [Equation]

Original: From equation  15,16, the expected cardinality of conditional DPPs given the appearance of set A is:
Revised: From equation 14, 15, the expected cardinality of conditional DPPs given the appearance of set A is:

Original: [Equation]
Revised: [Equation]

Original: Derivative For the proof, we will start with a following lemma:
Revised: Derivative For the proof, we will start with a following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: Given a matrix E and non-singular matrix A , following equation holds:
Revised: Given a matrix E and non-singular matrix A , following equation holds:

Original: [Equation]
Revised: [Equation]

Original: Proof .
Revised: Proof .

Original: First, consider ∂ ∂A ij t r ( E T A − 1 E ) .
Revised: First, consider ∂ ∂A ij t r ( E T A − 1 E ) .

Original: Since trace and derivative operator are interchangeable,
Revised: Since trace and derivative operator are interchangeable,

Original: [Equation]
Revised: [Equation]

Original:  By setting ∂A∂A ij = E ij where E ij denotes the matrix whose ( i, j ) component is 1 and 0 elsewhere and C = − E T A − 1 E ij A − 1 E ,
Revised: [Equation] By setting ∂A∂A ij = E ij where E ij denotes the matrix whose ( i, j ) component is 1 and 0 elsewhere and C = − E T A − 1 E ij A − 1 E ,

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: =

Original: [Equation]
Revised: [Equation]

Original:  with respect to set a A whose cardinality is p and matrix L ∈ R ( p + q ) ×
Revised: Now, with respect to set a A whose cardinality is p and matrix L ∈ R ( p + q ) ×

Original: ( p + q )
Revised: ( p + q )

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where E denotes
Revised: where E denotes

Original: [Equation]
Revised: [Equation]

Original:  q .
Revised: ∈ R ( p + q ) × q .

Original: Then by Lemma 1.
Revised: Then by Lemma 1.

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition 1 is now ﬁnished.
Revised: The proof of Proposition 1 is now ﬁnished.

Original: C A LGORITHMS FOR PDM TRAINING AND INFERENCE OF DPP-TTS C. PDM TRAINING
Revised: C A LGORITHMS FOR PDM TRAINING AND INFERENCE OF DPP-TTS C. PDM TRAINING

Original: Require:
Revised: Require:

Original: TextEncoder f ( · ) , PDM parameterized θ , a prosody predictor (trained ﬂow) g ( · ) , number of candidates n c , noise scale (cid:15) 1: while not converged do 2: h text ← f ( text ) 3:
Revised: TextEncoder f ( · ) , PDM parameterized θ , a prosody predictor (trained ﬂow) g ( · ) , number of candidates n c , noise scale (cid:15) 1: while not converged do 2: h text ← f ( text ) 3:

Original: Split h text into [ h target , h context ] 4:
Revised: Split h text into [ h target , h context ] 4:

Original: Sample latent code z context ∈ R T with noise scale (cid:15) 5:
Revised: Sample latent code z context ∈ R T with noise scale (cid:15) 5:

Original: Get prosodic features of contexts:
Revised: Get prosodic features of contexts:

Original: d context ← g − 1 ( h context , z context ) 6:
Revised: d context ← g − 1 ( h context , z context ) 6:

Original: Get quality of contexts: q context =
Revised: Get quality of contexts: q context =

Original: Density estimation ( d context ) 7:
Revised: Density estimation ( d context ) 7:

Original: Sample latent codes z target with noise scale (cid:15) 8:
Revised: Sample latent codes z target with noise scale (cid:15) 8:

Original: Get latent codes after PDM: z target ← PDM( z target ) ∈
Revised: Get latent codes after PDM: z target ← PDM( z target ) ∈

Original: R n c × T 9:
Revised: R n c × T 9:

Original: Get n c prosodic features of targets: ( d 1 target , d 2 target , ...
Revised: Get n c prosodic features of targets: ( d 1 target , d 2 target , ...

Original: d n c target ) 10:
Revised: d n c target ) 10:

Original: Get quality of targets: q target =
Revised: Get quality of targets: q target =

Original: Density estimation ( d target ) 11:
Revised: Density estimation ( d target ) 11:

Original: Concatenate contexts and targets: [ q context , q target ] , [ d context , d target ] 12:
Revised: Concatenate contexts and targets: [ q context , q target ] , [ d context , d target ] 12:

Original: Build the kernel of conditional DPPs:
Revised: Build the kernel of conditional DPPs:

Original: L ← Build kernel ([ q context , q target ] , [ d Context , d target ]) 13:
Revised: L ← Build kernel ([ q context , q target ] , [ d Context , d target ]) 13:

Original: Calculate the loss function: L diversity ← − tr( I − [( L + I ¯A ) − 1 ] ¯A ) 14:
Revised: Calculate the loss function: L diversity ← − tr( I − [( L + I ¯A ) − 1 ] ¯A ) 14:

Original: Update θ with the gradient ∇ L diversity 15: end while
Revised: Update θ with the gradient ∇ L diversity 15: end while

Original: C. I NFERENCE OF DPP-TTS
Revised: C. I NFERENCE OF DPP-TTS

Original: Require:
Revised: Require:

Original: TextEncoder f ( · ) , Decoder h ( · ) , PDM, a prosody predictor g ( · ) , noise scale (cid:15) 1: h text ← f ( text ) 2: Split h text into [ h target , h context ] 3: Sample latent code z context ∈ R rmT with noise scale (cid:15) 4:
Revised: TextEncoder f ( · ) , Decoder h ( · ) , PDM, a prosody predictor g ( · ) , noise scale (cid:15) 1: h text ← f ( text ) 2: Split h text into [ h target , h context ] 3: Sample latent code z context ∈ R rmT with noise scale (cid:15) 4:

Original: Get prosodic features of
Revised: Get prosodic features of

Original: D S AMPLE PARAGRAPH FOR THE SIDE - BY - SIDE COMPARISON TEST
Revised: D S AMPLE PARAGRAPH FOR THE SIDE - BY - SIDE COMPARISON TEST

Original: Now, cut to the future, about 10 days or so after all these events. Hale is in the Forge with Bernard in her custody.
Revised: Known individually and collectively as Shai-Hulud, the sandworms are these supermassive beings that plow through the deserts of Arrakis, consuming everything that dares venture unprepared into their territory.

Original: She’s still searching for an encryption key that will allow Delos to decode all the guest data now apparently in their possession.
Revised: The worms are what make harvesting spice so difficult because they tend to eat whatever tools off-worlders use to mine it.

Original: She suggest that it’s hidden in the dead Dolores. What she doesn’t know is that it’s not Bernard didn’t transmit the guest data but data about the hosts, and Bernard has created a robot replica of Hale and installed it with Dolores’s code.
Revised: They are also sacred to the Fremen, who seem to know ways to navigate around them, and, somehow, they’re linked to the creation of spice. Think of them as big honking metaphors for the sublime powers of nature that loom beyond human understanding, like a desert full of Moby Dicks

Original: The new Hale appears from stage left, kills the old one
Revised: E A DJUSTING THE EXTENT OF VARIATION

Original: and it’s check mate to the hosts.
Revised: [Figure]