Revisiting Over-Smoothness in Text to Speech (2022)

Abstract: Non-autoregressive text to speech (NAR-TTS) models have attracted much attention from both academia and industry due to their fast generation speed. One limitation of NAR-TTS models is that they ignore the correlation in time and frequency domains while generating speech mel-spectrograms, and thus cause blurry and over-smoothed results. In this work, we revisit this over-smoothing problem from a novel perspective: the degree of over-smoothness is determined by the gap between the complexity of data distributions and the capability of modeling methods. Both simplifying data distributions and improving modeling methods can alleviate the problem. Accordingly, we first study methods reducing the complexity of data distributions. Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods. Based on these studies, we find that 1) methods that provide additional condition inputs reduce the complexity of data distributions to model, thus alleviating the over-smoothing problem and achieving better voice quality. 2) Among advanced modeling methods, Laplacian mixture loss performs well at modeling multimodal distributions and enjoys its simplicity, while GAN and Glow achieve the best voice quality while suffering from increased training or model complexity. 3) The two categories of methods can be combined to further alleviate the over-smoothness and improve the voice quality. 4) Our experiments on the multi-speaker dataset lead to similar conclusions as above and providing more variance information can reduce the difficulty of modeling the target data distribution and alleviate the requirements for model capacity.

## introduction non-autoregressive text to speech (nar-tts) models (ren et al., 2019, 2020; peng et al., 2020; vainer and dušek, 2020; łańcucki, 2020; kim * corresponding author @xcite have shown much faster inference speed than their autoregressive counterparts @xcite @xcite , while achieving comparable or even better voice quality @xcite . the text-to-speech mapping can be formulated as a conditional distribution p (y|x) where x and y are the text and speech sequences, respectively. textto-speech mapping is a one-to-many mapping problem @xcite , since multiple possible speech sequences correspond to a text sequence due to speech variations such as pitch, duration and prosody. furthermore, speech mel-spectrograms are strongly correlated in time and frequency dimensions (see section 2.1 for detailed analyses). therefore, p (y|x) is actually a dependent and multimodal distribution @xcite 1 . early non-autoregressive tts models @xcite ) use mean absolute error (mae) or mean square error (mse) as loss function to model speech mel-spectrograms, implicitly assuming that data points in mel-spectrograms are independent to each other and follow a unimodal distribution foot_1 . consequently, the melspectrograms following dependent and multimodal distributions cannot be well modeled by the mae or mse loss, which presents great challenges in non-autoregressive tts modeling and causes over-smoothed (blurred) predictions in melspectrograms @xcite . in this work, we conduct a comprehensive study on the over-smoothing problem in tts. we find that the over-smoothness is closely related to the mismatch between the complexity of data distri-butions (e.g., dependent and multimodal distributions are more complex than independent and unimodal distributions) and the power of modeling methods (e.g., simple losses such as mae and mse are less powerful than gan or glow-based methods). both simplifying data distributions and enhancing modeling methods can alleviate the oversmoothing problem. from this perspective, we categorize recent methods combating over-smoothness into two classes: 1. simplifying data distributions: the data distribution p (y|x) can be simplified by providing more conditional input information. we review two main methods: 1) providing the previous mel-spectrogram frames @xmath0, i.e., factorizing the complex dependent distribution p (y|x) into a simpler conditional distribution t p (y t |@xmath1, x), as used in autoregressive tts models @xcite . 2) providing more variance information v foot_2 such as pitch, duration, and energy to predict mel-spectrogram in parallel, i.e., modeling p (y|x, v) rather than p (y|x), as done in some non-autoregressive tts models @xcite . 2. enhancing modeling methods: generally speaking, the modeling method should be powerful enough to fit complex data distributions. we review methods based on different distribution assumptions, including laplacian mixture loss, structural similarity index (ssim) @xcite loss, generative adversarial network (gan) @xcite and glow @xcite . by studying those methods, we have the following findings. we hope that our studies and findings can inspire the community to design better models for tts. besides, we find that glow has poor modeling ability in multi-speaker scenarios due to limited model capacity and more complex target data distributions compared with the single-speaker scenario, while it can be greatly alleviated by simplifying data distributions (introducing more variance information). ## preliminary study text-to-speech mapping is a one-to-many mapping since multiple speech sequences can possibly correspond to a text sequence with different pitch, duration and prosody, making the mel-spectrograms distribution p (y|x) multimodal. and due to the continuous nature of the mel-spectrograms, adjacent data points are dependent to each other. in this section, we first empirically characterize the distribution of p (y|x) in tts foot_3 through visualization (section 2.1), and then provide a novel perspective to study the over-smoothing problem in tts (section 2.2). ## simplifying data distributions p (y|x) simplifying data distributions p (y|x) is usually achieved by providing more conditional input information in the tts literature @xcite @xcite 7 . in this way, more conditional information in input can alleviate the one-to-many mapping issue, and thus the distribution becomes less multimodal and the correlation along time and frequency is reduced given more condition. there are mainly two methods to provide more conditional input information: 1) autoregressive factorization along the time @xcite or frequency axis; 2) providing more variance information to predict mel-spectrogram in parallel, as used in some non-autoregressive tts models @xcite . in this section, we first overview these two kinds of methods in detail and conduct the experiment analyses to measure their effectiveness in solving the over-smoothing problem. ## enhancing modeling methods most previous non-autoregressive tts models @xcite @xcite use mean absolute error (mae) or mean square error (mse) as training loss. however, they fail to capture dependent and multimodal distributions. mae loss is derived from the laplace distribution and mse from the gaussian distribution @xcite , which means min-imizing mae/mse will maximize the data loglikelihood under a laplace/gaussian distribution. both of these distributions are unimodal and thus encourage the model to predict a single mode in each data point. as a result, the model just learns an average of all modes, which leads to over-smoothed results. another problem brought by mae and mse is that they are independent across time and frequency for mel-spectrogram output, which ignores the correlation across time and frequency axes in mel-spectrogram. in this section, we first introduce several enhanced modeling methods to directly model the dependent and multimodal distribution p (y|x) (section 4.1), and then conduct experiments to compare and analyze these methods (section 4.2). ## further explorations and extensions in this section, we first explore the advantages of combining methods from two categories and then perform extensional analyses on multi-speaker dataset. after studying the two categories to combat oversmoothing problems, we have demonstrated that both simplifying data distributions and enhancing modeling methods can alleviate this problem and improve the voice quality of tts. a natural thought is to combine the methods from two categories, which may integrate the advantages of both aspects to reduce the gap between the complexity of data distribution and the power of the modeling method, resulting in better voice quality and further alleviating the over-smoothing problem. to demonstrate this idea, we choose the fastspeech 2 as the model from the first category since it achieves better perceptual voice quality than autoregressive modeling models according to ## conclusion in this paper, we revisited the over-smoothing problem in tts with a novel perspective: the degree of over-smoothness is determined by the gap between the complexity of data distribution and the capability of the modeling method. under this perspective, we classified existing methods combating over-smoothness into two categories: simplifying data distributions and enhancing modeling methods, and conducted comprehensive analyses and studies on these methods. for simplifying data distributions, we found that both ar factorization and providing more variance information as input (e.g., fastspeech 2) can alleviate the over-smoothing problem, and fastspeech 2 enjoys the advantage of fast generation over ar factorization. for enhancing modeling methods, we found that laplacian mixture loss can improve the generation quality and enjoy its simplicity, while gan and glow can further achieve better quality at the cost of increased training or model complexity. based on the above findings, we further combined these two categories of methods and found that the over-smoothing problem is further alleviated, and the generated speech quality is further improved, which shows that these two categories are complementary to each other. when performing our analyses on the multi-speaker dataset, we drew similar conclusions and found that providing more variance information can reduce the difficulty of modeling the target data distribution and alleviate the requirements for model capacity. we hope that our studies can inspire the community and industry to develop more powerful tts models. besides, since we are the first to discuss the over-smoothing problem systematically in the speech domain. we hope our analysis methodology as well as the findings can be extended to other tasks and inspire other domains.