Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework (2020)

Abstract: Text-to-speech synthesis (TTS) has witnessed rapid progress in recent years, where neural methods became capable of producing audios with high naturalness. However, these efforts still suffer from two types of latencies: (a) the computational latency (synthesizing time), which grows linearly with the sentence length, and (b) the input latency in scenarios where the input text is incrementally available (such as in simultaneous translation, dialog generation, and assistive technologies). To reduce these latencies, we propose a neural incremental TTS approach using the prefix-to-prefix framework from simultaneous translation. We synthesize speech in an online fashion, playing a segment of audio while generating the next, resulting in an O(1) rather than O(n) latency. Experiments on English and Chinese TTS show that our approach achieves similar speech naturalness compared to full sentence TTS, but only with a constant (1-2 words) latency.

## introduction text-to-speech synthesis (tts) generates speech from text, and is an important task with wide applications in dialog systems, speech translation, natural language user interface, assistive technologies, etc. recently, it has benefited greatly from deep learning, with neural tts systems becoming capable of generating audios with high naturalness @xcite . state-of-the-art neural tts systems generally consist of two stages: the text-to-spectrogram stage which generates an intermediate acoustic representation (linear-or mel-spectrogram) from the https://inctts.github.io/ . to reduce these latencies, we propose a neural incremental tts approach borrowing the recently proposed prefix-to-prefix framework for simultaneous translation @xcite . our idea is based on two observations: (a) in both stages, the dependencies on input are very local (see fig. 1 for the monotonic attention between text and spectrogram, for example); and (b) audio playing is inherently sequential in nature, but can be done simultaneously with audio generation, i.e., playing a segment of audio while generating the next. in a nutshell, we start to generate the spectrogram for the first word after receiving the first two words, and this spectrogram is fed into the vocoder right away to generate the waveform for the first word, which is also played immediately (see fig. 2 ). this results in an o(1) rather than o(n) latency. experiments on english and chinese tts show that our approach achieves similar speech naturalness compared to full sentence methods, but only with a constant (1-2 words) latency. 1 this paper makes following contributions: ## preliminaries: neural tts we briefly review the full-sentence neural tts pipeline to set up the notations. as shown in fig. 3 , the neural-based text-to-speech synthesis system generally has two main steps: (1) the textto-spectrogram step which converts a sequence of textual features (e.g. characters, phonemes, words) into another sequence of spectrograms (e.g. melspectrogram or linear-spectrogram); and (2) the spectrogram-to-wave step, which takes the predicted spectrograms and generates the audio wave by a vocoder. ## implementation details in this section, we provide some implementation details for the two steps (spectrogram and wave). we assume the given text input is normalized, and we use an existing grapheme-to-phoneme tool foot_0 to generate phonemes for the given text. for some languages like chinese, we need to use an existing tool foot_1 to do text segmentation before generating phonemes. in the following, we assume the pre-trained models for both steps are given, and we only perform inference-time adaptations. for the first step, we use the tacotron 2 model @xcite , which takes generated phonemes as input, and for the second step we use the parallel wavegan vocoder @xcite . ## related work there are some existing work about incremental tts based hidden markov model (hmm). baumann and schlangen (2012c) propose an incremental spoken dialogue system architecture and toolkit called inprotk, including recognition, dialogue management and tts modules. with this toolkit, baumann and schlangen (2012b) present a component for incremental speech synthesis, which is not fully incremental on the hmm level. @xcite propose a training strategy based on hmm with unknown linguistic features for incremental tts. baumann (2014a,b) proposes use linguistic features and choose default values when they are not available. the above works all focus on stress-timed languages, such as english and german, while @xcite propose a system for japanese, a mora-timed language. these systems require full context labels of linguistic features, making it difficult to improve the audio quality when input text is revealed incrementally. further, each component in their systems is trained and tuned separately, resulting in error propagation. there is parallel work from @xcite , which introduced a different neural approach for segment-based incremental tts. their proposed solution synthesizes each segment (could be as long as half sentence) at a time, thus not strictly incremental on the word level. when they perform word-level synthesis, as it is shown in their paper, there is a huge performance drop from 3.01 (full-sentence) to 2.08. their proposed approach has to retain the basic full-sentence model with segmented texts and audios which were obtained from forced alignment (different models for different latencies), while we only make adaptations to the decoder at inference time with an existing welltrained full-sentence model. our model not only uses previous context, but also use limited, a few lookahead words for better prosody and pronunciation. the above advantages of our model guarantee that our model achieves similar performance with full-sentence model with much lower latency on word-level inference. on the contrary, the model from @xcite did not use lookahead information at all, which can be problematic in the cases when word has multiple pronunciation that depends on following word. for example, there are two pronunciations for the word "the" which are "dh iy" and "dh ah". when the word after "the" starts with vowel sound, "dh iy" is the b731afe6-@xmath0. lookabead information is more important in liaison, where the final consonant of one word links with the first vowel of the next word, e.g., "an apple", "think about it", and "there is a". this problem is even more severe in other languages like french. more generally, co-articulation is common in most languages, where lookahead is needed. ## conclusions we have presented a prefix-to-prefix inference framework for incremental tts system, and a lookahead-k policy that the audio generation is always k words behind the input. we show that this policy can achieve good audio quality compared with full-sentence method but with low latency in different scenarios: when all the input are available and when input is given incrementally.