  Text to speech (TTS) is widely used to synthesize personal voice for a target
speaker, where a well-trained source TTS model is fine-tuned with few paired
adaptation data (speech and its transcripts) on this target speaker. However,
in many scenarios, only untranscribed speech data is available for adaptation,
which brings challenges to the previous TTS adaptation pipelines (e.g.,
AdaSpeech). In this paper, we develop AdaSpeech 2, an adaptive TTS system that
only leverages untranscribed speech data for adaptation. Specifically, we
introduce a mel-spectrogram encoder to a well-trained TTS model to conduct
speech reconstruction, and at the same time constrain the output sequence of
the mel-spectrogram encoder to be close to that of the original phoneme
encoder. In adaptation, we use untranscribed speech data for speech
reconstruction and only fine-tune the TTS decoder. AdaSpeech 2 has two
advantages: 1) Pluggable: our system can be easily applied to existing trained
TTS models without re-training. 2) Effective: our system achieves on-par voice
quality with the transcribed TTS adaptation (e.g., AdaSpeech) with the same
amount of untranscribed data, and achieves better voice quality than previous
untranscribed adaptation methods. Synthesized speech samples can be found at
https://speechresearch.github.io/adaspeech2/.
