  Cross-lingual voice conversion (VC) is an important and challenging problem
due to significant mismatches of the phonetic set and the speech prosody of
different languages. In this paper, we build upon the neural text-to-speech
(TTS) model, i.e., FastSpeech, and LPCNet neural vocoder to design a new
cross-lingual VC framework named FastSpeech-VC. We address the mismatches of
the phonetic set and the speech prosody by applying Phonetic PosteriorGrams
(PPGs), which have been proved to bridge across speaker and language
boundaries. Moreover, we add normalized logarithm-scale fundamental frequency
(Log-F0) to further compensate for the prosodic mismatches and significantly
improve naturalness. Our experiments on English and Mandarin languages
demonstrate that with only mono-lingual corpus, the proposed FastSpeech-VC can
achieve high quality converted speech with mean opinion score (MOS) close to
the professional records while maintaining good speaker similarity. Compared to
the baselines using Tacotron2 and Transformer TTS models, the FastSpeech-VC can
achieve controllable converted speech rate and much faster inference speed.
More importantly, the FastSpeech-VC can easily be adapted to a speaker with
limited training utterances.
