  This paper proposes a novel voice conversion (VC) method based on
non-autoregressive sequence-to-sequence (NAR-S2S) models. Inspired by the great
success of NAR-S2S models such as FastSpeech in text-to-speech (TTS), we extend
the FastSpeech2 model for the VC problem. We introduce the
convolution-augmented Transformer (Conformer) instead of the Transformer,
making it possible to capture both local and global context information from
the input sequence. Furthermore, we extend variance predictors to variance
converters to explicitly convert the source speaker's prosody components such
as pitch and energy into the target speaker. The experimental evaluation with
the Japanese speaker dataset, which consists of male and female speakers of
1,000 utterances, demonstrates that the proposed model enables us to perform
more stable, faster, and better conversion than autoregressive S2S (AR-S2S)
models such as Tacotron2 and Transformer.
