  Building cross-lingual voice conversion (VC) systems for multiple speakers
and multiple languages has been a challenging task for a long time. This paper
describes a parallel non-autoregressive network to achieve bilingual and
code-switched voice conversion for multiple speakers when there are only
mono-lingual corpora for each language. We achieve cross-lingual VC between
Mandarin speech with multiple speakers and English speech with multiple
speakers by applying bilingual bottleneck features. To boost voice cloning
performance, we use an adversarial speaker classifier with a gradient reversal
layer to reduce the source speaker's information from the output of encoder.
Furthermore, in order to improve speaker similarity between reference speech
and converted speech, we adopt an embedding consistency loss between the
synthesized speech and its natural reference speech in our network.
Experimental results show that our proposed method can achieve high quality
converted speech with mean opinion score (MOS) around 4. The conversion system
performs well in terms of speaker similarity for both in-set speaker conversion
and out-set-of one-shot conversion.
