  While generative adversarial networks (GANs) based neural text-to-speech
(TTS) systems have shown significant improvement in neural speech synthesis,
there is no TTS system to learn to synthesize speech from text sequences with
only adversarial feedback. Because adversarial feedback alone is not sufficient
to train the generator, current models still require the reconstruction loss
compared with the ground-truth and the generated mel-spectrogram directly. In
this paper, we present Multi-SpectroGAN (MSG), which can train the
multi-speaker model with only the adversarial feedback by conditioning a
self-supervised hidden representation of the generator to a conditional
discriminator. This leads to better guidance for generator training. Moreover,
we also propose adversarial style combination (ASC) for better generalization
in the unseen speaking style and transcript, which can learn latent
representations of the combined style embedding from multiple mel-spectrograms.
Trained with ASC and feature matching, the MSG synthesizes a high-diversity
mel-spectrogram by controlling and mixing the individual speaking styles (e.g.,
duration, pitch, and energy). The result shows that the MSG synthesizes a
high-fidelity mel-spectrogram, which has almost the same naturalness MOS score
as the ground-truth mel-spectrogram.
