  Emotional voice conversion models adapt the emotion in speech without
changing the speaker identity or linguistic content. They are less data hungry
than text-to-speech models and allow to generate large amounts of emotional
data for downstream tasks. In this work we propose EmoCat, a language-agnostic
emotional voice conversion model. It achieves high-quality emotion conversion
in German with less than 45 minutes of German emotional recordings by
exploiting large amounts of emotional data in US English. EmoCat is an
encoder-decoder model based on CopyCat, a voice conversion system which
transfers prosody. We use adversarial training to remove emotion leakage from
the encoder to the decoder. The adversarial training is improved by a novel
contribution to gradient reversal to truly reverse gradients. This allows to
remove only the leaking information and to converge to better optima with
higher conversion performance. Evaluations show that Emocat can convert to
different emotions but misses on emotion intensity compared to the recordings,
especially for very expressive emotions. EmoCat is able to achieve audio
quality on par with the recordings for five out of six tested emotion
intensities.
