  We consider the challenging problem of audio to animated video generation. We
propose a novel method OneShotAu2AV to generate an animated video of arbitrary
length using an audio clip and a single unseen image of a person as an input.
The proposed method consists of two stages. In the first stage, OneShotAu2AV
generates the talking-head video in the human domain given an audio and a
person's image. In the second stage, the talking-head video from the human
domain is converted to the animated domain. The model architecture of the first
stage consists of spatially adaptive normalization based multi-level generator
and multiple multilevel discriminators along with multiple adversarial and
non-adversarial losses. The second stage leverages attention based
normalization driven GAN architecture along with temporal predictor based
recycle loss and blink loss coupled with lipsync loss, for unsupervised
generation of animated video. In our approach, the input audio clip is not
restricted to any specific language, which gives the method multilingual
applicability. OneShotAu2AV can generate animated videos that have: (a) lip
movements that are in sync with the audio, (b) natural facial expressions such
as blinks and eyebrow movements, (c) head movements. Experimental evaluation
demonstrates superior performance of OneShotAu2AV as compared to U-GAT-IT and
RecycleGan on multiple quantitative metrics including KID(Kernel Inception
Distance), Word error rate, blinks/sec
