  Audio to Video generation is an interesting problem that has numerous
applications across industry verticals including film making, multi-media,
marketing, education and others. High-quality video generation with expressive
facial movements is a challenging problem that involves complex learning steps
for generative adversarial networks. Further, enabling one-shot learning for an
unseen single image increases the complexity of the problem while
simultaneously making it more applicable to practical scenarios. In the paper,
we propose a novel approach OneShotA2V to synthesize a talking person video of
arbitrary length using as input: an audio signal and a single unseen image of a
person. OneShotA2V leverages curriculum learning to learn movements of
expressive facial components and hence generates a high-quality talking-head
video of the given person. Further, it feeds the features generated from the
audio input directly into a generative adversarial network and it adapts to any
given unseen selfie by applying fewshot learning with only a few output
updation epochs. OneShotA2V leverages spatially adaptive normalization based
multi-level generator and multiple multi-level discriminators based
architecture. The input audio clip is not restricted to any specific language,
which gives the method multilingual applicability. Experimental evaluation
demonstrates superior performance of OneShotA2V as compared to Realistic
Speech-Driven Facial Animation with GANs(RSDGAN) [43], Speech2Vid [8], and
other approaches, on multiple quantitative metrics including: SSIM (structural
similarity index), PSNR (peak signal to noise ratio) and CPBD (image
sharpness). Further, qualitative evaluation and Online Turing tests demonstrate
the efficacy of our approach.
