Neural stylepreserving visual dubbing Dubbing is a technique for translating video content from one language to another. However, stateoftheart visual dubbing techniques directly copy facial expressions from source to target actors without considering identityspecific idiosyncrasies such as a unique type of smile. We present a stylepreserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages. At the heart of our approach is the concept of motion style, in particular for facial expressions, i.e., the personspecific expression change that is yet another essential factor beyond visual accuracy in face editing applications. Our method is based on a recurrent generative adversarial network that captures the spatiotemporal coactivation of facial expressions, and enables generating and modifying the facial expressions of the target actor while preserving their style. We train our model with unsynchronized source and target videos in an unsupervised manner using cycleconsistency and mouth expression losses, and synthesize photorealistic video frames using a layered neural face renderer. Our approach generates temporally coherent results, and handles dynamic backgrounds. Our results show that our dubbing approach maintains the idiosyncratic style of the target actor better than previous approaches, even for widely differing source and target actors.

Textbased Editing of Talkinghead Video Editing talkinghead video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talkinghead video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audiovisual flow (i.e. no jump cuts). Our method automatically annotates an input talkinghead video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis.