. Our model jointly encodes and decodes geometry and view-dependent appearance into a latent code z, from data captured from a multi-camera rig, enabling highly realistic data-driven facial rendering. We use this rich data to drive our avatars from cameras mounted on a head-mounted display (HMD). We do this by creating synthetic HMD images through image-based rendering, and using another variational autoencoder to learn a common representation y of real and synthetic HMD images. We then regress from y to the latent rendering code z and decode into mesh and texture to render. Our method enables high-fidelity social interaction in virtual reality. We introduce a deep appearance model for rendering the human face. Inspired by Active Appearance Models, we develop a data-driven rendering pipeline that learns a joint representation of facial geometry and appearance from a multiview capture setup. Vertex positions and view-specific textures are modeled using a deep variational autoencoder that captures complex nonlinear effects while producing a smooth and compact latent representation. View-specific texture enables the modeling of view-dependent effects such as specularity. In addition, it can also correct for imperfect geometry stemming from biased or low resolution estimates. This is a significant departure from the traditional graphics pipeline, which requires highly accurate geometry as well as all elements of the shading model to achieve realism through physically-inspired light transport. Acquiring such a high level of accuracy is difficult in practice, especially for complex and intricate parts of the face, Authors' addresses: Stephen Lombardi, Facebook Reality Labs, Pittsburgh, PA, stephen. lombardi@fb.com; Jason Saragih, Facebook Reality Labs, Pittsburgh, PA, jason.saragih@ fb.com; Tomas Simon, Facebook Reality Labs, Pittsburgh, PA, tomas.simon@fb.com; Yaser Sheikh, Facebook Reality Labs, Pittsburgh, PA, yaser.sheikh@fb.com. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). such as eyelashes and the oral cavity. These are handled naturally by our approach, which does not rely on precise estimates of geometry. Instead, the shading model accommodates deficiencies in geometry though the flexibility afforded by the neural network employed. At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering. The resulting system can be implemented simply using existing rendering engines through dynamic textures with flat lighting. This representation, together with a novel unsupervised technique for mapping images to facial states, results in a system that is naturally suited to real-time interactive settings such as Virtual Reality (VR).
Deep Appearance Models REF learn dynamic viewdependent texture maps for the human head.
Deep Appearance Models for Face Rendering
ACM Transactions on Graphics (SIGGRAPH 2018) 37, 4, Article 68
Computer Science