  Telecommunication with photorealistic avatars in virtual or augmented reality
is a promising path for achieving authentic face-to-face communication in 3D
over remote physical distances. In this work, we present the Pixel Codec
Avatars (PiCA): a deep generative model of 3D human faces that achieves state
of the art reconstruction performance while being computationally efficient and
adaptive to the rendering conditions during execution. Our model combines two
core ideas: (1) a fully convolutional architecture for decoding spatially
varying features, and (2) a rendering-adaptive per-pixel decoder. Both
techniques are integrated via a dense surface representation that is learned in
a weakly-supervised manner from low-topology mesh tracking over training
images. We demonstrate that PiCA improves reconstruction over existing
techniques across testing expressions and views on persons of different gender
and skin tone. Importantly, we show that the PiCA model is much smaller than
the state-of-art baseline model, and makes multi-person telecommunicaiton
possible: on a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered
in realtime in the same scene.
