  We present a learning-based method for estimating 4D reflectance field of a
person given video footage illuminated under a flat-lit environment of the same
subject. For training data, we use one light at a time to illuminate the
subject and capture the reflectance field data in a variety of poses and
viewpoints. We estimate the lighting environment of the input video footage and
use the subject's reflectance field to create synthetic images of the subject
illuminated by the input lighting environment. We then train a deep
convolutional neural network to regress the reflectance field from the
synthetic images. We also use a differentiable renderer to provide feedback for
the network by matching the relit images with the input video frames. This
semi-supervised training scheme allows the neural network to handle unseen
poses in the dataset as well as compensate for the lighting estimation error.
We evaluate our method on the video footage of the real Holocaust survivors and
show that our method outperforms the state-of-the-art methods in both realism
and speed.
