  Existing scene understanding systems mainly focus on recognizing the visible
parts of a scene, ignoring the intact appearance of physical objects in the
real-world. Concurrently, image completion has aimed to create plausible
appearance for the invisible regions, but requires a manual mask as input. In
this work, we propose a higher-level scene understanding system to tackle both
visible and invisible parts of objects and backgrounds in a given scene.
Particularly, we built a system to decompose a scene into individual objects,
infer their underlying occlusion relationships, and even automatically learn
which parts of the objects are occluded that need to be completed. In order to
disentangle the occluded relationships of all objects in a complex scene, we
use the fact that the front object without being occluded is easy to be
identified, detected, and segmented. Our system interleaves the two tasks of
instance segmentation and scene completion through multiple iterations, solving
for objects layer-by-layer. We first provide a thorough experiment using a new
realistically rendered dataset with ground-truths for all invisible regions. To
bridge the domain gap to real imagery where ground-truths are unavailable, we
then train another model with the pseudo-ground-truths generated from our
trained synthesis model. We demonstrate results on a wide variety of datasets
and show significant improvement over the state-of-the-art.
