  By estimating 3D shape and instances from a single view, we can capture
information about an environment quickly, without the need for comprehensive
scanning and multi-view fusion. Solving this task for composite scenes (such as
object stacks) is challenging: occluded areas are not only ambiguous in shape
but also in instance segmentation; multiple decompositions could be valid. We
observe that physics constrains decomposition as well as shape in occluded
regions and hypothesise that a latent space learned from scenes built under
physics simulation can serve as a prior to better predict shape and instances
in occluded regions. To this end we propose SIMstack, a depth-conditioned
Variational Auto-Encoder (VAE), trained on a dataset of objects stacked under
physics simulation. We formulate instance segmentation as a centre voting task
which allows for class-agnostic detection and doesn't require setting the
maximum number of objects in the scene. At test time, our model can generate 3D
shape and instance segmentation from a single depth view, probabilistically
sampling proposals for the occluded region from the learned latent space. Our
method has practical applications in providing robots some of the ability
humans have to make rapid intuitive inferences of partially observed scenes. We
demonstrate an application for precise (non-disruptive) object grasping of
unknown objects from a single depth view.
