  Given only a few glimpses of an environment, how much can we infer about its
entire floorplan? Existing methods can map only what is visible or immediately
apparent from context, and thus require substantial movements through a space
to fully map it. We explore how both audio and visual sensing together can
provide rapid floorplan reconstruction from limited viewpoints. Audio not only
helps sense geometry outside the camera's field of view, but it also reveals
the existence of distant freespace (e.g., a dog barking in another room) and
suggests the presence of rooms not visible to the camera (e.g., a dishwasher
humming in what must be the kitchen to the left). We introduce AV-Map, a novel
multi-modal encoder-decoder framework that reasons jointly about audio and
vision to reconstruct a floorplan from a short input video sequence. We train
our model to predict both the interior structure of the environment and the
associated rooms' semantic labels. Our results on 85 large real-world
environments show the impact: with just a few glimpses spanning 26% of an area,
we can estimate the whole area with 66% accuracy -- substantially better than
the state of the art approach for extrapolating visual maps.
