  In this paper, we rethink the problem of scene reconstruction from an
embodied agent's perspective: While the classic view focuses on the
reconstruction accuracy, our new perspective emphasizes the underlying
functions and constraints such that the reconstructed scenes provide
\em{actionable} information for simulating \em{interactions} with agents. Here,
we address this challenging problem by reconstructing an interactive scene
using RGB-D data stream, which captures (i) the semantics and geometry of
objects and layouts by a 3D volumetric panoptic mapping module, and (ii) object
affordance and contextual relations by reasoning over physical common sense
among objects, organized by a graph-based scene representation. Crucially, this
reconstructed scene replaces the object meshes in the dense panoptic map with
part-based articulated CAD models for finer-grained robot interactions. In the
experiments, we demonstrate that (i) our panoptic mapping module outperforms
previous state-of-the-art methods, (ii) a high-performant physical reasoning
procedure that matches, aligns, and replaces objects' meshes with best-fitted
CAD models, and (iii) reconstructed scenes are physically plausible and
naturally afford actionable interactions; without any manual labeling, they are
seamlessly imported to ROS-based simulators and virtual environments for
complex robot task executions.
