Simulationbased physics reasoning for consistent scene estimation in an HRI context Reasoning about spatial and geometric relations between objects in a tabletop humanrobot interaction is a challenge due to the perception not being always consistent: objects placed on a table seem to be slightly in the air; they overlap; they disappear due to occlusions. Yet, interpreting and anchoring perceptual data in a physically consistent estimation of the scene is a crucial ability for humans, and thus robots in HRI context. In this paper we present a simulationbased physics reasoner integrated in a lightweight situationassessment framework called Underworlds, that allows the robot to stabilize objects and build at runtime a consistent estimation of the scene, even for entirely hidden objects, while inferring the actions performed by its human partner.

Learning PhysicsBased Manipulation in Clutter: Combining ImageBased Generalization and LookAhead Planning Physicsbased manipulation in clutter involves complex interaction between multiple objects. In this paper, we consider the problem of learning, from interaction in a physics simulator, manipulation skills to solve this multistep sequential decision making problem in the real world. Our approach has two key properties: (i) the ability to generalize and transfer manipulation skills (over the type, shape, and number of objects in the scene) using an abstract imagebased representation that enables a neural network to learn useful features; and (ii) the ability to perform lookahead planning in the image space using a physics simulator, which is essential for such multistep problems. We show, in sets of simulated and realworld experiments (video available on https:youtu.beEmkUQfyvwkY), that by learning to evaluate actions in an abstract imagebased representation of the real world, the robot can generalize and adapt to the object shapes in challenging realworld environments.