Pixels to Plans: Learning NonPrehensile Manipulation by Imitating a Planner We present a novel method enabling robots to quickly learn to manipulate objects by leveraging a motion planner to generate xe2x80x9cexpertxe2x80x9d training trajectories from a small amount of humanlabeled data. In contrast to the traditional senseplanact cycle, we propose a deep learning architecture and training regimen called PtPNet that can estimate effective endeffector trajectories for manipulation directly from a single RGBD image of an object. Additionally, we present a data collection and augmentation pipeline that enables the automatic generation of large numbers (millions) of training image and trajectory examples with almost no human labeling effort.We demonstrate our approach in a nonprehensile toolbased manipulation task, specifically picking up shoes with a hook. In hardware experiments, PtPNet generates motion plans (openloop trajectories) that reliably (89% success over 189 trials) pick up four very different shoes from a range of positions and orientations, and reliably picks up a shoe it has never seen before. Compared with a traditional senseplanact paradigm, our system has the advantages of operating on sparse information (single RGBD frame), producing highquality trajectories much faster than the expert planner (300ms versus several seconds), and generalizing effectively to previously unseen shoes. Video available at https:youtu.bevoIkyiBtwn4.

Learning PhysicsBased Manipulation in Clutter: Combining ImageBased Generalization and LookAhead Planning Physicsbased manipulation in clutter involves complex interaction between multiple objects. In this paper, we consider the problem of learning, from interaction in a physics simulator, manipulation skills to solve this multistep sequential decision making problem in the real world. Our approach has two key properties: (i) the ability to generalize and transfer manipulation skills (over the type, shape, and number of objects in the scene) using an abstract imagebased representation that enables a neural network to learn useful features; and (ii) the ability to perform lookahead planning in the image space using a physics simulator, which is essential for such multistep problems. We show, in sets of simulated and realworld experiments (video available on https:youtu.beEmkUQfyvwkY), that by learning to evaluate actions in an abstract imagebased representation of the real world, the robot can generalize and adapt to the object shapes in challenging realworld environments.