Omnipush: accurate, diverse, realworld dataset of pushing dynamics with RGBD video Pushing is a fundamental robotic skill. Existing work has shown how to exploit models of pushing to achieve a variety of tasks, including grasping under uncertainty, inhand manipulation and clearing clutter. Such models, however, are approximate, which limits their applicability.Learningbased methods can reason directly from raw sensory data with accuracy, and have the potential to generalize to a wider diversity of scenarios. However, developing and testing such methods requires richenough datasets. In this paper we introduce Omnipush, a dataset with high variety of planar pushing behavior.In particular, we provide 250 pushes for each of 250 objects, all recorded with RGBD and a high precision tracking system. The objects are constructed so as to systematically explore key factors that affect pushingxe2x80x93the shape of the object and its mass distributionxe2x80x93which have not been broadly explored in previous datasets, and allow to study generalization in model learning.Omnipush includes a benchmark for metalearning dynamic models, which requires algorithms that make good predictions and estimate their own uncertainty. We also provide an RGB video prediction benchmark and propose other relevant tasks that can be suited with this dataset. Data and code are available at https:web.mit.edumcubeomnipushdataset.

The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints A robot can now grasp an object more effectively than ever before, but once it has the object what happens next? We show that a mild relaxation of the task and workspace constraints implicit in existing object grasping datasets can cause neural network based grasping algorithms to fail on even a simple block stacking task when executed under more realistic circumstances. To address this, we introduce the JHU CoSTAR Block Stacking Dataset (BSD), where a robot interacts with 5.1 cm colored blocks to complete an orderfulfillment style block stacking task. It contains dynamic scenes and real timeseries data in a less constrained environment than comparable datasets. There are nearly 12,000 stacking attempts and over 2 million frames of real data. We discuss the ways in which this dataset provides a valuable resource for a broad range of other topics of investigation. We find that handdesigned neural networks that work on prior datasets do not generalize to this task. Thus, to establish a baseline for this dataset, we demonstrate an automated search of neural network based models using a novel multipleinput HyperTree MetaModel, and find a final model which makes reasonable 3D pose predictions for grasping and stacking on our dataset. The CoSTAR BSD, code, and instructions are available at sites.google.comsitecostardataset.