Programming by Visual Demonstration for PickandPlace Tasks using Robot Skills In this paper, we present a visionbased robot programming system for pickandplace tasks that can generate programs from human demonstrations. The system consists of a detection network and a program generation module. The detection network leverages convolutional pose machines to detect the keypoints of the objects. The network is trained in a simulation environment in which the train set is collected and autolabeled. To bridge the gap between reality and simulation, we propose a design method of transform function for mapping a real image to synthesized style. Compared with the unmapped results, the Mean Absolute Error (MAE) of the model completely trained with synthesized images is reduced by 23% and the False Negative Rate FNR (FNR) of the model finetuned by the real images is reduced by 42.5% after mapping. The program generation module provides a humanreadable program based on the detection results to reproduce a realworld demonstration, in which a longshort memory (LSM) is designed to integrate current and historical information. The system is tested in the real world with a UR5 robot on the task of stacking colored cubes in different orders.

Synthesizing Robot Manipulation Programs from a Single Observed Human Demonstration Programming by Demonstration (PbD) lets users with little technical background program a wide variety of manipulation tasks for robots, but it should be as intuitive as possible for users while requiring as little time as possible. In this paper, we present a Programming by Demonstration system that synthesizes manipulation programs from a single observed demonstration, allowing users to program new tasks for a robot simply by performing the task once themselves. A humanintheloop interface helps users make corrections to the perceptual state as needed. We introduce Object Interaction Programs as a representation of multiobject, bimanual manipulation tasks and present algorithms for extracting programs from observed demonstrations and transferring programs to a robot to perform the task in a new scene. We demonstrate the expressivity and generalizability of our approach through an evaluation on a benchmark of complex tasks.