Learning Actions from Human Demonstration Video for Robotic Manipulation Learning actions from human demonstration is an emerging trend for designing intelligent robotic systems, which can be referred as video to command. The performance of such approach highly relies on the quality of video captioning. However, the general video captioning methods focus more on the understanding of the full frame, lacking of consideration on the specific object of interests in robotic manipulations. We propose a novel deep model to learn actions from human demonstration video for robotic manipulation. It consists of two deep networks, grasp detection network (GNet) and video captioning network (CNet). GNet performs two functions: providing grasp solutions and extracting the local features for the object of interests in robotic manipulation. CNet outputs the captioning results by fusing the features of both full frames and local objects. Experimental results on UR5 robotic arm show that our method could produce more accurate command from video demonstration than stateoftheart work, thereby leading to more robust grasping performance.

An Object Attribute Guided Framework for Robot Learning Manipulations from Human Demonstration Videos Learning manipulations from videos is an inspiriting way for robots to acquire new skills. In this paper, we propose a framework that can generate robotic manipulation plans by observing human demonstration videos without special marks or unnatural demonstrated behaviors. More specifically, the framework contains a video parsing module and a robot execution module. The first module recognizes the demonstratorxe2x80x99s actions using twostream convolution neural networks, and classifies the operated objects by adopting a Mask RCNN. After that, two XGBoost classifiers are applied to further classify the objects into subject object and patient object respectively, according to the demonstratorxe2x80x99s actions. In the second module, a grammarbased parser is used to summarize the videos and generate the common instructions for robot execution. Extensive experiments are conducted on a publicly available video datasets consisting of 273 videos and manifest that our approach is able to learn manipulation plans from demonstration videos with high accuracy (73.36%). Furthermore, we integrate our framework with a humanoid robot Baxter to perform the manipulation learning from demonstration videos, which effectively verifies the performance of our framework.