Simultaneous Multitask Determination And Manipulation With Visionbased Selfreasoning Network It remains a great challenge for robots to simultaneously determine the task objective and schedule the corresponding manipulation, which requires robots to autonomously perceive and understand the environments, think up a response strategy and plan a set of trajectories or actions. This paper proposes a mutitask selfreasoning neural network (MSRnet), which is an endtoend imitation learning framework consisting of a selfreasoning module and a control module. The selfreasoning module supports the task objective inference, while the control module predicts the robot motor angles for manipulation. With multitask learning, MSRnet enables robots to accomplish multiple tasks with one model without updating parameters. This paper takes three tasks as an example, involving pouring water into a cup of coffee powder, taking a spoon and stirring coffee with the spoon, which simulate the application scene of making coffee. We employ a lowcost robotic arm(less than 300) to evaluate our MSRnet on coffee maker(CM) dataset which is collected by ourselves. MSRnet with 640*480 resolution inputs can achieve 83.3% success rate on multitask test, 76.7% success rate on complex environment test. The result verifies that MSRnet can accurately infer the task objective and generate corresponding task actions simultaneously.

A Multitask Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes Autonomous robotic grasping plays an important role in intelligent robotics. However, how to help the robot grasp specific objects in object stacking scenes is still an open problem, because there are two main challenges for autonomous robots: (1) it is a comprehensive task to know what and how to grasp; (2) it is hard to deal with the situations in which the target is hidden or covered by other objects. In this paper, we propose a multitask convolutional neural network for autonomous robotic grasping, which can help the robot find the target, make the plan for grasping and finally grasp the target step by step in object stacking scenes. We integrate visionbased robotic grasping detection and visual manipulation relationship reasoning in one single deep network and build the autonomous robotic grasping system. Experimental results demonstrate that with our model, Baxter robot can autonomously grasp the target with a success rate of 90.6%, 71.9% and 59.4% in object cluttered scenes, familiar stacking scenes and complex stacking scenes respectively.