Deep Learning based Image Reconstruction from Brain Data during ObjectGrasping Task with a BMI Brainmachine interface systems (BMI) have allowed the control of prosthetics and robotic arms using brainwaves alone to do simple tasks such as grasping an object. However, the low throughput information of braindata decoding does not allow the robotic arm to achieve diverse grasp configurations for different types of objects. On the other hand, computer vision researchers have mostly solved the problem of robot hand configuration for objectgrasping given visual object recognition. It is then natural to think that if the robotic arm could decode from brain data the image of the object that the user intends to grasp, then it could automatically decide the type of grasping to execute. For this reason, in this paper, we propose a method to decode visual representations of objects from brain data towards improving robot arm grasp configurations. More specifically, we recorded EEG data during an objectgrasping experiment in which the participant had to control a robotic arm using a BMI to grasp an object. We also recorded images of the object and developed a multimodal representation of the encoded brain data and object image. Given this representation, the objective was to reconstruct the image given that only half of the image (the brain data encoding) was provided. To achieve this goal, we developed a deep convolutional autoencoder that learned a noisefree joint manifold of brain data encoding and the object image. After training, the autoencoder was able to reconstruct the missing part of the object image given that only brain data encoding was provided.

Object Recall from NaturalLanguage Descriptions for Autonomous Robotic Grasping Humans acquire grasping skills through repeated interaction with the objects; and they internalize the knowledge of various physical attributes of such objects. Even blindfolded, humans can reasonably estimate a suitable grasp pose given only the objectxe2x80x99s description. Human brain relies on the knowledge of having seen such objects and recalling its physical features such as shape, size, weight to estimate feasible grasps. In such a scenario, knowledge of an objectxe2x80x99s features is key to executing a successful grasp. This paper aims to provide this xe2x80x99recallxe2x80x99 ability to robots by proposing a methodology to represent human memory of objects with a dataset of objects and their physical features. A joint probability distance metric is derived, which can match the natural language descriptions to reference object features so as to recall and identify a particular object from the reference dataset, thereby facilitating better grasp planning. Experiments were performed to evaluate the accuracy of object recall, and simulation of an anthropomorphic robot hand was conducted for object grasping based on the recalled object features. The experiment results showed the accuracy of the proposed metric and effectiveness in object grasping.