Human-in-the-loop Robotic Grasping Using BERT Scene Representation (2022)

Abstract: Current NLP techniques have been greatly applied in different domains. In this paper, we propose a human-in-the-loop framework for robotic grasping in cluttered scenes, investigating a language interface to the grasping process, which allows the user to intervene by natural language commands. This framework is constructed on a state-of-the-art grasping baseline, where we substitute a scene-graph representation with a text representation of the scene using BERT. Experiments on both simulation and physical robot show that the proposed method outperforms conventional object-agnostic and scene-graph based methods in the literature. In addition, we find that with human intervention, performance can be significantly improved. Our dataset and code are available on our project website https://sites.google.com/view/hitl-grasping-bert .

## introduction grasping @xcite is a fundamental task for robot systems. it is useful for warehousing, manufacturing, medicine, retail, and service robots. one setting in robotic grasping is to grasp object orderly without disturbing the remaining in cluttered scenes @xcite @xcite (called collision-free grasp). to solve this problem, a typical method @xcite parses the input into a scene graph first @xcite ), in order to infer the spatial relation between objects and select a collisionfree object for grasping. in particular, as shown in figure . 2(a), nodes in a scene graph represent objects and edges represent their spatial relationship. such structural knowledge can effectively improve the grasping performance as compared to an endto-end model without scene structure information @xcite ). as a structured representation of the scene, the scene graph has a few limitations. for example, it can be costly to manually label scene graphs, and the amount of existing labeled data is quite small. in practice, due to variance in the working environment, it can be necessary to calibrate a scene understanding model when deployed on physical robots @xcite . this problem can be regarded as a domain adaptation task, which requires a certain amount of labeled scene graph data @xcite . in addition, graphs are relatively abstract and thus inconvenient for human-robot interaction. we consider a natural language representation of the scene for substituting a scene graph structure representation. as shown in figure . 2(a), small texts such as "notebook placed under pliers" and "apple on notebook" are used to indicate the recognized objects and their stacking relations. compared with a scene graph structure, natural language scene representation has the following advantages. first, the cost of manual labeling is relatively lower thanks to the availability of speech recognition systems @xcite and the relative independence from labeling gui @xcite . second, the state-of-theart pre-trained representation models @xcite can be used to improve scene understanding, which contains external knowledge beyond a scene graph structure. third, online human interaction can be achieved by using human input of the natural language scene representation to replace incorrect robot scene understanding through speech communication 2 . as shown in figure . 1(c), we adopt the model of @xcite by substituting the scene graph with a text description of the object to grasp and its spatial context, and using a neural image-to-text model for scene understanding and a pre-trained language model to represent the scene text for visual language grounding in subsequent grasping decisions. we compare the model performance with a dominant two-stage end-to-end planar grasping baseline @xcite ) and the baseline scene graph model ( figure. 1(b) ). for all models, the grasping backbone is implemented using an extended version model of @xcite with extra scene knowledge input. for training and evaluation, we make extensions to the visual manipulation relationship dataset (vmrd) @xcite ) by manually adding text descriptions and scene graphs to the scenes, resulting in a new dataset l-vmrd, as shown in figure. 2(a). experimental results show that (1) human language description can be a highly competitive alternative to the scene graph representation, giving better results for grasping; (2) online human language intervention is useful for improving the final grasping results, which is a new form of human-inthe-loop grasping. this indicates the promise of nlp models, especially pre-trained language models, for human-robot interaction. to our knowledge, we are the first to consider explicit textual scene representation and human intervention correction for robot grasping decisions, where bert (kenton and toutanova, 2019) is firstly introduced into the internal structure of a robotic model as a state representation. ## task and dataset the input of the robotic grasping task is an image from robotic camera observation and the output is a grasping configuration (a grasping bounding box). as shown in figure .1(c), we introduce a language description for the scene (image) during the inference process. we take a human-in-the2 we adopt the typed text to simulate the process here since voice recognition is beyond our research scope loop setting, where the language description can be obtained from a scene understanding model (selfexplanation) or human (human-intervention). existing grasping datasets (e.g., @xcite ) cannot be employed directly, because they do not include scene knowledge in human language. hence, we develop an extended language version of vmrd, named l-vmrd, to evaluate our method. l-vmrd is an integrated dataset, and each sample is organized as a 6-tuple (image, language descriptions, scene graph, object bounding box, grasping bounding box, surface) shown in figure language description in l-vmrd object pairs in an image are sampled and labeled with a scene description. there exist many factors that affect collision-free grasping, in which stacking is the most significant one @xcite . we first label the objects with the stacking relationship, e.g., "apple on notebook" or "notebook sitting under pliers" in figure . 2(a), and then label the horizontal relationship between non-stacked. then considering the distribution of the horizontal relationships ("left", "right", "front", "back") in our dataset, we use "left" and "right" to indicate the scene relationships. scene graph the original vmrd dataset includes partial relationships for adjacent objects, encoded by a relationship tree that only reveals the stacking relationship between two objects but is not able to indicate the relationship between objects stacked directly. to facilitate inference, we add a full scene graph to encode the pair-wise relationships of objects per image, where nodes and edges present objects and relationships, respectively. detail is available in appendix a.1.3. ## our approach the overall framework is shown in figure . 3. the input scene image is fed into convolutional-based grasping model (sec. 3.1) and scene understanding model (sec. 3.2; sec. 3.3), respectively. the scene understanding model is an image-to-text component that produces a sentence that describes the object to grasp and its context, such as "toothpaste put above box". that scene description is then fused with intermediate grasping results to select objectrelated grasping candidates by pre-trained language model and language grounding model. the final grasping output is selected with high probability after re-optimization. a grasp command is sent to a real robot to complete a collision-free grasping operation. for the human-in-the-loop scenario, an extra conditional input from human-intervention will be given to correct scene description when the description from the image-to-text model is incorrect. ## evaluation we construct both simulation and physical experiments to investigate four research questions: q1: how much natural language (unstructured) scene description perform better than scene graph (structured) knowledge in collision-free grasping task? q2: how much does the pre-trained model perform better than the randomly initialized model in our task? q3: how and where does human intervention using nlp improve grasping performance under the proposed human-in-the-loop framework? q4: how does our method perform on a physical robot? does data collection from our proposed human-in-the-loop framework improve efficiency during the fine-tuning process? 4.1 settings scene and platform implementation in simulation experiments, we use the test set split in sec. 2 to evaluate our method. in the physical experiment, we collect objects and set up the placement as similar as possible to the training set, in which 2-6 objects are randomly placed (stacked) on a white table. but the reality gap is usually inevitable, especially in a robotic environment, which also tests the robustness of our method implicitly. other details of the robot and training framework are described in appendix a.5. evaluation metrics we take benchmark performance and success rate as the main evaluation metrics. the first is used in both simulation and physical experiments, and the second one is only used in the physical robot experiment: ## conclusion we investigate language scene representation to robotic grasping, which enables a robot to explain its object selection to the user and allows the user to intervene with the selection by natural language. experiments show that the proposed explainable textual scene representation outperforms both object-agnostic and scene-graph based methods. by human language intervention, the performance can be broadly increased. our results indicate the promise of using nlp models in a robotic system both as a representation and for human intervention. to our knowledge, we are the first to consider textual scene encoding and human correction in robotic grasping tasks, which can improve grasping performance using natural language (vs. w/o human-in-the-loop) and robustness (by pretrained language model).