OCID -Ref: A 3 D Robotic Dataset With Embodied Language For Clutter Scene Grounding (2021)

Abstract: To effectively apply robots in working environments and assist humans, it is essential to develop and evaluate how visual grounding (VG) can affect machine performance on occluded objects. However, current VG works are limited in working environments, such as offices and warehouses, where objects are usually occluded due to space utilization issues. In our work, we propose a novel OCID-Ref dataset featuring a referring expression segmentation task with referring expressions of occluded objects. OCID-Ref consists of 305,694 referring expressions from 2,300 scenes with providing RGB image and point cloud inputs. To resolve challenging occlusion issues, we argue that it’s crucial to take advantage of both 2D and 3D signals to resolve challenging occlusion issues. Our experimental results demonstrate the effectiveness of aggregating 2D and 3D signals but referring to occluded objects still remains challenging for the modern visual grounding systems. OCID-Ref is publicly available at https://github.com/lluma/OCID-Ref

## introduction visual grounding (vg), which aims to locate the object according to a structured language query, is a crucial task in natural language processing (nlp), computer vision (cv), and robotics. recent vg studies most focus on web-crawled images such as @xcite @xcite . however, vg for human-robot interaction (hri) is less explored. most of the images in existing vg datasets are people and daily necessities, e.g., refcoco contains mainly persons, cars, and cats, which are separated and therefore easier to detect. nevertheless, working spaces such as offices or warehouses, where robots are usually applied to assist works, are usually crowded, and objects are overlapped with each figure 1 : a hard case where visual grounding (vg) network fails to predict the occluded object in clutter scene. our dataset provides more such cases than other datasets, which are commonly seen in the working spaces, like offices and warehouses. previous work @xcite suggested that a system that uses language for humancomputer interaction can help non-professionals instruct robots to complete technical work and collaborate. recent research pointed out that vg plays an important role in hri. @xcite utilized vg to resolve ambiguity in grasping tasks. @xcite studied how the robot learns about objects and tasks in an environment via nature language queries. therefore, explicit language instructions and good referring (grounding) expressions are pivotal in human-robot interaction and improve communication between non-expert humans and robots. some efforts have been made to collect vg datasets. refcoco @xcite and cops-ref @xcite utilize web-crawled images and manually label language expressions. a limitation is that images alone do not provide precise position cues, which are essential for various downstream robotic tasks such as grasping. a recent work, sun-spot @xcite , utilizes a depth channel for object detection and referring expression segmentation tasks. another existing dataset, scanrefer @xcite , uses more accurate multi-view point clouds for 3d signals. however, both sun-spot and scanrefer do not address occlusion issues, which is ordinary in working spaces and more challenging due to more compositions of shapes of each object. as shown in figure 1 , when an object (the red plastic bag) is blocked in an occluded environment, the shape of the object could be deformed and increase vg difficulty. observing this, we propose a novel ocid-ref dataset with two key features: (1) for each scene, we utilize both rgb image and point cloud to provide multi-modal signals for learning system development. (2) ocid-ref scenes have higher clutter level compared to existing datasets, as shown in figure 2 . hence, the model capability for resolving challenging occlusion issues could be evaluated. to the best of our knowledge, ocid-ref is the only existing dataset supporting the above features, and therefore allows vg task in grasping scenario. experimental results demonstrate that occluded scenes are more challenging to modern vg baselines. we observe 27% to 34% performance drops on referring expression segmentation tasks. also, utilizing 3d information continually improves performance across all clutter levels. furthermore, fusing 2d and 3d features reach the best performance on all clutter levels. we suggest that ocid-ref dataset could pave a new path for vg research in hri and benefit the research community and application developments. ## dataset and task to open up a new way for vg research in hri, we collect a novel ocid-ref dataset by the following steps: (1) we leverage a robotic object cluttered indoor dataset, ocid @xcite , which consists of complex clutter-level scenes with rich 3d point cloud data and the point-wise instance labels for each occluded objects. (2) we manually annotate fine-grained attributes and relations such as color, shape, size relation or spatial relation. ## experiments we conduct referring expression segmentation experiments on our collected ocid-ref dataset and scanrefer @xcite dataset. we compare different modalities, clutter levels, and regular expression lengths and provide a comprehensive analysis to pave a new path for future research. we also conduct the grasp experiment using different modality data as input, and the details are described in appendix a. all free(f) touching(t) stacked(s) decrease rate(f->s) ↓ 2d 0.512 / 0.501 0.673 / 0.660 0.450 / 0.496 0.450 / 0.433 33.23% / 34.29% 3d 0.588 / 0.580 0.745 / 0.751 0.589 / 0.583 0.507 / 0.489 31.97% / 34.92% fusion 0.634 / 0.637 0.763 / 0.769 0.64 / 0.651 0.551 / 0.540 27.71% / 29.75% evaluation metric we use acc@0.25iou as our metric to measure the thresholded accuracy where the positive predictions have a higher intersection over union (iou) with the ground truths than the thresholds.