ARK it S cene R efer: Text-based Localization of Small Objects in Diverse Real-World 3 D Indoor Scenes (2023)

Abstract: 3D referring expression comprehension is a task to ground text representations onto objects in 3D scenes. It is a crucial task for indoor household robots or augmented reality devices to localize objects referred to in user instructions. However, existing indoor 3D referring expression comprehension datasets typically cover larger object classes that are easy to localize, such as chairs, tables, or doors, and often overlook small objects, such as cooking tools or office supplies. Based on the recently proposed diverse and high-resolution 3D scene dataset of ARKitScenes, we construct the ARKitSceneRefer dataset focusing on small daily-use objects that frequently appear in real-world indoor scenes. ARKitSceneRefer contains 15k objects of 1,605 indoor scenes, which are significantly larger than those of the existing 3D referring datasets, and covers diverse object classes of 583 from the LVIS dataset. In empirical experiments with both 2D and 3D state-of-the-art referring expression comprehension models, we observed the task difficulty of the localization in the diverse small object classes.

## introduction however, when we develop robots that follow instructions and perform indoor household tasks, such robots are expected to find out and localize typically small objects that are required for household tasks. for example, for developing cooking robots, robots are expected to find cooking tools, foods, and ingredients. similarly, to develop autonomous laundry robots, they need to find small objects such as clothes. we assume that finding these small objects in 3d scenes is meaningful for household robots, although these objects are difficult to be captured and often overlooked from the existing real-world scan-based 3d scene rec datasets. in this study, we propose a new rec dataset named arkitscenerefer in 3d scenes in that we concentrate on the fineness and diversity of the objects referred to in instructions. arkitscenerefer is based on the recently proposed arkitscenes dataset. arkitscenes @xcite is a finegrained photo-realistic 3d scan for diverse 1, 661 venues and 5, 047 scenes. based on arkitscenes, we extract small objects that are not covered in the previous 3d scene datasets well. we first apply the 2d object detector detic @xcite with lvis @xcite object classes for the video frames (i.e., 2d images) from which 3d scenes in arkitscenes are constructed. next, we extract object labels and positions and map them to 3d scenes with ray-casting and clustering by db-scan @xcite . we confirm that most small objects are detected in 3d scenes with this approach. we then annotate referring expressions that people use to locate the objects via amazon mechanical turk, while manually revising the incorrectly detected object labels. figure 1 shows an example of our arkitscenerefer dataset. we finally collect 1, 605 scenes with 583 object classes and more than 15k objects and their corresponding referring expressions. in addition, we conduct experiments with both 2d and 3d models to localize objects in ark-itscenerefer. our 2d models are based on the state-of-the-art 2d rec models mdetr @xcite and ofa @xcite . our 3d models are based on an adaptation of the stateof-the-art 3d rec models scanrefer @xcite and 3dvg-transformer @xcite on our dataset. our contributions are as follows: (i) creating the first object localization dataset concentrating on the small objects in daily indoor scenes upon the high-resolution 3d scene dataset of arkitscenes, (ii) attaching more than 15k referring expressions with human annotations with a significantly large number of object classes, and (iii) comparisons with the state-of-the-art 2d and 3d rec models on arkitscenerefer. ## dataset we describe the methods to construct the ark-itscenerefer dataset in this section. ## model following the previous 3d referring expression studies @xcite @xcite , we compare 2d to 3d rec models. 4 ## conclusion in this paper, we introduced a new 3d rec dataset, arkitscenerefer, for small objects. ark-itscenerefer consists of 15, 553 referring expressions for 1, 605 scenes in arkitscenes. we found that conventional 3d models cannot get high accuracy on our dataset. we also confirmed that the performance of the 2d models varied significantly depending on the input video frames. in the future, we plan to use the confidence scores of 2d models for image selection. we hope that our dataset will be useful in the 3d rec community.