3 DRP -Net: 3 D Relative Position-aware Network for 3 D Visual Grounding (2023)

Abstract: 3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description. Typically, the sentences describing the target object tend to provide information about its relative relation between other objects and its position within the whole scene. In this work, we propose a relation-aware one-stage framework, named 3D Relative Position-aware Network (3DRP-Net), which can effectively capture the relative spatial relationships between objects and enhance object attributes. Specifically, 1) we propose a 3D Relative Position Multi-head Attention (3DRP-MA) module to analyze relative relations from different directions in the context of object pairs, which helps the model to focus on the specific object relations mentioned in the sentence. 2) We designed a soft-labeling strategy to alleviate the spatial ambiguity caused by redundant points, which further stabilizes and enhances the learning process through a constant and discriminative distribution. Extensive experiments conducted on three benchmarks (i.e., ScanRefer and Nr3D/Sr3D) demonstrate that our method outperforms all the state-of-the-art methods in general.

## introduction visual grounding aims to localize the desired objects based on the given natural language description. with the rapid development and wide applications of 3d vision @xcite @xcite in recent years, 3d visual grounding task has received more and more attention. compared to the wellstudied 2d visual grounding @xcite @xcite @xcite @xcite , the input sparse point clouds in the 3d visual grounding task are more figure 1 : 3d visual grounding is the task of grounding a description in a 3d scene. in the sentences, all the words indicating the relative positions of the target object are bolded. notice that relative position relations between objects are crucial for distinguishing the target object, and the relative position-related descriptions in 3d space are complex (e.g., "above", "on the left", "in front of", and "next to", etc.) irregular and more complex in terms of spatial positional relationships, which makes it much more challenging to locate the target object. in the field of 3d visual grounding, the previous methods can be mainly categorized into two groups: the two-stage approaches @xcite @xcite @xcite @xcite and the one-stage approaches @xcite . the former ones follow the detection-and-rank paradigm, and thanks to the flexibility of this architecture, they mainly explore the benefits of different object relation modeling methods for discriminating the target object. the latter fuse visual-text features to predict the bounding boxes of the target objects directly, and enhance the object attribute representation by removing the unreliable proposal generation phase. however, these two methods still have limitations. for two-stage methods, the model performance is highly dependent on the quality of the object proposals. however, due to the sparsity and irregularity of the input 3d point cloud, sparse proposals may leave out the target object, while dense proposals will bring redundant computational costs and make the matching stage too complicated to distinguish the target object. as for the one-stage methods, although the existing approach @xcite achieves better performance, they can not capture the relative spatial relationships between objects, which makes it often fail in samples that rely on relative relation reasoning. as shown in fig. 1 , the majority of sentences in 3d visual grounding contain relative spatial relation descriptions. furthermore, due to the spatial complexity of the 3d scene, there are various relative position-related descriptions from different orientations. to further illustrate that relative position is a general and fundamental issue in 3d visual grounding tasks, we analyze the frequency of relative position words in scanrefer and nr3d/sr3d, and the results show that at least 90% of the sentences describe the relative position of objects, and most of them contain multiple spatial relations. detailed statistics can be found in supplementary materials. to alleviate above problems, we propose a onestage 3d visual grounding framework, named 3d relative position-aware network (3drp-net). our 3drp-net combines and enhances the advantages of the two-stage approaches for relations modeling and the one-stage approaches for proposalfree detection while avoiding the shortcomings of both methods. for the relations modeling, we devise a novel 3d relative position multi-head attention (3drp-ma) module, which can capture object relations along multiple directions and fully consider the interaction between the relative position and object pairs which is ignored in previous two-stage methods @xcite @xcite . specifically, we first extract features from the point cloud and description, and select key points. then, the language and visual features interacted while considering the relative relations between objects. for the relation modeling, we introduce learnable relative position encoding in different heads of the multi-head attention to capture object pair relations from different orientations. moreover, in sentences, the relative relations between objects are usually described as "object 1-relation-object 2", such as "tv is on the tv cabinet" and "curtain is hanging on the window" in fig. 1 . the relation is meaningful only in the context of object pairs, thus our relative position encoding would interact with the object pairs' feature, to better capture and focus on the mentioned relations. besides, as discussed in @xcite , point clouds only capture surface of object, and the 3d object centers are likely to be far away from any point. to accurately reflect the location of objects and learn comprehensive object relation knowledge, we sample multiple key points of each object. however, redundant key points may lead to ambiguity. to achieve disambiguation while promoting a more stable and discriminative learning process, we propose a soft-labeling strategy that uses a constant and discriminative distribution as the target label instead of relying on unstable and polarized hardlabel or iou scores. our main contributions can be summarized as follows: ## method this section introduces the proposed 3d relative position-aware network (3drp-net) for 3d visual grounding. in sec.3.1, we present an overview of our method. in sec.3.2, we dive into the techni-cal details of the 3d relative position multi-head attention (3drp-ma) module and how to comprehensively and efficiently exploit the spatial position relations in the context of object pairs. in sec.3.3 and sec.3.4, we introduce our soft-labeling strategy and the training objective function of our method. ## conclusion in this paper, we propose a relation-aware onestage model for 3d visual grounding, referred to as 3d relative position-aware network (3drp-net). 3drp-net contains novel 3drp-ma modules to exploit complex 3d relative relations within point clouds. besides, we devise a soft-labeling strategy to achieve disambiguation while promoting a stable and discriminative learning process. comprehensive experiments reveal that our 3drp-net outperforms other methods.