See It All: Contextualized Late Aggregation for 3 D Dense Captioning (2024)

Abstract: 3D dense captioning is a task to localize objects in a 3D scene and generate descriptive sentences for each object. Recent approaches in 3D dense captioning have adopted transformer encoder-decoder frameworks from object detection to build an end-to-end pipeline without hand-crafted components. However, these approaches struggle with contradicting objectives where a single query attention has to simultaneously view both the tightly localized object regions and contextual environment. To overcome this challenge, we introduce SIA (See-It-All), a transformer pipeline that engages in 3D dense captioning with a novel paradigm called late aggregation. SIA simultaneously decodes two sets of queries—context query and instance query. The instance query focuses on localization and object attribute descriptions, while the context query versatilely captures the region-of-interest of relationships between multiple objects or with the global scene, then aggregated afterwards (i.e., late aggregation) via simple distance-based measures. To further enhance the quality of contextualized caption generation, we design a novel aggregator to generate a fully informed caption based on the surrounding context, the global environment, and object instances. Extensive experiments on two of the most widely-used 3D dense captioning datasets demonstrate that our proposed method achieves a significant improvement over prior methods.

## introduction vote2cap-detr @xcite has emulated the transformer encoder-decoder pipeline from object detection @xcite to alleviate these issues and fashioned an end-to-end pipeline for 3d dense captioning. powered by transformer attentions, this method contextualizes individual objects (i.e., self-attention with other proposals throughout the global scene) to generate dense captions. nevertheless, compared to the notable advancements that object detection has experienced, the direct application of this architecture has failed to fully leverage the contextual information required for 3d dense captioning. dense captioning has to perform precise object localization while generating captions that either independently describe an object's attributes (e.g., a wooden chair) or describe the object within its contextual environment (e.g., a chair in front of the tv). this presents a challenging scenario where the feature representation for a single query must encompass both accurate local features for localization or attribute-based caption generation, alongside incorporating contextual features that dynamically span neighboring regions or the broader global scene. focusing attention on local features can enhance localization and detailed attribute description but reduce sensitivity to the surrounding context. conversely, spreading attention to include the context can improve understanding of the environmental description but at the cost of localization accuracy. in this paper, we propose a pipeline engaging a novel late aggregation paradigm called sia (i.e., ## preliminary in this preliminary, we start with a basic transformer-based end-to-end 3d dense captioning pipeline @xcite . the caption head is attached to the top of the existing 3d object detection pipeline @xcite with vote queries that establish captions for each pinpointed object throughout the scene in an objectby-object manner. afterward, we discuss why this object-centric application of transformer attention is unsuitable for 3d dense captioning. ## method in this section, we introduce a transformer encoderdecoder pipeline that engages our novel late aggregation paradigm for 3d dense captioning. in previous methods, the transformer attention aggregates contextual information per-object, where a single feature is used to perform localization, generate localized attributes and simultaneously capture the surrounding context area. sia is designed to capture the unique region of interest for each caption. local attribute descriptions are generated with localized features. in contrast, contextualized captions that include relationships with other objects or the entire scene are generated with a separately decoded feature irrelevant to localization objectives. then, captions involving identical objects are aggregated via distance (i.e., late aggregation) to consist of the final caption. the overall pipeline is illustrated in figure 2 . ## conclusion in the 3d dense captioning task, the description of an object within a 3d scene encompasses not only the intrinsic characteristics of the object but also the relationship with surrounding objects and the spatial relationship of the object with respect to the overall space. in this work, we propose a novel approach that independently generates captions with different region of interests and aggregates them afterwards to enhance local-global sensitivity of descriptions. through extensive experiments on benchmark datasets, our method significantly improves 3d dense captioning over previous approaches, demonstrating the importance of an integrated understanding of objects, surroundings, and overall space for caption generation.