Voxel-informed Language Grounding (2022)

Abstract: Natural language applied to natural 2D images describes a fundamentally 3D world. We present the Voxel-informed Language Grounder (VLG), a language grounding model that leverages 3D geometric information in the form of voxel maps derived from the visual input using a volumetric reconstruction model. We show that VLG significantly improves grounding accuracy on SNARE, an object reference game task. At the time of writing, VLG holds the top place on the SNARE leaderboard, achieving SOTA results with a 2.0% absolute improvement.

## introduction embodied robotic agents hold great potential for providing assistive technologies in home environments @xcite , and natural language provides an intuitive interface for users to interact with such systems @xcite . for these systems to be effective, they must be able to reliably ground language in perception @xcite . despite typically being paired with 2d images, natural language that is grounded in vision describes a fundamentally 3d world. for example, consider the grounding task in figure 1 , where the agent must select a target chair against a distractor given the description "the swivel chair with 6 wheels." although the agent is provided with multiple images revealing all of the wheels on each chair, it must be able to properly aggregate information across images to successfully differentiate them, something that requires reasoning about their 3d geometry at some level. in this work, we show how language grounding performance may be improved by leveraging 3d prior knowledge. our model, voxel-informed language grounder (vlg), extracts 3d voxel maps using a pre-trained volumetric reconstruction model, which it fuses with multimodal features from a large-scale vision and language model in order to reason jointly over the visual and 3d geometric properties of objects. we focus our investigation within the context of snare @xcite , an object reference game where an agent must ground natural language describing common household objects by their geometric and visual properties, showing that grounding accuracy significantly improves by incorporating information from predicted 3d volumes of objects. at the time of writing, vlg achieves sota performance on snare, attaining an absolute improvement of 2.0% over the next closest baseline. code to replicate our results is publicly available. 2 based representations (hudson and manning, 2019; @xcite , which do not encode 3d properties of objects. most prior work tying language to 3d representations has largely focused on generating 3d structures conditioned on language, rather than using them as intermediate representations for language grounding as we do here. specifically, prior work has performed language conditioned generation at the scene @xcite , pose @xcite , or object @xcite level. more recently, a line of work has explored referring expression grounding in 3d by mapping referring expressions of objects to 3d bounding boxes localizing them in point clouds of indoor scenes @xcite @xcite . standard approaches follow a twotiered process where an object proposal system will first provide bounding boxes for candidate objects, and a scoring module will then compute a compatibility score between each box and the referring expression in order to ground it. at a more granular level, @xcite learn alignments from language to object parts by training agents on a reference game over point cloud representations of objects. in contrast, in this work we focus on augmenting language grounding over 2d rgb images using structured 3d representations derived from them. for the task of visual language navigation, prior work has shown how a persistent 3d semantic map may be used as an intermediate representation to aid in selecting navigational waypoints @xcite . the semantic maps, however, represent entire scenes with individual voxels representing object categories, rather than their geometry. in this work, we show how a more granular occupancy map representing objects' geometry can improve language grounding performance. closest to our work is that of @xcite , which presents a method for mapping language to 3d features within scenes from the clevr @xcite dataset. their system generates 3d feature maps inferred from images and then grounds language directly to 3d bounding boxes or coordinates. their method assumes, however, that dependency parse trees are provided for the natural language inputs, and it is trained with supervised alignments between noun phrases and the 3d representations, which vlg does not require. ## voxel-informed language grounder we consider a task where an agent must correctly predict a target object v t against a distractor v c given a natural language description w @xmath0, ..., w m } of the target. for each object, the agent is provided with n 2d views an agent for this task is represented by a scoring function s(v, w) ∈ [0, 1], computing the compatibility between the target description and the 2d views of each object, and is used to select the maximally scoring candidate. we first use unimodal encoders to encode the language description into e @xmath1; e w ]). prior approaches to this problem @xcite directly input this fused representation to a scoring module to produce a score s(e vw ). they do not explicitly reason about the 3d properties of the observed objects, requiring the models to learn them implicitly. in contrast, our voxel-informed language grounder augments the scoring function s with explicit 3d volumetric information e @xmath2. the volumetric information (in the form of a factorization of a voxel occupancy map in r w ×h×d ) is first fused into a joint representation with the language using a multimodal voxellanguage module e @xmath3; e w ]). the scoring function then produces a score based on all three modalities s([e vw ; e ow ]). ## language grounding evaluation evaluation. we test our method on the snare benchmark @xcite . snare is a language grounding dataset which augments acronym @xcite , a grasping dataset built off of shapenetsem @xcite , with natural language annotations of objects. snare presents an object reference game where an agent must correctly guess a target object against a distractor. in each instance of the game, the agent is provided with a language description of the target as well as multiple 2d views of each object. snare differentiates between visual and blindfolded object descriptions. visual descriptions primarily include attributes such as name, shape, and color (e.g. "classic armchair with white seat"). in contrast, blindfolded descriptions include attributes such as shape and parts (e.g. "oval back and vertical legs"). the train/validation/test sets were generated by splitting over (207 / 7 / 48) shapenetsem object categories, respectively containing (6,153 / 371 / 1,357) unique object instances and (39,104 / 2,304 / 8,751) object pairings with referring expressions. renderings are provided for each object extracting features from image regions, with the ground truth bounding boxes for each region (i.e. view) being provided. because this baseline is not open-source, we report the original numbers from @xcite . ## results we present test set performance for vlg and the snare baselines reported by @xcite . we also present average performance for trained models over 3 seeds with standard deviations on the validation set. ## discussion we have presented the voxel-informed language grounder (vlg), a model which leverages explicit 3d information from predicted volumetric voxel maps to improve language grounding performance. vlg achieves sota results on snare, and ablations demonstrate the effectiveness of using this 3d information for grounding. we hope this paper may inspire future work on integrating structured 3d representations into language grounding tasks.