3D Concept Grounding on Neural Fields
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we address the challenging problem of 3D concept grounding ( i.e . segmenting and learning visual concepts) by looking at RGBD images and reasoningabout paired questions and answers.
Existing visual reasoning approaches typicallyutilize supervised methods to extract 2D segmentation masks on which concepts aregrounded.
In contrast, humans are capable of grounding concepts on the underlying3D representation of images.
However, traditionally inferred 3D representations( e.g. , point clouds, voxelgrids and meshes) cannot capture continuous 3D featuresflexibly, thus making it challenging to ground concepts to 3D regions based on thelanguage description of the object being referred to.
To address both issues, wepropose to leverage the continuous, differentiable nature of neural fields to segmentand learn concepts.
Specifically, each 3D coordinate in a scene is representedas a high dimensional descriptor.
Concept grounding can then be performed bycomputing the similarity between the descriptor vector of a 3D coordinate andthe vector embedding of a language concept, which enables segmentations andconcept learning to be jointly learned on neural fields in a differentiable fashion.
As a result, both 3D semantic and instance segmentations can emerge directly fromquestion answering supervision using a set of defined neural operators on top ofneural fields ( e.g. , filtering and counting).
Experimental results on our collected
P ART N ET -R
EASONING dataset show that our proposed framework outperformsunsupervised / language-mediated segmentation models on semantic and instancesegmentation tasks, as well as outperforms existing models on the challenging 3Daware visual reasoning tasks.
Furthermore, our framework can generalize well tounseen shape categories and real scans.
Introduction
Visual reasoning, the ability to utilize compositional operators to perform complex visual questionanswering tasks ( e.g., counting, comparing and logical reasoning), has become a challenging problemthese years since they go beyond pattern recognition and bias exploitation [19, 22, 2].
Consideran image of a table pictured in Figure 1.
We wish to construct a method that is able to accuratelyground the concepts and reason about the image such as the number of legs the pictured table has.
Existing works typically address this problem by utilizing a supervised segmentation model for legs,and then applying a count operator on extracted segmentation masks [28].
However, as illustrated in Figure 1, in many visual reasoning problems, the correct answer depends on a very small portion ofa given image, such as a highly occluded back leg, which many existing 2D segmentation systemsmay neglect.
Humans are able to accurately ground the concepts from images and answer suchquestions by reasoning on the underlying 3D representation of the image [43].
In this underlying3D representation, the individual legs of a table are roughly similar in size and shape, withoutunderlying issues of occlusion of different legs, making reasoning significantly easier and moreflexible.
In addition, such an intermediate 3D representation further abstracts confounding factorstowards reasoning, such as the underlying viewing direction from which we see the input shape.
To
[Figure]
Question
[Figure]
resemble more human-like reasoning capability, in this paper, we propose a novel concept groundingframework by utilizing an intermediate 3D representation.
A natural question arises – what makes a good intermediate 3D representation for concept grounding?
Such an intermediate representation should be discovered in a weakly-supervised manner and efficientto infer, as well as maintain correspondences between 3D coordinates and feature maps in a fullydifferentiable and flexible way, so that segmentation and concept learning can directly emergefrom this inferred 3D representation with supervision of question answering.
While traditional 3Drepresentations ( e.g. , point clouds, voxelgrids and meshes) are efficient to infer, they can not providecontinuous 3D feature correspondences flexibly, thus making it challenging to ground concepts to3D coordinates based on the language descriptions of objects being referred to.
To address bothissues, in this work, we propose to leverage the continuous, differentiable nature of neural fields asthe intermediate 3D representation, which could be easily used for segmentation and concept learningthrough question answering.
To parameterize a neural field for reasoning, we utilize recently proposed Neural Descriptor Fields(NDFs) [41] which assign each 3D point in a scene a high dimensional descriptor.
Such descriptorsare learned in a weakly-supervised manner, and implicitly capture the hierarchical nature of a scene,by leveraging implicit feature hierarchy learned by a neural network.
Portions of a scene relevant toa particular concept could be then differentiably extracted through a vector similarity computationbetween each descriptor and a corresponding vector embedding of the concept, enabling conceptsegmentations on NDFs to be differentiably learned.
In contrast, existing reasoning approaches utilizesupervised segmentation models to pre-defined concepts, which prevents reasoning on conceptsunseen by the segmentation model [28], and further prevents models from adapting segmentationsbased on language descriptions.
On top of NDFs, we define a set of neural operators for visual reasoning.
First, we construct a filteroperator, and predict the existence of a particular concept in an image.
We also have a query operatorwhich queries about an attribute of the image.
In addition, we define a neural counting operator,which quantifies the number of instances of a particular concept.
To evaluate the performances of our 3D Concept Grounding (3D-CG) framework, we conductexperiments on our collected P ART N ET -R EASONING dataset, which contains approximately 3Kimages of 3D shapes and 20K quesion-answer pairs.
We find that by integrating our neural operatorswith NDF, our framework is able to effectively and robustly answer a set of visual reasoning questions.
Simultaneously, we observe the emergence of 3D segmentations of concepts, at the semantic andinstance level, directly from the underlying supervision of downstream visual question answering.
Our contributions can be summed up as follows: 1) we propose 3D-CG, which utilizes the differentiable nature of neural descriptor fields (NDF) to ground concepts and perform segmentations; 2)we define a set of neural operators, including a neural counting operator on top of the NDF; 3) with3D-CG, semantic and instance segmentations can emerge from question answering supervision; 4)our 3D-CG outperforms baseline models in both segmentation and reasoning tasks; 5) it can alsogeneralize well to unseen shape categories and real scans.
Related works
Visual Reasoning.
There have been different tasks focusing on learning visual concepts from naturallanguage, such as visually-grounded question answering [11, 12] and text-image retrieval [45].
Visualquestion answering that evaluates machine’s reasoning abilities stands out as a challenging task as itrequires human-like understanding of the visual scene.
Numerous visual reasoning models have beenproposed in recent years.
Specifically, MAC [16] combined multi-modal attention for compositional reasoning.
LCGN [15] built contextualized representations for objects to support relational reasoning.
These methods model the reasoning process implicitly with neural networks.
Neural-symbolicmethods [53, 28, 3] explicitly perform symbolic reasoning on the objects representations and languagerepresentations.
Specifically, they use perception models to extract 2D masks for 3D shapes as afirst step, and then execute operators and ground concepts on these pre-segmented masks, but arelimited to a set of pre-defined concepts.
In this paper, we present an approach towards grounding 3Dconcepts in a fully differentiable manner, with which 3D segmentations can emerge from questionanswering supervision.
Neural Fields.
Our approach utilizes neural fields also known as neural implicit representations,to parameterize an underlying 3D geometry of a shape for reasoning.
Implicit fields have beenshown to accurately represent shape topology and 3D geometry [1, 20, 32].
Recent works improvetraditional shape fields by using continuous neural networks [33, 5, 36, 38, 39, 51, 13].
These worksare typically used for reconstruction [33, 36, 29] or geometry processing [51, 37, 8].
We use a neuraldescriptor field (NDF) similar to [29] to infer the 3D representations from 2D images, and leveragethe features learnt from the NDF for visual reasoning and language grounding.
Neural fields havealso been used to represent dynamic scenes [34, 10], appearance [42, 30, 35, 52, 40], physics [23],robotics [18], acoustics [27] and more general multi-modal signals [9].
Conditional neural field suchas [25, 48] allows us to encode instance-specific information into the field, which is also utilized byour framework to encode the RGBD data of 3D shapes.
There are also some works that integratesemantics or language in neural fields [17, 46].
However, they mainly focus on using language formanipulation, editing or generation.
In this work, we utilize descriptors defined on neural fields [41]to differentiably reason in 3D given natural language questions.
We refer readers to recent surveys[49, 44] for more related works on neural fields.
Language-driven Segmentation.
Recent works have been focused on leveraging language forsegmentation.
Specifically, BiLD [14] distills the knowledge from a pre-trained image classificationmodel into a two-stage detector.
MDETR [21] is an end-to-end modulated detector that detectsobjects in an image conditioned on a raw text query, like a caption or a question.
LSeg [24] uses a textencoder to compute embeddings of descriptive input labels together with a transformer-based imageencoder that computes dense per-pixel embeddings of the input image.
GroupViT [50] learns to groupimage regions into progressively larger arbitrary-shaped segments with a text encode.
These methodstypically use an encoder to encode the text and do not have the ability to modify the segmentationsbased on the question answering loss.
LORL [47] uses the part-centric concepts derived fromlanguage to facilitate the learning of part-centric representations.
However, they can only improvethe segmentation results but cannot generate segmentations from scratch with language supervision.
3D Concept Grouding
In Figure 2, we show an overall framework of our 3D Concept Grounding (3D-CG), which seamlesslybridges the gap between 3D perception and reasoning by using Neural Descriptor Field (NDF).
Specifically, we first use NDF to assign a high dimensional descriptor vector for each 3D coordinate,and run a semantic parsing module to translate the question into neural operators to be executed on theneural field.
The concepts, also the parameters of the operators, possess concept embeddings as well.
Upon executing the operators and grounding concepts in the neural field, we perform dot productattention between the descriptor vector of each coordinate and the concept embedding vector tocalculate the score (or mask probability) of a certain concept being grounded on a coordinate.
We alsopropose a count operator which assigns point coordinates into different slots for counting the numberof instances of a part category.
Both the NDF and the neural operator have a fully differentiabledesign.
Therefore, the entire framework can be end-to-end trained, and the gradient from the questionanswering loss can be utilized to optimize both the NDF perception module and the visual reasoningmodule and jointly learn all the embeddings and network parameters.
3.1 Model Details
3.1.1 Neural Descriptor Fields
In this paper, we utilize a neural field to map a 3D coordinate x using a descriptor function f ( x ) andextract the feature descriptor of that 3D coordinate:
[Equation]
[Figure]
values.
Concepts are also mapped to embedding vectors.
Attention is calculated between the descriptors and concept embeddings to yield new mask probabilities and filter the set of coordinates being referred as the concept.
We also define a count operator which segments a part category into different slots.
The sum of the maxpooled mask probabilities of all slots can be used for counting, and the loss can be back propagated to optimize NDF as well as the embeddings.
Semantic and instance segmentations emerge from question-answering supervision with 3D-CG.
where v is the descriptor representation which encodes information about shape properties ( e.g. ,geometry, appearance, etc ).
In our setting, we condition the neural field on the partial point cloud P derived from RGB-Dimages.
We use a PointNet-based encoder E to encode P .
The descriptor function then becomesf ( x , E ( P ))
= v .
This continuous and differentiable representation maps each 3D coordinate x to adescriptor vector v .
Since the concepts to be grounded in most reasoning tasks focus on geometry andappearance, we leverage two pre-training tasks to parameterize f and learn the features concerningthese properties.
Shape Reconstruction.
Inspired by recent works on using occupancy networks [29] to reconstructshapes and learn shape representations, we also use an MLP decoder D 1 which maps each descriptorvector v to an occupancy value: D 1 : v (cid:55)→ o ∈ [0 , 1] , where the occupancy value indicates whetherthe 3D coordinate is at the surface of a shape.
Color Reconstruction.
We use another MLP D 2 to decode an RGB color of each coordinate:
D 2 : v (cid:55)→ c ∈ R 3 , where the color value indicates the color of the 3D coordinate on the shape.
After learning decoders D i through these pre-training tasks, to construct the Neural Descriptor Field(NDF)
f ( x ) , we concatenate all the intermediate activations of D i when decoding a 3D point x[41].
The resultant descriptor vector v can then be used for concept grounding.
Since v consists ofintermediate activations of D i , they implicitly capture the hierarchical nature of a scene, with earlieractivations corresponding to lower level features and later activations corresponding to higher levelfeatures.
3.1.2 Concept Quantization
Visual reasoning requires determining the attributes ( e.g. , color, category) of a shape or shape part,where each attribute contains a set of visual concepts ( e.g. , blue, leg).
As shown in Figure 2, visualconcepts such as legs, are represented as vectors in the embedding space of part categories.
Theseconcept vectors are also learned by our framework.
To ground concepts in the neural fields, wecalculate the dot products ⟨· , ·⟩ between the concept vectors and the descriptor vectors from the neuralfield.
For example, to calculate the score of a 3D coordinate belonging to the category leg , we takeits descriptor vector v and compute p = ⟨ v , v leg ⟩ , where v leg is the concept embedding of leg .
3.1.3 Semantic Parsing
To transform natural language questions into a set of primitive operators that can be executed on theneural field, a semantic parsing module similar to that in [53] is incorporated into our framework.
This module utilizes a LSTM-based Seq2Seq to translate questions into a set of fundamental operatorsfor reasoning.
These operators takes concepts and attributes as their parameters ( e.g. , Filter(leg) ,
Query(color) ).
3.1.4 Neural operators on Neural Descriptor Fields
The operators extracted from natural language questions can then be executed on the neural field.
Dueto the differentiable nature of the neural field, the whole execution process is also fully-differentiable.
We represent the intermediate results in a probabilistic manner: for the i -th sampled coordinate x i ,it is represented by a descriptor vector v i and has an attention mask m i ∈ [0 , 1] .
m i denotes theprobability that a coordinate belongs to a certain set, and is initialized using the occupancy valueoutput by D 1 : m i
= o i .
There are three kinds of operators output by the semantic parsing module, we will illustrate how eachoperator executes on the descriptor vectors to yield the outputs.
Filter Operator.
The filter operator “selects outs" a set of coordinates belonging to the concept cby outputting new mask probabilities:
[Equation]
182 Query Operator.
The query operator asks about an attribute a on selected coordinates and outputsconcept ˆ c of the maximum probability:
[Equation]
Count Operator.
The count operator intends to segment the coordinates in the same part categoryinto instances and count the number of instances.
Inspired by previous unsupervised segmentationmethods which output different parts in various slots [26, 4], we also allocate a set of slots fordifferent instances of a category.
The j -th slot s j has its own embedding vector v sj .
The score s ijof the i -th coordinate belonging to the j -th slot is also computed by dot product s ij = ⟨ v , v s j ⟩ .
Wefurther take softmax to normalize the probabilities across slots.
Since we want to count the instancesof a part that was previously selected out, we also take the minimum of previous mask probabilitiesand the mask probabilities of each slot.
The result of counting ( i.e. , the number of part instances n ) isobtained by summing up the maximum probability in each slot.
[Equation]
3.1.5 Segmentation
Note that based on the above operators, not only visual reasoning can be performed, but we can alsodo semantic segmentation and instance segmentation.
Semantic Segmentation.
We can achieve semantic segmentations by executing Query(category)on each coordinate and output the category ˆ c with the maximum probability for each point.
Instance Segmentation.
We can perform instance segmentations by first doing semanticsegmentation, and then for each output ˆ c we further execute count (ˆ c ) .
3.2 Training Paradigm
Optimization Objective.
During training, we jointly optimize the NDF and the concept embeddingsby minimizing the loss as:
[Equation]
Specifically, the NDF loss consists of two parts: the binary cross entropy classification loss betweenthe ground-truth occupancy and the predicted occupancy, and the MSE loss between the ground-truthrgb value and the predicted rgb value.
We also define three kinds of losses for three types of questions: 1) For questions that ask aboutwhether a part exist, we use an MSE loss ∥ a − max( m i )
∥ 2 where a is the ground-truth answer andmax( m i ) takes the maximum mask probability among all sampled 3D coordinates; 2) For questionsthat query about an attribute, we take the cross entropy classification loss between the predicted concept category ˆ c and the ground-truth concept category; 3) For counting questions, we first use an
MSE loss ∥ a − n ∥ 2 between the answer and the number output by summing up the maxpool valuesof all the slots, and further add a loss to ensure that the mask probabilities of the top K values inthe top a slots ( i.e. , the slots with the maximum maxpool values) should be close to 1, where K is ahyper-parameter.
During training, we first train the NDF module only for N 1 epochs and then jointlyoptimize the NDF module and the concept embeddings.
The Seq2seq model for semantic parsing istrained independently with supervision.
Curriculum Learning.
Motivated by previous curriculum strategies for visual reasoning [28], weemploy a curriculum learning strategy for training.
We split the questions according to the length ofoperators they are parsed into.
Therefore, we start with questions with only one neural operator ( e.g. ,“is there a yellow part of the chair" can be parsed into Filter(Color) ).
Experiments
4.1 Experimental Setup
4.1.1 Dataset
Instead of object-based visual reasoning [19, 53, 28] where objects are spatially disentangled, whichmakes segmentations quite trivial, we seek to explore part-based visual reasoning where segmentations and reasoning are both harder.
To this end, we collect a new dataset, PartNet-Reasoning ,which focuses on visual question answering on the PartNet [31] dataset.
Specifically, we render approximately 3K RGB-D images from shapes of 4 categories: Chair, Table, Bag and Cart , with 8question-answer pairs on average for each shape.
We have three question types:
exist_part queriesabout whether a certain part exists by having the filter operator as the last operator; query_part usesquery operator to query about an attribute ( e.g. , part category or color) of a filtered part; count_partutilizes the count operator to count the number of instances of a filtered part.
We are interested inwhether the reasoning tasks can benefit the segmentations of the fine-grained parts, as well as whetherthe latent descriptor vectors by NDF can result in better visual reasoning.
4.1.2 Evaluation Tasks
Reasoning.
We report the visual question answering accuracy on the PartNet-Reasoning datasetw.r.t the three types of questions on all four categories.
Segmentation.
We further evaluate both the performances of semantic segmentation and instancesegmentation on our dataset.
As stated in 3.1.5, semantic segmentation can be performed by queryingthe part category with the maximum mark probability of each coordinate, and filtering the coordinatesaccording to category labels, and instance segmentation can be achieved by using the count operatoron each part category.
We report the mean per-label Intersection Over Union (IOU).
4.1.3 Baselines
We compare our approach to a set of different baselines, with details provided in the supp.
material.
Unsupervised Segmentation Approaches.
We compare 3D-CG with two additional 3D unsupervised segmentation models, and further consider how we may integrate such approaches with conceptgrounding.
First, we consider CVXNet [7], which decomposes a solid object into a collection ofconvex polytope using neural implicit fields.
In this baseline, a number of primitives are selected anda convex part is generated in each primitive.
Next, we consider BAENet [4] which decomposes ashape into a set of pre-assigned branches, each specifying a portion of a whole shape.
In practice,we tuned with the original codes of BAENet on our dataset and found that nearly all coordinateswould be assigned to a single branch.
This is probably because the original model is trained on voxelmodels, while ours is on partial point cloud and contains more complex shapes, thus posing morechallenges.
Therefore, for BAENet we use an additional loss which enforces that at least some of thebranches should have multiple point coordinates assigned to positive occupancy value.
The primitivesin CVXNet and the branches in BAENet are similar to slots in our paper.
Reasoning Baselines.
We compare our approach to a set of different reasoning baselines.
Firstwe consider PointNet+LSTM and NDF+LSTM , which use the features by PointNet or our NDFmodule, concatenated with the features of the questions encoded by a LSTM model. The finalanswer distribution is predicted with an MLP. Next, we compare with MAC , a commonly-used
[Table] [Table]
Question: Is there any other part of the same color as the legs?
Shape & Question
Operator, Reference & Output Answer
Step1: Filter (Leg)
Step2: Query (Color)
Answer:
Yes
Dark Blue Step3: Other & Filter (Dark Blue)
Filter (Yellow)
Answer:
Yes
Shape & Question
Operator, Reference & Output Answer
Question : What is the category of the green part of the chair?
Step1: Filter (Green)
Step2: Query (Category) Seat
Answer: Seat
Question :
How many legs does this table have?
Step1: Filter (Leg)
Step2: Count Answer: attention-based model for visual reasoning.
We add a depth channel to the input to the model.
Finallywe compare with CVX+L and BAE+L are neural-symbolic models that use the similar languagemediated segmentation framework from [47].
Specifically, they first initiate the segmentations inthe slots, and each slot has a slot feature vector.
The operators from the questions are executedsymbolically on the slot features.
Question answering loss can be also propagated back to finetunethe slot segmentations and features.
Segmentation Baselines.
For the segmentation tasks, we compare our approach with unsupervisedsegmentation approaches CVX and BAE described above.
We further construct language-mediatedvariants CVX-L and BAE-L .
For semantic segmentation by CVX and BAE, we use the manualgrouping methods as in [7].
For semantic segmentation by language-mediated models, we use thefilter operator to filter the slots belonging to the part categories.
4.2 Experimental Results
4.2.1 Reasoning & Concept Grounding
Table 1 shows the VQA accuracy among all models on our dataset.
We can see that overall, our3D-CG outperforms baseline modeles by a large margin.
Language-mediated neural-symbolicmethods (CVX+L & BAE+L) are better than neural methods in the exist_part and query_partquestion types, but in general they are far worse than our method.
This is because the slot-basedrepresentations pre-select the parts and make it easier for the concepts to attend to specific regions.
However, wrong pre-segmentations are also hard to be corrected during the training of reasoning.
The accuracies for the count_part type further demonstrates this point, where language-mediatedbaselines have extremely low accuracies, and segmentations from Figure 4a also show that even
Shapes
Semantic Segmentation
[Figure]
Instance
[Figure]
Generalization to
New Categories
Count (Wheel)
Filter (Leg)
Generalization to
Real Scans
Instance Segmentation
Filter (Back)
[Figure]
Visualization of segmentation results.
Shapes
Semantic Segmentation
BAE
[Figure]
Instance Segmentation
[Figure]
CVX+L
[Figure]
Generalization to New Categories
Count (Wheel)
Filter (Leg)
Generalization to
[Figure]
Scans
Instance Segmentation
[Figure]
with supervision from question answering loss, BAE and CVX cannot segment the instances ofa part category ( e.g. , legs and wheels).
Neural methods such as MAC and NDF+LSTM performwell in the count_part type in some categories, probably due to some shortcuts of the PartNetdataset ( e.g. , most bags have two shoulder straps).
In comparison to both neural and neural-symbolicmethods, our method performs well in all three question types.
This is because the regions attendedby concepts are dynamically improved with the question answering loss, while the advantage ofexplicit disentanglement of concept learning and reasoning process is maintained.
Figure 3 shows some qualitative examples of the reasoning process of our method.
Specifically,the questions are parsed into a set of neural operators.
For each step, the neural operator takes theoutput of the last step as its input.
For the filter operator, we visualize the attended regions with maskprobabilities greater than 0.5.
From the figure, we can see that our method can attend to the correctregion to answer the questions, as well as segment the instances correctly for counting problems.
This attention-based reasoning pipeline is also closer than previous neural-symbolic methods to the [Table] way that humans perform reasoning.
For example, when asked the question “What is the category ofthe green part of the chair?", humans would directly pay attention to the green region regardless ofthe segmentations of the rest of the chair.
However, previous neural symbolic methods [53, 28, 47]segment the chair into different parts first and then select the green part, which is very unnatural.
Table 2 shows the semantic segmentation and instance segmentation results
[Table]
and Figure 4a visualizessome qualitative examples.
We can see that overall, our 3D-CG can better segment parts thanunsupervised or language-mediated methods.
Other methods experience some common problemssuch as failing to segment the instances within a part category ( e.g. , one of the legs is alwaysmerged with the seat), or experience unclear boundaries and segment one part into multiple parts( e.g. , segmenting the chair back or the tabletop into two parts).
In general, language-mediatedmethods are better than pure unsupervised methods, which indicates that the supervision of questionanswering does benefit the segmentation modules.
However, a lot of the wrong segmentationscannot be corrected by reasoning, resulting in bad results for counting questions for these models.
Incontrast, our 3D-CG can learn segmentations well using question answering loss, mainly because thesegmentations emerge from question-answering supervision without any restrictions.
4.2.3 Generalization
In Figure 4b, we show some qualitative examples of how our 3D-CG trained on seen categories canbe generalized directly to unseen categories and real scans.
We show results of semantic and instancesegmentations, as well as visualize the parts that are referred to in the questions.
For generalizing to new categories, we use the model trained on carts to ground and count theinstances of the concept “wheel" in cars.
We can see that the instance segmentation results on thewheels are not perfect because one wheel is in wrong position.
However, most of the wheels aredetected and the model manages to output the right count.
We also use the model trained on chairs tofilter the legs of a bed.
All the legs are successfully selected out by our 3D-CG.
We also use real 3D scans from the RedWood dataset [6] to estimate 3D-CG’s ability to generalize toreal scenes.
We use a single-view scan to reconstruct partial point cloud and remove the backgroundsuch as the floor.
We input the partial point cloud into our 3D-CG and output the segmentations onthe ground-truth voxels.
For generalizing to bicycles, we use the 3D-CG model trained on carts.
Wecan see that 3D-CG can find all instances in the bicycle and detect both wheels.
Furthermore, 3D-CGtrained on chairs can also be generalized to chairs in real scans.
Discussion
Conclusion.
In this paper, we propose 3D-CG, which leverages the continuous and differentiablenature of neural descriptor fields to segment and learn concepts in the 3D space.
We define aset of neural operators on top of the neural field, with which not only can semantic and instancesegmentations emerge from question-answering supervision, but visual reasoning can also be wellperformed.
Limitations and Future Work.
A limitation of our underlying framework with 3D-CG is thatwhile we show transfer results on real scenes, our approach is only trained with synthetic scenes andquestions.
While we believe our proposed operations is general-purpose, an interesting directionof future work would be scale our framework to directly train on complex real-world scenes, andground more concepts from natural language on these real scenes is worth delving into in the future.
A promising direction of future work would be to explore the combination of large pre-trainedvisual-language models and volumetric rendering on multi-view images.