Original: A NALOGY -F ORMING T RANSFORMERS FOR F EW -S HOT 3D P ARSING
Revised: A NALOGY -F ORMING T RANSFORMERS FOR F EW -S HOT 3D P ARSING

Original: We present Analogical Networks, a model that casts fine-grained 3D visual parsing as analogy-forming inference: instead of mapping input scenes to part labels, which is hard to adapt in a few-shot manner to novel inputs, our model retrieves related scenes from memory and their corresponding part structures, and predicts analogous part structures in the input object 3D point cloud, via an end-to-end learnable modulation mechanism.
Revised: We present Analogical Networks, a model that casts fine-grained 3D visual parsing as analogy-forming inference: instead of mapping input scenes to part labels, which is hard to adapt in a few-shot manner to novel inputs, our model retrieves related scenes from memory and their corresponding part structures, and predicts analogous part structures in the input object 3D point cloud, via an end-to-end learnable modulation mechanism.

Original: By conditioning on more than one retrieved memories, compositions of structures are predicted, that mix and match parts across the retrieved memories.
Revised: By conditioning on more than one retrieved memories, compositions of structures are predicted, that mix and match parts across the retrieved memories.

Original: One-shot, few-shot or many-shot learning are treated uniformly in Analogical Networks, by conditioning on the appropriate set of memories, whether taken from a single, few or many memory exemplars, and inferring analogous parses.
Revised: One-shot, few-shot or many-shot learning are treated uniformly in Analogical Networks, by conditioning on the appropriate set of memories, whether taken from a single, few or many memory exemplars, and inferring analogous parses.

Original: We show Analogical Networks are competitive in many-shot settings and outperform existing state-of-the-art detection transformer models on part segmentation in few-shot scenarios, as well as paradigms of metalearning and few-shot learning.
Revised: We show Analogical Networks are competitive in many-shot settings and outperform existing state-of-the-art detection transformer models on part segmentation in few-shot scenarios, as well as paradigms of metalearning and few-shot learning.

Original: Our model successfully parses instances of novel object categories simply by expandingthe its memory, without any weight updates.
Revised: Our model successfully parses instances of novel object categories simply by expanding its memory, without any weight updates.

Original: Ask not what it is, ask what it is like.
Revised: Ask not what it is, ask what it is like.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The dominant paradigm in existing deep visual learning is to train high-capacity networks that map input observations to task-specific output labels.
Revised: The dominant paradigm in existing deep visual learning is to train high-capacity networks that map input observations to task-specific output labels.

Original: Despite their success across a plethora of tasks, these models struggle to perform well in few-shot settings where only a small set of examples are available for learning.
Revised: Despite their success across a plethora of tasks, these models struggle to perform well in few-shot settings where only a small set of examples are available for learning.

Original: Meta-learning approaches provide one promising solution to this by enabling efficient task-specific adaptation of generic models but this specialization comes at the cost of poor performance on the original tasks as well as the need to adapt separate models for each novel task.
Revised: Meta-learning approaches provide one promising solution to this by enabling efficient task-specific adaptation of generic models, but this specialization comes at the cost of poor performance on the original tasks as well as the need to adapt separate models for each novel task.

Original: We introduce Analogical Networks, a semi-parametric learning framework for 3D scene parsing that pursues analogy-driven prediction: instead of mapping the input scene to part segments and labels directly, the model reasons analogically and maps the input to modifications and compositions of past labelled visual experiences.
Revised: We introduce Analogical Networks, a semi-parametric learning framework for 3D scene parsing that pursues analogy-driven prediction: instead of mapping the input scene to part segments and labels directly, the model reasons analogically and maps the input to modifications and compositions of past labelled visual experiences.

Original: Analogical Networks encode domain knowledge explicitly in a collection of structured labelled scene memories as well as implicitly, in model parameters.
Revised: Analogical Networks encode domain knowledge explicitly in a collection of structured labelled scene memories as well as implicitly, in model parameters.

Original: Given an input 3D scene, the model retrieves relevant memories and uses them to modulate inference and segment object parts in the input point cloud.
Revised: Given an input 3D scene, the model retrieves relevant memories and uses them to modulate inference and segment object parts in the input point cloud.

Original: During modulation, the input scene and the retrieved memories are jointly encoded and contextualized via cross-attention operations.
Revised: During modulation, the input scene and the retrieved memories are jointly encoded and contextualized via cross-attention operations.

Original: The contextualized memory part features are then used to segment and label analogous parts in the 3D input scene, binding the predicted part structure to the one from memory, as shown in Figure 1.
Revised: The contextualized memory part features are then used to segment and label analogous parts in the 3D input scene, binding the predicted part structure to the one from memory, as shown in Figure 1.

Original: Given the same input scene, the output prediction changes with varying conditioning memories.
Revised: Given the same input scene, the output prediction changes with varying conditioning memories.

Original: For example, conditioned on ∗ visual memories of varying label granularity, the model segments the input in a granularity analogous to the one of the retrieved memory.
Revised: For example, conditioned on ∗ visual memories of varying label granularity, the model segments the input in a granularity analogous to the one of the retrieved memory.

Original: One-shot, few-shot or many-shot learning are treated uniformly in Analogical Networks, by conditioning on the appropriate set of memories.
Revised: One-shot, few-shot or many-shot learning are treated uniformly in Analogical Networks, by conditioning on the appropriate set of memories.

Original: This is very beneficial over methods that specifically target few-shot only scenarios, since, at test time, an agent usually cannot know whether a scene is an example of many-shot or few-shot categories.
Revised: This is very beneficial over methods that specifically target few-shot only scenarios, since, at test time, an agent usually cannot know whether a scene is an example of many-shot or few-shot categories.

Original: Analogical Networks learn to bind memory part features to input scene part segments.
Revised: Analogical Networks learn to bind memory part features to input scene part segments.

Original: Fine-grained part correspondence across two 3D scenes is not easily available.
Revised: Fine-grained part correspondence across two 3D scenes is not easily available.

Original: We devise a novel within-scene pre-training scheme to encourage correspondence learning across scenes.
Revised: We devise a novel within-scene pre-training scheme to encourage correspondence learning across scenes.

Original: We augment (rotate and deform) a given scene in two distinct ways and train the modulator to parse one of them given the other as its modulating memory, bypassing the retrieval process.
Revised: We augment (rotate and deform) a given scene in two distinct ways and train the modulator to parse one of them given the other as its modulating memory, bypassing the retrieval process.

Original: During this within-scene pre-training, we have access to the part correspondence between the memory and the input scene and we supervise the query-to-part assignment process.
Revised: During this within-scene pre-training, we have access to the part correspondence between the memory and the input scene and we supervise the query-to-part assignment process.

Original: We show this self-supervised pre-training helps our model learn to associate memory queries to similarly labelled parts across scenes without ever using cross-scene part correspondence information, as shown in Figure 1.
Revised: We show this self-supervised pre-training helps our model learn to associate memory queries to similarly labelled parts across scenes without ever using cross-scene part correspondence information, as shown in Figure 1.

Original: We test our model on the PartNet benchmark of Mo et al.
Revised: We test our model on the PartNet benchmark of Mo et al.

Original: for 3D object segmentation.
Revised: for 3D object segmentation.

Original: We compare against state-of-the-art (SOTA) 3D object segmentors, as well as meta-learning and fewshot learning (Snell et al., 2017)
Revised: We compare against state-of-the-art (SOTA) 3D object segmentors, as well as meta-learning and fewshot learning (Snell et al., 2017)

Original: baselines adapted for the task of 3D parsing.
Revised: baselines adapted for the task of 3D parsing.

Original: Our experiments show that Analogical Networks perform similarly to the parametric alone baselines in the standard many-shot train-test split and particularly shine over the parametric baselines in the few-shot setting:
Revised: Our experiments show that Analogical Networks perform similarly to the parametric alone baselines in the standard many-shot train-test split and particularly shine over the parametric baselines in the few-shot setting:

Original: Analogical Networks segment and label novel instances much better than parametric existing models, simply by expanding the memory repository with encodings of a few exemplars, even without any weight updates.
Revised: Analogical Networks segment and label novel instances much better than parametric existing models, simply by expanding the memory repository with encodings of a few exemplars, even without any weight updates.

Original: We further compare against variations of our model that consider memory retrieval and attention without memory query binding, and thus lack explicit analogy formation, as well as other ablative versions of our model to quantify the contribution of the retriever and the proposed within-scene memory-augmented pre-training.
Revised: We further compare against variations of our model that consider memory retrieval and attention without memory query binding, and thus lack explicit analogy formation, as well as other ablative versions of our model to quantify the contribution of the retriever and the proposed within-scene memory-augmented pre-training.

Original: Our code and models are publicly available in the project webpage: http://analogicalnets.github.io/ .
Revised: Our code and models are publicly available in the project webpage: http://analogicalnets.github.io/ .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Few-shot prediction: meta-learning and learning associations
Revised: Few-shot prediction: meta-learning and learning associations

Original: A key goal for our approach is to enable accurate inference in few-shot settings.
Revised: A key goal for our approach is to enable accurate inference in few-shot settings.

Original: Previous approaches (Finn et al., 2017; Rusu et al., 2018;
Revised: Previous approaches (Finn et al., 2017; Rusu et al., 2018;

Original: Snell et al., 2017; Wang et al., 2018b; Bar et al., 2022; Mangla et al., 2020; Wang et al., 2020; Nguyen & Todorovic, 2019; Liu et al., 2020; Tian et al., 2020;
Revised: Snell et al., 2017; Wang et al., 2018b; Bar et al., 2022; Mangla et al., 2020; Wang et al., 2020; Nguyen & Todorovic, 2019; Liu et al., 2020; Tian et al., 2020;

Original: Yang et al., 2020) that target similar settings can be broadly categorized as relying on either meta-learning or learning better associations.
Revised: Yang et al., 2020) that target similar settings can be broadly categorized as relying on either meta-learning or learning better associations.

Original: Meta-learning approaches tackle few-shot prediction by learning a generic model that can be efficiently adapted to a new task of interest from a few labelled examples.
Revised: Meta-learning approaches tackle few-shot prediction by learning a generic model that can be efficiently adapted to a new task of interest from a few labelled examples.

Original: While broadly applicable, these methods result in catastrophic forgetting of the original task during adaptation and thus require training a new model for each task of interest.
Revised: While broadly applicable, these methods result in catastrophic forgetting of the original task during adaptation and thus require training a new model for each task of interest.

Original: Moreover, the goal of learning generic and rapidly adaptable models can lead to suboptimal performance over the base tasks with abundant data.
Revised: Moreover, the goal of learning generic and rapidly adaptable models can lead to suboptimal performance over the base tasks with abundant data.

Original: An alternative approach for the few-shot setting is to learn better associations.
Revised: An alternative approach for the few-shot setting is to learn better associations.

Original: For example, the category of a new example maybe inferred by transferring the label(s) from the one (or few) closest samples (Snell et al., 2017; Sung et al., 2018).
Revised: For example, the category of a new example maybe inferred by transferring the label(s) from the one (or few) closest samples (Snell et al., 2017; Sung et al., 2018).

Original: While this approach obviates the need for adapting models and can allow prediction in few-shot and many-shot settings, the current approaches are only applicable to global prediction, e.g., image labels.
Revised: While this approach obviates the need for adapting models and can allow prediction in few-shot and many-shot settings, the current approaches are only applicable to global prediction, e.g., image labels.

Original: Our work can be viewed as extending such association-based methods to allow predicting fine-grained and generic visual structures using our proposed modulation-based prediction mechanism.
Revised: Our work can be viewed as extending such association-based methods to allow predicting fine-grained and generic visual structures using our proposed modulation-based prediction mechanism.

Original: Memory-augmented neural networks Analogical Networks is a type of memory-augmented neural network.
Revised: Memory-augmented neural networks Analogical Networks is a type of memory-augmented neural network.

Original: Memory-augmentation of parametric models permits fast learning with few examples, where data are saved in and can be accessed from the memory immediately after their acquisition (Santoro et al., 2016), while learning via parameter update is slow and requires multiple gradient iterations on de-correlated examples.
Revised: Memory-augmentation of parametric models permits fast learning with few examples, where data are saved in and can be accessed from the memory immediately after their acquisition (Santoro et al., 2016), while learning via parameter update is slow and requires multiple gradient iterations on de-correlated examples.

Original: External memories have recently been used to scale up language models (Borgeaud et al., 2021; Khandelwal et al., 2019), and alleviate from the limited context window of parametric transformers (Wu et al., 2022), as well as to store knowledge in the form of entity mentions (de Jong et al., 2021), knowledge graphs (Das et al., 2022) and question-answer pairs (Chen et al., 2022).
Revised: External memories have recently been used to scale up language models (Borgeaud et al., 2021; Khandelwal et al., 2019), and alleviate from the limited context window of parametric transformers (Wu et al., 2022), as well as to store knowledge in the form of entity mentions (de Jong et al., 2021), knowledge graphs (Das et al., 2022) and question-answer pairs (Chen et al., 2022).

Original: Memory attention layers in these models are used to influence the computation of transformer layers and have proven very successful for factual question answering, but also for sentence completion over their parametric counterparts.
Revised: Memory attention layers in these models are used to influence the computation of transformer layers and have proven very successful for factual question answering, but also for sentence completion over their parametric counterparts.

Original: In-context learning
Revised: In-context learning

Original: In-context learning (ICL) (Brown et al., 2020) aims to induce a model to perform a task by feeding in input-output examples along with an unlabeled query example.
Revised: In-context learning (ICL) (Brown et al., 2020) aims to induce a model to perform a task by feeding in input-output examples along with an unlabeled query example.

Original: The primary advantage of in-context learning is that it enables a single model to perform many tasks immediately without fine-tuning.
Revised: The primary advantage of in-context learning is that it enables a single model to perform many tasks immediately without fine-tuning.

Original: Analogical Networks are in-context learners in that they infer the part segmentation of an object 3D point cloud in context of retrieved labelled object 3D point clouds.
Revised: Analogical Networks are in-context learners in that they infer the part segmentation of an object 3D point cloud in context of retrieved labelled object 3D point clouds.

Original: While in language models ICL emerges at test time while training unsupervised (and out-of-context) for language completion, Analogical Networks are trained in-context with related examples using supervision and self-supervision.
Revised: While in language models ICL emerges at test time while training unsupervised (and out-of-context) for language completion, Analogical Networks are trained in-context with related examples using supervision and self-supervision.

Original: Although in prompted language models (Huang et al., 2022; Brown et al., 2020)
Revised: Although in prompted language models (Huang et al., 2022; Brown et al., 2020)

Original: the input-output pairs are currently primarily decided by the engineer, in Analogical Networks they are automatically inferred by the retriever.
Revised: the input-output pairs are currently primarily decided by the engineer, in Analogical Networks they are automatically inferred by the retriever.

Original:  A NALOGICAL N ETWORKS FOR 3D OBJECT PARSING
Revised:  A NALOGICAL N ETWORKS FOR 3D OBJECT PARSING

Original: The architecture of Analogical Networks is illustrated in Figure 2.
Revised: The architecture of Analogical Networks is illustrated in Figure 2.

Original: Analogical Networks are analogyforming transformer networks for part segmentation where queries that decode entities in the input scene are supplied by retrieved part-encoded labelled scenes, as well as by the standard set of scene-agnostic parametric queries of detection transformers (Carion et al., 2020).
Revised: Analogical Networks are analogyforming transformer networks for part segmentation where queries that decode entities in the input scene are supplied by retrieved part-encoded labelled scenes, as well as by the standard set of scene-agnostic parametric queries of detection transformers (Carion et al., 2020).

Original: When a memory part query is used to decode a part segmentation in the input scene, we say the two parts, in the memory and the input, are put into correspondence.
Revised: When a memory part query is used to decode a part segmentation in the input scene, we say the two parts, in the memory and the input, are put into correspondence.

Original: By using memory queries to decode parts in the input, our model forms analogies between detected part graphs in the input scene and part graphs in the modulating memories.
Revised: By using memory queries to decode parts in the input, our model forms analogies between detected part graphs in the input scene and part graphs in the modulating memories.

Original: Then, metadata, such as semantic labels, attached on memory queries automatically propagate to the detected parts.
Revised: Then, metadata, such as semantic labels, attached on memory queries automatically propagate to the detected parts.

Original: Analogical Networks are comprised of two main modules: (i) A retriever, that takes as input a 3D object point cloud and a memory repository of labelled 3D object point clouds, and outputs a set of relevant memories for the scene at hand, and (ii) a modulator, that jointly encodes the memories and the input scene and predicts its 3D part segmentation and labelling.
Revised: Analogical Networks are comprised of two main modules: (i) A retriever, that takes as input a 3D object point cloud and a memory repository of labelled 3D object point clouds, and outputs a set of relevant memories for the scene at hand, and (ii) a modulator, that jointly encodes the memories and the input scene and predicts its 3D part segmentation and labelling.

Original: Retriever The retriever has access to a memory repository of labelled 3D object point clouds.
Revised: Retriever The retriever has access to a memory repository of labelled 3D object point clouds.

Original: Each labelled training example is a memory in this repository.
Revised: Each labelled training example is a memory in this repository.

Original: Examples labelled with different label granularities constitute separate memories.
Revised: Examples labelled with different label granularities constitute separate memories.

Original: The retriever encodes each memory example as well as the input point cloud into distinct normalized 1D feature encodings.
Revised: The retriever encodes each memory example as well as the input point cloud into distinct normalized 1D feature encodings.

Original: The top- k memories are retrieved by computing an inner product between the input point cloud feature and the memory features.
Revised: The top- k memories are retrieved by computing an inner product between the input point cloud feature and the memory features.

Original: Modulator
Revised: Modulator

Original: The modulator takes as input the retrieved memories and the unlabelled input point cloud and predicts part segments and semantic labels for the present scene.
Revised: The modulator takes as input the retrieved memories and the unlabelled input point cloud and predicts part segments and semantic labels for the present scene.

Original: The input scene is encoded into a set of 3D point features.
Revised: The input scene is encoded into a set of 3D point features.

Original: Each memory scene is encoded into a set of 1D part encodings, one for each annotated part in the memory, which we call memory part queries f m ∈ R c × M , in accordance to parametric queries used to decode objects in DETR (Carion et al., 2020).
Revised: Each memory scene is encoded into a set of 1D part encodings, one for each annotated part in the memory, which we call memory part queries f m ∈ R c × M , in accordance to parametric queries used to decode objects in DETR (Carion et al., 2020).

Original: The input points, memory part queries, and parametric queries are contextualized via a set of cross and self-attention operations, that iteratively update all queries and point features.
Revised: The input points, memory part queries, and parametric queries are contextualized via a set of cross and self-attention operations, that iteratively update all queries and point features.

Original: Then, each of the contextualized queries predicts a segmentation mask proposal through inner-product with the contextualized point features, an associated confidence score and a semantic label for the predicted part.
Revised: Then, each of the contextualized queries predicts a segmentation mask proposal through inner-product with the contextualized point features, an associated confidence score and a semantic label for the predicted part.

Original: For parts decoded by parametric queries f p we predict semantic labels through a categorical softmax layer.
Revised: For parts decoded by parametric queries f p we predict semantic labels through a categorical softmax layer.

Original: Parts decoded by memory part queries f m simply inherit the semantic label of the corresponding memory query.
Revised: Parts decoded by memory part queries f m simply inherit the semantic label of the corresponding memory query.

Original: These part segmentation proposals are matched to ground-truth part binary masks using Hungarian matching (Carion et al., 2020).
Revised: These part segmentation proposals are matched to ground-truth part binary masks using Hungarian matching (Carion et al., 2020).

Original: For the mask proposals that are matched to a ground-truth mask, we compute the segmentation loss, which is a per-point binary cross-entropy loss (Vu et al., 2022; Cheng et al., 2021b).
Revised: For the mask proposals that are matched to a ground-truth mask, we compute the segmentation loss, which is a per-point binary cross-entropy loss (Vu et al., 2022; Cheng et al., 2021b).

Original: We also supervise the confidence score of all queries and the predicted semantic labels of the parametric queries f p .
Revised: We also supervise the confidence score of all queries and the predicted semantic labels of the parametric queries f p .

Original: For more implementation details, please see the Appendix, Section 6.1.
Revised: For more implementation details, please see the Appendix, Section 6.1.

Original: Our modulator network resembles detection and segmentation transformers for 2D images (Carion et al., 2020; Cheng et al., 2021a) where a set of learnable 1D vectors, termed parametric queries, iteratively cross-attend to input image features and self-attend among themselves to predict object segmentation masks and their semantic labels in the input scene.
Revised: Our modulator network resembles detection and segmentation transformers for 2D images (Carion et al., 2020; Cheng et al., 2021a) where a set of learnable 1D vectors, termed parametric queries, iteratively cross-attend to input image features and self-attend among themselves to predict object segmentation masks and their semantic labels in the input scene.

Original: The key difference between our modulator network and existing detection transformers are that retrieved memory entity encodings are used as queries, i.e., candidates for decoding parts in the input point cloud, alongside the standard set of scene-agnostic parametric queries, as shown in Figure 2.
Revised: The key difference between our modulator network and existing detection transformers are that retrieved memory entity encodings are used as queries, i.e., candidates for decoding parts in the input point cloud, alongside the standard set of scene-agnostic parametric queries, as shown in Figure 2.

Original: Additional differences are that we update the point features alongside the queries in the cross-attention layers, which we found helped performance.
Revised: Additional differences are that we update the point features alongside the queries in the cross-attention layers, which we found helped performance.

Original: Finally, we operate in 3D point clouds as opposed to 2D images.
Revised: Finally, we operate in 3D point clouds as opposed to 2D images.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: Analogical Networks aim to learn to associate parts in the retrieved memory graphs with parts in the input point cloud.
Revised: Analogical Networks aim to learn to associate parts in the retrieved memory graphs with parts in the input point cloud.

Original: Semantic part labels can only partially supervise this fine-grained part association, since retrieved memories can be too different from the input scene.
Revised: Semantic part labels can only partially supervise this fine-grained part association, since retrieved memories can be too different from the input scene.

Original: We train our model in two stages to facilitate this fine-grained association learning:
Revised: We train our model in two stages to facilitate this fine-grained association learning:

Original: [Figure]
Revised: [Figure]

Original: Within-scene pre-training: We apply two distinct augmentations (rotations and deformations (Kim et al., 2021)) to each training 3D point cloud, and use one as the input scene and the other as the modulating memory, as shown in Figure 3.
Revised: Within-scene pre-training: We apply two distinct augmentations (rotations and deformations (Kim et al., 2021)) to each training 3D point cloud, and use one as the input scene and the other as the modulating memory, as shown in Figure 3.

Original: In this case, we have access to groundtruth part associations between the parts of the memory and the input scene, which we use to supervise each memory part query to decode the corresponding part in the input cloud, and we do not use Hungarian matching.
Revised: In this case, we have access to groundtruth part associations between the parts of the memory and the input scene, which we use to supervise each memory part query to decode the corresponding part in the input cloud, and we do not use Hungarian matching.

Original: We call this within-scene pre-training since both the input and memory come from the same scene instance.
Revised: We call this within-scene pre-training since both the input and memory come from the same scene instance.

Original: Cross-scene training: During cross-scene training, the modulating memories are sampled from the top- k retrieved memories per input scene.
Revised: Cross-scene training: During cross-scene training, the modulating memories are sampled from the top- k retrieved memories per input scene.

Original: We
Revised: We

Original: show in
Revised: show in

Original: our experimental
Revised: our experimental

Original: section that
Revised: section that

Original: accurate memory query to input part associations emerge during Hungarian matching in cross-scene training, even without using part association ground-truth, thanks to the within-scene pre-training.
Revised: accurate memory query to input part associations emerge during Hungarian matching in cross-scene training, even without using part association ground-truth, thanks to the within-scene pre-training.

Original: This is important, as often we cannot have finegrained annotations of structure correspondence across scene exemplars.
Revised: This is important, as often we cannot have finegrained annotations of structure correspondence across scene exemplars.

Original: In Analogical Networks, the retrieval process is not end-to-end differen tiable with respect to the downstream scene parsing task.
Revised: In Analogical Networks, the retrieval process is not end-to-end differen tiable with respect to the downstream scene parsing task.

Original: This is because i) we have no ground-truth annotations for retrieval, and ii) the retrieved memories are contextualized with the input through dense cross-attention operations which do not allow gradients to flow to the retriever’s encoding process.
Revised: This is because i) we have no ground-truth annotations for retrieval, and ii) the retrieved memories are contextualized with the input through dense cross-attention operations which do not allow gradients to flow to the retriever’s encoding process.

Original: We pre-train the encoder and modulator parameters independently of the retriever using within-scene pretraining.
Revised: We pre-train the encoder and modulator parameters independently of the retriever using within-scene pretraining.

Original: We then use the modulator’s encoder weights as the retriever’s frozen encoder (adding a parameter-free average pooling layer on top).
Revised: We then use the modulator’s encoder weights as the retriever’s frozen encoder (adding a parameter-free average pooling layer on top).

Original: In the Appendix, we include pseudo-code for within-scene training in
Revised: In the Appendix, we include pseudo-code for within-scene training in

Original: Algorithm 1 and cross-scene training in Algorithm 2.
Revised: Algorithm 1 and cross-scene training in Algorithm 2.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test Analogical Networks on PartNet (Mo et al., 2019), an established benchmark for 3D object segmentation.
Revised: We test Analogical Networks on PartNet (Mo et al., 2019), an established benchmark for 3D object segmentation.

Original: PartNet contains 3D object instances from multiple object categories, annotated with parts in three different levels of granularity.
Revised: PartNet contains 3D object instances from multiple object categories, annotated with parts in three different levels of granularity.

Original: We split PartNet object categories into base and novel categories.
Revised: We split PartNet object categories into base and novel categories.

Original: Our base categories are Chair, Display, Storage Furniture, Bottle, Clock, Door, Earphone,
Revised: Our base categories are Chair, Display, Storage Furniture, Bottle, Clock, Door, Earphone,

Original: Faucet, Knife, Lamp, Trash Can, Vase and our novel categories are Table, Bed, Dishwasher
Revised: Faucet, Knife, Lamp, Trash Can, Vase and our novel categories are Table, Bed, Dishwasher

Original: and
Revised: and

Original: Refrigerator.
Revised: Refrigerator.

Original: We consider two experimental paradigms: 1.
Revised: We consider two experimental paradigms: 1.

Original: Many-shot: For the exemplars of the base categories we consider the standard PartNet train/test splits.
Revised: Many-shot: For the exemplars of the base categories we consider the standard PartNet train/test splits.

Original: Our model and baselines are trained in the base category training sets, and tested on segmenting instances of the base categories in the test set.
Revised: Our model and baselines are trained in the base category training sets, and tested on segmenting instances of the base categories in the test set.

Original: Few-shot: K labelled examples from each novel category are given and the model is tasked to segment new examples of these categories.
Revised: Few-shot: K labelled examples from each novel category are given and the model is tasked to segment new examples of these categories.

Original: This tests few-shot adaptation.
Revised: This tests few-shot adaptation.

Original: We consider
Revised: We consider

Original: K = 1 and K = 5 .
Revised: K = 1 and K = 5 .

Original: We aggregate results across multiple K -shot learning tasks (episodes).
Revised: We aggregate results across multiple K -shot learning tasks (episodes).

Original: We test two versions of our model: i)
Revised: We test two versions of our model: i)

Original: AnalogicalNets single-mem , which is Analogical Networks with a single modulating memory, and ii) AnalogicalNets multi-mem , with five modulating memories.
Revised: AnalogicalNets single-mem , which is Analogical Networks with a single modulating memory, and ii) AnalogicalNets multi-mem , with five modulating memories.

Original: Unless mentioned explicitly, Analogical Networks will imply the single-memory version.
Revised: Unless mentioned explicitly, Analogical Networks will imply the single-memory version.

Original: Our experiments aim to answer the following questions:
Revised: Our experiments aim to answer the following questions:

Original: How do Analogical Networks compare against parametric-alone state-of-the-art models in many-shot and few-shot 3D object segmentation?
Revised: How do Analogical Networks compare against parametric-alone state-of-the-art models in many-shot and few-shot 3D object segmentation?

Original: How well do Analogical Networks adapt few-shot via memory expansion, without any weight update?
Revised: How well do Analogical Networks adapt few-shot via memory expansion, without any weight update?

Original: How do Analogical Networks compare against alternative memory-augmented networks where retrieved memory part encodings are attended to but not used for decoding parts? 4.
Revised: How do Analogical Networks compare against alternative memory-augmented networks where retrieved memory part encodings are attended to but not used for decoding parts? 4.

Original: How well do Analogical Networks learn part-based associations across scenes without part association ground-truth or semantic label supervision?
Revised: How well do Analogical Networks learn part-based associations across scenes without part association ground-truth or semantic label supervision?

Original: Evaluation metrics We use the Adjusted Random Index (ARI) as our label-agnostic segmentation quality metric (Rand, 1971), which is a clustering score ranging from − 1 (worst) to 1 (best).
Revised: Evaluation metrics We use the Adjusted Random Index (ARI) as our label-agnostic segmentation quality metric (Rand, 1971), which is a clustering score ranging from − 1 (worst) to 1 (best).

Original: ARI calculates the similarity between two point clusterings while being invariant to the order of the cluster centers.
Revised: ARI calculates the similarity between two point clusterings while being invariant to the order of the cluster centers.

Original: We compute 100 × ARI using the publicly available implementation of (Kabra et al., 2019).
Revised: We compute 100 × ARI using the publicly available implementation of (Kabra et al., 2019).

Original: We use mean Panoptic Quality (PQ) (Kirillov et al., 2019) for semantic part instance segmentation and mean intersection over union (mIoU) for 3D point semantic segmentation.
Revised: We use mean Panoptic Quality (PQ) (Kirillov et al., 2019) for semantic part instance segmentation and mean intersection over union (mIoU) for 3D point semantic segmentation.

Original: We compute PQ and mIoU per-shape , by treating each input scene as a mini-dataset, then aggregating the scores over the objects of a single category and then averaging over the categories.
Revised: We compute PQ and mIoU per-shape , by treating each input scene as a mini-dataset, then aggregating the scores over the objects of a single category and then averaging over the categories.

Original: Baselines We compare our model to existing models in the literature.
Revised: Baselines We compare our model to existing models in the literature.

Original: We further compare against strong parametric baseline models we develop; the latter ended up outperforming all previous existing works in the many-shot settings.
Revised: We further compare against strong parametric baseline models we develop; the latter ended up outperforming all previous existing works in the many-shot settings.

Original: We consider the following baseline models:
Revised: We consider the following baseline models:

Original: • PartNet of (Mo et al., 2019) is a 3D segmentation network with the same backbone as our model.
Revised: • PartNet of (Mo et al., 2019) is a 3D segmentation network with the same backbone as our model.

Original: This model implements “queries” as a fixed number of sets of MLPs that operate over the encoded input point cloud.
Revised: This model implements “queries” as a fixed number of sets of MLPs that operate over the encoded input point cloud.

Original: Each set contains three MLPs: one for per-pixel confidence, one for objectness and one for semantic class.
Revised: Each set contains three MLPs: one for per-pixel confidence, one for objectness and one for semantic class.

Original: The same losses used by our model are used to supervise this baseline.
Revised: The same losses used by our model are used to supervise this baseline.

Original: • DETR3D is a 3D segmentation transformer network with the same backbone as our model and similar segmentation prediction heads and losses, but without any memory retrieval, akin to the 3D equivalent of a state-of-the-art 2D image segmentor (Carion et al., 2020; Athar et al., 2021).
Revised: • DETR3D is a 3D segmentation transformer network with the same backbone as our model and similar segmentation prediction heads and losses, but without any memory retrieval, akin to the 3D equivalent of a state-of-the-art 2D image segmentor (Carion et al., 2020; Athar et al., 2021).

Original: We update the point features in the decoder layers, same as in our model, which we found to boost performance.
Revised: We update the point features in the decoder layers, same as in our model, which we found to boost performance.

Original: Contrary to DETR3D , Analogical Networks attend to external memories.
Revised: Contrary to DETR3D , Analogical Networks attend to external memories.

Original: • PrototypicalNets is an adaptation of the episodic prototypical networks for image classification of (Snell et al., 2017) to the task of 3D object part segmentation.
Revised: • PrototypicalNets is an adaptation of the episodic prototypical networks for image classification of (Snell et al., 2017) to the task of 3D object part segmentation.

Original: Specifically, given a set of N labelled point clouds, we form average feature vectors for each semantic labelled part and use them as queries to segment points into corresponding part masks through inner-product decoding.
Revised: Specifically, given a set of N labelled point clouds, we form average feature vectors for each semantic labelled part and use them as queries to segment points into corresponding part masks through inner-product decoding.

Original: Contrary to PrototypicalNets , Analogical Networks contextualize the memory queries and the input scene.
Revised: Contrary to PrototypicalNets , Analogical Networks contextualize the memory queries and the input scene.

Original: PrototypicalNets require a retriever that knows the category label of the input scene, so it has access to privileged information.
Revised: PrototypicalNets require a retriever that knows the category label of the input scene, so it has access to privileged information.

Original: Without this assumption, we got very low performance from this model.
Revised: Without this assumption, we got very low performance from this model.

Original: Ablations We compare our model to the following variants and ablative versions:
Revised: Ablations We compare our model to the following variants and ablative versions:

Original: • Re-DETR3D (Retrieval-DETR3D) is a variant of Analogical Networks that attends to retrieved part memory encodings but does not use them to decode parts.
Revised: • Re-DETR3D (Retrieval-DETR3D) is a variant of Analogical Networks that attends to retrieved part memory encodings but does not use them to decode parts.

Original: Instead, all parts are decoded from the parametric scene-agnostic queries.
Revised: Instead, all parts are decoded from the parametric scene-agnostic queries.

Original: Different from Analogical Networks, analogies cannot emerge between a memory and the input scene since this model does not represent such correspondence explicitly, but only implicitly, in the attention operations.
Revised: Different from Analogical Networks, analogies cannot emerge between a memory and the input scene since this model does not represent such correspondence explicitly, but only implicitly, in the attention operations.

Original: • AnalogicalNets w/o pretrain is our AnalogicalNets single-mem model without any within-scene (pre)training. • AnalogicalNets OrclCategoryRetrv is our AnalogicalNets single-mem model with a privileged retriever that has access to the ground-truth category label of the input and only retrieves memories of the same category label.
Revised: • AnalogicalNets w/o pretrain is our AnalogicalNets single-mem model without any within-scene (pre)training. • AnalogicalNets OrclCategoryRetrv is our AnalogicalNets single-mem model with a privileged retriever that has access to the ground-truth category label of the input and only retrieves memories of the same category label.

Original: • LabelConstr AnalogicalNets is our AnalogicalNets single-mem model where semantic part labels are used to constrain memory query to ground-truth part assignments to have matching semantic part category label during training.
Revised: • LabelConstr AnalogicalNets is our AnalogicalNets single-mem model where semantic part labels are used to constrain memory query to ground-truth part assignments to have matching semantic part category label during training.

Original: [Table]
Revised: [Table]

Original: M ANY AND FEW - SHOT 3D OBJECT SEGMENTATION
Revised: M ANY AND FEW - SHOT 3D OBJECT SEGMENTATION

Original: The PartNet benchmark provides three levels of segmentation annotations per object instance where level 3 is the most fine-grained.
Revised: The PartNet benchmark provides three levels of segmentation annotations per object instance where level 3 is the most fine-grained.

Original: We train and test our model and baselines on all three levels.
Revised: We train and test our model and baselines on all three levels.

Original: We use a learnable level embedding as additional input for our baselines PartNet and DETR3D , as is usually the case in multi-task models (Jang et al., 2022).
Revised: We use a learnable level embedding as additional input for our baselines PartNet and DETR3D , as is usually the case in multi-task models (Jang et al., 2022).

Original: In the many-shot setting, we train our model and baselines jointly across all base categories and test them across all of them as well, using the standard PartNet train/test splits.
Revised: In the many-shot setting, we train our model and baselines jointly across all base categories and test them across all of them as well, using the standard PartNet train/test splits.

Original: For Analogical Networks and Re-DETR3D , all examples in the training set become part of their memory repository.
Revised: For Analogical Networks and Re-DETR3D , all examples in the training set become part of their memory repository.

Original: In the few-shot setting, PartNet and DETR3D adapt by weight finetuning on the K -shot task.
Revised: In the few-shot setting, PartNet and DETR3D adapt by weight finetuning on the K -shot task.

Original: Analogical Networks and Re-DETR3D adapt in two ways: i) by expanding the memory of the model with the novel K -shot support examples, and ii) by further adapting the weights via fine-tuning to segment the K examples.
Revised: Analogical Networks and Re-DETR3D adapt in two ways: i) by expanding the memory of the model with the novel K -shot support examples, and ii) by further adapting the weights via fine-tuning to segment the K examples.

Original: In this case, the memory set is only the novel labelled support set instances.
Revised: In this case, the memory set is only the novel labelled support set instances.

Original: The retriever does not have access to the object category information in any of the many-shot or few-shot settings unless explicitly stated so.
Revised: The retriever does not have access to the object category information in any of the many-shot or few-shot settings unless explicitly stated so.

Original: 4.1. S EMANTICS - FREE SEGMENTATION
Revised: 4.1. S EMANTICS - FREE SEGMENTATION

Original: In this section, we train all models and baselines for semantics-free object part segmentation, without any semantic labelling objectives .
Revised: In this section, we train all models and baselines for semantics-free object part segmentation, without any semantic labelling objectives .

Original: We show quantitative semantic-agnostic many-shot and few-shot part segmentation results in Table 1.
Revised: We show quantitative semantic-agnostic many-shot and few-shot part segmentation results in Table 1.

Original: For the few-shot setting, we show both fine-tuned and non-finetuned models.
Revised: For the few-shot setting, we show both fine-tuned and non-finetuned models.

Original: For the few-shot performance, we report mean and standard deviation over 10 tasks that consist of a different subset of the K -shot support set.
Revised: For the few-shot performance, we report mean and standard deviation over 10 tasks that consist of a different subset of the K -shot support set.

Original: Our conclusions are as follows:
Revised: Our conclusions are as follows:

Original: (i) Analogical Networks dramatically outperform DETR3D in few-shot part segmentation.
Revised: (i) Analogical Networks dramatically outperform DETR3D in few-shot part segmentation.

Original: While in the many-shot setting the two models have similar performance ( ∼ 71.5% ARI), when adapting few-shot to novel categories, Analogical Networks and all their variants dramatically outperform parametric alone DETR3D , both before and after fine-tuning.
Revised: While in the many-shot setting the two models have similar performance ( ∼ 71.5% ARI), when adapting few-shot to novel categories, Analogical Networks and all their variants dramatically outperform parametric alone DETR3D , both before and after fine-tuning.

Original: (ii) Analogical networks adapt few-shot simply by memory expansion, without weight updates.
Revised: (ii) Analogical networks adapt few-shot simply by memory expansion, without weight updates.

Original: Indeed, the 5-shot performance of our multi-memory model is very close before and after fine-tuning in the novel categories
Revised: Indeed, the 5-shot performance of our multi-memory model is very close before and after fine-tuning in the novel categories

Original: (56.2% versus 57.6% ARI).
Revised: (56.2% versus 57.6% ARI).

Original: (iii) AnalogicalNets multi-mem outperform the single-memory version in few-shot learning.
Revised: (iii) AnalogicalNets multi-mem outperform the single-memory version in few-shot learning.

Original: (iv) Re-DETR3D adapt few-shot better than DETR3D .
Revised: (iv) Re-DETR3D adapt few-shot better than DETR3D .

Original: Still, before weight fine-tuning in the few-shot test set, this memory-augmented variant exhibits worse performance than our single-memory model (43.9% versus 50.3% ARI).
Revised: Still, before weight fine-tuning in the few-shot test set, this memory-augmented variant exhibits worse performance than our single-memory model (43.9% versus 50.3% ARI).

Original: (v) Both within-scene pre-training and label constraining help many-shot and few-shot learning.
Revised: (v) Both within-scene pre-training and label constraining help many-shot and few-shot learning.

Original: [Table]
Revised: [Table]

Original: provide a performance
Revised: provide a performance

Original: 4.1. S EMANTIC AND INSTANCE SEGMENTATION
Revised: 4.1. S EMANTIC AND INSTANCE SEGMENTATION

Original: We show quantitative semantic and instance segmentation results in Table 2.
Revised: We show quantitative semantic and instance segmentation results in Table 2.

Original: In this section, we train models and baselines for both object part segmentation and part labelling , except for the AnalogicalNets w/o semantics variants that are only trained for part segmentation.
Revised: In this section, we train models and baselines for both object part segmentation and part labelling , except for the AnalogicalNets w/o semantics variants that are only trained for part segmentation.

Original: AnalogicalNets w/o semantics predicts semantic part labels only via propagation from memory part queries f m that decode parts in the scene, and does not produce any semantic labels for parts decoded by parametric queries f p , so by default it will make a mistake each time a parametric query is used.
Revised: AnalogicalNets w/o semantics predicts semantic part labels only via propagation from memory part queries f m that decode parts in the scene, and does not produce any semantic labels for parts decoded by parametric queries f p , so by default it will make a mistake each time a parametric query is used.

Original: We found more than 80% of 3D points to be on parts decoded by memory part queries on average.
Revised: We found more than 80% of 3D points to be on parts decoded by memory part queries on average.

Original: For the few-shot settings in Table 2, all models are fine-tuned on the few given examples except for AnalogicalNets w/o finetune .
Revised: For the few-shot settings in Table 2, all models are fine-tuned on the few given examples except for AnalogicalNets w/o finetune .

Original: Similar to our model, PrototypicalNets propagate semantic labels of the prototypical part features.
Revised: Similar to our model, PrototypicalNets propagate semantic labels of the prototypical part features.

Original: We evaluate PrototypicalNets only for semantic segmentation since it cannot easily produce instance segments: if multiple part instances share the same semantic label, this model assumes they belong to the same semantic prototype.
Revised: We evaluate PrototypicalNets only for semantic segmentation since it cannot easily produce instance segments: if multiple part instances share the same semantic label, this model assumes they belong to the same semantic prototype.

Original: Our conclusions are as follows:
Revised: Our conclusions are as follows:

Original: (i) Analogical Networks outperform DETR3D in semantic and instance segmentation accuracy in the few-shot settings.
Revised: (i) Analogical Networks outperform DETR3D in semantic and instance segmentation accuracy in the few-shot settings.

Original: (ii) DETR3D outperforsm Analogical Networks in semantic and instance segmentation accuracy in the many-shot settings.
Revised: (ii) DETR3D outperforsm Analogical Networks in semantic and instance segmentation accuracy in the many-shot settings.

Original: A retriever that better recognizes object categories boosts performance of Analogical Networks over DETR3D in both few-shot and many-shot settings.
Revised: A retriever that better recognizes object categories boosts performance of Analogical Networks over DETR3D in both few-shot and many-shot settings.

Original: (iii) AnalogicalNets w/o semantics show very competitive semantic and instance segmentation accuracy via label propagation through memory part queries.
Revised: (iii) AnalogicalNets w/o semantics show very competitive semantic and instance segmentation accuracy via label propagation through memory part queries.

Original: This shows our model learns cross-scene part associations without any semantic information.
Revised: This shows our model learns cross-scene part associations without any semantic information.

Original: (iv) AnalogicalNets w/o semantics w/o pretrain has much worse semantic segmentation performance which suggests within-scene pretraining much helps cross-scene associations to emerge without semantic information.
Revised: (iv) AnalogicalNets w/o semantics w/o pretrain has much worse semantic segmentation performance which suggests within-scene pretraining much helps cross-scene associations to emerge without semantic information.

Original: (v) AnalogicalNets
Revised: (v) AnalogicalNets

Original: w/o finetune adapts few-shot simply by expanding its memory, and shows competitive performance against all the fine-tuned models.
Revised: w/o finetune adapts few-shot simply by expanding its memory, and shows competitive performance against all the fine-tuned models.

Original: (vi) PrototypicalNets achieves high 1-shot performance but does not scale with more data and is unable to handle both few-shot and many-shot settings efficiently.
Revised: (vi) PrototypicalNets achieves high 1-shot performance but does not scale with more data and is unable to handle both few-shot and many-shot settings efficiently.

Original: 4.1. M ULTI - CATEGORY TRAINING HELPS FEW - SHOT ADAPTATION IN A NALOGICAL N ETWORKS
Revised: 4.1. M ULTI - CATEGORY TRAINING HELPS FEW - SHOT ADAPTATION IN A NALOGICAL N ETWORKS

Original: We compare our model and baselines on their ability to improve few-shot learning performance with more diverse training data in Table 3.
Revised: We compare our model and baselines on their ability to improve few-shot learning performance with more diverse training data in Table 3.

Original: We train each model under two setups: i) training only on instances of “Chair”, which is the most common category—approximately 40% of our training examples fall into this category—(refer to Table 5 for statistics of the training data distribution) and, ii) training on “all” categories.
Revised: We train each model under two setups: i) training only on instances of “Chair”, which is the most common category—approximately 40% of our training examples fall into this category—(refer to Table 5 for statistics of the training data distribution) and, ii) training on “all” categories.

Original: We test the performance of each model on 5-shot learning.
Revised: We test the performance of each model on 5-shot learning.

Original: Our conclusions are as follows: (i) Models trained on a single category usually fail to few-shot
Revised: Our conclusions are as follows: (i) Models trained on a single category usually fail to few-shot

Original: [Table]
Revised: [Table]

Original: Analogical Networks learn better few-shot when trained across all categories, while DETR3D does not improve its few shot learning performance when trained across more categories.
Revised: Analogical Networks learn better few-shot when trained across all categories, while DETR3D does not improve its few shot learning performance when trained across more categories.

Original: generalize to other classes with or w/o fine-tuning, despite their strong performance on the training class.
Revised: generalize to other classes with or w/o fine-tuning, despite their strong performance on the training class.

Original: (ii) DETR3D does not improve in its ability to adapt few-shot with more diverse training data, in contrast to Analogical Networks.
Revised: (ii) DETR3D does not improve in its ability to adapt few-shot with more diverse training data, in contrast to Analogical Networks.

Original: In the Appendix, we show the effect of different retrieval mechanisms (6.2, 6.3) and qualitative results on more benchmarks (6.4).
Revised: In the Appendix, we show the effect of different retrieval mechanisms (6.2, 6.3) and qualitative results on more benchmarks (6.4).

Original: 4. L IMITATIONS
Revised: 4. L IMITATIONS

Original: A set of future directions that are necessary for Analogical Networks to scale beyond segmentation of single-object scenes are the following: (i) The retriever in Analogical Networks operates currently over whole object memories and is not end-to-end differentiable with respect to the downstream task.
Revised: A set of future directions that are necessary for Analogical Networks to scale beyond segmentation of single-object scenes are the following: (i) The retriever in Analogical Networks operates currently over whole object memories and is not end-to-end differentiable with respect to the downstream task.

Original: Sub-object part-centric memory representations would permit fine-grained retrieval of visual memory scenes.
Revised: Sub-object part-centric memory representations would permit fine-grained retrieval of visual memory scenes.

Original: We further plan to explore alternative supervision for the retriever module inspired by works in the language domain (Izacard et al., 2022; Izacard & Grave, 2021).
Revised: We further plan to explore alternative supervision for the retriever module inspired by works in the language domain (Izacard et al., 2022; Izacard & Grave, 2021).

Original: (ii) Scaling Analogical Networks to segmentation of complete, multi-object 3D scenes in realistic home environments requires scaling up the size of memory collection.
Revised: (ii) Scaling Analogical Networks to segmentation of complete, multi-object 3D scenes in realistic home environments requires scaling up the size of memory collection.

Original: It would further necessitate bootstrapping fine-grained object part annotations, missing from 3D scene datasets (Dai et al., 2017), by transferring knowledge of object part compositions from PartNet.
Revised: It would further necessitate bootstrapping fine-grained object part annotations, missing from 3D scene datasets (Dai et al., 2017), by transferring knowledge of object part compositions from PartNet.

Original: Such semi-supervised fine-grained scene parsing is an exciting avenue of future work.
Revised: Such semi-supervised fine-grained scene parsing is an exciting avenue of future work.

Original: (iii)
Revised: (iii)

Original: So far we have assumed the input to Analogical Networks to be a complete 3D point cloud.
Revised: So far we have assumed the input to Analogical Networks to be a complete 3D point cloud.

Original: However, this is hardly ever the case in reality.
Revised: However, this is hardly ever the case in reality.

Original: Humans and machines need to make sense of single-view, incomplete and noisy observations.
Revised: Humans and machines need to make sense of single-view, incomplete and noisy observations.

Original: Extending Analogical Networks with generative heads that not only detect analogous parts in the input, but also inpaint or complete missing parts, is a direct avenue for future work.
Revised: Extending Analogical Networks with generative heads that not only detect analogous parts in the input, but also inpaint or complete missing parts, is a direct avenue for future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented Analogical Networks, a semi-parametric model for associative 3D visual parsing that puts forward an analogical paradigm of corresponding input scenes to compositions and modifications of memory scenes and their labels, instead of mapping scenes to segmentations and semantic labels directly.
Revised: We presented Analogical Networks, a semi-parametric model for associative 3D visual parsing that puts forward an analogical paradigm of corresponding input scenes to compositions and modifications of memory scenes and their labels, instead of mapping scenes to segmentations and semantic labels directly.

Original: By casting visual parsing as analogical correspondence, Analogical Networks can few-shot learn better than parametric alone models.
Revised: By casting visual parsing as analogical correspondence, Analogical Networks can few-shot learn better than parametric alone models.

Original: One-shot, few-shot or many-shot learning are treated uniformly in Analogical Networks, by conditioning to the appropriate set of memories, whether taken from a single, few or many memory exemplars, and inferring analogous parses.
Revised: One-shot, few-shot or many-shot learning are treated uniformly in Analogical Networks, by conditioning to the appropriate set of memories, whether taken from a single, few or many memory exemplars, and inferring analogous parses.

Original: We showed
Revised: We showed

Original: Analogical Networks outperform SOTA parametric and meta-learning baselines in few-shot 3D parsing.
Revised: Analogical Networks outperform SOTA parametric and meta-learning baselines in few-shot 3D parsing.

Original: We further showed correspondences emerge across scenes without semantic supervision, as a by-product of the analogical inductive bias and our within-scene pre-training.
Revised: We further showed correspondences emerge across scenes without semantic supervision, as a by-product of the analogical inductive bias and our within-scene pre-training.

Original: In his seminal work “the proactive brain” (Bar, 2007), Moshe Bar argues for the importance of analogies and associations in human reasoning—highlighting how associations of novel inputs to analogous representations in memory can drive perceptual inference.
Revised: In his seminal work “the proactive brain” (Bar, 2007), Moshe Bar argues for the importance of analogies and associations in human reasoning—highlighting how associations of novel inputs to analogous representations in memory can drive perceptual inference.

Original: Analogical Networks operationalise these insights in a retrieval-augmented in-context 3D parsing framework, with analogy formation between retrieved memory graphs and input scene segmentations.
Revised: Analogical Networks operationalise these insights in a retrieval-augmented in-context 3D parsing framework, with analogy formation between retrieved memory graphs and input scene segmentations.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure the reproducibility of the empirical results, we include a pseudo-code of the main model components and training pipeline in the Appendix.
Revised: To ensure the reproducibility of the empirical results, we include a pseudo-code of the main model components and training pipeline in the Appendix.

Original: We have made our code publicly available on
Revised: We have made our code publicly available on

Original: GitHub.
Revised: GitHub.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work is supported by Sony AI, a DARPA Young Investigator Award, an NSF CAREER award, an AFOSR Young Investigator Award, a Verisk AI Faculty Research Award and DARPA Machine
Revised: This work is supported by Sony AI, a DARPA Young Investigator Award, an NSF CAREER award, an AFOSR Young Investigator Award, a Verisk AI Faculty Research Award and DARPA Machine

Original: Common Sense.
Revised: Common Sense.

Original:  A PPENDIX
Revised:  A PPENDIX

Original: Our Appendix is organized as follows:
Revised: Our Appendix is organized as follows:

Original: In Section 6.1, we provide implementation details and pseudocode for training Analogical Networks.
Revised: In Section 6.1, we provide implementation details and pseudocode for training Analogical Networks.

Original: In Section 6.2 we ablate Analogical Networks’ performance under varying memory retrieval schemes in 5 -shot setting and show qualitative results for the retriever in Section 6.3.
Revised: In Section 6.2 we ablate Analogical Networks’ performance under varying memory retrieval schemes in 5 -shot setting and show qualitative results for the retriever in Section 6.3.

Original: We show more results on noisy point clouds (ScanObjectNN (Uy et al., 2019) dataset) in 6.4.
Revised: We show more results on noisy point clouds (ScanObjectNN (Uy et al., 2019) dataset) in 6.4.

Original: We provide extensive qualitative visual object parsing results for single and multi-memory variants of Analogical Networks in section 6.5.
Revised: We provide extensive qualitative visual object parsing results for single and multi-memory variants of Analogical Networks in section 6.5.

Original: Lastly, we discuss additional related literature in
Revised: Lastly, we discuss additional related literature in

Original: Section 6.6.
Revised: Section 6.6.

Original: 6. I MPLEMENTATION D ETAILS AND T RAINING P SEUDO C ODE
Revised: 6. I MPLEMENTATION D ETAILS AND T RAINING P SEUDO C ODE

Original: The modulator encodes the input scene S and each retrieved memory scene M into a set of 3D point features using PointNet++ backbone.
Revised: The modulator encodes the input scene S and each retrieved memory scene M into a set of 3D point features using PointNet++ backbone.

Original: We encode positional information using rotary 3D positional encodings (Su et al., 2021; Li & Harada, 2021), which have the property that P ( x ) T P ( y ) =
Revised: We encode positional information using rotary 3D positional encodings (Su et al., 2021; Li & Harada, 2021), which have the property that P ( x ) T P ( y ) =

Original: P ( y − x ) , where P the positional encoding function and x, y two 3D points.
Revised: P ( y − x ) , where P the positional encoding function and x, y two 3D points.

Original: These embeddings are multiplied with queries and keys in the attention operations, thus making attention translation-invariant.
Revised: These embeddings are multiplied with queries and keys in the attention operations, thus making attention translation-invariant.

Original: For memory queries, we use the centroid of the corresponding memory part to compute positional encodings.
Revised: For memory queries, we use the centroid of the corresponding memory part to compute positional encodings.

Original: For scene-agnostic queries, we use the center of the input object.
Revised: For scene-agnostic queries, we use the center of the input object.

Original: Each labelled part p in each memory M is encoded into an 1D feature vector f
Revised: Each labelled part p in each memory M is encoded into an 1D feature vector f

Original: Mp by average pooling its point features.
Revised: Mp by average pooling its point features.

Original: Next, the queries (memory and scene-agnostic combined) self-attend and crossattend to the input point features and update themselves; point features self-attend and cross-attend to queries to also update themselves.
Revised: Next, the queries (memory and scene-agnostic combined) self-attend and crossattend to the input point features and update themselves; point features self-attend and cross-attend to queries to also update themselves.

Original: This input feature update is a difference from existing detection transformers, where only the queries are updated.
Revised: This input feature update is a difference from existing detection transformers, where only the queries are updated.

Original: We consider 6 layers of self and cross-attention.
Revised: We consider 6 layers of self and cross-attention.

Original: Lastly, we upsample the point features to the original resolution using convolutional layers (Qi et al., 2017) and compute an inner product between each query and point feature to compute the segmentation mask for each query.
Revised: Lastly, we upsample the point features to the original resolution using convolutional layers (Qi et al., 2017) and compute an inner product between each query and point feature to compute the segmentation mask for each query.

Original: The output of the modulator is a set of N q segmentation mask proposals and corresponding confidence scores, where N q is the total number of queries.
Revised: The output of the modulator is a set of N q segmentation mask proposals and corresponding confidence scores, where N q is the total number of queries.

Original: At training time, these proposals are matched to ground-truth instance binary masks using the Hungarian matching algorithm (Carion et al., 2020).
Revised: At training time, these proposals are matched to ground-truth instance binary masks using the Hungarian matching algorithm (Carion et al., 2020).

Original: For the proposals that are matched to a ground-truth instance, we compute the segmentation loss, which is a per-point binary cross-entropy loss (Vu et al., 2022; Cheng et al., 2021b) and a Softmax loss over the semantic classes.
Revised: For the proposals that are matched to a ground-truth instance, we compute the segmentation loss, which is a per-point binary cross-entropy loss (Vu et al., 2022; Cheng et al., 2021b) and a Softmax loss over the semantic classes.

Original: We also supervise the confidence score of each query, similar to (Carion et al., 2020).
Revised: We also supervise the confidence score of each query, similar to (Carion et al., 2020).

Original: The target labels are 1 for the proposals matched with a ground-truth part and 0 for non-matched.
Revised: The target labels are 1 for the proposals matched with a ground-truth part and 0 for non-matched.

Original: We found it beneficial to apply these losses after every cross-attention layer in the modulator.
Revised: We found it beneficial to apply these losses after every cross-attention layer in the modulator.

Original: At test time, we multiply the per point mask occupancy probability with tiled confidence scores to get a N p × N q tensor ( N p is the number of points); then each point is assigned to the highest scoring query by computing inner product and taking per-point argmax over the queries.
Revised: At test time, we multiply the per point mask occupancy probability with tiled confidence scores to get a N p × N q tensor ( N p is the number of points); then each point is assigned to the highest scoring query by computing inner product and taking per-point argmax over the queries.

Original: The modulator’s weights are trained with within-scene and cross-scene training where the modulating memories are sampled from the top-k retrieved memories.
Revised: The modulator’s weights are trained with within-scene and cross-scene training where the modulating memories are sampled from the top-k retrieved memories.

Original: During cross-scene training, we further co-train with within-scene training data.
Revised: During cross-scene training, we further co-train with within-scene training data.

Original: For both stages of training (i.e. within-scene correspondence pre-training and cross-scene training), we use AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 2e − 4 and batch size of 16 .
Revised: For both stages of training (i.e. within-scene correspondence pre-training and cross-scene training), we use AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 2e − 4 and batch size of 16 .

Original: We train the model for 40 epochs within-scene and 60 cross-scene.
Revised: We train the model for 40 epochs within-scene and 60 cross-scene.

Original: For few-shot fine-tuning/evaluation, we use AdamW optimizer with an initial learning rate of 1e − 5 and batch size of 8 .
Revised: For few-shot fine-tuning/evaluation, we use AdamW optimizer with an initial learning rate of 1e − 5 and batch size of 8 .

Original: We fine-tune it for 40 epochs and we report the performance across 10 different episodes, where each episode has a different set of K support samples.
Revised: We fine-tune it for 40 epochs and we report the performance across 10 different episodes, where each episode has a different set of K support samples.

Original: We describe Analogical Networks’ training details in pseudo-code for within (Algorithm 1) and cross-scene training (Algorithm 2) respectively.
Revised: We describe Analogical Networks’ training details in pseudo-code for within (Algorithm 1) and cross-scene training (Algorithm 2) respectively.

Original: For our DETR3D baseline we use same hyperparameters and train for 60 epochs.
Revised: For our DETR3D baseline we use same hyperparameters and train for 60 epochs.

Original: Training takes approximately 15 and 20 minutes per epoch on a single NVIDIA A100 gpu for DETR3D and
Revised: Training takes approximately 15 and 20 minutes per epoch on a single NVIDIA A100 gpu for DETR3D and

Original: Analogical Networks respectively.
Revised: Analogical Networks respectively.

Original: 6. P ERFORMANCE UNDER V ARYING R ETRIEVAL S CHEMES
Revised: 6. P ERFORMANCE UNDER V ARYING R ETRIEVAL S CHEMES

Original: In the few-shot setting, we evaluate the performance of Analogical Networks under varying memory retrieval schemes in Table 4.
Revised: In the few-shot setting, we evaluate the performance of Analogical Networks under varying memory retrieval schemes in Table 4.

Original: We compare against a hypothetical oracle retriever that can fetch the most helpful (in terms of resulting ARI) memory for each input.
Revised: We compare against a hypothetical oracle retriever that can fetch the most helpful (in terms of resulting ARI) memory for each input.

Original: All examined retrievers are
Revised: All examined retrievers are

Original: # S: input point cloud, M: memory point cloud, Np: numbers of points in S or M, N: sub-sampled points, C: number of feature channels, P: number of parts in M # augment: a sequence of standard 3D point cloud augmentations # pc_encoder: point cloud encoder # Xp(M): ground-truth label assignment of points in parts, copied directly from the memory # part_encoder: Computes the part features using mean pooling # pos_encode:
Revised: # S: input point cloud, M: memory point cloud, Np: numbers of points in S or M, N: sub-sampled points, C: number of feature channels, P: number of parts in M # augment: a sequence of standard 3D point cloud augmentations # pc_encoder: point cloud encoder # Xp(M): ground-truth label assignment of points in parts, copied directly from the memory # part_encoder: Computes the part features using mean pooling # pos_encode:

Original: Adds positional encoding # upsampler:
Revised: Adds positional encoding # upsampler:

Original: Upsamples point cloud # Segmentation_Loss: Cross entropy loss to assign each point to the Hungarian matched query.
Revised: Upsamples point cloud # Segmentation_Loss: Cross entropy loss to assign each point to the Hungarian matched query.

Original: for S in dataloader: # load a batch with B samples M = S
Revised: for S in dataloader: # load a batch with B samples M = S

Original: # the memory is the un-augmented version
Revised: # the memory is the un-augmented version

Original: S = augment(S)
Revised: S = augment(S)

Original: # the input is augmented # S : B x Np x 3 and M: B x Np x# Compute point features FˆS = pc_encoder(S)
Revised: # the input is augmented # S : B x Np x 3 and M: B x Np x# Compute point features FˆS = pc_encoder(S)

Original: # B x N x C FˆM = pc_encoder(M)
Revised: # B x N x C FˆM = pc_encoder(M)

Original: # B x N x C
Revised: # B x N x C

Original: # Initialize memory part queries fˆM =
Revised: # Initialize memory part queries fˆM =

Original: part_encoder(FˆM)
Revised: part_encoder(FˆM)

Original: # B x P x C # Compute positional embedding x_pos = pos_encode(FˆS) y_pos = pos_encode(fˆM)
Revised: # B x P x C # Compute positional embedding x_pos = pos_encode(FˆS) y_pos = pos_encode(fˆM)

Original: # B x P x C
Revised: # B x P x C

Original: Loss = 

Revised: Loss =

Original: # Do multiple layers of modulation using Self-Attn and Cross-Attn for layer in num_layers: x = Cross-Attn(x, y, x_pos, y_pos)
Revised: # Do multiple layers of modulation using Self-Attn and Cross-Attn for layer in num_layers: x = Cross-Attn(x, y, x_pos, y_pos)

Original: # B x N x C y
Revised: # B x N x C y

Original: =
Revised: =

Original: Cross-Attn(y, x, y_pos, x_pos)
Revised: Cross-Attn(y, x, y_pos, x_pos)

Original: # B x P x C x = Self-Attn(x, x_pos)
Revised: # B x P x C x = Self-Attn(x, x_pos)

Original: # B x N x C y =
Revised: # B x N x C y =

Original: Self-Attn(y, y_pos)
Revised: Self-Attn(y, y_pos)

Original: # B x P x C
Revised: # B x P x C

Original: X = upsampler(x)
Revised: X = upsampler(x)

Original: # B x Np x
Revised: # B x Np x

Original: C point_query_similarity = matmul(normalize(X), normalize(y.T))
Revised: C point_query_similarity = matmul(normalize(X), normalize(y.T))

Original: # B
Revised: # B

Original: [Table]
Revised: [Table]

Original: category-constrained, i.e., they have access to the object category of the input and retrieve an object of the same category.
Revised: category-constrained, i.e., they have access to the object category of the input and retrieve an object of the same category.

Original: Our conclusions are as follows: (i) Analogical Networks with an oracle memory retriever perform better than Analogical Networks using memories retrieved by the retriever .
Revised: Our conclusions are as follows: (i) Analogical Networks with an oracle memory retriever perform better than Analogical Networks using memories retrieved by the retriever .

Original: This suggests that better training of our retriever or exploring its co-training with the rest of our model could have significant impact in improving its performance.
Revised: This suggests that better training of our retriever or exploring its co-training with the rest of our model could have significant impact in improving its performance.

Original: (ii) Considering any of the 5-shot exemplars randomly does worse than using memories retrieved by our retriever.
Revised: (ii) Considering any of the 5-shot exemplars randomly does worse than using memories retrieved by our retriever.

Original: 6. Q UALITATIVE P ERFORMANCE OF THE R ETRIEVER
Revised: 6. Q UALITATIVE P ERFORMANCE OF THE R ETRIEVER

Original: We show qualitative results of the retriever on multiple classes, both seen (Figure 4) during training and unseen (Figure 5).
Revised: We show qualitative results of the retriever on multiple classes, both seen (Figure 4) during training and unseen (Figure 5).

Original: We observe the following: (i)
Revised: We observe the following: (i)

Original: The retriever considers fine-grained object similarities and not only class information.
Revised: The retriever considers fine-grained object similarities and not only class information.

Original: To illustrate this, we include two examples for the “Chair” and “Earphone” classes in Figure 4, as well as the “Bed” and “Refrigerator” classes in Figure 5.
Revised: To illustrate this, we include two examples for the “Chair” and “Earphone” classes in Figure 4, as well as the “Bed” and “Refrigerator” classes in Figure 5.

Original: Different instances of the same category retrieve very different memories, that share both structural and semantic similarities with the respective input point cloud.
Revised: Different instances of the same category retrieve very different memories, that share both structural and semantic similarities with the respective input point cloud.

Original: (ii)
Revised: (ii)

Original: The retriever generalizes to novel classes, not seen during training , as shown in Figure 5.
Revised: The retriever generalizes to novel classes, not seen during training , as shown in Figure 5.

Original: 6. E VALUATION ON S CAN O BJECT NN D ATASET (U Y ET AL ., 2019)
Revised: 6. E VALUATION ON S CAN O BJECT NN D ATASET (U Y ET AL ., 2019)

Original: We test Analogical Networks on ScanObjectNN (Uy et al., 2019), which contains noisy and incomplete real-world point clouds.
Revised: We test Analogical Networks on ScanObjectNN (Uy et al., 2019), which contains noisy and incomplete real-world point clouds.

Original: We split the training into 11 classes seen during training (bag, bin, box,
Revised: We split the training into 11 classes seen during training (bag, bin, box,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: = Self-Attn(x)
Revised: = Self-Attn(x)

Original: # B x N x C y =
Revised: # B x N x C y =

Original: Self-Attn(y)
Revised: Self-Attn(y)

Original: # B x P x C
Revised: # B x P x C

Original: X = upsampler(x)
Revised: X = upsampler(x)

Original: # B x Np x
Revised: # B x Np x

Original: C point_query_similarity = matmul(normalize(X), normalize(y.T))
Revised: C point_query_similarity = matmul(normalize(X), normalize(y.T))

Original: # B x Np x
Revised: # B x Np x

Original: (P +
Revised: (P +

Original: Q)
Revised: Q)

Original: Loss += Segmentation_Loss(armax(point_query_similarity, -1), Xp_Hungarian) + Objectness_Loss(y, yp_Hungarian) + Semantic_Loss(y[matched], target_classes)
Revised: Loss += Segmentation_Loss(armax(point_query_similarity, -1), Xp_Hungarian) + Objectness_Loss(y, yp_Hungarian) + Semantic_Loss(y[matched], target_classes)

Original: # optimizer step loss.backward() optimizer.step()
Revised: # optimizer step loss.backward() optimizer.step()

Original: Method
Revised: Method

Original: [Table]
Revised: [Table]

Original: Novel Category:
Revised: Novel Category:

Original: [Table]
Revised: [Table]

Original: single
Revised: single

Original: [Table]
Revised: [Table]

Original: cabinet, chair, desk, door, pillow, shelf, sink, sofa) and 4 unseen (bed, display, table, toilet).
Revised: cabinet, chair, desk, door, pillow, shelf, sink, sofa) and 4 unseen (bed, display, table, toilet).

Original: Note that ScanObjectNN is not consistently labelled.
Revised: Note that ScanObjectNN is not consistently labelled.

Original: For example, as we show in rows 5 and 6 of Figure 12, the legs of a chair may be annotated as a single part or multiple parts in the dataset.
Revised: For example, as we show in rows 5 and 6 of Figure 12, the legs of a chair may be annotated as a single part or multiple parts in the dataset.

Original: Although the PartNet dataset provides the “level” information, there is no such information in ScanObjectNN.
Revised: Although the PartNet dataset provides the “level” information, there is no such information in ScanObjectNN.

Original: Therefore we only qualitatively evaluate our model in Figure 12 for base classes and in Figure 13 for
Revised: Therefore we only qualitatively evaluate our model in Figure 12 for base classes and in Figure 13 for

Original: [Table]
Revised: [Table]

Original: novel classes.
Revised: novel classes.

Original: We can see that our predictions are always plausible and consistent with the retrieved memory, even if the expected label space is different.
Revised: We can see that our predictions are always plausible and consistent with the retrieved memory, even if the expected label space is different.

Original: We additionally visualize retrieval results in
Revised: We additionally visualize retrieval results in

Original: Figure 14.
Revised: Figure 14.

Original: 6. Q UALITATIVE R ESULTS FOR S INGLE -M EMORY AND M ULTI -M EMORY A NALOGICAL N ETWORKS
Revised: 6. Q UALITATIVE R ESULTS FOR S INGLE -M EMORY AND M ULTI -M EMORY A NALOGICAL N ETWORKS

Original: In this section, we show our model’s qualitative results for object parsing.
Revised: In this section, we show our model’s qualitative results for object parsing.

Original: In the Figures 6, 7, 8, 9, 10, 11, 12, 13 we use the following 5 -column pattern:
Revised: In the Figures 6, 7, 8, 9, 10, 11, 12, 13 we use the following 5 -column pattern:

Original: • Unlabelled input point cloud • Memory used for modulation • Object parsing generated using only the memory part queries (not the scene-agnostic queries).
Revised: • Unlabelled input point cloud • Memory used for modulation • Object parsing generated using only the memory part queries (not the scene-agnostic queries).

Original: In this column, regions that are not decoded by a memory part query are colored in black.
Revised: In this column, regions that are not decoded by a memory part query are colored in black.

Original: • Final predicted segmentation parsing using both memory-initialized queries and sceneagnostic queries.
Revised: • Final predicted segmentation parsing using both memory-initialized queries and sceneagnostic queries.

Original: Regions that are colored in black in the third column but colored differently in the fourth column are decoded by scene-agnostic queries.
Revised: Regions that are colored in black in the third column but colored differently in the fourth column are decoded by scene-agnostic queries.

Original: • Input point cloud’s ground truth segmentation at the granularity level of the memory.
Revised: • Input point cloud’s ground truth segmentation at the granularity level of the memory.

Original: We qualitatively show the emergence of part correspondence between retrieved memory (column 2) and the input point cloud parsed using memory queries (column 3).
Revised: We qualitatively show the emergence of part correspondence between retrieved memory (column 2) and the input point cloud parsed using memory queries (column 3).

Original: Parts having the same color in columns 2 and 3 demonstrate correspondence, i.e. a part in column 2 decodes the part with the same color in column 3.
Revised: Parts having the same color in columns 2 and 3 demonstrate correspondence, i.e. a part in column 2 decodes the part with the same color in column 3.

Original: Analogical Networks promote correspondence of parts on both base (Figure 6 bottom and 7) and novel (Figure 6 top and 8) categories.
Revised: Analogical Networks promote correspondence of parts on both base (Figure 6 bottom and 7) and novel (Figure 6 top and 8) categories.

Original: This correspondence is semantic but also geometric, as can be seen in Figure 10.
Revised: This correspondence is semantic but also geometric, as can be seen in Figure 10.

Original: When multiple memories are available, Analogical Networks mix and match parts of different memories to parse the input.
Revised: When multiple memories are available, Analogical Networks mix and match parts of different memories to parse the input.

Original: Furthermore, we show parsing results for AnalogicalNets single-mem w/o pretrain in Figure 11.
Revised: Furthermore, we show parsing results for AnalogicalNets single-mem w/o pretrain in Figure 11.

Original: We observe that all of memory part query are inactive in the parsing stage.
Revised: We observe that all of memory part query are inactive in the parsing stage.

Original: This demonstrates the utility of within-scene pre-training, as without this pre-training part correspondence does not emerge, as shown in Figure 11.
Revised: This demonstrates the utility of within-scene pre-training, as without this pre-training part correspondence does not emerge, as shown in Figure 11.

Original: Lastly, we show that Analogical Networks generalize to noisy and incomplete point clouds in
Revised: Lastly, we show that Analogical Networks generalize to noisy and incomplete point clouds in

Original: ScanObjectNN.
Revised: ScanObjectNN.

Original: 6. A DDITIONAL RELATED WORK
Revised: 6. A DDITIONAL RELATED WORK

Original: Neural-symbolic models Analogical Networks are a type of neural-symbolic model that represents knowledge explicitly, in terms of structured visual memories, where each one is a graph of partentity neural embeddings.
Revised: Neural-symbolic models Analogical Networks are a type of neural-symbolic model that represents knowledge explicitly, in terms of structured visual memories, where each one is a graph of partentity neural embeddings.

Original: A structured visual memory can be considered the neural equivalent of a FRAME introduced in (Minsky, 1985), “a graph of nodes and their relations for representing a stereotyped situation, like being in a certain kind of living room, or going to a child’s birthday party” , to quote Minsky (1985).
Revised: A structured visual memory can be considered the neural equivalent of a FRAME introduced in (Minsky, 1985), “a graph of nodes and their relations for representing a stereotyped situation, like being in a certain kind of living room, or going to a child’s birthday party” , to quote Minsky (1985).

Original: FRAME nodes would operate as “slots” to be filled with specific entities, or symbols, in the visual scene.
Revised: FRAME nodes would operate as “slots” to be filled with specific entities, or symbols, in the visual scene.

Original: Symbol detection would be carried out by a separate state estimation process such as general-purpose object detectors (Zhou et al., 2022), employed also by recent neuro-symbolic models (Yi et al., 2018; Mao et al., 2019;
Revised: Symbol detection would be carried out by a separate state estimation process such as general-purpose object detectors (Zhou et al., 2022), employed also by recent neuro-symbolic models (Yi et al., 2018; Mao et al., 2019;

Original: Yi et al., 2019).
Revised: Yi et al., 2019).

Original: In-the-wild detection of symbols (e.g., chair handles, faucet tips, fridge doors) typically fails, which is the reason why these earlier symbolic systems of knowledge, largely disconnected from the sensory input, have not been widely adopted.
Revised: In-the-wild detection of symbols (e.g., chair handles, faucet tips, fridge doors) typically fails, which is the reason why these earlier symbolic systems of knowledge, largely disconnected from the sensory input, have not been widely adopted.

Original: Analogical Networks take a step towards addressing these shortcomings by including symbol detection as part of inference itself, through a top-down modulation that uses the context represented in the memory graph, to jointly search for multiple entities and localize them in context of one another.
Revised: Analogical Networks take a step towards addressing these shortcomings by including symbol detection as part of inference itself, through a top-down modulation that uses the context represented in the memory graph, to jointly search for multiple entities and localize them in context of one another.

Original: 3D instance segmentation has been traditionally approached as a clustering problem (Chen et al., 2009; Sidi et al., 2011).
Revised: 3D instance segmentation has been traditionally approached as a clustering problem (Chen et al., 2009; Sidi et al., 2011).

Original: Point-based methods learn either translation vectors mapping every point to its instance’s center (Jiang et al., 2020; Chen et al., 2021; Vu et al., 2022) or similarities across points (Wang et al., 2018a; Zhang & Wonka, 2021), followed by one or more stages of clustering.
Revised: Point-based methods learn either translation vectors mapping every point to its instance’s center (Jiang et al., 2020; Chen et al., 2021; Vu et al., 2022) or similarities across points (Wang et al., 2018a; Zhang & Wonka, 2021), followed by one or more stages of clustering.

Original: Similarly, (Wang et al., 2021; Jones et al., 2022) oversegment the point cloud into small regions and then merge them into parts.
Revised: Similarly, (Wang et al., 2021; Jones et al., 2022) oversegment the point cloud into small regions and then merge them into parts.

Original: Yu et al.
Revised: Yu et al.

Original: (2019) recursively decompose a point cloud into segments of finer resolution.
Revised: (2019) recursively decompose a point cloud into segments of finer resolution.

Original: (Mo et al., 2019; Sun et al., 2022) learn representative vectors that form clusters by voting for each point.
Revised: (Mo et al., 2019; Sun et al., 2022) learn representative vectors that form clusters by voting for each point.

Original: However, these approaches usually assume a fixed label space and need to train a separate model for each sub-task.
Revised: However, these approaches usually assume a fixed label space and need to train a separate model for each sub-task.

Original: In contrast, we employ Detection Transformers (Carion et al., 2020) for instance segmentation by repurposing the query vectors to act as representative vectors.
Revised: In contrast, we employ Detection Transformers (Carion et al., 2020) for instance segmentation by repurposing the query vectors to act as representative vectors.

Original: We extend this set of queries with memory-initialized queries, enabling in-context reasoning.
Revised: We extend this set of queries with memory-initialized queries, enabling in-context reasoning.

Original: This allows us to train one model across all categories.
Revised: This allows us to train one model across all categories.

Original: As our results show, in absence of such memory contextualization, training one model across multiple categories hurts generalization (Table 3).
Revised: As our results show, in absence of such memory contextualization, training one model across multiple categories hurts generalization (Table 3).