Original: A NALOGICAL N ETWORKS FOR M EMORY -M ODULATED 3D P ARSING
Revised: A NALOGICAL N ETWORKS FOR M EMORY -M ODULATED 3D P ARSING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite recent breakthroughs in the applications of deep neural networks in visual perception, onesetting that presents a persistent challenge is that of “few-shot learning.”
Revised: Despite recent breakthroughs in the applications of deep neural networks in visual perception, one persistent challenge is few-shot learning .

Original: Works in the area of few shot visual learning mostly address the task of coarse image classification. Fine-grain visual parsing is necessary for scene understanding and action recognition.
Revised: Works in the area of fewshot visual learning mostly address the task of coarse image classification, while fine-grained 3D visual parsing, although useful for scene understanding and interaction with the physical world, remains underexplored.

Original: Thus far, a separate neural model is trained to parse each semantic category, which hinders knowledge sharing across objects, let alone few shot visual parsing.
Revised: Thus far, a separate neural model is trained to parse each semantic category, which hinders knowledge sharing across objects and few-shot generalization.

Original: We present Analogical Networks, a model that casts fine-grained visual parsing into analogical inference: instead of mapping input scenes to part labels, which is hard to adapt in a few-shot manner to novel inputs, our model retrieves related scenes from memory and their corresponding part structures, and predicts analogous part structures in the input scene, via an end-to-end learnable modulation mechanism.
Revised: We present Analogical Networks, a model that casts fine-grained visual parsing into analogical inference: instead of mapping input scenes to part labels, which is hard to adapt in a few-shot manner to novel inputs, our model retrieves related scenes from memory and their corresponding part structures, and predicts analogous part structures in the input scene, via an end-to-end learnable modulation mechanism.

Original: By conditioning on more than one memory, compositions of structures are predicted, that mix and match parts from different visual experiences.
Revised: By conditioning on more than one memory, compositions of structures are predicted, that mix and match parts from different visual experiences.

Original: We show  Analogical Networks excel at few-shot learning, where instances of novel object categories are successfully parsed simply by expanding the model’s memory, without any weight updates.
Revised: We show that Analogical Networks excel at few-shot 3D parsing, where instances of novel object categories are successfully parsed simply by expanding the model’s memory, without any weight updates.

Original: Analogical Networks outperform existing state-of-the-art detection transformer models at part segmentation, as well as paradigms of meta-learning and few-shot learning.
Revised: Analogical Networks outperform existing state-of-the-art detection transformer models for part segmentation, as well as paradigms of meta-learning and few-shot learning.

Original: We show  part correspondences emerge across memory and input scenes by simply training for a label-free segmentation objective, as a byproduct of the analogical inductive bias.
Revised: We show that part correspondences emerge across memory and input scenes by simply training for a label-free segmentation objective, as a byproduct of the analogical inductive bias.

Original: Ask not what it is, ask what it is like.
Revised: Ask not what it is, ask what it is like.

Original: Moshe Bar
Revised: Moshe Bar

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The dominant paradigm in existing deep visual learning is to train high capacity networks that map input visual observations to task-specific output labels.
Revised: The dominant paradigm in existing deep visual learning is to train high capacity networks that map input visual observations to task-specific output labels.

Original: Despite their success across a plethora of tasks, these models struggle to perform well in few-shot settings where only a small set of examples are available for learning.
Revised: Despite their success across a plethora of tasks, these models struggle to perform well in few-shot settings where only a small set of examples are available for learning.

Original: Meta-learning approaches provide one promising solution to this setting by enabling efficient task-specific adaptation of generic models, but this specialization comes at the cost of poor performance on the original tasks as well as the need to adapt separate models for each novel task considered.
Revised: Meta-learning approaches provide one promising solution to this setting by enabling efficient task-specific adaptation of generic models, but this specialization comes at the cost of poor performance on the original tasks as well as the need to adapt separate models for each novel task considered.

Original: In this work, we introduce an alternate approach that unifies inference in both, few-shot and many-shot settings.
Revised: In this work, we introduce an alternative approach that unifies inference in both, few-shot and many-shot settings.

Original: Instead of pursuing prediction of task-specific outputs, we formulate an approach for analogy-driven prediction, where the desired prediction can be dynamically specified at inference given labelled examples from memory.
Revised: Instead of pursuing prediction of task-specific outputs, we formulate an approach for analogy-driven prediction, where the desired prediction can be dynamically specified at inference given labelled examples from memory.

Original: In his seminal work (Bar, 2007), Moshe Bar argued for the importance of analogies and associations in human reasoning—highlighting how associations of novel inputs to analogous representations in memory can drive perceptual inference.
Revised: In his seminal work (Bar, 2007), Moshe Bar argued for the importance of analogies and associations in human reasoning—highlighting how associations of novel inputs to analogous representations in memory can drive perceptual inference.

Original: In this work, we operationalize these insights and propose
Revised: In this work, we operationalize these insights and propose

Original: Analogical Networks, an analogical learning framework for 3D visual parsing.
Revised: Analogical Networks, an analogical learning framework for 3D visual parsing.

Original: Our approach encodes domain knowledge explicitly in a collection of structured parsed scene memories as well as implicitly, in model parameters.
Revised: Our approach encodes domain knowledge explicitly in a collection of structured parsed scene memories as well as implicitly, in model parameters.

Original: Given an input 3D shape, the model retrieves relevant memories on-the-fly and uses them to modulate inference and segment object parts in the input point cloud.
Revised: Given an input 3D shape, the model retrieves relevant memories on-the-fly and uses them to modulate inference and segment object parts in the input point cloud.

Original: The modulation mechanism operates as top-down feedback for the model to focus onentities present in the retrievedRetrieved memories 
Revised: The modulation mechanism operates as top-down guidance for the model to focus on the retrieved memories and infer the analogous parts in the input scene.

Original: ) modulate segmentation set of memories, and infer the analogous parts in the input scene.

Original: During modulation, the input scene and the retrieved memories are jointly encoded and contextualized via cross-attention operations.
Revised: During modulation, the input scene and the retrieved memories are jointly encoded and contextualized via cross-attention operations.

Original: Each memory is represented as a spatial graph of point and part feature embeddings.

Original: The memory part features are then used to decode corresponding parts in the input 3D scene.
Revised: The memory part features are then used to decode corresponding parts in the input.

Original: Instead of mapping the input point cloud to semantic class labels, the model reasons analogically, and maps the input to modifications and compositions of past visual experiences, without any explicit semantic label prediction step.
Revised: Instead of mapping the input point cloud to semantic class labels, the model reasons analogically and maps the input to modifications and compositions of past visual experiences, without any explicit semantic label prediction step.

Original: Memory to input scene part associations emerge in our model without association supervision, simply as a byproduct of the analogical inductive biases of the modulation process, as shown in Figure 1.
Revised: Memory to input scene part associations emerge in our model without association supervision, simply as a byproduct of the analogical inductive biases of the modulation process, as shown in Figure 1.

Original: In Analogical Networks, the encoder and modulator parameters are trained end-to-end for the visual parsing task. However, the  retrieval process is not end-to-end differentiable.
Revised: In Analogical Networks the encoder and modulator parameters are trained end-to-end for the visual parsing task, but the memory retrieval process is not end-to-end differentiable.

Original: We devise a novel pre-training scheme where we train theparameters of the encoder and modulator sub-modules to parse an augmented version of a 3D scene given the original, un-augmented scene as the modulating memory, bypassing the retrieval process.
Revised: We devise a novel pre-training scheme where we train the encoder and modulator sub-modules to parse an augmented version of a 3D scene given the original, un-augmented scene as the modulating memory, bypassing the retrieval process.

Original: We show this pre-training helpsperformance of our model, particularly in the few shot setting.
Revised: We show this pre-training helps our model, particularly in the few-shot setting.

Original: We test our model on the 3D object segmentation PartNet benchmark (Mo et al., 2019).
Revised: We test our model on the 3D object segmentation PartNet benchmark of Mo et al.

Original: We compare against state-of-the-art (SOTA) 3D object segmentors,  as well as meta-learning and few-shot learning (Snell et al., 2017)
Revised: We compare against state-of-the-art (SOTA) 3D object segmentors, approaches designed specifically for part segmentation, as well as meta-learning and few-shot learning (Snell et al., 2017)

Original: baselines adapted for the task of 3Dobject parsing.
Revised: baselines adapted for the task of 3D parsing.

Original: Analogical Networks outperform the baselines in the standard many shot train-test split, and particularly shine over the baselines in the few-shot setting: simply by expanding the memory repository with encodings of a few exemplars, and even without any weight updates, Analogical Networks segment novel instances much better than the baselines.
Revised: Analogical Networks outperform the baselines in the standard many-shot train-test split and particularly shine over the baselines in the few-shot setting: simply by expanding the memory repository with encodings of a few exemplars, and even without any weight updates, Analogical Networks segment novel instances much better than the baselines.

Original: We extensively ablate design choices of our model to quantify contributions of the modulation architecture, the pre-training scheme, and the memory retrieval.
Revised: We extensively ablate design choices of our model to quantify contributions of the modulation architecture, the pre-training scheme and the memory retrieval.

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: A mechanism for in-context memory-modulated learning.
Revised: A mechanism for in-context memory-modulated learning.

Original: We introduce a model that given a set of relevant examples predicts analogical part structures in the present input scene.
Revised: We introduce a model that, given a set of relevant examples, predicts analogical part structures in the present input scene.

Original: The memory context is a natural way to communicate the task to the model, such as the segmentation granularity or object category.
Revised: The memory context is a natural way to communicate the task to the model. In this work, a task refers to parsing objects of different categories or segmentation granularity.

Original: Analogical Networks cast every task as an analogical correspondence problem to an appropriate set of memories and their labels.
Revised: Analogical Networks cast every task as an analogical correspondence problem to an appropriate set of memories and their labels.

Original: This allows us to train one model across all tasks , where each inference is modulated with a different set of relevant memories, of appropriate category and segmentation granularity.
Revised: This allows us to train one model across all tasks , where each inference is modulated with a different set of relevant memories, of appropriate category and segmentation granularity.

Original: In contrast, existing neural network models that are trained to map input to output (Carion et al., 2020; Wu et al., 2019; Yi et al., 2018b; Deng et al., 2019; Yang et al., 2019) break the general task of point cloud segmentation into sub-tasks, one per object category (e.g., chair, table,keyboard, etc.) and part label granularity (levels 1, 2, 3 in the PartNet benchmark (Mo et al., 2019)) and a different neural network is trained in each sub-task.
Revised: In contrast, existing neural network models that are trained to map input to output (Carion et al., 2020; Wu et al., 2019; Yi et al., 2018b; Deng et al., 2019; Yang et al., 2019; Yu et al., 2019; Sun et al., 2022) break the general task of point cloud segmentation into sub-tasks and train different networks per object category (e.g., chair, table, etc.) and part label granularity (levels 1, 2, 3 in the PartNet benchmark (Mo et al., 2019)).

Original: Few-shot transfer via analogical prediction.
Revised: Few-shot transfer via analogical prediction.

Original: In parametric learning by gradient descent, any change to neural parameters impacts all aspects of agent’s future behavior, leading to catastrophic forgetting (Kaushik et al., 2021).
Revised: In parametric learning by gradient descent, any change to neural parameters impacts all aspects of agent’s future behavior, leading to catastrophic forgetting (Kaushik et al., 2021).

Original: Analogical Networks learn from a few examples through memory expansion, without necessarily updating the weights of the networks, which resolves the interference problems of parametric models (Greff et al., 2020).
Revised: Analogical Networks learn from a few examples through memory expansion, without needing to update the weights of the networks, which resolves the interference problems of parametric models (Greff et al., 2020).

Original: Emergent correspondences without semantic supervision.
Revised: Emergent correspondences without semantic supervision.

Original: Correspondences emerge across exemplars through the proposed modulation mechanism that uses memory part features as expert queries during part detection, in place of learnable scene-agnostic queries used 
Revised: Correspondences emerge across exemplars through the proposed modulation mechanism that uses memory part features as expert queries during part detection, in place of learnable scene-agnostic queries used in SOTA transformer visual detectors (Carion et al., 2020).

Original: in

Original: SOTA transformer visual

Original: detectors

Original: (Carion et al., 2020).

Original: Our code  will be publicly available upon acceptance.
Revised: Our code and trained checkpoints will be publicly available upon acceptance.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Frames, Schemas, and neural-symbolic models Analogical Networks represent knowledge explicitly, in terms of structured visual memories, where each one is a graph of part entity neural embeddings.

Original: A structured visual memory can be considered the neural equivalent of the earlier concept of a FRAME introduced in (Minsky, 1985), “a graph of nodes and their relations for representing a stereotyped situation, like being in a certain kind of living room, or going to a child’s birthday party”, to quote Minsky (1985).

Original: FRAME nodes would operate as “slots” to be filled with specific entities, or symbols, in the visual scene.

Original: Symbol detection would be carried out by a separate state estimation process such as, general-purpose object detectors (Zhou et al., 2022), employed also by recent neuro-symbolic models (Yi et al., 2018a; Mao et al., 2019;

Original: Yi et al., 2019).

Original: In the wild detection of domain symbols (e.g., chair handles, table legs, faucet tips, fridge doors) typically fails, which is precisely the reason why these earlier symbolic systems of knowledge, largely disconnected from the sensory input, have not been widely adopted.

Original: Analogical Networks take a step towards addressing these shortcomings by including symbol detection as part of the FRAME inference itself, through a top-down modulation that uses the context represented in the FRAME memory graph, to jointly search for multiple entities and localize them in context of one another.

Original: Few-shot Prediction: Meta-learning and Learning associations
Revised: Few-shot Prediction: Meta-learning and Learning associations

Original: A key goal for our approach is to enable accurate inference in few-shot settings.
Revised: A key goal for our approach is to enable accurate inference in few-shot settings.

Original: Previous approaches (Finn et al., 2017; Rusu et al., 2018;
Revised: Previous approaches (Finn et al., 2017; Rusu et al., 2018;

Original: Snell et al., 2017; Wang et al., 2018; Bar et al., 2022; Mangla et al., 2020; Wang et al., 2020; Nguyen & Todorovic, 2019; Liu et al., 2020; Tian et al., 2020; Yang et al., 2020) that target similar settings can be broadly categorized as relying on either meta-learning or learning better associations.
Revised: Snell et al., 2017; Wang et al., 2018b; Bar et al., 2022; Mangla et al., 2020; Wang et al., 2020; Nguyen & Todorovic, 2019; Liu et al., 2020; Tian et al., 2020; Yang et al., 2020) that target similar settings can be broadly categorized as relying on either meta-learning or learning better associations.

Original: Meta-learning approaches tackle few-shot prediction by learning a generic model that can be efficiently adapted to a new task of interest from a few labelled examples.
Revised: Meta-learning approaches tackle few-shot prediction by learning a generic model that can be efficiently adapted to a new task of interest from a few labelled examples.

Original: While broadly applicable, these methods result in catastrophic forgetting of the original task due to adaptation and thus require training a new model for each task of interest.
Revised: While broadly applicable, these methods result in catastrophic forgetting of the original task during adaptation and thus require training a new model for each task of interest.

Original: Moreover, the goal of learning generic and rapidly adaptable models can lead to suboptimal performance over the base tasks with abundant data.
Revised: Moreover, the goal of learning generic and rapidly adaptable models can lead to suboptimal performance over the base tasks with abundant data.

Original: An alternate approach for the few-shot setting is to learn better associations.
Revised: An alternative approach for the few-shot setting is to learn better associations.

Original: For example, the category of a new example maybe inferred by transferring the label(s) from the one (or few) closest samples (Snell et al., 2017; Sung et al., 2018).
Revised: For example, the category of a new example maybe inferred by transferring the label(s) from the one (or few) closest samples (Snell et al., 2017; Sung et al., 2018).

Original: While this approach obviates the need for adapting models and can allow prediction in few-shot and many-shot settings, the current approaches are only applicable to global prediction, e.g., image labels.
Revised: While this approach obviates the need for adapting models and can allow prediction in few-shot and many-shot settings, the current approaches are only applicable to global prediction, e.g., image labels.

Original: Our work can be viewed as extending such association-based methods to allow predicting fine-grained and generic visual structures using our proposed modulation-based prediction mechanism.
Revised: Our work can be viewed as extending such association-based methods to allow predicting fine-grained and generic visual structures using our proposed modulation-based prediction mechanism.

Original: Memory-augmented neural networksMemory-augmented neural networks (Santoro et al., 2016) are equipped with explicit external key-value memories, which they can attend on and retrieve key-value pairs relevant to the input.
Revised: Memory-augmented neural networks (Santoro et al., 2016) are equipped with explicit external key-value memories, which they can attend on and retrieve key-value pairs relevant to the input.

Original: Memory-augmentation of parametric models permit fast learning with few examples, where the data are saved in the explicit memory immediately after their acquisition, while learning via parameter update is slow and requires multiple gradient iteration on de-correlated examples.
Revised: Memory-augmentation of parametric models permit fast learning with few examples, where the data are saved in the explicit memory immediately after their acquisition, while learning via parameter update is slow and requires multiple gradient iterations on de-correlated examples.

Original: Another benefit of explicit memories over implicit memories of model parameters is that computation only scales sub-linearly if approximate nearest neighbors schemes are used, while it scales quadratically with the number of model parameters.

Original: External memories have recently been used in scaling up language models (Borgeaud et al., 2021; Khandelwal et al., 2019) to alleviate from the limited context window of parametric transformers (Wu et al., 2022), to store knowledge in the form of entity mentions (de Jong et al., 2021), knowledge graphs (Das et al., 2022), and question answer pairs (Chen et al., 2022).
Revised: External memories have recently been used in scaling up language models (Borgeaud et al., 2021; Khandelwal et al., 2019), to alleviate from the limited context window of parametric transformers (Wu et al., 2022), to store knowledge in the form of entity mentions (de Jong et al., 2021), knowledge graphs (Das et al., 2022) and question-answer pairs (Chen et al., 2022).

Original: Memory attention layers in these models are used to influence the computation of transformer layers, and have proven critical for factual question answering, but also for sentence completion over their parametric counterparts.
Revised: Memory attention layers in these models are used to influence the computation of transformer layers and have proven critical for factual question answering, but also for sentence completion over their parametric counterparts.

Original: The architecture of Analogical Networks for 3D part segmentation is illustrated in Figure 2.
Revised: 3D instance segmentation has been traditionally approached as a clustering problem (Chen et al., 2009; Sidi et al., 2011). Point-based methods learn either translation vectors mapping every point to its instance’s center (Jiang et al., 2020; Chen et al., 2021; Vu et al., 2022) or similarities across points (Wang et al., 2018a; Zhang & Wonka, 2021), followed by one or more stages of clustering. Similarly, (Wang et al., 2021; Jones et al., 2022) oversegment the point cloud into small regions and then merge them into parts. Yu et al.

Revised: (2019) recursively decompose a point cloud into segments of finer resolution.

Revised: Most of these approaches rely on domain-specific or even class-specific priors that do not generalize across sub-tasks, let alone few-shot adaptation to novel setups.

Revised: A more general solution is to learn representative vectors that form clusters by voting for each point (Mo et al., 2019;

Revised: Sun et al., 2022).

Revised: However, these approaches are limited to learning a fixed set of vectors that are not contextualized with the input and need to train a separate model for each sub-task.

Revised: In contrast, we employ Detection Transformers (Carion et al., 2020) for instance segmentation by repurposing the query vectors to act as representative vectors.

Revised: The attention to the input point cloud allows for flexible scene-aware specialization.

Revised: We extend this set of queries using memory-initialized ones, enabling in-context reasoning.

Revised: Analogical Networks lack a task-specific design, can support multiple classes and segmentation levels simultaneously and generalizes to novel classes in few-shot manner.

Revised: Neural-symbolic models Analogical Networks represent knowledge explicitly, in terms of structured visual memories, where each one is a graph of part-entity neural embeddings.

Revised: A structured visual memory can be considered the neural equivalent of a FRAME introduced in (Minsky, 1985), “a graph of nodes and their relations for representing a stereotyped situation, like being in a certain kind of living room, or going to a child’s birthday party”, to quote Minsky (1985).

Revised: FRAME nodes would operate as “slots” to be filled with specific entities, or symbols, in the visual scene.

Revised: Symbol detection would be carried out by a separate state estimation process such as general-purpose object detectors (Zhou et al., 2022), employed also by recent neuro-symbolic models (Yi et al., 2018a; Mao et al., 2019;

Revised: Yi et al., 2019).

Revised: In-the-wild detection of symbols (e.g., chair handles, faucet tips, fridge doors) typically fails, which is precisely the reason why these earlier symbolic systems of knowledge, largely disconnected from the sensory input, have not been widely adopted.

Revised: Analogical Networks take a step towards addressing these shortcomings by including symbol detection as part of the FRAME inference itself, through a top-down modulation that uses the context represented in the FRAME memory graph, to jointly search for multiple entities and localize them in context of one another.

Revised:  A NALOGICAL N ETWORKS FOR 3D OBJECT PARSING

Revised: The architecture of Analogical Networks for 3D part segmentation is illustrated in Figure 2.

Original: It is comprised of an encoder, a retriever and a modulator sub-network, and a collection of labelledobject 3D  point clouds of different categories and segmentation granularities as the memory set.
Revised: It is comprised of an encoder, a retriever and a modulator sub-network, as well as a collection of labelled 3D object point clouds of different categories and segmentation granularity as the memory set.

Original: The same object with finer or coarser segmentation labelling is represented in different memories.
Revised: The same object with finer or coarser segmentation labelling is represented in different memories.

Original: In a nutshell, Analogical Networks work as follows: First, the encoder encodes the input point cloud to a set of point features, and each memory point cloud to point and part features.
Revised: In a nutshell, Analogical Networks work as follows: First, the encoder encodes the input point cloud to a set of point features, and each memory point cloud to point and part features.

Original: The retriever takes as input an 1D encoding of the input3D point cloud and retrieves similar labelled objects from memory.
Revised: The retriever encodes the input and all memory point clouds into 1D vectors, to retrieve similar labelled objects.

Original: The retrieved memory parts initialize queries that are tasked to decode the parts of the input scene.
Revised: The retrieved memory parts initialize queries, tasked to decode the parts of the input scene.

Original: The modulator uses multiple iterations of self and cross-attention operations to jointly contextualize the encodings of the input point cloud and the retrieved memory part queries.
Revised: The modulator uses multiple iterations of attention operations to jointly contextualize the encodings of the input point cloud and the retrieved memory part queries.

Original: Queries that are found confident decode the “analogous” parts in the present 3D point cloud and a Hungarian loss is computed between predicted and ground-truth parts, similar todetection transformers (Carion et al., 2020).
Revised: Confident queries decode the “analogous” parts in the present 3D point cloud and a Hungarian loss is computed between predicted and groundtruth parts, similar to (Carion et al., 2020).

Original: When a memory part query is used to decode an analogous part in the input point cloud, we say  the two parts are put in correspondence by the model.
Revised: When a memory part query is used to decode an analogous part in the input point cloud, we say that the two parts are put in correspondence by the model.

Original: The colors in columns 2, 3 and 5, 6 of Figure 2 denote part correspondence between the retrieved memory parts and the input point cloud.
Revised: The colors in columns 2, 3 and 5, 6 of Figure 1 denote part correspondence between the retrieved memory parts and the input point cloud.

Revised: Note that more than one memories can be retrieved and used, allowing us to mix parts of different memories to explain the current input, as in Figure 3 right.

Original: Encoder The encoder  is a PointNet++ (Qi et al., 2017) and encodes the point cloud scene S and each memory scene M to a set of 3D point features F S , F M ∈ R N × C (where N is the number of sub-sampled points and C is the feature dimension).
Revised: Encoder The encoder (pink in Figure 2) is a PointNet++ (Qi et al., 2017) and encodes the point cloud scene S and each memory scene M to a set of 3D point features F S , F M ∈ R N × C (where N is the number of sub-sampled points and C is the feature dimension).

Original: 3D positional encodings are then added to the 3D point features.
Revised: 3D positional encodings are then added to the 3D point features.

Original: Each labelled part p of a memory scene M is then encoded into a 1D feature vector f
Revised: Each labelled part p of a memory scene M is then encoded into a 1D feature vector f

Original: Mp by average pooling of its point features.
Revised: Mp by average pooling of its point features.

Original: For retrieval, each scene (memory or the input) is summarized in an 1D vector by  global average pooling.
Revised: The encoder is trained end-to-end with the modulator, i.e. gradients of the part segmentation objectives back-propagate to the encoder. Retriever The retriever module first extracts features for the input object point cloud and each memory object point cloud. The feature extractor is a PointNet++ (green encoder in Figure 2) that shares the same architecture as the encoder. Then, each memory and input feature cloud is summarized in an 1D vector by a global average pooling of its features.

Original: Retriever Theretriever retrieves top- k memories for the input scene S by computing an inner product between  input scene query and memorykey features.
Revised: The top- k memories are retrieved by computing an inner product between the input point cloud feature and memory features.

Original: Modulator
Revised: Modulator

Original: The modulator sub-network resembles the decoder of a detection transformer  (Carion et al., 2020) with two key differences: First, instead of only using a set of scene agnostic learnable query vectors (64 in our case), it also considers the retrieved memory part features f Mp (where M = 1 ..k ) as queries, i.e. candidates for decoding parts in the input point cloud scene.
Revised: The modulator sub-network resembles the decoder of a detection transformer (DETR) (Carion et al., 2020) with two key differences: First, instead of only using a set of scene-agnostic learnable query vectors (64 in our case), it also considers the retrieved memory part features f Mp (where M = 1 ..k ) as queries, i.e. candidates for decoding parts in the input point cloud scene.

Original: Second, during the iterative self and cross-attention layers, both the queries and the point features of the input scene are updated, while in DETR(Carion et al., 2020) only the queries are updated.
Revised: Second, during the iterative self and cross-attention layers, both the queries and the point features of the input scene are updated, while in DETR only the queries are updated.

Original: We considerlayers of self and cross-attention.
Revised: We consider 6 layers of self and cross-attention.

Original: More details and pseudocode are provided in the supplementary.
Revised: More details and pseudocode are provided in the supplementary.

Original: Lastly, we upsample the point features to the original resolution (from N to N p ) using convolutional layers (Qi et al., 2017) and compute an inner product between each query and point feature to compute the segmentation mask for each query.
Revised: Lastly, we upsample the point features to the original resolution (from N to N p ) using convolutional layers (Qi et al., 2017) and compute an inner product between each query and point feature to compute the segmentation mask for each query.

Original: The output of the modulator is a set of N q segmentation mask proposals and corresponding confidence scores, where N q is the total number of queries.
Revised: The output of the modulator is a set of N q segmentation mask proposals and corresponding confidence scores, where N q is the total number of queries.

Original: At training time, these proposals are matched to ground-truth instance binary masks using the Hungarian matching algorithm (Carion et al., 2020).
Revised: At training time, these proposals are matched to ground-truth instance binary masks using the Hungarian matching algorithm (Carion et al., 2020).

Original: For the proposals that are matched to a ground-truth instance, we compute the segmentation loss, which is a per-point binary cross-entropy loss (Vu et al., 2022; Cheng et al., 2021).
Revised: For the proposals that are matched to a ground-truth instance, we compute the segmentation loss, which is a per-point binary cross-entropy loss (Vu et al., 2022; Cheng et al., 2021).

Original: We also supervise the confidence score of each query, similar to (Carion et al., 2020).
Revised: We also supervise the confidence score of each query, similar to (Carion et al., 2020).

Original: The target labels are 1 for the proposals matched with a ground-truth part and 0 for non-matched.
Revised: The target labels are 1 for the proposals matched with a ground-truth part and 0 for non-matched.

Original: We found it beneficial to apply these losses after every cross-attention layer in the modulator.
Revised: We found it beneficial to apply these losses after every cross-attention layer in the modulator.

Original: At test time, we multiply the per point mask occupancy probability with tiled confidence scores to get a N p × N q tensor ( N p is the number of points); then each point is assigned to the most confident query by computing inner product and taking per-point argmax over the queries.
Revised: At test time, we multiply the per point mask occupancy probability with tiled confidence scores to get a N p × N q tensor ( N p is the number of points); then each point is assigned to the highest scoring query by computing inner product and taking per-point argmax over the queries.

Original: Within-instance correspondence pre-training In Analogical Networks, the retrieval process is not end-to-end differentiable with respect to the downstream scene parsing task.
Revised: Within-instance correspondence pre-training In Analogical Networks, the retrieval process is not end-to-end differentiable with respect to the downstream scene parsing task.

Original: This isthe case because  retrieved memories are contextualized with the inputscene through dense cross-attention operations, instead of bi-encoder fusion (Miech et al., 2021); the former, though a stronger performer than bi-encoder models, it is not differentiable with respect to the memory feature vectors used by the retriever, as also explained in (Chen et al., 2022).
Revised: This is because i) we have no ground-truth annotations for retrieval and ii) the retrieved memories are contextualized with the input through dense cross-attention operations, instead of bi-encoder fusion (Miech et al., 2021); the former, although a stronger performer than bi-encoder models, is costly to compute for the whole memory set. Thus, our solution is a hard top- k selection of memories that is not differentiable with respect to the memory feature vectors used by the retriever, as also explained in (Chen et al., 2022).

Original: We devise a novel pre-training scheme where we pre-train the encoder and modulator parameters independent of the retriever, by augmenting (rotating and deforming (Kim et al., 2021))
Revised: We devise a novel pre-training scheme where we pre-train the encoder and modulator parameters independently of the retriever, by augmenting (rotating and deforming (Kim et al., 2021))

Original: each training 3D point cloud, and using the original point cloud as the modulating memory, as shown in the left side of Figure 3.
Revised: each training 3D point cloud, and using the original point cloud as the modulating memory, as shown in the left side of Figure 3.

Original: In this special case, we actually have ground-truth information regarding part association between the original and the augmented point cloud.
Revised: In this special case, we actually have ground-truth information regarding part association between the original and the augmented point cloud.

Original: We thus supervise each memory part query to decode the corresponding deformed part in the input cloud and do not use Hungarian matching.
Revised: We thus supervise each memory part query to decode the corresponding deformed part in the input cloud and do not use Hungarian matching.

Original: We do not have such part association ground-truth during regular (cross-instance) training.
Revised: We do not have such part association ground-truth during regular (cross-instance) training.

Revised: After within-instance pre-training (Algorithm 1), we maintain two copies of the encoder weights: one copy acts as the encoder of the retriever and is frozen.

Original: Upon within-instance pre-training, we train our model using as  modulating memory set sampled from the retriever’s top K memories.
Revised: The other copy is used as the encoder and is further trained cross-instance, i.e. with the modulating memory being sampled from the top-k retrieved memories.

Original: Duringregular training (i.e. cross-instance training), we further co-train with our within instance training data.
Revised: During cross-instance training (Algorithm 2), we further co-train with within-instance training data.

Original: We found that in this second training stage using the part association supervision in within-instance examples does not further contribute to performance, so we supervise with our standard losses and Hungarian matching.
Revised: We found that in this second training stage, using the part association supervision in within-instance examples does not further contribute to performance, so we use our standard losses and Hungarian matching.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test Analogical Networks on PartNet (Mo et al., 2019), an established benchmark for 3D object segmentation.
Revised: We test Analogical Networks on PartNet (Mo et al., 2019), an established benchmark for 3D object segmentation.

Original: PartNet contains multiple 3D object instances from 24 object categories, each segmented into three different segmentation granularities.
Revised: PartNet contains multiple 3D object instances from 24 object categories, each segmented into three different levels of granularity.

Original: We consider two experimental paradigms:
Revised: We consider two experimental paradigms:

Original: • Many-shot: In this setup, we split the exemplars of each of (base) categories into train and test sets.
Revised: Many-shot: We split the exemplars of the base categories into train and test sets.

Original: The model is tested on segmenting instances of the same categories in the test set.
Revised: The model is tested on segmenting instances of the same categories in the test set.

Original: Our base categories are Chair, Display, StorageFurniture, Bottle, Clock, Door, Earphone, Faucet, Knife, Lamp, TrashCan, Vase.
Revised: Our base categories are Chair, Display, Storage Furniture, Bottle, Clock, Door, Earphone, Faucet, Knife, Lamp, Trash Can and Vase.

Original: • Few-shot:In this setup, K examples of novel categories are shown to the model, and the model is tasked to segment new examples of these categories.
Revised: Few-shot: K examples of novel categories are given and the model is tasked to segment new examples of these categories.

Original: This tests the ability of the model for few shot adaptation.
Revised: This tests few-shot adaptation.

Original: We consider K = 1 and K = 5 setups.
Revised: We consider K = 1 and K = 5 setups.

Original: Our novel categories are Table, Bed, Dishwasher, Refrigerator.
Revised: Our novel categories are Table, Bed, Dishwasher and Refrigerator.

Original: Our experiments aim to answer the following questions: 
Revised: Our experiments aim to answer the following questions: Q1 :

Original: How do Analogical Networks compare to parametric SOTA neural networks and metalearning networks for 3D object parsing in the many-shot and few-shot paradigms?
Revised: How do Analogical Networks compare to parametric SOTA neural networks and meta-learning networks for 3D object parsing in the many-shot and few-shot paradigms?

Original: How well Analogical Networks learn part-based associations across exemplars without part association or part semantic label supervision?
Revised: (Section 4.1) Q2 :

Original: How much  weight updating  over memory updating help few shot adaptation in Analogical Networks?4.
Revised: How much does weight updating (fine-tuning) over memory expansion help few-shot adaptation in Analogical Networks?

Revised: (Section 4.1) Q3 :

Original: How much within-instance correspondence pretraining helps Analogical Networks?
Revised: How much within-instance correspondence pre-training helps Analogical Networks?

Original: [Table] Our few shot experiments use four held-out (novel) categories which are Table, Bed, Dishwasher, Refrigerator. We report mean and standard deviation for few-shot ARI performance over 10 tasks (each task consists of a different set of K-shot support set).
Revised: Q4 : How does the proposed query-centric modulation compare against more conventional memoryaugmented neural networks (Wu et al., 2022) where memories are attended to but not used directly as decoding slot queries?

Original: Without any finetuning, Analogical Networks outperform the baseline by 26% in the few shot setup. Though weight finetuning helps both models, it brings a performance boost of 18% for the baseline and only 2% for our model.
Revised: (Section 4.1) Q5 :

Original: This means Analogical Networks can generalize and adapt few shot to new categories by expanding their memory, without any weight interference.
Revised: How well do Analogical Networks learn part-based associations across exemplars without part association or part semantic label supervision?

Original: Even upon finetuning, Analogical Networks outperform the 3D-DETR baseline by 10 % ARI. How the proposed query-centric modulation compares against more conventional memoryaugmented neural networks (Wu et al., ) where memories are attended on but not used directly as decoding slot queries?
Revised: (Section 4.2)

Original: How much the performance ofAnalogical Networks varies with different retrieval mechanisms?
Revised: In the appendix, we show the effect of different retrieval mechanisms (6.4, 6.5), the importance of training using multiple classes (6.6), and results under more setups and benchmarks (6.3, 6.7).

Original: Evaluation metrics We use  Adjusted Random Index (ARI) as ourevaluation metric for segmentation accuracy (Rand, 1971), which is acategory label-agnostic clustering score ( 1 being the best and − 1 being the worst).
Revised: Evaluation metrics We use the Adjusted Random Index (ARI) as our segmentation quality metric (Rand, 1971), which is a label-agnostic clustering score ranging from − 1 (worst) to 1 (best).

Original: ARI calculates the similarity between two point clusters while being invariant to the ordering of the cluster centers.
Revised: ARI calculates the similarity between two point clusters while being invariant to the order of the cluster centers.

Original: We report 100 × ARI as our metric. For this, we use the publicly available implementation of (Kabra et al., 2019).
Revised: We compute 100 × ARI using the publicly available implementation of (Kabra et al., 2019).

Original: Baselines We compare our model to the following models in the literature:  3D-DETR is a 3D detection transformerneural network with the same backbone as our model, and similar segmentation prediction head and loss, but without any memory retrieval, akin to the 3D equivalent of a stateof-the-art 2D image segmentor (Carion et al., 2020; Athar et al., 2021).
Revised: Baselines We compare our model to the following models in the literature: PartNet (Mo et al., 2019) is an instance segmentation model with the same input encoder as our model. 3D-DETR is a 3D detection transformer network with the same backbone as our model and similar segmentation prediction head and loss, but without any memory retrieval, akin to the 3D equivalent of a stateof-the-art 2D image segmentor (Carion et al., 2020; Athar et al., 2021).

Original: Note that point features are updated during the decoder layers, same as  our model, which we found to help performance.
Revised: Note that point features are updated during the decoder layers, same as in our model, which we found to help performance.

Original: Prototypical Networks is an adaptation of the episodic prototypical networks for image classification of (Snell et al., 2017) to the task of 3D object part segmentation.
Revised: Prototypical Networks is an adaptation of the episodic prototypical networks for image classification of (Snell et al., 2017) to the task of 3D object part segmentation.

Original: Specifically, given a set of N labelled point clouds, we form average feature vectors for each semantic labelled part and use them as queries to segment points into corresponding part masks.
Revised: Specifically, given a set of N labelled point clouds, we form average feature vectors for each semantic labelled part and use them as queries to segment points into corresponding part masks.

Original: Contrary to 3D-DETR , Analogical Networks attend on external memories.
Revised: Contrary to 3D-DETR, Analogical Networks attend on external memories.

Original: Contrary to Prototypical Networks , Analogical Networks consider feature contextualization between memory prototypes and the input scene.
Revised: Contrary to Prototypical Networks, our model considers feature contextualization between memory prototypes and the input scene.

Original: We further consider the following ablated versions of our model, each one aims to evaluate a specific aspect of its design: Analogical Networks w/o memory part queries, which is a model similar to the parametric 3D-DETR baseline augmented with attention to retrieved memories (Wu et al., 2022).
Revised: We further consider the following ablated versions of our model, each one aims to evaluate a specific aspect of its design: Analogical Networks w/o memory part queries is a model similar to the parametric 3D-DETR baseline augmented with attention to retrieved memories (Wu et al., 2022).

Original: Instead of using the part memory features as queries, this baseline updates the scene-agnostic query vectors by iteratively attending to the input feature cloud and the memory part features.
Revised: Instead of using the part memory features as queries, this baseline updates the scene-agnostic query vectors by iteratively attending to the input feature cloud and the memory part features.

Original: Different from Analogical Networks, correspondence cannot emerge between a memory and the input scene, since the model does not encode such correspondence explicitly, but only implicitly, in the attention operations.
Revised: Different from Analogical Networks, correspondence cannot emerge between a memory and the input scene, since the model does not encode such correspondence explicitly, but only implicitly, in the attention operations.

Original: Analogical Networks single memory, which is Analogical Networks with a single modulating memory.
Revised: Analogical Networks single memory is Analogical Networks with a single modulating memory, in contrast to Analogical Networks multi-memory that uses five memories.

Original: Analogical Networks w/o pretrain, which is our model without the self-supervised within-instance correspondence objective for training of the modulator and encoder weights.
Revised: Analogical Networks w/o pretrain is our model without the self-supervised within-instance correspondence objective for training of the modulator and encoder weights.

Original: 4. M ANY AND FEW  SHOT 3D OBJECT PART SEGMENTATION
Revised: 4. M ANY AND FEW - SHOT 3D OBJECT PART SEGMENTATION

Original: The PartNet benchmark supplies three levels of segmentation per object instance where level 3 is the most fine-grained.
Revised: The PartNet benchmark supplies three levels of segmentation per object instance where level 3 is the most fine-grained.

Original: We train and test our model and baselines on all three levels of segmentation.
Revised: We train and test our model and baselines on all three levels of segmentation.

Original: We use a learnable level embedding concatenated to the input for our baseline 3D-DETR , as is usually the case in multi-task models (Jang et al., 2022).
Revised: We use a learnable level embedding concatenated to the input for our baselines PartNet and 3D-DETR, as is usually the case in multi-task models (Jang et al., 2022).

Original: In the many shot setting, all examples in the training set become part of our model’s memory.
Revised: In the many-shot setting, all examples in the training set become part of our model’s memory.

Original: In the few shot setting, Analogical Networks adapt in two ways i) by expanding the memory of the model with the novel K -shot support examples, and ii) by further adapting the weights via finetuning to parse the K examples using both within-instance pre-training and cross-instance training.
Revised: In the few-shot setting, Analogical Networks adapt in two ways i) by expanding the memory of the model with the novel K -shot support examples, and ii) by further adapting the weights via fine-tuning to parse the K examples.

Original: In this case, the memory set is only the novel labelled support set instances.
Revised: In this case, the memory set is only the novel labelled support set instances.

Original: We show quantitative results in Table 1.
Revised: We show quantitative results in Table 1.

Original: Our conclusions are as follows: (i) Analogical Networks dramatically outperform parametric DETR segmentors in few-shot learning  .
Revised: Our conclusions are as follows: (i) Analogical Networks dramatically outperform parametric DETR segmentors in few-shot learning (Q1) .

Original: While in the base category test set the two models have similar performance (70.5 versus 70.6), when adapting  few-shot to the novel categories, Analogical Networks dramatically outperforms 3D-DETR (48.
Revised: While in the base category test set the two models have similar performance (70.5 versus 70.6), when adapting [Table] tuning, Analogical Networks outperform 3D-DETR by 10% ARI. few-shot to the novel categories, Analogical Networks dramatically outperforms 3D-DETR (48.

Original: versus 58.4 in 5-shot ARI).
Revised: versus 58.4 in 5-shot ARI).

Original: (ii) Analogical networks can adapt few-shot to novel categories without weight updates, simply by memory expansion  .
Revised: (ii) Analogical networks can adapt few-shot to novel categories without weight updates, simply by memory expansion (Q2) .

Original: Indeed, the 5-shot performance of our multi-memory model is very close before and after finetuning in the novel categories.
Revised: Indeed, the 5-shot performance of our multi-memory model is very close before and after fine-tuning in the novel categories.

Original: (iii) Multimemory Analogical Networks outperform single memory Analogical Networks in few-shot adaptation .
Revised: (iii) Multimemory Analogical Networks outperform single memory Analogical Networks in few-shot adaptation .

Original: (iv) Within-instance correspondence pre-training helps Analogical Networks to adapt few-shot to the novel categories.
Revised: (iv) Within-instance correspondence pre-training helps Analogical Networks to adapt few-shot to the novel categories.

Original:  (v) Attending on retrieved memory part features and contextualizing them with the input, as done in Analogical Networks w/o memory part queries, helps over parametricalone segmentors  inboth many and few-shot settings.
Revised: (Q3) (v) Attending on retrieved memory part features and contextualizing them with the input, as done in Analogical Networks w/o memory part queries, helps over parametric segmentors (Q4) in few-shot settings.

Original: Our interpretation for this interesting result is that attending on relevant memories permits a model to operate in context and overcome challenges caused by imbalance of examples across categories.
Revised: Our interpretation for this interesting result is that attending on relevant memories permits a model to operate in context and overcome challenges caused by imbalance of examples across categories.

Original: Rare instances are not forgotten as they are explicitly saved in the external memory, and upon retrieval and modulation with the input, can steer inference.
Revised: Rare instances are not forgotten as they are explicitly saved in the external memory and, upon retrieval and modulation with the input, can steer inference.

Original: In parametric alone networks, rare examples are in competition with examples in the mode of the training distribution for their representation in the parameters.
Revised: In parametric-only networks, rare examples are in competition with examples in the mode of the training distribution for their representation in the parameters.

Original: We show per-class ARI scores in the supplementary.

Original: Still, this memory-augmented model exhibits much worse performance than our complete model without weight finetuning (49% versus 56.4 % ARI).
Revised: Still, this memory-augmented model exhibits much worse performance than our complete model without weight fine-tuning (49% versus 56.4 % ARI).

Original: Please, see our supplementary file for performance of our model under varying retrieval schemes.
Revised: We show per-class ARI scores in the supplementary.

Original:  E VALUATION OF EMERGENT PART CORRESPONDENCES
Revised: 4. E VALUATION OF E MERGENT P ART C ORRESPONDENCES

Original: In this section, we evaluate the quality of cross-exemplar part correspondences that emerge in
Revised: In this section, we evaluate the quality of cross-exemplar part correspondences that emerge in

Original: Analogical Networks .
Revised: Analogical Networks .

Original: Our model is trained solely for segmentation of 3D objects, and no semantic part labels are used during training.
Revised: Our model is trained solely for segmentation of 3D objects, and no semantic part labels are used during training.

Original: The only cross-part association (correspondence) supervision is during pre-training, where the parts in a 3D point cloud are used as queries to decode(upon modulation) the corresponding part of the augmented 3D point cloud.
Revised: The only cross-part association (correspondence) supervision is during pre-training, where the parts in a 3D point cloud are used as queries to decode the corresponding part of the augmented 3D point cloud.

Original: Yet, through the use of memory part queries shown in Figure 2, our model can “label” input scenes with the labels of the corresponding memory parts.
Revised: Yet, through the use of memory part queries shown in Figure 2, our model can “label” input scenes with the labels of the corresponding memory parts.

Original: If the parts of visual memories are semantically labelled, those semantic labels will propagate to the input scene.
Revised: If the parts of visual memories are semantically labelled, those semantic labels will propagate to the input scene.

Original: Parts decoded from scene agnostic queries are not put in correspondence with any memory in our model.
Revised: Parts decoded from scene agnostic queries are not put in correspondence with any memory in our model.

Original: We found 80% of 3D points to be on parts decoded by memory part queries on average.
Revised: We found 80% of 3D points to be on parts decoded by memory part queries on average.

Original: Here we evaluate the quality of label propagation through emergent correspondences by evaluating  part instance segmentation and semantic segmentation in the few shot setting.
Revised: Here we evaluate the quality of label propagation through emergent correspondences by evaluating few-shot part instance segmentation and semantic segmentation.

Original: We compare against 3D-DETR semantic which isprecisely our 3D-DETR baseline additionally supervised for classifying detected 3D parts into semantic part labels of the category considered in each case.
Revised: We compare against 3D-DETR semantic which is our 3D-DETR baseline additionally supervised for predicting semantic part labels.

Original: Our baseline Prototypical Networks , similar to Analogical Networks, propagates semantic labels of the prototypical part features.
Revised: Similar to our model, Prototypical Networks propagate semantic labels of the prototypical part features.

Original: We evaluate Prototypical Networks only for semantic segmentation since it cannot easily produce instance segments: if multiple part instances belong to the same label, e.g., chair leg, Prototypical Networks assume they belong to the same semantic prototypical part.
Revised: We evaluate Prototypical Networks only for semantic segmentation since it cannot easily produce instance segments: if multiple part instances belong to the same label, this model assumes they belong to the same semantic prototype.

Original: We further train  a version of Analogical Networks whereonly the scene agnostic queries are explicitly supervised for semantic part labels; we call it Analogical Networks semantic .
Revised: We also train Analogical Networks semantic, a version of our model where the scene-agnostic queries are explicitly supervised for semantic part labels.

Original:  Evaluation metric We evaluate the part instance segmentation using mean Panoptic Quality (PQ)as our metric (Kirillov et al., 2019).
Revised: [Table] Evaluation metrics We use mean Panoptic Quality (PQ) (Kirillov et al., 2019) for part instance segmentation.

Original: We evaluate the 3D point semantic segmentation using mean intersection over union (mIoU) as our metric.
Revised: We use mean intersection over union (mIoU) for 3D point semantic segmentation.

Original: We show quantitative semantic and instance segmentation results in
Revised: We show quantitative semantic and instance segmentation results in

Original: Table 2.
Revised: Table 2.

Original: Analogical Networks show very high semantic and instance segmentation accuracy via label propagation through memory part queries, without any supervision.
Revised: Analogical Networks show very high semantic and instance segmentation accuracy via label propagation through memory part queries, without any supervision (Q5).

Original: We report two sets of numbers for our model.
Revised: We report two sets of numbers for our model.

Original: The first penalizes points that belong on parts decoded by scene agnostic queries and assumes they have wrong labels, the second only evaluates segmentation scores on the 3D points on parts decoded by part memory queries.
Revised: The first penalizes points that belong on parts decoded by scene agnostic queries and assumes they have wrong labels, the second only evaluates segmentation scores on the 3D points on parts decoded by part memory queries.

Original: [Table]

Original: 3D object segmentation on PartNet dataset (Mo et al., 2019).

Original: Few-shot performance is calculated by averaging over the 10 different K − shot tasks (each task consists of a different support set, we observed std of 0 .

Original: 03 for all the reported values).

Original: * performance is calculated on the subset of input 3D points that were classified by memory part queries (and not by the scene agnostic queries since in the latter case semantic labels cannot be propagated.)

Original: 4. L IMITATIONS - F UTURE DIRECTIONS
Revised: 4. L IMITATIONS - F UTURE DIRECTIONS

Original: Below we present a set of future directions that are necessary for Analogical Networks to scale beyond segmentation of single object scenes.
Revised: Below we present a set of future directions that are necessary for Analogical Networks to scale beyond segmentation of single object scenes.

Original: (i)
Revised: (i)

Original: The retriever in Analogical Networks operates currently over whole object memories and it is not end-to-end differentiable with respect to the downstream task.
Revised: The retriever in Analogical Networks operates currently over whole object memories and it is not end-to-end differentiable with respect to the downstream task.

Original: Sub-object part-centric memory representations would permit fine-grained retrieval of visual memory scenes.
Revised: Sub-object part-centric memory representations would permit fine-grained retrieval of visual memory scenes.

Original: We further plan to explore alternative supervision for the retriever module inspired by works in the language domain (Izacard et al., 2022; Izacard & Grave, 2021).
Revised: We further plan to explore alternative supervision for the retriever module inspired by works in the language domain (Izacard et al., 2022; Izacard & Grave, 2021).

Original: (ii) Scaling Analogical Networks to segmentation of complete, multi-object 3D scenes in realistic home environments requires scaling up the size of memory collection.
Revised: (ii) Scaling Analogical Networks to segmentation of complete, multi-object 3D scenes in realistic home environments requires scaling up the size of memory collection.

Original: It would further necessitate bootstrapping fine-grained object part annotations, missing from 3D scene datasets (Dai et al., 2017), by transferring knowledge of object part compositions from PartNet.
Revised: It would further necessitate bootstrapping fine-grained object part annotations, missing from 3D scene datasets (Dai et al., 2017), by transferring knowledge of object part compositions from PartNet.

Original: Such semi-supervised finegrained scene parsing is an exciting avenue of future work.
Revised: Such semi-supervised finegrained scene parsing is an exciting avenue of future work.

Original: (iii)
Revised: (iii)

Original: So far we have assumed the input to Analogical Networks to be a complete 3D point cloud.
Revised: So far we have assumed the input to Analogical Networks to be a complete 3D point cloud.

Original: However, this is hardly ever the case in reality.
Revised: However, this is hardly ever the case in reality.

Original: Humans and machines need to make sense of single view, incomplete and noisy observations.
Revised: Humans and machines need to make sense of single view, incomplete and noisy observations.

Original: Extending Analogical Networks with generative heads that not only detect analogous parts in the input, but also inpaint or complete missing parts, is a direct avenue for future work.
Revised: Extending Analogical Networks with generative heads that not only detect analogous parts in the input, but also inpaint or complete missing parts, is a direct avenue for future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented Analogical Networks, a model for associative 3D visual parsing that encodes domain knowledge explicitly, in a set of past labelled visual scenes, and implicitly, in model parameters.
Revised: We presented Analogical Networks, a model for associative 3D visual parsing that encodes domain knowledge explicitly, in a set of past labelled memories, and implicitly, in model parameters.

Original: Analogical Networks forgo theestablished paradigm of mapping visual scenes to semantic labels and instead puts forward an analogical paradigm of mapping  scenes to past scenes, their parts labels and their compositions.
Revised: Analogical Networks forgo the paradigm of mapping visual scenes to semantic labels and instead put forward an analogical paradigm of mapping input scenes to past scenes and their parts labels.

Original: Given an inputvisual scene, the model retrieves a set of labelled memory point clouds, and contextualized those with the input throughself and cross attention layers.
Revised: Given an input scene, the model retrieves a set of labelled memory point clouds and contextualizes those with the input through attention layers.

Original: The contextualized memory part features decode corresponding parts in the input point cloud.
Revised: The contextualized memory part features decode corresponding parts in the input point cloud.

Original: The encoder and modulation parameters of the model are trained end-to-end for the segmentation task, and the retriever parameters are trained with supervised contrastive feature learning.

Original: By casting visual parsing as analogical correspondence, Analogical Networks can be trainedacross and share knowledge across multiple tasks, where each task is communicated via the features and labels of appropriate conditioning memories.
Revised: By casting visual parsing as analogical correspondence, Analogical Networks can be trained and share knowledge across multiple tasks, where each task is communicated via the features and labels of appropriate conditioning memories.

Original: One shot, few shot or many shot learning are treatedin a uniformly, by conditioning to the appropriate set of memories, whether taken from a single, few or many memory exemplars, and inferring analogous parses.
Revised: One-shot, few-shot or many-shot learning are treated uniformly, by conditioning to the appropriate set of memories, whether taken from a single, few or many memory exemplars, and inferring analogous parses.

Original: We showed Analogical Networks outperform SOTA baselines in both many and few shot parsing setting.
Revised: We showed Analogical Networks outperform SOTA baselines in both many and few-shot parsing setting.

Original: We further showed part correspondences emerge across exemplars without supervision, as a by-product of the analogical inductive bias.
Revised: We further showed part correspondences emerge across exemplars without supervision, as a by-product of the analogical inductive bias.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure the reproducibility of the empirical results, we include a pseudo-code of the main model components and training pipeline in the Appendix.
Revised: To ensure the reproducibility of the empirical results, we include a pseudo-code of the main model components and training pipeline in the Appendix.

Original: Upon publication, we will make our code publicly available on GitHub.
Revised: Upon publication, we will make our code publicly available on GitHub.

Original: [Figure]
Revised: [Figure]

Original:  A PPENDIX
Revised:  A PPENDIX

Original: Our Appendixmaterial is organized as follows:
Revised: Our Appendix is organized as follows:

Original: In Section 6.1, we provide implementation detailssuch as pseudo code and hyper-parameters for training Analogical Networks.
Revised: In Section 6.1, we provide implementation details and pseudocode for training Analogical Networks.

Original: Section 6. reports the  ARI scores for eachcategory involved in base training phase.
Revised: Section 6.2 reports the many-shot ARI scores for each training category.

Original: In section 6.3 we ablate
Revised: We show more quantitative results for part instance segmentation using AP 50 metric and compare against SOTA baselines in Section 6.3.

Original:  Analogical Networks performance under varying memory retrieval schemes in 5 -shot setting.
Revised: In section 6.4 we ablate Analogical Networks’ performance under varying memory retrieval schemes in 5 -shot setting and show qualitative results for the retriever in Section 6.5. In Section 6.6 we examine the importance of training on multiple classes for emergent in-context behavior. We show more results on noisy point clouds (ScanObjectNN (Uy et al., 2019) dataset) in 6.7.

Original: We provide  qualitative visual object parsing results for single and multi memory variants of Analogical Networks in section 6.4.
Revised: Lastly, we provide extensive qualitative visual object parsing results for single and multi memory variants of Analogical Networks in section 6.8.

Original: 6. T RAINING P SUEDO C ODE AND I MPLEMENTATION D ETAILS
Revised: 6. T RAINING P SEUDO C ODE AND I MPLEMENTATION D ETAILS

Original: For both stages of training (i.e. within-instance correspondence pre-training and cross-instance training), we use AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 1e − 4 , l 2 weight decay of 1e − 2 and batch size of 8 .
Revised: For both stages of training (i.e. within-instance correspondence pre-training and cross-instance training), we use AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 1e −and batch size of 8 .

Original: We train it for 100 epochs  with a learning rate schedule decay of 0 .
Revised: We train the model for 40 epochs within-instance and 25 cross-instance with a learning rate schedule decay of 0 .

Original: 5 at every 30 th epoch.
Revised: 5 at every 30 th epoch.

Original: For few-shot finetuing/evaluation, we use AdamW optimizer(Loshchilov & Hutter, 2017) with an initial learning rate of 1e − 5, l 2 weight decay of 1e − 2 and batch size of 4 .
Revised: For few-shot fine-tuning/evaluation, we use AdamW optimizer with an initial learning rate of 1e − 5 and batch size of 4 .

Original: We finetune it for 30 epochs and we report the performance across 10 different task runs (each task has a different set of K-shot support samples).
Revised: We fine-tune it forepochs and we report the performance across 10 different task runs (each task has a different set of K support samples).

Original: We describeour Analogical Networks training details in the following pseudo code:
Revised: We describe Analogical Networks’ training details in the following pseudo-code for within (Algorithm 1) and cross-instance training (Algorithm 2) respectively: [Table] for S in dataloader

Original: [Table]
Revised: [Table]

Revised: with

Revised: [Table]

Revised: # the

Revised: [Table]

Revised: is augmented

Revised: [Table]

Revised: Do

Revised: [Table]

Revised: layers of modulation using

Original: [Table]
Revised: [Table]

Revised: num

Revised: [Table]

Revised: [Table]

Revised: +

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: 6. ARI P ERFORMANCE FOR B ASE C ATEGORIES
Revised: [Table] [Table] optimizer. [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: for S

Revised: [Table]

Revised: M in dataloader

Revised: [Table]

Revised: batch with B samples

Revised: [Table]

Revised: [Table]

Revised: 6. ARI P ERFORMANCE FOR B ASE C ATEGORIES

Original: In this section, we provide category specific ARI scores for different approaches as mentioned in table 3 that belongs to the base category set.
Revised: We provide category-specific many-shot ARI scores for our model and baselines on the training categories in Table 3. Note that all models in this table are trained on all classes and segmentation levels jointly [Table]

Revised: 6. E VALUATION ON THE L ABELED I NSTANCE S EGMENTATION S ETUP OF (S UN ET AL ., 2022)

Revised: We compare our Analogical Networks semantic model with baselines such as PartNet (Mo et al., 2019), GSPN (Yi et al., 2018b), PE (Zhang & Wonka, 2021) and Semantic Segmentation-Assisted

Original: 6. P ERFORMANCE UNDER VARYING RETRIEVAL SCHEMES .
Revised: Instance Feature Fusion (SAIF) (Sun et al., 2022) for AP 50 metric in Table 4. We observe that [Table] P E

Revised: Coarse

Revised: [Table]

Revised: Middle

Revised: [Table]

Revised: - Fine

Original: In few-shot setting
Revised: [Table] [Table] our model Analogical Networks semantic outperforms existing approach by a large margin. Note that our model is at disadvantage because it is tested without fine-tuning on novel classes (Bed,

Original: [Table]
Revised: Dishwasher, Fridge and Table) and as a result has never seen any examples of these classes at training time, apart from in-context usage of labeled memories at test time. In contrast, all other approaches in Table 4 use all available training data for each class. Still, we can see that it largely outperforms other methods in three out of four novel classes. 6. P ERFORMANCE UNDER V ARYING R ETRIEVAL S CHEMES

Original:  we evaluate the performance of Analogical Networks under varying memory retrieval schemes in Table 4.
Revised: In few-shot setting, we evaluate the performance of Analogical Networks under varying memory retrieval schemes in Table 5.

Original: We compare against a hypothetical oracle retriever that can fetch the most helpful (in terms of resulting ARI) memory for each input.
Revised: We compare against a hypothetical oracle retriever that can fetch the most helpful (in terms of resulting ARI) memory for each input.

Original: Our conclusions are as follows: (i)
Revised: Our conclusions are as follows: (i)

Original: Analogical Networks with an oracle memory perform better than Analogical Networks using memories retrieved by the retriever .
Revised: Analogical Networks with an oracle memory perform better than Analogical Networks using memories retrieved by the retriever .

Original: This suggests that better training of our retriever or exploring its co-training with the rest of our model would have significant impact in improving its performance.
Revised: This suggests that better training of our retriever or exploring its co-training with the rest of our model would have significant impact in improving its performance.

Original: (ii) Considering any of the 5 shot exemplars randomly does slightly worse than using memories retrieved by our retriever. 
Revised: (ii) Considering any of the 5-shot exemplars randomly does slightly worse than using memories retrieved by our retriever. (iii)

Revised: The distance metric used (cosine, L1, L2) has minimal effect

Revised: [Table]

Original: Note that while in the few-shot setting we operate in each object category separately for both our model and baselines, in the base category case, our retriever needs to learn to retrieve memories of the correct category, which is critical for the good performance of our model.
Revised: on performance. This is probably because we do not optimize for the retrieval task explicitly, thus the feature space is not regularized for a specific distance metric. This however could be a more important choice if we train the retriever end-to-end with the modulator. Lastly, note that while in the few-shot setting we operate in each object category separately for both our model and baselines, in the base category case, our retriever needs to learn to retrieve memories of the correct category, which is critical for the good performance of our model.

Revised: 6. Q UALITATIVE P ERFORMANCE OF THE R ETRIEVER

Original: [Table]
Revised: We show qualitative results of the retriever on multiple classes, both seen (Figure 4) during training and unseen (Figure 5). We observe the following: (i) The retriever considers fine-grained object similarities and not only class information. To illustrate this, we include two examples for the “Chair” and “Earphone” classes in Figure 4, as well as the “Bed” and “Refrigerator” classes in Figure 5.

Revised: Different instances of the same category retrieve very different memories, that share both structural and semantic similarities with the respective input point cloud.

Revised: (ii)

Revised: The retriever generalizes to novel classes, not seen during training , as shown in Figure 5.

Revised: 6. T HE E FFECT OF T RAINING ON M ULTIPLE C LASSES

Revised: One advantage of Analogical Networks is that it can adapt in different context, simply by retrieving memories of different object classes or segmentation granularity.

Revised: This allows our model to i) generalize few-shot to novel classes without fine-tuning and ii) improve many-shot performance across all classes in a single model .

Revised: We quantitatively show this in Table 7, where we compare against the parametric models PartNet and 3D-DETR.

Revised: We train each model under three setups: i) only on “Chair”, which is the most common category (see Table 6, approximately half of our training examples fall into that category), ii) only on “Faucet”, which is a class with relatively few examples (Table 6), iii) on all classes.

Revised: We test the performance of each variant on the two selected classes as well as the four novel classes we use throughout our paper.

Revised: We can observe the following patterns: (i) Models trained on a single category fail to generalize to other classes even after fine-tuning, despite their strong performance on the training class.

Revised: (ii)

Revised: Analogical Networks trained on all classes outperform category-specific models on all tested setups.

Revised: This is not true for the other baselines.

Revised: Both PartNet and 3D-DETR are better when trained on a single class and tested on the same class.

Revised: Analogical Networks generalize better with more diverse data, as the model learns to use labelled memory as input context.

Revised: 6. E VALUATION ON S CAN O BJECT NN D ATASET (U Y ET AL ., 2019)

Revised: We train and test Analogical Networks on ScanObjectNN (Uy et al., 2019), which contains noisy and incomplete point clouds.

Revised: Analogical Networks maintain high performance on both base and novel categories and outperforms the baselines PartNet and 3D-DETR .

Revised: We show qualitative results in Figure 12 for base classes and in Figure 13 for novel classes.

Revised: We additionally visualize retrieval results in Figure 14.

Original:  6. Q UALITATIVE P ARSING RESULTS FOR SINGLE - MEMORY AND MULTI - MEMORY A NALOGICAL N ETWORKS
Revised: [Table] [Table] 6. Q UALITATIVE P ARSING R ESULTS FOR S INGLE -M EMORY AND M ULTI -M EMORY A NALOGICAL N ETWORKS

Original: In this section, we show our model’s qualitative results for object parsing.
Revised: In this section [Table] we show our model’s qualitative results for object parsing

Original:  In the figures below (4, 5, 6 and 7), we visualize the figures with the following 5 columns: (a) Unlabelled input point cloud (b)
Revised: [Table] In the Figures 6, 7, 8, 9, 10, 11, 12, 13 we use the following 5 -column pattern:

Original:  Memory used for modulating the object parsing (c) Object parsing generated using only the memory part queries (not the scene-agnostic queries).
Revised: • Unlabelled input point cloud • Memory used for modulation • Object parsing generated using only the memory part queries (not the scene-agnostic queries).

Original: In this column, black region of areas denote it would be eventually decoded by scene-agnostic query.
Revised: In this column, regions that are not decoded by a memory part query are colored in black.

Original: (d) Final predicted segmentation parsing uses both memory initialized query and scene-agnostic query.
Revised: • Final predicted segmentation parsing using both memory-initialized queries and sceneagnostic queries.

Revised: Regions that are colored in black in the third column but colored differently in the fourth column are decoded by scene-agnostic queries.

Original: (e) Input point cloud’s ground truth segmentation at thesegmentation granularity level of the memory.
Revised: • Input point cloud’s ground truth segmentation at the granularity level of the memory.

Original: We qualitatively show the emergence of part correspondence between retrieved memory (column 2) and the input point cloud parsed using memory query (column 3).
Revised: We qualitatively show the emergence of part correspondence between retrieved memory (column 2) and the input point cloud parsed using memory queries (column 3).

Original: Parts having the same color in column 2 and 3 demonstrates correspondence.
Revised: Parts having the same color in columns 2 and 3 demonstrate correspondence, i.e. a part in column 2 decodes the part with the same color in column 3. Analogical Networks promote correspondence of parts on both base (Figure 6 bottom and 7) and novel (Figure 6 top and 8) categories. This correspondence is semantic but also geometric, as can be seen in Figure 10. When multiple memories are available,

Revised: Analogical Networks mix and match parts of different memories to parse the input.

Original: Furthermore, we show parsing results for Analogical Networks single memory w/o pretrain.
Revised: Furthermore, we show parsing results for Analogical Networks single memory w/o pretrain in Figure 11.

Original: We observe that all of memory part query are inactive in the parsing stage.
Revised: We observe that all of memory part query are inactive in the parsing stage.

Original: This demonstrate the utility of within-instance pretraining as without this pretraining part correspondence doesn’t emerge as shown in Figure 8.
Revised: This demonstrates the utility of within-instance pre-training, as without this pre-training part correspondence does not emerge, as shown in Figure 11.

Revised: Lastly, we show that Analogical Networks generalize to noisy and incomplete point clouds in ScanObjectNN.