Original: SE(3) -E QUIVARIANT A TTENTION N ETWORKS FOR
Revised: SE(3) -E QUIVARIANT A TTENTION N ETWORKS FOR

Original: S HAPE R
Revised: S HAPE R

Original: ECONSTRUCTION IN
Revised: ECONSTRUCTION IN

Original: F UNCTION S PACE
Revised: F UNCTION S PACE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a method for 3D shape reconstruction from unoriented point clouds.
Revised: We propose a method for 3D shape reconstruction from unoriented point clouds.

Original: Our method consists of a novel SE(3)-equivariant coordinate-based network, that parametrizes the occupancy field of the shape and respects the inherent symmetries of the problem.
Revised: Our method consists of a novel SE(3)-equivariant coordinate-based network (TFONet), that parametrizes the occupancy field of the shape and respects the inherent symmetries of the problem.

Original: In contrast to previous shape reconstruction methods that align the input to a regular grid, we operate directly on the irregular point cloud.
Revised: In contrast to previous shape reconstruction methods that align the input to a regular grid, we operate directly on the irregular point cloud.

Original: Our architecture leverages equivariant attention layers that operate on local tokens.
Revised: Our architecture leverages equivariant attention layers that operate on local tokens.

Original: This mechanism enables local shape modelling, a crucial property for scalability to large scenes.
Revised: This mechanism enables local shape modelling, a crucial property for scalability to large scenes.

Original: Given an unoriented, sparse, noisy point cloud as input, we produce equivariant features for each point.
Revised: Given an unoriented, sparse, noisy point cloud as input, we produce equivariant features for each point.

Original: These serve as keys and values for the subsequent equivariant cross-attention blocks that parametrize the occupancy field.
Revised: These serve as keys and values for the subsequent equivariant cross-attention blocks that parametrize the occupancy field.

Original: By querying an arbitrary point in space, we predict its occupancy score.
Revised: By querying an arbitrary point in space, we predict its occupancy score.

Original: We show that our method outperforms previous SO(3)-equivariant methods, as well as non-equivariant methods trained on SO(3)-augmented datasets.
Revised: We show that our method outperforms previous SO(3)-equivariant methods, as well as non-equivariant methods trained on SO(3)-augmented datasets.

Original: More importantly, local modelling together with SE(3)-equivariance create an ideal setting for SE(3) scene reconstruction.
Revised: More importantly, local modelling together with SE(3)-equivariance create an ideal setting for SE(3) scene reconstruction.

Original: We show that by training only on single, aligned objects and without any pre-segmentation, we can reconstruct novel scenes containing arbitrarily many objects in random poses without any performance loss.
Revised: We show that by training only on single, aligned objects and without any pre-segmentation, we can reconstruct novel scenes containing arbitrarily many objects in random poses without any performance loss.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the advent of range sensors in robotics and in medical applications, research in shape reconstruction from point clouds hasbeen seen  increasing activity (Berger et al., 2017).
Revised: - struction from point clouds has seen an increasing activity (Berger et al., 2017).

Original: The performance of classical optimization methods tends to degrade when point clouds become sparser, noisier, unoriented, or untextured.
Revised: The performance of classical optimization methods tends to degrade when point clouds become sparser, noisier, unoriented, or untextured.

Original: Deep learning methods have been proven useful to encode shape priors, and solve the reconstruction problem end to end (Riegler et al., 2017).
Revised: Deep learning methods have been proven useful in encoding shape priors, and solving the reconstruction problem end to end (Riegler et al., 2017).

Original: Many of these deep learning methods operate on meshes (Wang & Zhang, 2022;
Revised: Many of these deep learning methods operate on meshes (Wang & Zhang, 2022;

Original: Gong et al., 2019), voxels (Riegler et al., 2017), and point clouds (Qi et al., 2016).
Revised: Gong et al., 2019), voxels (Riegler et al., 2017), and point clouds (Qi et al., 2016).

Original: While voxels are easy to manipulate, shape resolution is limited by memory.
Revised: While voxels are easy to manipulate, shape resolution is limited by memory.

Original: On the other hand, meshes can guarantee watertight reconstructions, but they only handle a predefined topology.
Revised: On the other hand, meshes can guarantee watertight reconstructions, but they only handle a predefined topology.

Original: Point clouds are lightweight in terms of memory, but they discard topology.
Revised: Point clouds are lightweight in terms of memory, but they discard topology.

Original: Recently proposed deep learning methods represent the geometry via a learned occupancy map, or a signed distance function.
Revised: Recently proposed deep learning methods represent the geometry via a learned occupancy map, or a signed distance function (SDF).

Original: In particular, the seminal works of Mescheder et al.
Revised: In particular, the seminal works of Mescheder et al.

Original: ; Park et al.
Revised: ; Park et al.

Original: (2019) inspired many follow-up works (Chen & Zhang, 2019b; Genova et al., 2020; Sitzmann et al., 2019).
Revised: (2019) inspired many follow-up works (Chen & Zhang, 2019b; Genova et al., 2020; Sitzmann et al., 2019).

Original: Such representations can encode arbitrary topologies with an effectively infinite resolution.
Revised: Such representations can encode arbitrary topologies with an effectively infinite resolution.

Original: According to Kendall, “Shape is the geometry of an object modulo position, orientation, and scale” (Kendall, 1989).
Revised: According to Kendall, “Shape is the geometry of an object modulo position, orientation, and scale” (Kendall, 1989).

Original: While intensive research in the field (Niemeyer & Geiger, 2021; Peng et al., 2020;
Revised: While intensive research in the field (Niemeyer & Geiger, 2021; Peng et al., 2020;

Original: Niemeyer et al., 2020) has led to increasingly better results, very few of these methods incorporate symmetries as an inductive bias for learning.
Revised: Niemeyer et al., 2020) has led to increasingly better results, very few of these methods incorporate symmetries as an inductive bias for learning.

Original: Most translation-equivariant reconstruction methods build on the convolutional occupancy network (Peng et al., 2020), while most SO(3)-equivariant architectures (Zhu et al., 2021), and their extensions to SE(3)  (Chen et al., 2022), use the equivariant modules from Vector Neurons (Deng et al., 2021a).
Revised: Most translation-equivariant reconstruction methods build on the convolutional occupancy network (Peng et al., 2020), while most SO(3)-equivariant architectures (Zhu et al., 2021), and their extensions to SE(3) with GraphOnet (Chen et al., 2022), use the equivariant modules from Vector Neurons (Deng et al., 2021).

Original: We propose  a novel SE(3)-equivariant coordinate-based network for shape reconstruction.
Revised: We propose TF-Onet, a novel SE(3)-equivariant coordinate-based network for shape reconstruction.

Original: Motivated by the SE(3)-transformer (Fuchs et al., 2020), we design a two-level network that uses equivariant attention modules.
Revised: Motivated by the SE(3)- transformer (Fuchs et al., 2020), we design a two-level network that uses equivariant attention modules.

Original: The first level, acting as an encoder, extracts local features from the point cloud by applying self-attention in local neighborhoods around each point.
Revised: The first level, acting as an encoder, extracts local features from the point cloud by applying self-attention in local neighborhoods around each point.

Original: The second level, a cross-attention occupancy network, takes as input the extracted point features and the coordinates of a query point in space, and outputs the value of the occupancy function at the specified query point.
Revised: The second level, a cross-attention occupancy network, takes as input the extracted point features and the coordinates of a query point in space, and outputs the value of the occupancy function at the specified query point.

Original: In contrast to Deng et al. (2021b), our equivariant fully attention-based encoder captures local features for each point in space, thus facilitating local shape modeling.

Original: Even unique objectsusually consist of smaller primitive parts, whose subsets are subsequently composed to form large collections of objects.
Revised: Even unique objects consist of smaller primitive parts, whose subsets are subsequently composed to form large collections of objects.

Original: This property extends naturally to scenes that are created by a composition of objects.
Revised: This property extends naturally to scenes that are created by a composition of objects.

Original: Local shape modelling leverages this composability to generalize to unseen objects and  scenes  of arbitrarily placed objects.
Revised: Our method performs local shape modeling by leveraging the expressivity of equivariant local attention modules and generalizes to novel scenes with novel configurations of objects from classes unseen during training.

Original: Encoding local shape priors and leveraging compositionality is distinct from the typical encoder-decoder architectures that extract a global signature description of the input. In the experiments we show that our model (trained on single objects) can reconstruct scenes consisting of multiple objects (Fig. 1) without retraining, even if each object is individually rotated and translated. Moreover, our method differs from state of the art methods such as Peng et al.
Revised: This property distinguishes our method from similar equivariant works that either use global features (Deng et al., 2021) or per-point features that encode long-range dependencies by using subsampling to expand their receptive field (Chen et al., 2022). Additionally, as we describe in Section 3.3 the use of the Tensor Field framework allows our method to utilize higher order representations in contrast to the previous works which use Vector Neurons and thus are constrained to only use type-0 (scalars) and type-1 (vectors) representations.

Original: ; Tang et al. (2021), because it works directly on the irregular point cloud without any alignment via interpolation functions (Peng et al., 2020), or voxelization (Tang et al., 2021).
Revised: In Section 4, we provide experimental evidence showcasing how these differences benefit our method in the reconstruction of single objects in arbitrary poses and in the reconstruction of novel scenes.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows :

Original: • We propose  a novel SE(3)-equivariant, coordinate-based, attention network for learning occupancy fields from sparse point clouds, and use it for shape reconstruction.
Revised: • We propose TF-Onet, a novel SE(3)-equivariant, coordinate-based, attention network for learning occupancy fields from sparse point clouds and use it for surface reconstruction.

Original: • Experimentally, we outperform other equivariant coordinate-based networks (Vector Neurons) and non-equivariant networks (Occupancy Networks, Convolutional Occupancy Networks, IFNet, NeuralPull) trained with augmentations.
Revised: • Experimentally, we outperform other equivariant coordinate-based networks (Vector Neurons, GraphOnet) and non-equivariant networks (Occupancy Networks, Convolutional Occupancy Networks, IFNet, NeuralPull) trained with augmentations.

Original: • We show that equivariance and local shape modelling allows ourproposed network to output high quality reconstructions of novel scenes, while  trained only on single aligned objects. 
Revised: • The most compelling property of our method is that equivariance and local shape modeling allows our network to produce high-quality reconstructions of novel scenes while being trained only on single-aligned objects. These scenes contain an arbitrary number of objects in random poses. We show quantitative 5a and qualitative 6a performance gap over previous

Revised: methods in a synthetic dataset of randomly placed objects ( Seismic

Revised: dataset )

Original:   R ELATED W ORK
Revised: We also show qualitative results on the more challenging Matterport3D (Chang et al containing real scenes with unseen object classes.  R ELATED W ORK

Original: In this section, we discuss previous work on surface reconstruction from input point clouds.
Revised: In this section, we discuss previous work on surface reconstruction from input point clouds.

Original: We focus on methods that reconstruct the surface of an object by using a function defined in 3D space, such as the occupancy function or the signed distance function (SDF).
Revised: We focus on methods that reconstruct the surface of an object by using either an occupancy function or a SDF.

Original: For oriented point clouds (with known normal vectors), the occupancy function or the SDF can be constructed by classical methods that do not require learning (Alexa et al., 2003; Kazhdan & Hoppe, 2013).
Revised: For oriented point clouds (with known normal vectors), the occupancy function or the SDF can be constructed by classical methods that do not require learning (Alexa et al., 2003; Kazhdan & Hoppe, 2013).

Original: These methods tend to fail in the presence of noise, or when the input point cloud is sparse.
Revised: These methods tend to fail in the presence of noise, or when the input point cloud is sparse.

Original: To surpass such limitations, Mescheder et al.
Revised: To surpass such limitations, Mescheder et al.

Original: ; Chen & Zhang (2019a) proposed to learn the occupancy function for each input point cloud.
Revised: ; Chen & Zhang (2019a) proposed to learn the occupancy function for each input point cloud.

Original: Similarly, Park et al.
Revised: Similarly, Park et al.

Original: (2019) proposed to learn to infer the SDF of the object’s surface from a sparse set of SDF values.
Revised: (2019) proposed to learn to infer the SDF of the object’s surface from a sparse set of SDF values.

Original: A limitation of the above methods is that they use a global feature vector—or code—to represent the whole object (or scene), which limits their ability to generalize to novel scenes or objects.
Revised: A limitation of the above methods is that they use a global feature vector—or code—to represent the whole object (or scene), which limits their ability to generalize to novel scenes or objects.

Original: More recent methods leverage the similarities between local patches of different objects by learning either a combination of local and global features (Genova et al., 2020; Erler et al., 2020), or only local ones (Jiang et al., 2020; Chabra et al., 2020;
Revised: More recent methods leverage the similarities between local patches of different objects by learning either a combination of local and global features (Genova et al., 2020; Erler et al., 2020), or only local ones (Jiang et al., 2020; Chabra et al., 2020;

Original: Tretschk et al., 2020; Boulch & Marlet, 2022; Williams et al., 2022; Chen et al., 2022).
Revised: Tretschk et al., 2020; Boulch & Marlet, 2022; Williams et al., 2022; Chen et al., 2022).

Original: The composition of these local features allows the reconstruction of scenes containing a variety of different objects.
Revised: The composition of these local features allows the reconstruction of scenes containing a variety of different objects.

Original: To define the local neighborhoods for which the local features are extracted, Jiang et al.
Revised: To define the local neighborhoods for which the local features are extracted, Jiang et al.

Original: (2020); Chabra et al.
Revised: (2020); Chabra et al.

Original: (2020); Tretschk et al.
Revised: (2020); Tretschk et al.

Original: (2020) voxelize
Revised: (2020) voxelize

Original: [Figure]
Revised: [Figure]

Original: the space into a regular grid and learn a feature representation for each voxel, while Boulch & Marlet (2022); Chen et al. (2022) learns a local feature representation for each input point. In our work, we follow the latter approach of creating dynamic neighborhoods that depend on the input point cloud, by using local attention layers (Bahdanau et al., 2015) that dynamically change the attention weights of each point to its neighbors. Attention layers were popularized in natural language processing with the introduction of the transformer architecture (Vaswani et al., 2017), and were later applied in computer vision tasks suchas image classification (Dosovitskiy et al., 2021; Wu et al., 2021).
Revised: et al such classification (Dosovitskiy et al., 2021; Wu et al., 2021)

Original: Specifically for point cloud processing tasks, Pan et al.
Revised: Specifically for point cloud processing tasks, Pan et al.

Original: (2021) used both local and global attention for object detection, and Yu et al.
Revised: (2021) used both local and global attention for object detection, and Yu et al.

Original: (2021) used a transformer for point cloud completion.
Revised: (2021) used a transformer for point cloud completion.

Original: There is a large body of work on incorporating known symmetries into the learning process. A first major success of such methods can be traced back to Fukushima (1980); LeCun et al.
Revised: There is a large body of work on incorporating known symmetries into the learning process, tracing back to Fukushima (1980); LeCun et al.

Original: (1989) with the use of convolutional layers to build translation-equivariant networks.
Revised: (1989) with the use of convolutional layers to build translation-equivariant networks.

Original: More recent works extend this idea to discrete (Cohen & Welling, 2016) and continuous (Weiler et al., 2018b) rotationally invariant architectures.
Revised: More recent works extend this idea to discrete (Cohen & Welling, 2016) and continuous (Weiler et al., 2018b) rotationally invariant architectures.

Original: These methods have also been applied beyond Euclidean domains, to spheres (Esteves et al., 2018), graphs (Maron et al., 2019; Brandstetter et al., 2022), meshes (de Haan et al., 2021), and general manifolds (Cohen et al., 2019; Weiler et al., 2021).
Revised: These methods have also been applied beyond Euclidean domains, to spheres (Esteves et al., 2018), graphs (Maron et al., 2019; Brandstetter et al., 2022), meshes (de Haan et al., 2021), and general manifolds (Cohen et al., 2019; Weiler et al., 2021).

Original: Additionally, for transformer architectures, Fuchs et al.
Revised: For transformer architectures, Fuchs et al.

Original: (2020) propose an SE(3)-equivariant transformer by using the results of Tensor Field Networks (Thomas et al., 2018).
Revised: (2020) propose an SE(3)-equivariant transformer by using the results of Tensor Field Networks (Thomas et al., 2018).

Original: Romero & Cordonnier (2021) proposed a framework for constructing linear self-attention layers that are equivariant to arbitrary discrete groups.
Revised: Romero & Cordonnier (2021) proposed a framework for constructing linear self-attention layers that are equivariant to arbitrary discrete groups.

Original: Operating on point clouds, an SE(3)-equivariant network (Chen et al., 2021) performs pose estimation and classification using SE(3) convolutions.
Revised: Operating on point clouds, an SE(3)-equivariant network (Chen et al., 2021) performs pose estimation and classification using SE(3) convolutions.

Original: For the problem of surface reconstruction from point clouds, Convolutional Occupancy Networks (Peng et al., 2020) and later methods (Lionar et al., 2021; Tang et al., 2021; Boulch & Marlet, 2022) use convolutional layers to achieve translation-equivariance.
Revised: For the problem of surface reconstruction from point clouds, Convolutional Occupancy Networks (Peng et al., 2020) and later methods (Lionar et al., 2021; Tang et al., 2021; Boulch & Marlet, 2022) use convolutional layers to achieve translation-equivariance.

Original: Finally, Vector Neurons (Deng et al., 2021b) propose an architecture for SO(3)-equivariant reconstruction from point clouds, extended by  Chen et al.
Revised: Finally, Vector Neurons (Deng et al., 2021) propose an architecture for SO(3)-equivariant reconstruction from point clouds, extended by GraphOnet Chen et al.

Original: for the SE(3) equivariant case.
Revised: for the SE(3)-equivariant case.

Original:  M ETHOD 3. L EARNING THE O CCUPANCY F IELD
Revised:  M ETHOD 3. L EARNING THE O CCUPANCY F IELD

Original: Consider a 3D point cloud P = { ( ⃗x i , f i ) } Ni =1 , where ⃗x i ∈ R 3 denotes the spatial location of a point and
Revised: Consider a 3D point cloud P = { ( ⃗x i , f i ) } Ni =1 , where ⃗x i ∈ R 3 denotes the spatial location of a point and

Original: f i ∈ R M is an (optional) associated feature.
Revised: f i ∈ R M is an (optional) associated feature.

Original: If the features include the normal vectors, the point cloud is called oriented .
Revised: If the features include the normal vectors, the point cloud is called oriented .

Original: Otherwise it is called unoriented , which is the focus of this paper.
Revised: Otherwise it is called unoriented , which is the focus of this paper.

Original: We denote the matrix of coordinates by X = [ ⃗x 1 , · · · , ⃗x N ] ∈ R 3 × N and the matrix of features by
Revised: We denote the matrix of coordinates by X = [ ⃗x 1 , · · · , ⃗x N ] ∈ R 3 × N and the matrix of features by

Original: F = [ f 1 , · · · , f N ] ∈ R M × N .
Revised: F = [ f 1 , · · · , f N ] ∈ R M × N .

Original: Given a point cloud, our goal is to infer the shape from which it was sampled.
Revised: Given a point cloud, our goal is to infer the shape from which it was sampled.

Original: We represent this shape with an occupancy function o ∗ : R 3 → { 0 , 1 } whose 1 -level set, { ⃗x ∈ R 3 | o ∗ ( ⃗x ) = 1 } , encodes the volume occupied by the object and whose boundary encodes the surface of the object.
Revised: We represent this shape with an occupancy function o ∗ : R 3 → { 0 , 1 } whose 1 -level set, { ⃗x ∈ R 3 | o ∗ ( ⃗x ) = 1 } , encodes the volume occupied by the object and whose boundary encodes the surface of the object.

Original: We approximate this function by learning an operator T that acts on an input point cloud and produces a function ˆ o P : R 3 → [0 , 1] that describes the occupancy score ˆ o P ( ⃗q ) of each position ⃗q ∈ R 3 .
Revised: We approximate this function by learning an operator T that acts on an input point cloud and produces a function ˆ o P : R 3 → [0 , 1] that describes the occupancy score ˆ o P ( ⃗q ) of each position ⃗q ∈ R 3 .

Original: Following Mescheder et al.
Revised: Following Mescheder et al.

Original: (2019), we parametrize T by a coordinate-based neural net- work.
Revised: (2019), we parametrize T by a coordinate-based neural network.

Original: In other words, given a point cloud P =
Revised: In other words, given a point cloud P =

Original: ( X, F ) , and an arbitrary point ⃗q in space, we predict
Revised: ( X, F ) , and an arbitrary point ⃗q in space, we predict

Original: T [ X, F ]( ⃗q ) =
Revised: T [ X, F ]( ⃗q ) =

Original: ˆ o P ( ⃗q ) ∈ [0 , 1] .
Revised: ˆ o P ( ⃗q ) ∈ [0 , 1] .

Original: In contrast to Mescheder et al.
Revised: In contrast to Mescheder et al.

Original: (2019), our architecture works directly on the irregular point grid, extracting a local signature to describe the point cloud, and constraining the model to achieve SE(3)-equivariance, thus respecting the symmetries of the problem.
Revised: (2019), our architecture works directly on the irregular point grid, extracting a local signature to describe the point cloud, and constraining the model to achieve SE(3)-equivariance, thus respecting the symmetries of the problem.

Original: It is more common, especially in robotics applications, to receive featureless point clouds (for example, from Lidar sensors).
Revised: It is more common, especially in robotics applications, to receive featureless point clouds (for example, from Lidar sensors).

Original: Our paper focuses on sparse, noisy, and featureless (in particular, unoriented) point clouds.
Revised: Our paper focuses on sparse, noisy, and featureless (in particular, unoriented) point clouds.

Original: Since the performance of T depends on the expressivity of the input features F , we first design a feature extractor (or encoder)
Revised: Since the performance of T depends on the expressivity of the input features F , we first design a feature extractor (or encoder)

Original: E that produces a point cloud along with features, E
Revised: E that produces a point cloud along with features, E

Original: [ X ] =
Revised: [ X ] =

Original: ( X, F ) .
Revised: ( X, F ) .

Original: Overall, ˆ o P ( ⃗q ) =
Revised: Overall, ˆ o P ( ⃗q ) =

Original: [ T ( E ( X ; ϕ ))]
Revised: [ T ( E ( X ; ϕ ))]

Original: ( ⃗q ; θ ) .
Revised: ( ⃗q ; θ ) .

Original: Our design of T , E is founded on two basic principles: local shape modelling and equivariance .
Revised: Our design of T , E is founded on two basic principles: local shape modelling and equivariance .

Original: Objects can be seen as a composition of local parts and local surfaces.
Revised: Objects can be seen as a composition of local parts and local surfaces.

Original: At this level, even dissimilar objects may share structure.
Revised: At this level, even dissimilar objects may share structure.

Original: Local shape modelling leverages this compositionality to learn from multiple shapes in a more effective way.
Revised: Local shape modelling leverages this compositionality to learn from multiple shapes in a more effective way.

Original: This is particularly helpful for reconstructing novel object classes.
Revised: This is particularly helpful for reconstructing novel object classes.

Original: On the other hand, SE(3)-equivariance restricts models to produce consistent occupancy maps irrespective of the rigid transformations of the shape.
Revised: On the other hand, SE(3)-equivariance restricts models to produce consistent occupancy maps irrespective of the rigid transformations of the shape.

Original: This can further reduce sample complexity.
Revised: This can further reduce sample complexity.

Original: Together, these two properties create an ideal setting for scene reconstruction.
Revised: Together, these two properties create an ideal setting for scene reconstruction.

Original: Scenes are composed of objects which appear in different orientations and positions.
Revised: Scenes are composed of objects which appear in different orientations and positions.

Original: Without leveraging compositionality and equivariance, every new configuration of objects would be seen as a novel scene by the network, and thus learning would be hindered by the combinatorial explosion of these combinations.
Revised: Without leveraging compositionality and equivariance, every new configuration of objects would be seen as a novel scene by the network, and thus learning would be hindered by the combinatorial explosion of these combinations.

Original: We discuss how we impose the above properties on T , E in the next sections.
Revised: We discuss how we impose the above properties on T , E in the next sections.

Original: We note two more  properties that any processing on point clouds should take into account.
Revised: We note two more important properties of point cloud processing.

Original: First, there is no canonical ordering on the points, so a simultaneous permutation of the columns of X, F describes the same point cloud ( permutation equivariance ) and second, the number N can vary between point clouds.
Revised: First, there is no canonical ordering on the points, so a simultaneous permutation of the columns of X, F describes the same point cloud ( permutation equivariance ) and second, the number N can vary between point clouds.

Original: Thus the input has a set structure, irregularly embedded in a Euclidean domain.
Revised: Thus the input has a set structure, irregularly embedded in a Euclidean domain.

Original: Two main architectures that deal with such inputs are Point Cloud Convolutions (Wu et al., 2019) and Attention modules (Yu et al., 2021).
Revised: Two main architectures that deal with such inputs are Point Cloud Convolutions (Wu et al., 2019) and Attention modules (Yu et al., 2021).

Original: We opt for the latter in our design because point cloud convolutions, when constrained to be SE(3)-equivariant, result in very few free parameters (Thomas et al., 2018) and in particular they eliminate the angular degrees of freedom from learning.
Revised: We opt for the latter in our design because point cloud convolutions, when constrained to be SE(3)-equivariant, result in very few free parameters (Thomas et al., 2018) and in particular they eliminate the angular degrees of freedom from learning.

Original: Thus the performance is highly dependent on ad-hoc non-linearities (Poulenard & Guibas, 2021).
Revised: Thus the performance is highly dependent on ad-hoc non-linearities (Poulenard & Guibas, 2021).

Original: Attention modules on the other hand are inherently non-linear and the attention kernel can be viewed as a modulation to the angular profile of the weights, thus having more degrees of freedom (Fuchs et al., 2020).
Revised: Attention modules on the other hand are inherently non-linear and the attention kernel can be viewed as a modulation to the angular profile of the weights, thus having more degrees of freedom (Fuchs et al., 2020).

Original: In the next sections we will describe how we design attention blocks for the feature extractor

Original: E and the occupancy operator T to satisfy both locality and equivariance.

Original: 3. U SING A TTENTION FOR L OCAL S HAPE M ODELLING
Revised: 3. U SING A TTENTION FOR L OCAL S HAPE M ODELLING

Original: We propose to construct our networks E , T as a composition of equivariant attention modules.
Revised: We propose to construct our networks E , T as a composition of equivariant attention modules.

Original: In this section we discuss how we impose locality and in the next section we discuss equivariance.
Revised: Given

Original: Given N out query tokens Q i ∈ R d Q and N in key-value pairs K ∈ R d Q × N in , V ∈ R d V × N in , an attention module can be described by the formula (with subscripts indexing columns)
Revised: N out query tokens Q i ∈ R d Q and N in key-value pairs K ∈ R d Q [Equation] N in , V ∈ R d V × N in , an attention module can be described by the formula (with subscripts indexing columns)

Original: [Equation]
Revised: [Equation]

Original: For each query token Q i , the output A is a linear combination of the values V j , modulated by the similarity α ( Q i , K j ) of the query Q i to the corresponding key K j , as imposed by the attention kernel α .
Revised: For each query token Q i , the output A is a linear combination of the values V j , modulated by the similarity α ( Q i , K j ) of the query Q i to the corresponding key K j , as imposed by the attention kernel α .

Original: When all queries, keys and values are functions of the same set of feature vectors, the module is called self-attention . When the keys and values are functions of the same features, which differ from the query features, the module is called cross-attention .
Revised: In self-attention layers the queries, keys and values are functions of the same set of features, while in cross-attention the query features are distinct and only the keys and values are functions of the same features.

Original: We design the feature extractor E as a composition of L self-attention layers, and the occupancy operator T as a composition of L ′ cross-attention layers.
Revised: We design the feature extractor E as a composition of L self-attention layers, and the occupancy operator T as a composition of L ′ cross-attention layers.

Original: Self-attention feature extractor E : We associate each point  a query token with the same functional dependency
Revised: Self-attention feature extractor E : We associate each point with a query token with the same functional dependency

Original: [Equation]
Revised: [Equation]

Original: Moreover, we design the self-attention module so that not all queries have access to all the keys, but a query attends only keys depending on its position.
Revised: Moreover, we design the self-attention module so that not all queries have access to all the keys, but a query attends only keys depending on its position.

Original: In particular, if ⃗x j ∈ N ( ⃗x i ) , with N ( ⃗x i ) the[Figure] k nearest neighbors of x i , we assign to the pair ( ⃗x i , ⃗x j ) a key-value token.
Revised: In particular, if ⃗x j ∈ N ( ⃗x i ) , with N ( ⃗x i ) the k nearest neighbors of ⃗x i , we assign to the pair ( ⃗x i , ⃗x j ) a key-value token.

Original: The functional form of keys and values is
Revised: The functional form of keys and values is

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: The standard way to incorporate relative positional encoding is via concatenation  (Vaswani et al., 2017).
Revised: The standard way to incorporate relative positional encoding is via concatenation or addition (Vaswani et al., 2017).

Original: Then, W K ( f ⊕ γ ( ⃗y − ⃗x )) = W K, 1 f + W K, 2 γ ( ⃗y − ⃗x ) for some γ : R 3 → R n . Thus, the functional form W K ( ⃗y − ⃗x  ) f above is a generalization of this simple concatenation; as  the dimensionality of f is user-defined.
Revised: Our functional form W K ( ⃗x j − ⃗x i ) f can be understood as a more general way to encode the positions of the tokens.

Original: This generalization will be important in our case when we impose extra constraints to satisfy rotational equivariance.
Revised: This generalization will be important in our case when we impose extra constraints to satisfy rotational equivariance.

Original: The self-attention module of the encoder E at layer l > 1 thus is E l
Revised: The self-attention module of the encoder E at layer l ≥ 1 gets a feature-augmented point cloud ( X, F ) and produces a new feature-augmented pointcloud ( X, F ′ ) where F ′ is computed as:

Original: [Equation]
Revised: [Equation]

Original: This provides the feature output, and X passes through all layers. Since the first layer E 1 of the feature extractor does not have any features F , we design a function 
Revised: Since the input point cloud X includes only the locations of the points without additional features, we design a function ˜

Original: S that associates hand-designed (not learned) features to the points so that E 1 [ X ] =
Revised: E 0 that associates hand-designed (not learned) features to the points before passing them to the attention modules.

Original:  ˜ E 

Revised: In this work, ˜ E

Original: [ X, S ( X )] and ˜ E 1 is of the form above.
Revised: [ X ] = S ( X ) where S ( X ) i = ⃗x

Original: In this work S ( X ) i = ⃗x i − 1 / |N
Revised: i − 1 / |N

Original: ( x i ) | (cid:80) x j ∈N ( x i ) ⃗x
Revised: ( x i ) | (cid:80) x j ∈N ( x i ) ⃗x

Original: j is the relative position from the neighborhood’s centroid.
Revised: j is the relative position from the neighborhood’s centroid.

Original: Then, E = E L ◦ · · · ◦ E 1  .
Revised: Then, E = E L ◦ · · · ◦ E 1 ◦ ˜ E 0 .

Original: Cross-attention occupancy network T : Similarly, we design the occupancy operator T as a composition of L ′ cross-attention layers.
Revised: Cross-attention occupancy network T : We design the occupancy operator T as a composition of L ′ cross-attention layers.

Original: The input  to Tis not restricted to the point cloud, and can be any point ⃗q ∈ R 3 .
Revised: The input query to T can be any point ⃗q ∈ R 3 to which we assign a token

Original: This query location is associated with a query-token q 1 = ( ⃗q, f 1 q ) , before given as input to T .

Original: Here f 1 q is constructed as the position vector relative to the centroid of the neighborhood of its closest point in the point cloud i.e., f 1 q := S ′

Original:  ( X, q ) =⃗q −
Revised: Q ( ⃗q, f q ) =

Original: [Equation]
Revised: W ′ Q f q .

Original: where N X ( ⃗q ) = N (arg min ⃗x i ∈ X
Revised: The output of T is the occupancy value of that query location.

Original: ∥ ⃗q − ⃗x i ∥ 2 ) (details on ties are deferred to the Appendix).

Original: The keys and values in T are constructed from the feature-augmented point cloud learned by the feature extractor E [ X ] =
Revised: The keys and values in T are constructed from the output of the feature extractor E [ X ] =

Original: (X, F E :=
Revised: ( F E :=

Original: [ f E 1 , ..., f E N ]) .
Revised: [ f E 1 , ..., f E N ]) .

Original: In particular, if ⃗x j ∈ N X ( ⃗q ) , we create a key-value pair for ( ⃗q, x j ) with thesame functional form as before.
Revised: In particular, if ⃗x j ∈ N X ( ⃗q ) , we create a key-value pair for ( ⃗q, ⃗x j ) with the form K ( ⃗x j , ⃗q, f E j ) = W ′ K ( ⃗x j − ⃗q ) f E j and V ( ⃗x j , ⃗q, f E j ) = W ′ V ( ⃗x j − ⃗q )

Revised: f E j .

Original: At layer l > 1 , given a feature-augmented point q l := ( ⃗q, f lq ) , the cross-attention module has the form:{+T l outputs a new feature-augmented point q l +
 = ( ⃗q, f l +1 q ) . These new features are computed as:+}
Revised: Given a feature-augmented point q l := ( ⃗q, f lq ) , the cross-attention module {+T l outputs a new feature-augmented point q l +
 = ( ⃗q, f l +1 q ) . These new features are computed as:+}

Original: [Equation]
Revised: [Equation]

Original: where again we describe the output feature f lq but note that ⃗q passes through in all layers. As discussed, the first layer is T 1 [ X, F E ]( ⃗q ) = 
Revised: Since the input query includes only its location ⃗q and not additional features, first we associate it with a fixed (not learned) feature f 1 q := S ′ ( X, q ) = ⃗q − 1 |N X ( ⃗q )

Revised: | (cid:80) ⃗x

Original: ˜ T 1 [ X, F E ]( ⃗q, S ′ ( X, q )) and ˜ T 1 has the same form as in Eq. 2.
Revised: j ∈N X ( ⃗q ) ⃗x j , where N X ( ⃗q ) = N (arg min ⃗x i ∈ X ∥ ⃗q − ⃗x i ∥ 2 ) .

Original: The output ( ⃗q, f L ′ +1 q ) of T corresponds to the occupancy value at point ⃗q .
Revised: Then after passing the feature augmented point to the remaining layers of T we get the output ( ⃗q, f L ′ +1 q ) which corresponds to the occupancy value at point ⃗q .

Original: Thus the self-attention and cross-attention modules process the features on the point cloud and the 3d field respectively, without changing the topology and by using local key-tokens in the neighbourhood of every query-token.
Revised: Thus the self-attention and cross-attention modules process the features on the point cloud and the 3d field respectively, without changing the topology and by performing local attention.

Original: 3. E QUIVARIANT A TTENTION FOR S HAPE R ECONSTRUCTION
Revised: 3. E QUIVARIANT A TTENTION FOR S HAPE R ECONSTRUCTION

Original: Shape reconstruction should be independent of the coordinate system used, including of the position of the origin and the orientation of the coordinate axes.
Revised: Shape reconstruction should be independent of the coordinate system used, including of the position of the origin and the orientation of the coordinate axes.

Original: One way to capture this geometric consistency is via SE(3) -equivariance, which can be formulated in the language of group theory.
Revised: One way to capture this geometric consistency is via SE(3) -equivariance, which can be formulated in the language of group theory.

Original: In our setup, neither E nor T satisfy this property without additional constraints on ( W Q , W K , W V ) and ( W ′ Q , W ′ K , W ′ V ) .
Revised: In our setup, neither E nor T satisfy this property without additional constraints on ( W Q , W K , W V ) and ( W ′ Q , W ′ K , W ′ V ) .

Original: In this section we formulate and solve these constraints.
Revised: In this section we formulate and solve these constraints.

Original: We will assume familiarity with equivariance, and defer a more extensive discussion to the Appendix.
Revised: We will assume familiarity with equivariance, and defer a more extensive discussion to the Appendix.

Original: Equivariance Constraints:
Revised: Equivariance constraints:

Original: In our formulation, the equivariance constraint is that the occupancy field should be a type-0 (or scalar) field i.e., a simultaneous roto-translation of the point cloud and the query should result in an invariant prediction.
Revised: In our formulation, the equivariance constraint is that the occupancy field should be a type-0 (or scalar) field i.e., a simultaneous roto-translation of the point cloud and the query should result in an invariant prediction.

Original: Formally, for all ( T, R ) ∈ SE(3) , ⃗q ∈ R 3 ,
Revised: Formally, for all ( T, R ) ∈ SE(3) , ⃗q ∈ R 3 ,

Original: [Equation]
Revised: [Equation]

Original: This constraint has to be satisfied from input to output, not at every layer.
Revised: This constraint has to be satisfied from input to output, not at every layer.

Original: For intermediate layers we can relax the constraint and output more expressive vectorfields, orhigher dimensional tensor fields, with pre-specified transformation properties, so that the whole composition of layers results in a scalar field. We discuss this next.
Revised: For intermediate layers we can relax the constraint and output more expressive vector or tensor fields, with pre-specified transformation properties, so that the whole composition of layers results in a scalar field. Per-layer equivariance constraints .

Original: We can think of a feature-augmented point cloud ( X, F ) as a feature field in R 3 with finite support.
Revised: We can think of a feature-augmented point cloud ( X, F ) as a feature field in R 3 with finite support.

Original: Then, both E , T process fields in R 3 at each layer.
Revised: Then, both E , T process fields in R 3 at each layer.

Original: We need to specify how these fields transform under a roto-translation of X .
Revised: We need to specify how these fields transform under a roto-translation of X .

Original: Different transformation laws in the layers correspond to different constraints on their weights.
Revised: Different transformation laws in the layers correspond to different constraints on their weights.

Original: In this work,our transformation law has the form of the induced representation of SE(3) via SO(3) (Fulton & Harris, 2013). More concretely, each layer will be designed to satisfy, for all ( R, T ) ∈ SE (3) ,
Revised: In this work, the layers are designed to transform as,

Original: [Equation]
Revised: [Equation]

Original:  where ρ l E ( R ) , ρ l T ( R ) are SO(3)-representations i.e., square invertible matricesof the dimensionality of the features, satisfying ρ ( R 1 R 2 )
Revised: for all ( T, R ) ∈ SE (3) , where ρ l E ( R ) , ρ l T ( R ) are SO(3)-representations i.e., square invertible matrices satisfying ρ ( R 1 R 2 )

Original: =
Revised: =

Original: ρ ( R 1 ) ρ ( R 2 ) , for all R 1 , R 2 ∈ SO (3) .
Revised: ρ ( R 1 ) ρ ( R 2 ) , for all R 1 , R 2 ∈ SO (3) .

Original: In the first layers, the same constraints have to be satisfied by ˜ E 1 , ˜ T 1 , where we have already selected ρ 1 E ( R ) =
Revised: The layers ˜ E 0 , ˜ T 0 have also been designed to admit this transformation law since: [Equation] i.e [Equation]

Original:  ρ 1 T ( R ) =
Revised: , ρ [Equation]

Original: R , i.e., the inputs to the attention modules ˜ E1 , ˜ T 1 are type-1 (or vector) fields . This is due to
Revised: E

Original: S ( RX + ⊕ N T ) = 
Revised: [Equation] R ) = ρ

Original: RS ( X ) and S ′ ( RX + ⊕ N T, R⃗q + T ) =
Revised: [Equation] R ) =

Original: RS ′ ( X, ⃗q ) .
Revised: R (proof Appendix A.10).

Original: It is also clear that these per-layer constraints in Eqs.
Revised: These per-layer constraints in Eqs.

Original: 4, 5 are sufficient to solve Eq. 3, provided
Revised: 4, 5 are sufficient to solve Eq. 3, provided

Original: that ρ L ′ +1 T ( R ) =
Revised: that ρ L ′ +1 T ( R ) =

Original: I .
Revised: I .

Revised: These transformation laws have the form of the induced representation of SE(3) via SO(3)

Revised: (Fulton & Harris, 2013) (Appendix A.8.4).

Original:  Observe that the constraints above describe fields whose features stay invariant under a translation of their domains (i.e., the point cloud and R 3 ) but when this domain rotates they are multiplied by a square matrix ρ ( R ) .
Revised: Per-layer constraints on the weights : Observe that the transformation laws above describe fields whose features stay invariant under a translation of their domains (i.e., the point cloud and R 3 ) but when this domain rotates they are multiplied by a square matrix ρ ( R ) .

Original: Such fields are called ρ -fields.
Revised: Such fields are called ρ -fields.

Original: It is clear from Eq.
Revised: It is clear from Eq.

Original: that due to the use of relative positional encoding, each feature is translation invariant; and thus each field is translation equivariant.
Revised: that due to the use of relative positional encoding, each feature is translation invariant; and thus each field is translation equivariant.

Original: We focus on rotations next, discussing constraints on E , and adapting them to T .
Revised: We focus on rotations next, discussing constraints on E , and adapting them to T .

Original: We prove in the Appendix that to solve Eq.
Revised: We prove in the Appendix that to solve Eq.

Original: 4, it suffices to satisfy for all R ∈ SO (3) and ⃗x i , ⃗x j , j ̸
Revised: 4, it suffices to satisfy for all R ∈ SO (3) and ⃗x i , ⃗x j , j ̸

Original: = i : [Equation]
Revised: = i the following constraints on the weights:

Original: [Equation]
Revised: [Equation]

Original: This reduces the per-layer constraints in Eq.
Revised: This reduces the per-layer constraints in Eq.

Original: 4, 5 to constraints on the weights of each layer.
Revised: 4 to constraints on the weights of each layer.

Original: The solution indicates how we parametrize the weight matrices.
Revised: By solving these constraints, we uncover the free parameters in each layer.

Original: Before that, we introduce some useful terminology from representation theory. Feature types and irreducibles for shape reconstruction: Consider a representation ρ : SO (3) → R d × d , acting on a d -dimensional feature f . According to  the Peter-Weyl theorem (Appendix A.3.3)
Revised: Feature types and irreducibles: To solve equation 6 it is necessary to exploit the structure of the matrices ρ l E , ρ l T , which is an object of study of representation theory (Appendix A.8).

Original: , the matrices ρ ( R ) , R ∈ SO (3) , canbe simultaneously block diagonalized. Thus, ρ ( R ) = Q T [ ⊕ k D k
Revised: Specifically, according to Peter-Weyl theorem we can block diagonalize a representation ρ : SO (3) → R d × d as

Original: ( R )]
Revised: [Equation]

Original: Q where Q is a change of basis matrix and D k : SO (3) → R (2 k +1) × (2 k +1) , k ∈ N is another sub-representation.
Revised: where Q is a change of basis matrix and D k : SO (3) → R (2 k +1) × (2 k +1) , k ∈ N is the k -th irreducible representation—i.e., non-decomposable into smaller one—of SO(3).

Original: The matrices D k ( R )are the k -th Wigner-D matrices and correspond to the  k -th irreducible representation—i.e., non-decomposable into smaller ones—of SO(3).
Revised: Those D k ( R ) matrices produced by the decomposition are called k -th Wigner-D matrices.

Original: If only a single irreducible appears in the decomposition of a field, i.e., ρ = D k for some k ∈ N ,we call it  a type-k field.
Revised: If a field decomposes to a single irreducible D k , i.e., ρ = D k for some k ∈ N , it is called a type-k field.

Original: We already discussed a type-0 (or scalar) field ρ ( R ) =
Revised: We already discussed a type-0 (or scalar) field ρ ( R ) =

Original: D 0 ( R )
Revised: D 0 ( R )

Original: =
Revised: =

Original: I , the occupancy field, as well as a type-1 (or vector) field ρ ( R ) =
Revised: I , the occupancy field, as well as a type-1 (or vector) field ρ ( R ) =

Original: D 1 ( R )
Revised: D 1 ( R )

Original: =
Revised: =

Original: R on the input point cloud ( X, S ( X )) .
Revised: R on the input point cloud ( X, S ( X )) .

Original: In deep networks, the difference between stacked channels and those forming a ρfield is that the latter are equipped with transformation laws to mix channels, providing a geometric meaning to the representations. [Figure] O cc N e t V NN
Revised: In practice our feature fields f on each point are composed of multiple copies (or multiplicities ) of irreducibles of different types that form complex ρ − fields i.e., f = ⊕ k,m l f k,m l , where k indexes the type and m k its multiplicity.

Original: O u r s

Original: I npu t C o n v O n e t [Figure] Thus, our feature fields are composed of multiple copies (or multiplicities ) of different irreducible types.
Revised: Layer Parametrization :

Original: Each point in the domain has features f = ⊕ k,m l fk,m l , where k indexes the type and m k its multiplicity.
Revised: Given f l = (cid:76) k ∈K (cid:76) m k f lk,m k , and f l +1 =

Original: We select the irreducibles to describe our intuition about the geometric entities we look for. If we want to learn a feature that behaves like a 3D Euclidean vector under rotation (for example a normal vector) we need to construct a type-1 feature. If we want a feature that behaves like a symmetric matrix under rotations (such as the inertial matrix), we need a 6 -dimensional channel that comprises of one type-0 (scalar) channel and five channels that compose a type-2 (traceless symmetric matrix) feature. We incorporate such geometric entities in a distinct way from previous
Revised: (cid:76) k

Original: SO(3)-based methods in surface reconstruction, which only handle vector fields. (Deng et al., 2021b; Zhu et al., 2021; Chen et al., 2022). In particular, type-2 features can be useful in our problem to define the bending of the local surface in the receptive field of the query.
Revised: ′ ∈K ′

Original: Thus they are crucial to infer whether the query is inside the shape. Our intuitions are supported by the experimental performance gain compared to Deng et al. (2021b). Layer Parametrization : Given
Revised: (cid:76) m k ′

Original: [Equation] , we denote by W k ′ k a block of the matrix W that maps a k -type to a k ′ -type
Revised: f l +1 k ′ ,m k ′ , we denote by W k ′ k a block of the matrix W that maps a k -type to a k ′ -type

Original: (there are many such blocks depending on the multiplicities).
Revised: (there are many such blocks depending on the multiplicities).

Original: For clarity, we drop the layer index l from the matrices, since all constraints are similar.
Revised: For clarity, we drop the layer index l from the matrices, since all constraints are similar.

Original: By requiring that the attention kernel is a scalar field of the positions, we find the sufficient conditions:
Revised: By requiring that the attention kernel is a scalar field of the positions, we [Figure] VNN TF

Original: [Equation]
Revised: [Figure]

Revised: Onet

Revised: [Figure]

Revised: Input

Original: We refer to the Appendix for the details.
Revised: ConvOnet GraphOnet [Figure] find in Appendix A.10 the sufficient conditions :

Original:  The  first result follows from Schur’s Lemma.
Revised: [Equation]   (7) The solution of the first equation follows from Schur’s Lemma (Appendix A.8.1).

Original: The next two have been studied in Weiler et al.
Revised: The next two have been studied in Weiler et al.

Original: (2018a)
Revised: (2018a)

Original: ; Thomas et al.
Revised: ; Thomas et al.

Original: Finally, the inner product in the attention kernel can zero out some of the parameters in the keys:
Revised: Finally, the inner product in the attention kernel can zero out some of the parameters in the keys:

Original: [Equation]
Revised: [Equation]

Original:  where C k ′ k J (ˆ x ) =
Revised: ⃗x i ∥ ; θ K ) C k ′ k J ( ⃗x j − ⃗x i ∥ ⃗x j − ⃗x i ∥ ) , W k ′ k K = 0 , k ′ / ∈ K (8) where C k ′ k J (ˆ x ) =

Original: (cid:80)
Revised: (cid:80)

Original: Jm = − J Y
Revised: Jm = − J Y

Original: Jm (ˆ x ) Q k ′ k
Revised: Jm (ˆ x ) Q k ′ k

Original: Jm , ˆ x = ⃗x/ ∥ ⃗x ∥ and Q
Revised: Jm , ˆ x = ⃗x/ ∥ ⃗x ∥ and Q

Original: k ′ k
Revised: k ′ k

Original: Jm ∈ R (2 k ′ +1)
Revised: Jm ∈ R (2 k ′ +1)

Original: × (2 k +1) are slices from the Clebsch-Gordan matrices Q k ′ k , Y J : S 

Revised: × (2 k +1) are slices from the Clebsch-Gordan matrices Q k ′ k , Y J : S

Original: → R 2 J +1 is the j -th real-valued spherical harmonic and Y Jm (ˆ x ) =
Revised: → R 2 J +1 is the j -th real-valued spherical harmonic and Y Jm (ˆ x ) =

Original: [ Y J (ˆ x )] m is its m -th coordinate.
Revised: [ Y J (ˆ x )] m is its m -th coordinate.

Original: Also, ϕ J -s are parametrized with small MLPs.
Revised: The remaining free parameters are the, ϕ J -s which we parametrize with small MLPs.

Revised: (See Appendix A.7 for an example of these solutions)

Original: The constraints for the cross-attention module T are similar to Eq.
Revised: The constraints for the cross-attention module T are similar to Eq.

Original: However, we can  restrict to type-1 features for the keys in the first layer, i.e.,  W ijK = 0 , for ( i, j ) ̸ = (1 , 1) , without any loss.The [Figure]
Revised: However, we can further restrict to type-1 features for the keys in the first layer, i.e., [ W K ] i,j = 0 , for i ̸ = 1 , without any loss.

Original:  reason is that the input query field is of type-1.
Revised: The reason is that the input query field is of type-1.

Original: In Fig.
Revised: In Fig.

Original: 3 we visualize the equivariance constraint on T , E by using a commutative diagram.
Revised: 3 we visualize the equivariance constraint on T , E by using a commutative diagram.

Original: For additional expressivity, we perform equivariant multiheaded attention by splitting the channels, i.e., the multiplicities of the irreducible representations.
Revised: For additional expressivity, we perform equivariant multiheaded attention by splitting the channels, i.e., the multiplicities of the irreducible representations.

Original: We also use skip connections and equivariant layer normalization as in Thomas et al.
Revised: We also use skip connections and equivariant layer normalization as in Thomas et al.

Original: (2018), but defer the details to the Appendix.
Revised: (2018), but defer the details to the Appendix.

Original: See Fig.
Revised: See Fig.

Original: 2 for an overview of the method.
Revised: 2 for an overview of the method.

Revised: Irreducible types for shape reconstruction: In deep networks, the difference between stacked channels and those forming a ρ -field is that the latter are equipped with transformation laws to mix channels, providing a geometric meaning to the representations.

Revised: We select the irreducibles to describe our intuition about the geometric entities we look for.

Original: E XPERIMENTS
Revised: If we want to learn a feature that behaves like a 3D Euclidean vector under rotation (for example a normal vector) we need to construct a type-1 feature. If we want a feature that behaves like a symmetric matrix under rotations (such as the inertial matrix), we need a 6 -dimensional channel that comprises of one type-0 (scalar) channel and five channels that compose a type-2 (traceless symmetric matrix) feature. We incorporate such geometric entities in a distinct way from previous SO(3)-based methods in surface reconstruction, which only handle vector fields (Deng et al., 2021; Zhu et al., 2021; Chen et al., 2022). In particular, type-2 features can be useful in our problem to define the bending of the local surface in the receptive field of the query.

Revised: Thus they are useful to infer whether the query is inside the shape.

Revised: Our intuitions are supported by the experimental performance gain compared to Deng et al.

Revised: ; Chen et al.

Revised: (2022) and the ablation study on the types of representations used in our network, shown in Table 1.

Revised: [Figure]

Revised: [Table]

Revised:  E XPERIMENTS

Original: We perform experiments with surface reconstruction from  sparse and noisy input point clouds.
Revised: We perform experiments with surface reconstruction from unoriented sparse and noisy input point clouds.

Original: We show the importance of SE(3)-equivariance by evaluating objects in various poses and positions in space.
Revised: We show the importance of SE(3)-equivariance by evaluating objects in various poses and positions in space.

Original: Additionally, we show how local shape modeling and equivariance allows our method to train only on single aligned objects and generalize to scenes containing multiple objects in arbitrary locations and orientations.
Revised: Additionally, we show how local shape modeling and equivariance allows our method to train only on single aligned objects and generalize to scenes containing multiple objects in arbitrary locations and orientations

Original: We refer to Section A.2 of the Appendix for more details regarding the model architecture and the training/testing process.
Revised: [Table]

Original: 4. S INGLE O BJECT R ECONSTRUCTION FROM A S PARSE POINT CLOUD
Revised: 4. S INGLE O BJECT R ECONSTRUCTION FROM A S PARSE POINT CLOUD

Original: In our first experiment, we train and evaluate our model on sparse point clouds sampled from single objects in ShapeNet (Chang et al., 2015).
Revised: In our first experiment, we train and evaluate our model on sparse point clouds sampled from single objects in ShapeNet (Chang et al., 2015).

Original: For each input shape, we first uniformly sample 300 points from the ground truth mesh, and then we add normal noise with zero mean and standard deviation of 0.005.
Revised: For each input shape, we first uniformly sample 300 points from the ground truth mesh, and then we add normal noise with zero mean and standard deviation of 0.005.

Original: The network is trained with a binary cross-entropy loss between the predicted and the ground truth occupancy value at points sampled uniformly in R 3 , similarly to Mescheder et al.
Revised: We note that this experimental setting of 300 points is more challenging compared to previous works that used denser point clouds containing 3000 points.

Original: First we train and evaluate on the original objects from ShapeNet, where both the training and test data points are in their canonical position (the I/I case).
Revised: First we train and evaluate on the original objects from ShapeNet, where both the training and test data points are in their canonical position (the I/I case).

Original: Additionally, we follow Deng et al.
Revised: Additionally, we follow Deng et al.

Original: (2021b)

Original: and evaluate on test data points transformed by random SO(3) rotations.
Revised: and evaluate on test data points transformed by random SO(3) rotations.

Original: We evaluate models that were trained either on aligned training data (the I/ SO(3) case), or on data augmented by SO(3) rotations (the SO(3) / SO(3) case).
Revised: We evaluate models that were trained either on aligned training data (the I/ SO(3) case), or on data augmented by SO(3) rotations (the SO(3) / SO(3) case).

Revised: In the case of our method (TF-Onet), as shown in Table 1, we experiment with different choices for the type of the intermediate representations used by our encoder and decoder.

Original:  We compare with the Occupancy Network (OccNet) (Mescheder et al., 2019)—a non-equivariant network that extracts a global feature; with the Convolutional Occupancy Network (ConvOnet) (Peng et al., 2020) and the Implicit Feature Network (IF-Net) (Chibane et al., 2020)—two translation equivariant networks that extract local features; with Neural Pull (Baorui et al., 2021)—a method that focuses on reconstruction from dense point clouds by performing only test time optimization;and with Vector Neurons (VNN) 
Revised: By adjusting the number of channels we make sure that all of our models have the same number of learnable parameters. We compare with the Occupancy Network (OccNet) (Mescheder et al., 2019)—a non-equivariant network that extracts a global feature; with the Convolutional Occupancy Network (ConvOnet) (Peng et al., 2020) and the Implicit Feature Network (IFNet) (Chibane et al., 2020)—two translation equivariant networks that extract local features; with Neural Pull (Baorui et al., 2021)—a method that focuses on reconstruction from dense point clouds by performing only test time optimization; with Vector Neurons (VNN) (Deng et al., 2021), —a

Original: (Deng et al., 2021b)—a SO(3) equivariant network that extracts a global feature.
Revised: SO(3) equivariant network that extracts a global feature; and with GraphOnet (Chen et al., 2022), — a SE(3) equivariant network that subsamples the input point cloud to extract per point features with long range dependencies. We note that in contrast to our method both VNN and GraphOnet can only use up to type-1 equivariant features.

Original: In Table 1, we present the F-Score (Tatarchenko et al., 2019), the Chamfer-L1 distance (Fan et al., 2017) and the Intersection over Union (IoU) of the reconstruction, for models trained on the aligned and SO(3) -augmented datasets.
Revised: In Table 1, we present the F-Score (Tatarchenko et al., 2019), the Chamfer-L1 distance (Fan et al., 2017) and the Intersection over Union (IoU) of the reconstruction, for models trained on the aligned and SO(3) -augmented datasets.

Original: We refer to section A.1 of the Appendix for a more detailed description of the evaluation metrics.
Revised: We refer to section A.1 of the Appendix for a more detailed de- [Figure] scription of the evaluation metrics.

Original: Our method achieves consistent performance regardless of the rotation of the training or testing data points.
Revised: Our method achieves consistent performance regardless of the rotation of the training or testing data points.

Revised: Additionally, the best performance is achieved when we use up to type-2 representations in both the encoder and the decoder (E:0-2,D:0-2).

Original: When the testing data are transformed by random SO(3) rotations, our method consistently outperforms the compared methods, regardless of whether it is trained on rotated or aligned training data.
Revised: When the testing data are transformed by random SO(3) rotations, our method consistently outperforms the compared methods, regardless of whether it is trained on rotated or aligned training data.

Original: In figure 4a, we show qualitative comparisons between the methods.
Revised: In figure 4a, we show qualitative comparisons between the methods.

Original: Again, our model achieves consistently high quality reconstruction in all settings ( I/I , I/ SO(3) , SO(3) / SO(3) ).
Revised: Our model achieves consistently high quality reconstruction in all settings ( I/I , I/ SO(3) , SO(3) / SO(3) ).

Original: Finally, we study how the point cloud density and the added noise affects the reconstruction by training in the original setting(300 points and normal noise with 0. standard deviation) and evaluating on point clouds with different densities and levels of noise.
Revised: Finally, to study how the point cloud density and the added noise affects the reconstruction, we train in the original setting and evaluate on point clouds with different densities and levels of noise.

Original: As shown in figure 5b, our methodconsistently outperforms previous methods on sparser point clouds, and scales better to denser point clouds.[Figure] [Table] [Table]
Revised: As shown in figure 5b, our method outperforms previous methods on sparser point clouds, and scales better to denser point clouds.

Original:  S CENE R ECONSTRUCTION WITH S INGLE O BJECT T RAINING
Revised: 4. S CENE R ECONSTRUCTION WITH S INGLE O BJECT T RAINING

Original: In this section, we evaluate the ability of our model to reconstruct novel scenes with many different objects, while trained only on single objects .
Revised: In this section, we evaluate the ability of our model to reconstruct novel scenes with many different objects, while trained only on single objects .

Original: We are interested in scenes where the objects are rotated and translated.

Original: Due to SE(3)-equivariance, performance is consistent and independent of the pose and the position of the objects.
Revised: Due to SE(3)-equivariance, performance is consistent and independent of the pose and the position of the objects.

Original: Additionally, our method performs computations in local neighborhoods that usually contain points from a single object.
Revised: Additionally, our method performs computations in local neighborhoods that usually contain points from a single object.

Original: These two factors allow us to reconstruct scenes that contain objects in arbitrary poses and positions, without the need to train on similar scenes, or to segment into separate objects and reconstruct each one.
Revised: These two factors allow us to reconstruct scenes that contain objects in arbitrary poses and positions, without the need to train on similar scenes, or to segment into separate objects and reconstruct each one.

Original: We construct a dataset of synthetic rooms with multiple objects from ShapeNet in arbitrary locations and poses. This is similar to the dataset constructed in Peng et al. 
Revised: We construct a dataset of synthetic rooms with multiple objects from ShapeNet in arbitrary locations and poses, similarly to the dataset constructed in Peng et al. (2020), but

Original: but we also perform random SO(3) rotations on each object.
Revised: with the addition of random SO(3) rotations on each object.

Original: We call this dataset the Seismic dataset .
Revised: We call this dataset the Seismic dataset .

Original: For the evaluation on the Seismic dataset, we sample 300 points from each object inside a scene.

Original: In figure 5a we show a quantitative comparison between our method  and previous methods that lack either the local shape modeling or the equivariant property.
Revised: In figure 5a we show a quantitative comparison between our method (TF-Onet [E:0-2,D:0-2]) and previous methods that lack either the local shape modeling or the equivariant property.

Original: Figure 4b shows a qualitative comparison between the reconstructions achieved by these methods, while Figure 6a shows more examples of the reconstruction achieved by our model for scenes from the seismic dataset.
Revised: Figure 4b shows a qualitative comparison between the reconstructions achieved by these methods, while Figure 6a shows more examples of the reconstruction achieved by our model for scenes from the seismic dataset.

Original: Methods that perform global shape modeling (OccNet, VNN) and are trained on single objects fail to generalize to scenes containing multiple objects.
Revised: Methods that perform global shape modeling (OccNet, VNN) and are trained on single objects fail to generalize to scenes containing multiple objects.

Original: On the other hand, methods that perform local shape modeling but are not equivariant to SE(3) transforms (ConvOnet), can reconstruct objects inthe novel poses, but with reduced quality.
Revised: On the other hand, methods that perform local shape modeling but are not equivariant to SE(3) transforms (ConvOnet), can reconstruct objects in novel poses, but with reduced quality.

Revised: Finally, the SE(3) equivariant GraphOnet, that uses per-point features with long range dependencies, cannot generalize well on novel scenes when it is only trained on single objects.

Original: Our method benefits from both equivariance andthe local shape modeling, and can thus generalize to novel scenes. It reconstructs objects with a similar quality  to that on single object reconstruction.
Revised: Our method benefits from both equivariance and local shape modeling, and is able to generalize to novel scenes achieving quality similar to that on single object reconstruction.

Original: In Figure 6b, we also show examples of reconstructions of realistic scenes captured from the Matterport3D dataset (Chang et al., 2017).
Revised: In Figure 6b, we also show examples of reconstructions of realistic scenes captured from the Matterport3D dataset (Chang et al., 2017).

Original: These scenes contain between 6000 to 13000 points.
Revised: These scenes contain between 6000 to 13000 points.

Original: While our method has only been trained with aligned single objects from ShapeNet represented as point clouds of 300 points, it successfully reconstructs complicated realistic scenes containing multiple objects in arbitrary positions and poses, even from unseen classes.
Revised: While our method has only been trained with aligned single objects from ShapeNet represented as point clouds of 300 points, it successfully reconstructs complicated realistic scenes containing multiple objects in arbitrary positions and poses, even from unseen classes.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a novel SE(3) -equivariant coordinate-based model for shape reconstruction, consisting of two attention-based networks.
Revised: We proposed a novel SE(3) -equivariant coordinate-based model for shape reconstruction, consisting of two attention-based networks.

Original: By incorporating equivariance and local shape modeling, our method leverages the compositional structure of objects and scenes.
Revised: By incorporating equivariance and local shape modeling, our method leverages the compositional structure of objects and scenes.

Original: These two properties allows our model to train on single aligned objects and reconstruct novel objects and scenes.
Revised: These two properties allow our model to train on single-aligned objects and reconstruct novel objects and scenes.

Original: We evaluate our method against state of the art SO(3) -equivariant and non-equivariant methods trained with augmentations, and compare favorably in the single shape reconstruction category.
Revised: We evaluate our method against state-of-the-art SO(3) -equivariant and non-equivariant methods trained with augmentations and compare favorably in the single shape reconstruction category.

Original: Additionally, using our model trained on single aligned objects, we show that it can reconstruct novel scenes with quality similar to single object reconstruction, a task where other methods fail.
Revised: Additionally, using our model trained on single aligned objects, we show that it can reconstruct novel scenes with quality similar to single object reconstruction, a task where other methods fail.

Original: A A PPENDIX A. E VALUATION M ETRICS
Revised: A A PPENDIX A. E VALUATION M ETRICS

Original: In table 1, we present the F-Score, Chamfer-L1 distance and Intersection over Union (IoU) of the reconstruction achieved by various models.
Revised: In table 1, we present the F-Score, Chamfer-L1 distance and Intersection over Union (IoU) of the reconstruction achieved by various models.

Original: For the F-Score and Chamfer-L1 distance, we compute the reconstructed mesh using the Marching Cubes algorithm (Lorensen & Cline, 1987).
Revised: For the F-Score and Chamfer-L1 distance, we compute the reconstructed mesh using the Marching Cubes algorithm (Lorensen & Cline, 1987).

Original: Then, we uniformly sample 100,000 points from the reconstructed mesh, and compare them with the “ground truth” points sampled similarly from the “ground truth” mesh.
Revised: Then, we uniformly sample 100,000 points from the reconstructed mesh, and compare them with the “ground truth” points sampled similarly from the “ground truth” mesh.

Original: The standard deviation of the results for both metrics due to the randomness introduced by the sampling of the 100,000 points is smaller than 10 − 5 .
Revised: The standard deviation of the results for both metrics due to the randomness introduced by the sampling of the 100,000 points is smaller than 10 − 5 .

Original: As defined in Tatarchenko et al.
Revised: As defined in Tatarchenko et al.

Original: (2019), the F-Score here is the harmonic mean of the precision and the recall of the reconstruction Precision is the percentage of reconstructed points that lie within a certain distance τ of the ground truth points, and recall is the percentage of ground truth points that lie within a distance τ of the reconstructed points.
Revised: (2019), the F-Score here is the harmonic mean of the precision and the recall of the reconstruction Precision is the percentage of reconstructed points that lie within a certain distance τ of the ground truth points, and recall is the percentage of ground truth points that lie within a distance τ of the reconstructed points.

Original: We compute the F-Score with τ equal to 1% of the side length of the reconstructed volume and with τ equal to 2% of the side length of the reconstructed volume.
Revised: We compute the F-Score with τ equal to 1% of the side length of the reconstructed volume and with τ equal to 2% of the side length of the reconstructed volume.

Original: For the IoU metric, we uniformly sample 100,000 points from the whole space, and compare the predicted occupancy to the ground truth occupancy.
Revised: For the IoU metric, we uniformly sample 100,000 points from the whole space, and compare the predicted occupancy to the ground truth occupancy.

Original: The IoU is then computed by taking the ratio of points that are occupied according to both the predicted and the ground truth occupancy to the points that are occupied according to either the predicted or the ground truth occupancy.
Revised: The IoU is then computed by taking the ratio of points that are occupied according to both the predicted and the ground truth occupancy to the points that are occupied according to either the predicted or the ground truth occupancy.

Original: A. M ODEL A RCHITECTURE , T RAINING AND T ESTING D ETAILS
Revised: A. M ODEL A RCHITECTURE , T RAINING AND T ESTING D ETAILS

Original: For the self-attention feature extractor network E , we use ten multi-headed SE(3) attention layers, and for the cross-attention occupancy network T we use another two.
Revised: For the self-attention feature extractor network E , we use ten multi-headed SE(3) attention layers, and for the cross-attention occupancy network T we use another two.

Original: All multi-headed SE(3) attention layers use eight heads.
Revised: All multi-headed SE(3) attention layers use eight heads.

Original: For each point ⃗x i of the point cloud, we define the neighborhood N ( ⃗x i ) as its k nearest neighbors, where k is chosen to be 5% of the size of the point cloud (e.g., forpoints, k =15).
Revised: For each point ⃗x i of the point cloud, we define the neighborhood N ( ⃗x i ) as its k nearest neighbors, where k is chosen to be 5% of the size of the point cloud (e.g., forpoints, k =15).

Original: Each feature map in the layers of E , T uses irreducibles up to type-2.
Revised: Each feature map in the layers of E , T uses irreducibles up to type-2.

Original: We train our model on the ShapeNet (Chang et al., 2015) subset constructed in Choy et al.
Revised: We train our model on the ShapeNet (Chang et al., 2015) subset constructed in Choy et al.

Original: We use the Adam (Kingma & Ba, 2015) optimizer with learning rate that starts at 2 · 10 − 4 and linearly decreases to reach the value of 10 − 5 .
Revised: We use the Adam (Kingma & Ba, 2015) optimizer with learning rate that starts at 2 · 10 − 4 and linearly decreases to reach the value of 10 − 5 .

Original: We train for 200,000 iterations using a batch size of 64.
Revised: We train for 200,000 iterations using a batch size of 64.

Original: During training, we take as input a point cloud of 300 points and as ground truth the occupancy value of 2048 points that are sampled uniformly inside a box that bounds the object.
Revised: During training, we take as input a point cloud of 300 points and as ground truth the occupancy value of 2048 points that are sampled uniformly inside a box that bounds the object.

Original: As a training loss, we use the binary cross-entropy loss between the predicted and the ground truth occupancy of the 2048 randomly sampled points.
Revised: As a training loss, we use the binary cross-entropy loss between the predicted and the ground truth occupancy of the 2048 randomly sampled points.

Original: During inference, recalling that the output of our model for one query point is in the range of [0 , 1] , we classify a query point as occupied if the value of the learned occupancy function is above 0 .
Revised: During inference, recalling that the output of our model for one query point is in the range of [0 , 1] , we classify a query point as occupied if the value of the learned occupancy function is above 0 .

Original: After we query the occupancy of points throughout the space, we reconstruct the object’s mesh using the Marching Cubes algorithm (Lorensen & Cline, 1987).
Revised: After we query the occupancy of points throughout the space, we reconstruct the object’s mesh using the Marching Cubes algorithm (Lorensen & Cline, 1987).

Original: A. P RELIMINARIES
Revised: A. E XTRACTING D IFFERENT T YPES OF R EPRESENTATIONS AS F EATURES

Revised: An important quality of our method is its ability to use different types of representations as its intermediate features. These different types determine the way that the features transform as the input rotates. These transformation laws affect the shape properties that each type of features can encode.

Revised: For example type-0 features can encode properties of the shape that are invariant to the rotation of the shape, type-1 features can encode properties that must rotate as 3d vectors (e.g. the normal vectors at the surface) and type-2 features can encode properties that rotate as symmetric matrices (e.g. the inertial matrix of the shape).

Revised: A type l feature consists of a 2 l + 1 dimensional vector that rotates according to the corresponding

Revised: Wigner-D matrix D l : SO (3) → R (2 l +1) × (2 l +1) .

Revised: In figure 7 we visualize the different types of features extracted by our encoder for different cases of inputs.

Revised: The difference between the type of features can be clearly observed in the case of the symmetric sphere where features of higher type l correspond to higher frequencies on the sphere.

Revised: 𝑚𝑚 =𝑚𝑚 = −𝑚𝑚 =

Revised: 𝑚𝑚 =

Revised: 𝑚𝑚 = −𝑚𝑚 = −𝑚𝑚 = 2 𝑚𝑚 = 𝑚𝑚 =𝑡𝑡𝑦𝑦𝑡𝑡𝑡𝑡𝑡𝑡𝑦𝑦𝑡𝑡𝑡𝑡

Revised: [Figure]

Revised: A. R ECONSTRUCTIONS OF S CANNED S CENES

Revised: We further investigate the generalization of our method to novel scenes by reconstructing real objects of different domains and levels of clutter (Fig. 8).

Revised: The input is a sparse, unoriented point cloud scanned from scenes consisting of an arbitrary number of randomly placed objects.

Revised: The model is agnostic to the number of objects in the scene and is trained only on synthetic single objects.

Revised: The model outputs high-quality reconstructions even in difficult settings with objects very close to each other.

Revised: We observe small artifacts in very cluttered scenes, localized in regions of incidence between objects (e.g. mugs in line 4 of Fig.

Revised: A. R OBUSTNESS TO CLUTTER

Revised: We evaluate the model on the Seismic dataset (Sec. 4.2) which contains scenes with varying levels of clutter.

Revised: In Fig.

Revised: 9 we measure the Intersection over Union (IoU) of the reconstruction versus the minimum distance between any two objects in the scene (normalized with respect to the room size).

Revised: The minimum distances range from around 6% to 20% of the room.

Revised: The performance of the model is relatively stable across all settings with a maximum performance decrease of 0 .

Revised: 025 in IoU. respect to

Revised: A. L IMITATIONS AND F UTURE W ORK

Revised: A possible limitation of this work is the inherent memory overhead of the attention modules used in the network.

Revised: In our setting, this overhead is lowered due to the locality of the attention operation but it can be further minimized with additional optimization of the attention modules.

Revised: The optimization of the attention modules is an active research direction that can also benefit our method.

Revised: An additional possible direction for future work is to extend our method to the problem of scene completion from partial observations.

Revised: In this task, the model is required to hallucinate and reconstruct unobserved areas of a scene and thus requires a combination of local and global information about the scene configuration.

Revised: While a significant methodological extension is required for our method to tackle such a task we believe that the core principles of equivariance via higher-order representations and local shape modeling can provide useful tools.

Revised: A. W EIGHT P ARAMETRIZATION

Revised: Here, we present the parametrization of the weights of the queries, keys, and values that appear as the solutions of Equation 7 (and written analytically in Equation 8).

Revised: We focus on the specific case where both the input and output feature maps are ρ -fields consisting of up to type-2 irreducibles.

Revised: Then, each feature in the feature maps can be decomposed into

Revised: [Equation]

Revised: For clarity, we start with multiplicities 1 i.e., M k

Revised: = N k ′ =

Revised: 1 (and omit the index of the layer).

Revised: Then

Revised: The matrix W Q appearing in the query tokens Q ( ⃗x i , f in )

Revised: =

Revised: W Q f in has the form:

Revised: [Equation]

Revised: Similarly, we present the form of the function of matrices W K , that appear in the key tokens as

Revised: K ( ⃗x j , ⃗x i , f in ) =

Revised: W K ( ⃗x j − ⃗x i ) f in .

Revised: To simplify the notation we use ⃗x instead of ⃗x j − ⃗x i and ˆ x = ⃗x/ ∥ ⃗x ∥ .

Revised: To write W K in matrix form we first need to define:

Revised: [Equation]

Revised: The functions ϕ k ′ ,k

Revised: i : R + → R are parametrized by MLPs.

Revised: C k ′ ,k i (ˆ x ) ∈ R

Revised: [Equation]

Revised: 2 k ′ +1)

Revised: × (2 k +1) are fixed matrices defined in Section 8.

Revised: Given the definition above we can write W K as:  

Revised: [Equation]

Revised: ∥ ⃗x ∥

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised:   f 0 , 1 in f 1 , 1 in f

Revised: [Equation]

Revised: [Equation]

Revised: in  

Revised: The form of the matrices W V appearing in the value tokens as V ( ⃗x j , ⃗x i , f in ) =

Revised: W V ( ⃗x j − ⃗x i ) f in is similar to W K above.

Revised: In the general case, if the input representation contains M k copies of type-k irreducibles we need to stack the corresponding blocks of W Q , W K , M k times in the column dimension.

Revised: Similarly, if the output representation contains N ′ k copies of type- k ′ irreducibles we need to stack the corresponding blocks N k ′ times in the row dimension.

Revised: Specifically,

Revised: • for W Q and k = k ′ : (cid:2) w k,k

Revised: I 2 k +1 (cid:3) (cid:55)→ 

Revised:  w k

Revised: [Equation]

Revised: k (

Revised: [Equation]

Revised: 1) I 2 k +

Revised: [Equation]

Revised: [Equation]

Revised: k )

Revised: [Equation]

Revised: 2 k + ... ... ..

Revised: [Equation]

Revised: w k,k ( N k ′ , 1) I 2 k +

Revised: [Equation]

Revised: [Equation]

Revised: k ( N k ′ ,M k ) I 2 k +   • for W K and l = | k ′ − k

Revised: | , u = | k ′ + k | : (cid:104)

Revised: Φ k ′ ,k l − u ⊗ C k ′ ,k l − u (cid:105)

Revised: (cid:55)→ 

Revised: 

Revised: [Φ k ′ ,k l − u ] (1 , 1) ⊗ C k ′ ,k l − u · · · [Φ k ′ ,k l − u ] (1 ,M k ) ⊗ C k ′ ,k l − u ...

Revised: [Φ k ′ ,k l − u ] ( N k ′ , 1) ⊗ C k ′ ,k l − u · · · [Φ k ′ ,k l − u ] ( N k ′ ,M k )

Revised: ⊗ C k ′ ,k l − u  

Revised: A. P RELIMINARIES

Original: We recall some basic notions from group and representation theory, see e.g., Fulton & Harris (2013).
Revised: We recall some basic notions from group and representation theory, see e.g., Fulton & Harris (2013).

Original: A group ( G, · ) is a set G together with a binary operator “ · ” : G × G → G that satisfies the following axioms:
Revised: A group ( G, · ) is a set G together with a binary operator “ · ” : G × G → G that satisfies the following axioms:

Original: • Associativity: g · ( h · f ) =
Revised: • Associativity: g · ( h · f ) =

Original: ( g · h ) · f for all g, h, f ∈ G • Identity: there exists an element e ∈ G such that e · g = g · e = g • Inverse: for all g ∈ G , there exists g − 1 ∈ G such that g − 1 · g = g · g − 1 = e .
Revised: ( g · h ) · f for all g, h, f ∈ G • Identity: there exists an element e ∈ G such that e · g = g · e = g • Inverse: for all g ∈ G , there exists g − 1 ∈ G such that g − 1 · g = g · g − 1 = e .

Original: Given G , we say that each group element g ∈ G acts on the space X via an action L g :
Revised: Given G , we say that each group element g ∈ G acts on the space X via an action L g :

Original: X → X if
Revised: X → X if

Original: L g satisfy the following two properties:
Revised: L g satisfy the following two properties:

Original: • If e is the identity element of G then L e [ x ] = x for all x ∈ G ; • L g ◦ L h =
Revised: • If e is the identity element of G then L e [ x ] = x for all x ∈ G ; • L g ◦ L h =

Original: L g · h for all g, h ∈ G 
Revised: L g · h for all g, h ∈ G .

Original: If for any x, y ∈ X there exists g ∈ G such that L g [ x
Revised: If for any x, y ∈ X there exists g ∈ G such that L g [ x

Original: ] =
Revised: ] =

Original: y , then we call X a homogeneous space for the group G .
Revised: y , then we call X a homogeneous space for the group G .

Original: When X = V is a vector space, we can define the group action using a linear group representation ( V, ρ ) , where ρ : G → GL ( V ) is a map from group G to the general linear group GL ( V ) .
Revised: When X = V is a vector space, we can define the group action using a linear group representation ( V, ρ ) , where ρ : G → GL ( V ) is a map from group G to the general linear group GL ( V ) .

Original: This means that using the linear operator ρ ( g ) on V , we can define the group action L g on the vector space V as L g [ x ] = ρ ( g ) x for all x ∈ V , g ∈ G .
Revised: This means that using the linear operator ρ ( g ) on V , we can define the group action L g on the vector space V as L g [ x ] = ρ ( g ) x for all x ∈ V , g ∈ G .

Original: Then ( V, ρ ) is a linear group representation if ρ is a group homomorphism, i.e., ρ ( g · h )
Revised: Then ( V, ρ ) is a linear group representation if ρ is a group homomorphism, i.e., ρ ( g · h )

Original: =
Revised: =

Original: ρ ( g ) ρ ( h ) for all g, h ∈ G , and ρ ( e ) =
Revised: ρ ( g ) ρ ( h ) for all g, h ∈ G , and ρ ( e ) =

Original: I V is the identity operator over V .
Revised: I V is the identity operator over V .

Original: To simplify the notation, when the vector space V that the group acts on is easily inferred from the context, we will use ρ to denote the representation ( V, ρ ) .
Revised: To simplify the notation, when the vector space V that the group acts on is easily inferred from the context, we will use ρ to denote the representation ( V, ρ ) .

Original: Given a set of actions L g : X → X for g ∈ G , and a set of actions T g : Y → Y for g ∈ G , we say that a map f : X → Y is ( G, L, T ) -equivariant
Revised: Given a set of actions L g : X → X for g ∈ G , and a set of actions T g : Y → Y for g ∈ G , we say that a map f : X → Y is ( G, L, T ) -equivariant

Original: if for every g ∈ G :
Revised: if for every g ∈ G :

Original: T g [ f ( x )] = f ( L g [ x ]) for all g ∈ G, x ∈ X.
Revised: [Equation]

Original: If f is linear and equivariant (with respect to ( G, L, T ) ), then it is called an intertwiner (with respect to ( G, L, T ) ).
Revised: If f is linear and equivariant (with respect to ( G, L, T ) ), then it is called an intertwiner (with respect to ( G, L, T ) ).

Original: If there exists a subspace W ⊂ V such that for all g ∈ G and w ∈ W , we have that ρ ( g ) w ∈ W , then W is a G -invariant subspace of V , and ( W, ρ ) is a subrepresentation of ( V, ρ ) .
Revised: If there exists a subspace W ⊂ V such that for all g ∈ G and w ∈ W , we have that ρ ( g ) w ∈ W , then W is a G -invariant subspace of V , and ( W, ρ ) is a subrepresentation of ( V, ρ ) .

Original: All representations ( V, ρ ) have at least two subrepresentations: (0 , ρ ) and ( V, ρ ) .
Revised: All representations ( V, ρ ) have at least two subrepresentations: (0 , ρ ) and ( V, ρ ) .

Original: If a representation has no other subrepresentations, then it is called irreducible .
Revised: If a representation has no other subrepresentations, then it is called irreducible .

Original: Otherwise it is called reducible.
Revised: Otherwise it is called reducible.

Original: A fundamental result is the following (Fulton & Harris, 2013).
Revised: A fundamental result is the following (Fulton & Harris, 2013).

Original: A.3.
Revised: A.8. S CHUR ’ S L EMMA

Original: S CHUR ’ S L EMMA

Original: Let ( V, ρ V ) , ( W, ρ W ) be irreducible representations of G acting on V and W , respectively.
Revised: Let ( V, ρ V ) , ( W, ρ W ) be irreducible representations of G acting on V and W , respectively.

Original: • If V and W are not isomorphic, then there are no nontrivial intertwiners between them.
Revised: • If V and W are not isomorphic, then there are no nontrivial intertwiners between them.

Original: • If V = W are finite-dimensional vector spaces over C , and if ρ V =
Revised: • If V = W are finite-dimensional vector spaces over C , and if ρ V =

Original: ρ W , then all intertwiners are scalar multiples of the identity.
Revised: ρ W , then all intertwiners are scalar multiples of the identity.

Original: Now we list a number of fundamental examples, see e.g., Fulton & Harris (2013);
Revised: Now we list a number of fundamental examples, see e.g., Fulton & Harris (2013);

Original: Hall (2003).
Revised: Hall (2003).

Original: A.3. T HE TRANSLATION GROUP ( R 3 , +) :
Revised: A.8. T HE TRANSLATION GROUP ( R 3 , +) :

Original: R 3 equipped with the addition operator “ + ” is a group that is isomorphic to the group of translations in the 3D space.
Revised: R 3 equipped with the addition operator “ + ” is a group that is isomorphic to the group of translations in the 3D space.

Original: A.3. T HE SPECIAL ORTHOGONAL GROUP SO(3):
Revised: A.8. T HE SPECIAL ORTHOGONAL GROUP SO(3):

Original: SO(3) is the group of 3 × 3 orthogonal matrices with determinant +1 equipped with multiplication; and is isomorphic to the group of all 3D rotations about the origin.
Revised: SO(3) is the group of 3 × 3 orthogonal matrices with determinant +1 equipped with multiplication; and is isomorphic to the group of all 3D rotations about the origin.

Original: SO(3) is a compact group and as a result of the Peter-Weyl theorem, its linear representations can be decomposed into a direct sum of finite-dimensional, unitary, irreducible representations.
Revised: SO(3) is a compact group and as a result of the Peter-Weyl theorem, its linear representations can be decomposed into a direct sum of finite-dimensional, unitary, irreducible representations.

Original: Specifically a linear representation of SO(3) decomposes as: 
Revised: Specifically a linear representation of SO(3) decomposes as: [Equation]

Revised: for all

Original: [Equation]
Revised: [Equation]

Original: where Q is a change of basis matrix and J = 0 , 1 , . .
Revised: where Q is a change of basis matrix and J = 0 , 1 , . .

Original: D J are (2 J + 1) ×
Revised: D J are (2 J + 1) ×

Original: (2 J + 1) matrices known as the Wigner D-matrices.
Revised: (2 J + 1) matrices known as the Wigner D-matrices.

Original: The Wigner D-matrices are the irreducible representations of SO(3) .
Revised: The Wigner D-matrices are the irreducible representations of SO(3) .

Original: In the context of the features of a neural net, the representations (viewed as vectors) that transform according to D J are called type- J vectors.
Revised: In the context of the features of a neural net, the representations (viewed as vectors) that transform according to D J are called type- J vectors.

Original: A.3. T HE SPECIAL E UCLIDEAN GROUP SE(3) :
Revised: A.8. T HE SPECIAL E UCLIDEAN GROUP SE(3) :

Original: SE(3) is the group of proper rigid transformations of 3D space, and is isomorphic to the semidirect product ( R 3 , +) ⋊ SO(3) .
Revised: SE(3) is the group of proper rigid transformations of 3D space, and is isomorphic to the semidirect product ( R 3 , +) ⋊ SO(3) .

Original: An element of SE(3) can be represented as ( T, R ) where T is an element of the group of translations and R is an element of the group SO(3) of 3D rotations.
Revised: An element of SE(3) can be represented as ( T, R ) where T is an element of the group of translations and R is an element of the group SO(3) of 3D rotations.

Original: For two elements ( T 1 , R 1 ) , ( T 2 , R 2 ) ∈ SE(3) , the group law is defined as:
Revised: For two elements ( T 1 , R 1 ) , ( T 2 , R 2 ) ∈ SE(3) , the group law is defined as:

Original: [Equation]
Revised: [Equation]

Original: Since every point in R 3 can be transformed into any other point with a proper rigid transformation, we have that R 3 is a homogeneous space for SE(3) .
Revised: Since every point in R 3 can be transformed into any other point with a proper rigid transformation, we have that R 3 is a homogeneous space for SE(3) .

Original: In addition to the action of SE(3) on vectors in R 3 , we can also define the action of SE(3) on functions f : R 3 → R M , for any given integer M > 0 .
Revised: In addition to the action of SE(3) on vectors in R 3 , we can also define the action of SE(3) on functions f : R 3 → R M , for any given integer M > 0 .

Original: This action is called the induced representation π = Ind SE(3)SO(3) ρ of SE(3) .
Revised: This action is called the induced representation π = Ind SE(3)SO(3) ρ of SE(3) .

Original: It acts on f as follows:
Revised: It acts on f as follows:

Original: [Equation]
Revised: [Equation]

Original: where ( T, R ) ∈ SE(3) , x ∈ R 3 and ρ is a representation of SO (3) .
Revised: where ( T, R ) ∈ SE(3) , x ∈ R 3 and ρ is a representation of SO (3) .

Original: Especially in the context of neural nets where functions are viewed as feature fields, a function that transforms according to π = Ind SE(3)SO(3) ρ is called a ρ -field, and if ρ corresponds to the l -th irreducible representation of
Revised: Especially in the context of neural nets where functions are viewed as feature fields, a function that transforms according to π = Ind SE(3)SO(3) ρ is called a ρ -field, and if ρ corresponds to the l -th irreducible representation of

Original: SO(3) , it is also called a field of type- l .
Revised: SO(3) , it is also called a field of type- l .

Original: A. A RCHITECTURE D ETAILS
Revised: A. A RCHITECTURE D ETAILS

Original: A feature-augmented point cloud P =
Revised: A feature-augmented point cloud P =

Original: ( X, F ) where X :=
Revised: ( X, F ) where X :=

Original: [ ⃗x 1 · · · ⃗x N ] ∈ R 3 × N and F :=
Revised: [ ⃗x 1 · · · ⃗x N ] ∈ R 3 × N and F :=

Original: [ f 1 · · · f N ] ∈ R 3 × M for some M ∈ N can be associated with a 3d field f : R 3 → R M of finite support by writing f ( x )
Revised: [ f 1 · · · f N ] ∈ R 3 × M for some M ∈ N can be associated with a 3d field f : R 3 → R M of finite support by writing f ( x )

Original: = (cid:80) N i =
Revised: = (cid:80) Ni =

Original: 1 f i δ ( ⃗x − ⃗x i ) .
Revised: 1 f i δ ( ⃗x − ⃗x i ) .

Original: This association will be useful to define the group action on the feature-augmented point cloud and will unify the inputs and outputs of T , E in the sense that both of them process fields in R 3 and output fields in R 3 .
Revised: This association will be useful to define the group action on the feature-augmented point cloud and will unify the inputs and outputs of T , E in the sense that both of them process fields in R 3 and output fields in R 3 .

Original: We will interchangeably use ( X, F ) and f for the point cloud in the next sections.
Revised: We will interchangeably use ( X, F ) and f for the point cloud in the next sections.

Original: We will call f the point cloud function when the distinction is not clear.
Revised: We will call f the point cloud function when the distinction is not clear.

Original: The main module in our architecture is an SE(3) -equivariant attention block.
Revised: The main module in our architecture is an SE(3) -equivariant attention block.

Original: Each block consists of a multi-head SE(3) -equivariant attention module followed by a skip connection and an equivariant layer normalization step.
Revised: Each block consists of a multi-head SE(3) -equivariant attention module followed by a skip connection and an equivariant layer normalization step.

Original: A.4. M ULTI - HEAD SE(3) - EQUIVARIANT A TTENTION M ODULE :
Revised: A.9. M ULTI - HEAD SE(3) - EQUIVARIANT A TTENTION M ODULE :

Original: This module consists of multiple heads that implement either self-attention (input features for keys, values and queries are the same) or cross-attention (input features for keys and values are the same and different from the inputs for the queries).
Revised: This module consists of multiple heads that implement either self-attention (input features for keys, values and queries are the same) or cross-attention (input features for keys and values are the same and different from the inputs for the queries).

Original: We will first describe the self-attention variant of the module and then describe the changes that are required for the cross-attention version.
Revised: We will first describe the self-attention variant of the module and then describe the changes that are required for the cross-attention version.

Original: The self-attention SE(3) -equivariant module takes as input a function f (describing the point cloud as discussed above) defined by f ( ⃗x ) =
Revised: The self-attention SE(3) -equivariant module takes as input a function f (describing the point cloud as discussed above) defined by f ( ⃗x ) =

Original: (cid:80) Ni =
Revised: (cid:80) Ni =

Original: 1 f i δ ( ⃗x − ⃗x i ) .
Revised: 1 f i δ ( ⃗x − ⃗x i ) .

Original: Each one of the f i vectors can be decomposed into irreducible representations of SO(3) appearing with different multiplicities.
Revised: Each one of the f i vectors can be decomposed into irreducible representations of SO(3) appearing with different multiplicities.

Original: This means that a single vector f i can be decomposed as f i = (cid:76) l ≥ 

Revised: This means that a single vector f i can be decomposed as f i = (cid:76) l ≥

Original: (cid:76) m l ≥ 

Revised: (cid:76) m l ≥

Original: f i l,ml , where f i l,ml corresponds to the m l -th multiplicity of the irreducible component of type- l .
Revised: f i l,ml , where f i l,ml corresponds to the m l -th multiplicity of the irreducible component of type- l .

Original: Since we perform self-attention, the keys, values and queries are computed using the same input function f .
Revised: Since we perform self-attention, the keys, values and queries are computed using the same input function f .

Original: Specifically, for pairs of key and query points ( ⃗x j , ⃗x i ) , we compute the keys K ( ⃗x j , ⃗x i , f j ) =
Revised: Specifically, for pairs of key and query points ( ⃗x j , ⃗x i ) , we compute the keys K ( ⃗x j , ⃗x i , f j ) =

Original: W K ( ⃗x j − ⃗x i ) f j , the queries
Revised: W K ( ⃗x j − ⃗x i ) f j , the queries

Original: Q ( ⃗x i , f i ) =
Revised: Q ( ⃗x i , f i ) =

Original: W Q f i , and the values V ( ⃗x j , ⃗x i , f j ) =
Revised: W Q f i , and the values V ( ⃗x j , ⃗x i , f j ) =

Original: W V ( ⃗x j − ⃗x i ) f j .
Revised: W V ( ⃗x j − ⃗x i ) f j .

Original: To ensure equivariance, W K ,
Revised: To ensure equivariance, W K ,

Original: W Q , W V must satisfy the conditions described in Section 3.3.
Revised: W Q , W V must satisfy the conditions described in Section 3.3.

Original: Suppose that for the computed key and query features, the l -th irreducible appears with multiplicity
Revised: Suppose that for the computed key and query features, the l -th irreducible appears with multiplicity

Original: M l , and for the computed value features the k -th irreducible appears with multiplicity
Revised: M l , and for the computed value features the k -th irreducible appears with multiplicity

Original: N k .
Revised: N k .

Original: For each irreducible, we split its multiplicities across the different heads of the attention block.
Revised: For each irreducible, we split its multiplicities across the different heads of the attention block.

Original: Assuming that we have H heads in total, each head h receives keys K ( h ) ( ⃗x j , ⃗x i , f j ) and queries Q ( h ) ( ⃗x i , f i ) containing irreducibles with multiplicities M l /H and receives values V ( h ) ( ⃗x j , ⃗x i , f k ) containing irreducibles appearing with multiplicities N k /H .
Revised: Assuming that we have H heads in total, each head h receives keys K ( h ) ( ⃗x j , ⃗x i , f j ) and queries Q ( h ) ( ⃗x i , f i ) containing irreducibles with multiplicities M l /H and receives values V ( h ) ( ⃗x j , ⃗x i , f k ) containing irreducibles appearing with multiplicities N k /H .

Original: After this split, the output of the self-attention for each head can be computed as:
Revised: After this split, the output of the self-attention for each head can be computed as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Similar to the keys, values and queries, the output can have irreducibles and multiplicities that differ from the input and decompose as:
Revised: Similar to the keys, values and queries, the output can have irreducibles and multiplicities that differ from the input and decompose as:

Original: [Equation]
Revised: [Equation]

Original: where SA ( h ) [ X, f ] k,n k is the n k -th multiplicity of the k -th irreducible.
Revised: where SA ( h ) [ X, f ] k,n k is the n k -th multiplicity of the k -th irreducible.

Original: After the application of the self-attention layer, we concatenate the output of all the heads to create SA[ X, f ] = (cid:76) h SA ( h ) [ X, f ] .
Revised: After the application of the self-attention layer, we concatenate the output of all the heads to create SA[ X, f ] = (cid:76) h SA ( h ) [ X, f ] .

Original: Then we pass the concatenated output through a linear SE(3) - equivariant layer to take SA out [ X, f ] = W P SA[ X, f ] .
Revised: Then we pass the concatenated output through a linear SE(3) - equivariant layer to take SA out [ X, f ] = W P SA[ X, f ] .

Original: Since this linear layer also needs to be equivariant, it must follow the same constraints as the query matrix W Q .
Revised: Since this linear layer also needs to be equivariant, it must follow the same constraints as the query matrix W Q .

Original: By Schur’s lemma, it follows that W P can only mix feature vectors that correspond to the same irreducibles.
Revised: By Schur’s lemma, it follows that W P can only mix feature vectors that correspond to the same irreducibles.

Original: For implementing cross-attention, we use the same process as with self-attention, with the only difference that the inputs for the queries are different from the inputs for the keys and the values.
Revised: For implementing cross-attention, we use the same process as with self-attention, with the only difference that the inputs for the queries are different from the inputs for the keys and the values.

Original: As a result, the output of the cross-attention for a single head h is computed as:
Revised: As a result, the output of the cross-attention for a single head h is computed as:

Original: [Equation]
Revised: [Equation]

Original: A.4. S KIP CONNECTION :
Revised: A.9. S KIP CONNECTION :

Original: The skip connection concatenates the output features of the multi-headed SE(3) -equivariant module with the features of the input query.
Revised: The skip connection concatenates the output features of the multi-headed SE(3) -equivariant module with the features of the input query.

Original: To respect the type of each feature, this concatenation must happen between features corresponding to the same irreducible.
Revised: To respect the type of each feature, this concatenation must happen between features corresponding to the same irreducible.

Original: Suppose that f out is the output of the multi-head attention, decomposing into irreducibles and their multiplicities as f out = (cid:76)
Revised: Suppose that f out is the output of the multi-head attention, decomposing into irreducibles and their multiplicities as f out = (cid:76)

Original: k (cid:76) n k f out k,n k .
Revised: k (cid:76) n k f out k,n k .

Original: Similarly suppose that f q is the input query, decomposing into irreducibles and their multiplicities as f q = (cid:76) l (cid:76) m l f q, ( l,m l ) .
Revised: Similarly suppose that f q is the input query, decomposing into irreducibles and their multiplicities as f q = (cid:76) l (cid:76) m l f q, ( l,m l ) .

Original: The application of the skip connection gives an output
Revised: The application of the skip connection gives an output

Original: [Equation]
Revised: [Equation]

Original: To keep the output at the same types and multiplicities as specified by the user (i.e., to be a ρ outfield) we only concatenate multiplicities from the input that exist in the output.
Revised: To keep the output at the same types and multiplicities as specified by the user (i.e., to be a ρ outfield) we only concatenate multiplicities from the input that exist in the output.

Original: Also, after the skip connection we apply an equivariant linear layer between irreducibles of the same type to project the features to the correct dimensions.
Revised: Also, after the skip connection we apply an equivariant linear layer between irreducibles of the same type to project the features to the correct dimensions.

Original: A.4. E QUIVARIANT L AYER N ORM :
Revised: A.9. E QUIVARIANT L AYER N ORM :

Original: Suppose that f l,m denotes the m -th type- l irreducible of the input vector f .
Revised: Suppose that f l,m denotes the m -th type- l irreducible of the input vector f .

Original: As proposed in Fuchs et al.
Revised: As proposed in Fuchs et al.

Original: for each vector f l,m , we apply layer normalization and a nonlinearity on the norm of f l,m , leaving its direction unchanged.
Revised: for each vector f l,m , we apply layer normalization and a nonlinearity on the norm of f l,m , leaving its direction unchanged.

Original: Thus, the equivariant normalization layer can be written as: 
Revised: Thus, the equivariant normalization layer can be written as: [Equation]

Revised: l

Original: [Equation]
Revised: [Equation]

Original: where LayerNorm corresponds to layer normalization, proposed in Ba et al.
Revised: where LayerNorm corresponds to layer normalization, proposed in Ba et al.

Original: We use the SE(3) -equivariant attention block to construct both the network E that assigns to each point in the point cloud a learned feature, and the network T , that outputs the occupancy value of a query point in space.
Revised: We use the SE(3) -equivariant attention block to construct both the network E that assigns to each point in the point cloud a learned feature, and the network T , that outputs the occupancy value of a query point in space.

Original: The network E consists of ten SE(3) -equivariant self-attention blocks, and T consists of two SE(3) -equivariant cross-attention blocks.
Revised: The network E consists of ten SE(3) -equivariant self-attention blocks, and T consists of two SE(3) -equivariant cross-attention blocks.

Original: Figure 2 shows a diagram of this architecture.
Revised: Figure 2 shows a diagram of this architecture.

Original: In both networks, we use blocks with eight heads and intermediate representations that contain features up to type-2.
Revised: In both networks, we use blocks with eight heads and intermediate representations that contain features up to type-2.

Original: Additionally, for each type we set the multiplicity of the corresponding irreducible to 32.
Revised: Additionally, for each type we set the multiplicity of the corresponding irreducible to 32.

Original: Finally, for the computation of the local neighborhood N ( ⃗x ) around each point ⃗x , we use the k nearest neighbors, where k is chosen to be 5% of the size of the point cloud (e.g., forpoints, k =15).
Revised: Finally, for the computation of the local neighborhood N ( ⃗x ) around each point ⃗x , we use the k nearest neighbors, where k is chosen to be 5% of the size of the point cloud (e.g., forpoints, k =15).

Original: Although it is possible for T to output a single scalar that corresponds to the occupancy value at a queried point, we observe in the experiments an increase in performance when T outputs 32 scalar values that we then pass through a simple MLP to get the final occupancy value.
Revised: Although it is possible for T to output a single scalar that corresponds to the occupancy value at a queried point, we observe in the experiments an increase in performance when T outputs 32 scalar values that we then pass through a simple MLP to get the final occupancy value

Original:  A. P ROOFS ON E QUIVARIANCE FROM S EC .3. 
Revised: . A. P ROOFS ON E QUIVARIANCE FROM S EC .3.

Original: Input-output equivariance(Eq. 3):
Revised: Input-output equivariance

Original:  We will formalize the equivariance constraints in the language of group theory.
Revised: (Eq. 3): We will formalize the equivariance constraints in the language of group theory.

Original: We have a map ˆ o that takes as input a point cloud X ∈ R 3 × N and outputs an occupancy field
Revised: We have a map ˆ o that takes as input a point cloud X ∈ R 3 × N and outputs an occupancy field

Original: ˆ o ( X ) : R 3 → R .
Revised: ˆ o ( X ) : R 3 → R .

Original: First we need to define the actions of SE(3) on the input point cloud and the occupancy map which we denote by L in , L out respectively.
Revised: First we need to define the actions of SE(3) on the input point cloud and the occupancy map which we denote by L in , L out respectively.

Original: Those have the following form for ( T, R ) ∈ SE (3) :
Revised: Those have the following form for ( T, R ) ∈ SE (3) :

Original: [Equation]
Revised: [Equation]

Original: where ⊕ N on vectors denotes concatenation column-wise.
Revised: where ⊕ N on vectors denotes concatenation column-wise.

Original: The first equation describes N standard representations of SE(3) and the second the induced representation of SE(3) via SO(3) with ρ ( R ) =
Revised: The first equation describes N standard representations of SE(3) and the second the induced representation of SE(3) via SO(3) with ρ ( R ) =

Original: I , i.e., the map ˆ o is a scalar (or type-0) field.
Revised: I , i.e., the map ˆ o is a scalar (or type-0) field.

Original: For completeness we show that L in indeed describes an action of SE(3) on X .
Revised: For completeness we show that L in indeed describes an action of SE(3) on X .

Original: Letting ( I, ⊕ N 0) be the identity element of SE(3), we can check that
Revised: Letting ( I, ⊕ N 0) be the identity element of SE(3), we can check that

Original: [Equation]
Revised: [Equation]

Original: Also, for any ( T 1 , R 1 ) , ( T 2 , R 2 ) from SE(3), we can check that
Revised: Also, for any ( T 1 , R 1 ) , ( T 2 , R 2 ) from SE(3), we can check that

Original: [Equation]
Revised: [Equation]

Original: Using the vector space isomorphism associating X ∈ R 3 × N with the unrolled vector vec( X ) ∈ R 3 N , we can view this action as the direct sum of N standard representations.
Revised: Using the vector space isomorphism associating X ∈ R 3 × N with the unrolled vector vec( X ) ∈ R 3 N , we can view this action as the direct sum of N standard representations.

Original: Moreover, L out is also an action, and in fact an induced representation of SE(3) via SO(3) with ρ ( R )
Revised: Moreover, L out is also an action, and in fact an induced representation of SE(3) via SO(3) with ρ ( R )

Original: =
Revised: =

Original: I, R ∈ SO (3) , according to the definition A.3.4.
Revised: I, R ∈ SO (3) , according to the definition A.8.4.

Original: Now that we have the input and output actions, we can define the equivariance constraint for our problem.
Revised: Now that we have the input and output actions, we can define the equivariance constraint for our problem.

Original: Informally, a simultaneous roto-translation of the point cloud and the query results in an invariant prediction.
Revised: Informally, a simultaneous roto-translation of the point cloud and the query results in an invariant prediction.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: Now we recall that we have parametrized ˆ o as a composition of a feature extractor E and an occupancy network T , i.e., [ˆ o ( X )]( ⃗q ) =
Revised: Now we recall that we have parametrized ˆ o as a composition of a feature extractor E and an occupancy network T , i.e., [ˆ o ( X )]( ⃗q ) =

Original: [ T ( E ( X ))]( ⃗q ) .
Revised: [ T ( E ( X ))]( ⃗q ) .

Original: Thus we arrive to Eq.
Revised: Thus we arrive to Eq.

Original: [Equation]
Revised: [Equation]

Original: Per-layer equivariance (Eq. 4, 5):
Revised: Per-layer equivariance (Eq. 4, 5):

Original: Next, we need to prove that the per-layer equivariance constraints in Eq.
Revised: Next, we need to prove that the per-layer equivariance constraints in Eq.

Original: 4, 5 are sufficient to satisfy the input-output equivariance constraint from Eq.
Revised: 4, 5 are sufficient to satisfy the input-output equivariance constraint from Eq.

Original: 3, provided that ρ L ′ +1 T [Equation]
Revised: 3, provided that ρ L ′ +1 T ( R ) =

Original: = I, R [Equation] SO 
Revised: I, R ∈ SO (3) .

Original: [Equation]

Original: We recall that the forms of the feature extractor E and the occupancy network T are:
Revised: We recall that the forms of the feature extractor E and the occupancy network T are:

Original: [Equation]
Revised: [Equation]

Original: All˜ E 1 , E 2 , · · · , E L take as input and produce as output a feature field on the point cloud and satisfy the constraints in Eq. 4.
Revised: All E 1 , E 2 , · · · , E L take as input and produce as output a feature field on the point cloud and satisfy the constraints in Eq. 4.

Original: Similarly, all˜ T 1 , T 2 , · · · , T L ′ take as input a feature field on the point cloud and a feature field in R 3 and pass the point cloud feature field unaltered to the next layer.
Revised: Similarly, all T 1 , T 2 , · · · , T L ′ take as input a feature field on the point cloud and a feature field in R 3 and pass the point cloud feature field unaltered to the next layer.

Original: They transform the feature field in R 3 and satisfy the constraints in Eq. 5.
Revised: They transform the feature field in R 3 and satisfy the constraints in Eq. 5.

Original: Thus, by composing these constraints in Eq.
Revised: Thus, by composing these constraints in Eq.

Original: 4.5 we immediately find:
Revised: 4.5 we immediately find:

Original: [Equation]
Revised: [Equation]

Original: where ρ L ′ +1 T ( R )
Revised: where ρ L ′ +1 T ( R )

Original: =
Revised: =

Original: I as discussed.
Revised: I as discussed.

Original: The equations above show that if every layer transforms an input ρ -field to an output ρ -field, then the composition also transforms an input ρ -field to an output ρ -field.
Revised: The equations above show that if every layer transforms an input ρ -field to an output ρ -field, then the composition also transforms an input ρ -field to an output ρ -field.

Original: Thus, it remains to show that the inputs in ˜ E 1 , ˜ T 1 , namely ( X, S ( X )) and ( ⃗q, S ′ ( X, ⃗q )) respectively are ρ -fields as well.
Revised: Thus, it remains to show that the fixed layers ˜ E 0 , ˜ T 0 , namely ( X, S ( X )) and ( ⃗q, S ′ ( X, ⃗q )) respectively are ρ -fields as well.

Original: Then, the equations in Eq.
Revised: Then, the equations in Eq.

Original: 18 would not only apply from the first layer to the output, but from the input to the output.
Revised: 20 would not only apply from the first layer to the output, but from the input to the output.

Original: In other words, we need to check that the features produced by S, S ′ do not translate when the point cloud translates but they do rotate (under suitable representations) when the point cloud rotates.
Revised: In other words, we need to check that the features produced by S, S ′ do not translate when the point cloud translates but they do rotate (under suitable representations) when the point cloud rotates.

Original: Input point cloud field and input query field are type-1 fields : For the i -th point at position ⃗x i , we construct a feature f i as input to the first self-attention layer of E .
Revised: Input point cloud field and input query field are type-1 fields : For the i -th point at position ⃗x i , we construct a feature f i as input to the first self-attention layer of E .

Original: We select the feature f i as the relative position of the i -th point, ⃗x i , to the centroid of a neighborhood constructed from its k nearest neighbors in the point cloud in Euclidean norm, i.e.,
Revised: We select the feature f i as the relative position of the i -th point, ⃗x i , to the centroid of a neighborhood constructed from its k nearest neighbors in the point cloud in Euclidean norm, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Revised: N

Original: [Equation]
Revised: [Equation]

Original: is the neighborhood of the i -th point in the point cloud.
Revised: is the neighborhood of the i -th point in the point cloud.

Original: Also, ( ⃗x ( i ) j ) j ∈ [ N ] is a sorting of the points in the point cloud in increasing Euclidean distance to the i -th point, i.e, ∥ ⃗x i − ⃗x ( i ) 1 ∥ 2 ≤ · · · ≤ ∥ ⃗x i − ⃗x ( i ) N ∥ 2 .
Revised: Also, ( ⃗x ( i ) j ) j ∈ [ N ] is a sorting of the points in the point cloud in increasing Euclidean distance to the i -th point, i.e, ∥ ⃗x i − ⃗x ( i ) 1 ∥ 2 ≤ · · · ≤ ∥ ⃗x i − ⃗x ( i ) N ∥ 2 .

Original: In case of ties, we assign numbers randomly.
Revised: In case of ties, we assign numbers randomly.

Original: Due to the use of “ ≤ ” (instead of the strict inequality symbol “ < ” ), in the definition of the neighborhoods, if there are ties for ⃗x ( i ) k , we include all tied points in the neighborhood.
Revised: Due to the use of “ ≤ ” (instead of the strict inequality symbol “ < ” ), in the definition of the neighborhoods, if there are ties for ⃗x ( i ) k , we include all tied points in the neighborhood.

Original: Now we study how the features f i transform when the points in the point cloud transform via
Revised: Now we study how the features f i transform when the points in the point cloud transform via

Original: L in , ( T,R ) , discussed in the first step.
Revised: L in , ( T,R ) , discussed in the first step.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we find:
Revised: we find:

Original: [Equation]
Revised: [Equation]

Original: where we used the claim, proved below, that
Revised: where we used the claim, proved below, that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Thus, S ( L in , ( T,R )
Revised: Thus, S ( L in , ( T,R )

Original: [ X ]) i further equals: 
Revised: [ X ]) i further equals: [Equation]

Revised: R

Original: [Equation]
Revised: [Equation]

Revised: ∈

Revised: [Equation]

Revised: [Equation]

Original: We now prove 19.
Revised: We now prove 21.

Original: When all ⃗x i are mapped as ⃗x i L in
Revised: When all ⃗x i are mapped as ⃗x i L in

Original: , ( T,R ) −−−−−−→
Revised: , ( T,R ) −−−−−−→

Original: R⃗x i + T , the Euclidean distance between any two points is preserved, i.e., ∥ ( R⃗x i + T ) − ( R⃗x j + T ) ∥
Revised: R⃗x i + T , the Euclidean distance between any two points is preserved, i.e., ∥ ( R⃗x i + T ) − ( R⃗x j + T ) ∥

Original: 2 = ∥ ⃗x i − ⃗x j ∥ 2 .
Revised: 2 = ∥ ⃗x i − ⃗x j ∥ 2 .

Original: Thus, if before the transformation ∥ ⃗x i − ⃗x ( i ) k
Revised: Thus, if before the transformation ∥ ⃗x i − ⃗x ( i ) k

Original: ∥ 2 = d k , then after the transformation we also have ∥L in , ( T,R )
Revised: ∥ 2 = d k , then after the transformation we also have ∥L in , ( T,R )

Original: [ X ] i −
Revised: [ X ] i −

Original: L in , ( T,R )
Revised: L in , ( T,R )

Original: [ X ] ( i ) k
Revised: [ X ] ( i ) k

Original: ∥ 2 = d k .
Revised: ∥ 2 = d k .

Original: This is because all nearest neighbors preserve their distances, and thus sorting returns the same indices up to random tie breaking.
Revised: This is because all nearest neighbors preserve their distances, and thus sorting returns the same indices up to random tie breaking.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: The first and fourth equivalence hold because we include all tied neighbors in the neighborhood.
Revised: The first and fourth equivalence hold because we include all tied neighbors in the neighborhood.

Original: Thus, the neighborhood is defined by the distance d k , and not by the identity of the k -neighbors.
Revised: Thus, the neighborhood is defined by the distance d k , and not by the identity of the k -neighbors.

Original: Thus f i = S
Revised: Thus f i = S

Original: ( X ) i transforms according to the standard representation of SO(3) when each point in the point cloud transforms according to the standard representation of SE(3) .
Revised: ( X ) i transforms according to the standard representation of SO(3) when each point in the point cloud transforms according to the standard representation of SE(3) .

Original: If we view the features as a function on the point cloud extended to the homogeneous space R 3 ∼ = SE(3) / SO(3) , i.e., f ( x ) = (cid:80) N i =1 f i δ ( x − x i ) , then this function transforms according to the induced representation as:
Revised: If we view the features as a function on the point cloud extended to the homogeneous space R 3 ∼ = SE(3) / SO(3) , i.e., f ( x ) = (cid:80) Ni =1 f i δ ( x − x i ) , then this function transforms according to the induced representation as:

Original: [Equation]
Revised: [Equation]

Original: Recall that functions transforming according to the above law are called type-1 fields.
Revised: Recall that functions transforming according to the above law are called type-1 fields.

Original: We will keep the name, but use a matrix notation instead of the Dirac notation.
Revised: We will keep the name, but use a matrix notation instead of the Dirac notation.

Original: By concatenating the features column-wise, we find the map, described in the main text as S , i.e.,
Revised: By concatenating the features column-wise, we find the map, described in the main text as S , i.e.,

Original: S 
Revised: S ( RX + ⊕ N T ) = ⊕ Ni =1 Rf i = RS

Original: [Equation]
Revised: ( X ) .

Original: Now we turn to the second network, T .
Revised: Now we turn to the second network, T .

Original: For each point ⃗q ∈ R 3 whose occupancy value we wish to find, we first construct a feature f 1 q := S ′
Revised: For each point ⃗q ∈ R 3 whose occupancy value we wish to find, we first construct a feature f 1 q := S ′

Original: ( X, ⃗q ) and then use the pair ( ⃗q, f 1 q ) as the query to the first cross-attention layer of T .
Revised: ( X, ⃗q ) and then use the pair ( ⃗q, f 1 q ) as the query to the first cross-attention layer of T .

Original: We will show that, when the query ⃗q and the point cloud X transform according to the standard representation of SE(3) , this input feature
Revised: We will show that, when the query ⃗q and the point cloud X transform according to the standard representation of SE(3) , this input feature

Original: f 1 q also transforms according to the standard representation of SO(3) .
Revised: f 1 q also transforms according to the standard representation of SO(3) .

Original: We again construct the feature f 1 q as the relative position between ⃗q and the centroid of its neighborhood N QX ( ⃗q ) .
Revised: We again construct the feature f 1 q as the relative position between ⃗q and the centroid of its neighborhood N QX ( ⃗q ) .

Original: We consider N QX ( ⃗q ) to be the same as the neighborhood of its closest—in Euclidean distance—point in the point cloud, and write (if the closest point is defined uniquely)
Revised: We consider N QX ( ⃗q ) to be the same as the neighborhood of its closest—in Euclidean distance—point in the point cloud, and write (if the closest point is defined uniquely)

Original: [Equation]
Revised: [Equation]

Original: We discuss at the end of this step the case where the nearest neighbors are tied.
Revised: We discuss at the end of this step the case where the nearest neighbors are tied.

Original: At the moment, let the unique closest point be c = arg min i ∈
Revised: At the moment, let the unique closest point be c = arg min i ∈

Original: [ N ] ( ∥ ⃗q − ⃗x i ∥ 2 ) .
Revised: [ N ] ( ∥ ⃗q − ⃗x i ∥ 2 ) .

Original: Then, the query feature becomes:
Revised: Then, the query feature becomes:

Original: [Equation]
Revised: [Equation]

Original: The proof that
Revised: The proof that

Original: [Equation]
Revised: [Equation]

Original: is the same as the one for S before.
Revised: is the same as the one for S before.

Original: Viewed again as a function defined on R 3 , the map ⃗q (cid:55)→ S ′
Revised: Viewed again as a function defined on R 3 , the map ⃗q (cid:55)→ S ′

Original: ( X, ⃗q ) constitutes a type-1 field.
Revised: ( X, ⃗q ) constitutes a type-1 field.

Original: The transformation law of this field is depicted in Fig.
Revised: The transformation law of this field is depicted in Fig.

Revised: Remark: When there are ties, i.e., the set arg min i ∈

Revised: [ N ] ( ∥ ⃗q − ⃗x i ∥ 2 ) contains more than one point, we form all neighborhoods

Original: [Equation]
Revised: [Equation]

Original: Then, we consider a query token for each pair ( ⃗q c , f cq ) , where ⃗q c = ⃗q and f cq are constructed as in 20, using the neighborhood N kc ( X ) ∈ N QX ( ⃗q ) .
Revised: Then, we consider a query token for each pair ( ⃗q c , f cq ) , where ⃗q c = ⃗q and f cq are constructed as in 22, using the neighborhood N kc ( X ) ∈ N QX ( ⃗q ) .

Original: If f q transforms as a type-1 field, then, after the roto-translation of the point cloud and the query, we could identify the same c as a closest neighbor.
Revised: If f q transforms as a type-1 field, then, after the roto-translation of the point cloud and the query, we could identify the same c as a closest neighbor.

Original: However, since those points are equivalent as nearest neighbors, we will process the whole set of fields independently, producing a set of fields in the output.
Revised: However, since those points are equivalent as nearest neighbors, we will process the whole set of fields independently, producing a set of fields in the output.

Original: A different order of selection of nearest neighbors after the roto-translation corresponds to a permutation of the set of the output fields.
Revised: A different order of selection of nearest neighbors after the roto-translation corresponds to a permutation of the set of the output fields.

Original: Since attention modules are permutation equivariant, this permutation propagates to the output.
Revised: Since attention modules are permutation equivariant, this permutation propagates to the output.

Original: We only need to discuss how to combine these outputs on the tokens ⃗q c that correspond to the same position ⃗q in the occupancy field.
Revised: We only need to discuss how to combine these outputs on the tokens ⃗q c that correspond to the same position ⃗q in the occupancy field.

Original: Since the output is a scalar field (as we prove next), we can take the maximum across the same channels to construct a new scalar field that is also invariant to any permutation.
Revised: Since the output is a scalar field (as we prove next), we can take the maximum across the same channels to construct a new scalar field that is also invariant to any permutation.

Original: The idea is that taking the maximum corresponds to an “OR” operation, since both the usual non-linearities (such as the sigmoid) and the thresholding operations that follow are increasing functions of their inputs.
Revised: The idea is that taking the maximum corresponds to an “OR” operation, since both the usual non-linearities (such as the sigmoid) and the thresholding operations that follow are increasing functions of their inputs.

Original: Thus, when the network predicts that the position of the query is “occupied”, even based on one neighborhood, the position is likely to be considered occupied.
Revised: Thus, when the network predicts that the position of the query is “occupied”, even based on one neighborhood, the position is likely to be considered occupied.

Original: From per-layer constraints 4, 5 to constraints on the weights 6: We focus now on each layer of E , T separately.
Revised: From per-layer constraints 4, 5 to constraints on the weights 6: We focus now on each layer of E , T separately.

Original: Now we need to prove that Eq.
Revised: Now we need to prove that Eq.

Original: 6 provide sufficient conditions on the weights to satisfy the per-layer constraints 4, 5.
Revised: 6 provide sufficient conditions on the weights to satisfy the per-layer constraints 4, 5.

Original: Every layer is composed of a multi-headed attention layer, a skip connection and a normalization layer.
Revised: Every layer is composed of a multi-headed attention layer, a skip connection and a normalization layer.

Original: We prove the result for the self-attention and cross-attention layers (focusing on one attention head for simplicity).
Revised: We prove the result for the self-attention and cross-attention layers (focusing on one attention head for simplicity).

Original: It will be clear from the proof that a concatenation of the heads and a subsequent linear transformation mixing channels that correspond to the same irreducible preserves equivariance, as well as a skip connection that concatenates irreducibles of the same type.
Revised: It will be clear from the proof that a concatenation of the heads and a subsequent linear transformation mixing channels that correspond to the same irreducible preserves equivariance, as well as a skip connection that concatenates irreducibles of the same type.

Original: We start with the self-attention layers in E described in A.4.1.
Revised: We start with the self-attention layers in E described in A.9.1.

Original: We drop the layer index l for clarity and denote the actions ρ l E , ρ l +1 E by ρ in , ρ out .
Revised: We drop the layer index l for clarity and denote the actions ρ l E , ρ l +1 E by ρ in , ρ out .

Original: At the i -th token.
Revised: At the i -th token.

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the attention kernel α X takes the form:
Revised: where the attention kernel α X takes the form:

Original: [Equation]
Revised: [Equation]

Original: Then, we are given
Revised: Then [Equation] we are given

Original: [Equation]
Revised: [Equation]

Original: and we need to prove:
Revised: and we need to prove:

Original: [Equation]
Revised: [Equation]

Original: For the attention layer, we have for X = ⊕ Ni =1 [ ⃗x i ] and for each output tokeni ∈
Revised: For the attention layer [Equation] we have for X = ⊕ Ni =1 [ ⃗x i ] and for each output token

Revised: i ∈

Original: [ N ] :
Revised: [ N ] :

Original: [Equation]
Revised: [Equation]

Original: where we used 19 for the invariant neighborhoods, N ki ( RX + ⊕ N T )
Revised: where we used 21 for the invariant neighborhoods, N ki ( RX + ⊕ N T )

Original: =
Revised: =

Original: N ki
Revised: N ki

Original: ( X ) .
Revised: ( X ) .

Original: Now using the constraints for the matrices 21, we find for each term j ∈ N ki
Revised: Now using the constraints for the matrices 23, we find for each term j ∈ N ki

Original: ( X ) that the individual terms in the sum above equal:
Revised: ( X ) that the individual terms in the sum above equal:

Original: [Equation]
Revised: [Equation]

Original: ) where in the second equation we used that the attention kernel gives a scalar output.
Revised: where in the second equation we used that the attention kernel gives a scalar output.

Original: Now, the attention kernel from the last equation transforms as
Revised: Now, the attention kernel from the last equation transforms as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we used again the properties of the invariant neighborhoods and that ρ out is a unitary representation, i [Equation] e., ρ out ( R ) T ρ out ( R )
Revised: where we used again the properties of the invariant neighborhoods and that ρ out is a unitary representation, i.e., ρ out ( R ) T ρ out ( R )

Original: =
Revised: =

Original: I for all R ∈ SO(3) .
Revised: I for all R ∈ SO(3) .

Original: Using 23, 24, and 22, we find:
Revised: Using 25, 26, and 24, we find:

Original: [Equation]
Revised: [Equation]

Original: Thus each self-attention layer in E is equivariant.
Revised: Thus each self-attention layer in E is equivariant.

Original: The proof for each cross-attention layer in T is similar.
Revised: The proof for each cross-attention layer in T is similar.

Original: Again, note that if the closest neighbor of the query is not uniquely defined, then—as discussed above—we output the entire set of fields for every equivalent neighborhood.
Revised: Again, note that if the closest neighbor of the query is not uniquely defined, then—as discussed above—we output the entire set of fields for every equivalent neighborhood.

Original: Then, the output is also a set of fields and a roto-translation of the input will only result in a permutation of these fields.
Revised: Then, the output is also a set of fields and a roto-translation of the input will only result in a permutation of these fields.

Original: Then, the “max” operation in the output will eliminate the permutation, making the output equivariant.
Revised: Then, the “max” operation in the output will eliminate the permutation, making the output equivariant.

Original: From the constraints on the weights (Eq. 6) to their solutions (Eq. 7) :
Revised: From the constraints on the weights (Eq. 6) to their solutions (Eq. 7) :

Original: Again, we focus on each layer separately as we did in the previous section.
Revised: Again, we focus on each layer separately as we did in the previous section.

Original: The goal is to solve Eq.
Revised: The goal is to solve Eq.

Original: By the Peter-Weyl theorem, ρ in decomposes into unitary, irreducible representations of SO(3) , possibly with multiplicities.
Revised: By the Peter-Weyl theorem, ρ in decomposes into unitary, irreducible representations of SO(3) , possibly with multiplicities.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: where in our case Q in =
Revised: where in our case Q in =

Original: I , (cid:76) for matrices denotes a concatenation along the diagonal and l indexes the irreducible types and m l indexes the multiplicity of the l -th irreducible.
Revised: I , (cid:76) for matrices denotes a concatenation along the diagonal and l indexes the irreducible types and m l indexes the multiplicity of the l -th irreducible.

Original: Then, for each ⃗x in the point cloud (where we suppress the index i for clarity), we have f in = (cid:76) l (cid:76) m l f in l,m l .
Revised: Then, for each ⃗x in the point cloud (where we suppress the index i for clarity), we have f in = (cid:76) l (cid:76) m l f in l,m l .

Original: We also consider an output field transforming according to ρ out ( R ) =
Revised: We also consider an output field transforming according to ρ out ( R ) =

Original: Q T out ( (cid:76) k
Revised: Q T out ( (cid:76) k

Original: (cid:76) n k D k
Revised: (cid:76) n k D k

Original: ( R ))
Revised: ( R ))

Original: Q out —
Revised: Q out —

Original: Q out
Revised: Q out

Original: = I in our case—that decomposes as f out = (cid:76)
Revised: = I in our case—that decomposes as f out = (cid:76)

Original: k
Revised: k

Original: (cid:76) n k f out k,n k .
Revised: (cid:76) n k f out k,n k .

Original: Recall that each of the matrices
Revised: Recall that each of the matrices

Original: W Q , W K , W V are of dimension (cid:80) k n k (2 k + 1) ×
Revised: W Q , W K , W V are of dimension (cid:80) k n k (2 k + 1) ×

Original: (cid:80) l m l (2 l + 1) .
Revised: (cid:80) l m l (2 l + 1) .

Original: For W Q , we have for all R ∈ SO(3) :
Revised: For W Q , we have for all R ∈ SO [Equation]

Original: [Equation]
Revised: [Equation]

Original: By Schur’s Lemma (A.3.1), for each block [ W Q ] k,n k l,m l that multiplies f in l,m l to create f out k,n k (after adding all contributions), we have, for some constants c k,n k l,m l :
Revised: By Schur’s Lemma (A.8.1), for each block [ W Q ] k,n k l,m l that multiplies f in l,m l to create f out k,n k (after adding all contributions), we have, for some constants c k,n k l,m l :

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the solution above says that the query cannot transform an irreducible type to a different irreducible type (e.g., a scalar to vector), but only mix channels that correspond to the same irreducible.
Revised: Intuitively, the solution above says that the query cannot transform an irreducible type to a different irreducible type (e [Equation] g., a scalar to vector), but only mix channels that correspond to the same irreducible.

Original: For W V , the constraint is that for all R ∈ SO(3) , the following set of equivalent statements holds:
Revised: For W V , the constraint is that for all R ∈ SO(3) , the following set of equivalent statements holds:

Original: [Equation]
Revised: [Equation]

Original: The solution, as discussed in the main text and solved in Weiler et al. (2018a); Thomas et al.
Revised: The solution, as discussed in the main text and solved in Weiler et al. (2018a); Thomas et al.

Original: (2018) is:
Revised: (2018) is:

Original: [Equation]
Revised: [Equation]

Original: where C k,lJ (ˆ x ) =
Revised: where C k,l

Original: (cid:80) J m = − J Y Jm (ˆ x ) Q kl Jm , Q klJm ∈ R (2 k +1) ×
Revised: [Equation]

Original: (2 l +1) are the slices from the Q kl Clebsch-Gordan matrices, Y J : S 2 → R (2 J +1) , is the J -th real spherical harmonic,
Revised: l +1) are the slices from the Q kl Clebsch-Gordan matrices, Y J : S 2 → R (2 J +1) , is the J -th real spherical harmonic,

Original: Y Jm (ˆ x ) =
Revised: Y Jm (ˆ x ) =

Original: [ Y J (ˆ x )] m is its m -th coordinate, and ˆ x = ⃗x/ ∥ ⃗x ∥ .
Revised: [ Y J (ˆ x )] m is its m -th coordinate, and ˆ x = ⃗x/ ∥ ⃗x ∥ .

Original: For W K , we have the same equation as for W V above:
Revised: For W K , we have the same equation as for W V above:

Original: [Equation]
Revised: [Equation]

Original: but in addition we can constrain the blocks between different irreducible types to be zero without impacting the result.
Revised: but in addition we can constrain the blocks that transform to an irreducible that does not appear in the input to be zero without impacting the result.

Original: The reason is the form of W Q in Eq.
Revised: The reason is the form of W Q in Eq.

Original: In particular, if l, k are different irreducibles in the input and output respectively, then the blocks of W K  that transform a type- l  to a type- k only contribute as terms to the  inner product in the attention kernel:
Revised: In particular, if l ′ , k are different irreducibles in the input and output respectively, then the blocks [ W K ] k,n k l ′ ,m l ′ that transform a type- l ′ to a type- k only contribute as terms to the total inner product in the attention kernel as follows. For all ( l, m l ) :

Original: [Equation]
Revised: [Equation]

Original: The above inner product is zero  due to Eq.26.
Revised: The above inner product is zero if l ̸ = k due to Eq.

Revised: If k / ∈ K , where K is the set of irreducibles appearing in the input (i.e. the range of l ) then the inner product is zero for all l, m l .

Original: Thus we can also choose:
Revised: So, we might as well choose:

Original: [Equation]
Revised: [Equation]

Original:  For the same input/output types we have:
Revised: since parametrizing those blocks will not contribute to the result. For the rest of the blocks we have, similar as before,

Original: [Equation]
Revised: [Equation]

Original: The parametrization of the weights of the cross-attention module T is similar to Eq.
Revised: The parametrization of the weights of the cross-attention module T is similar to Eq.

Original: The only difference is that we can further reduce the number of parameters without impacting the result by using only the type-1 features of the keys in the first layer, i.e.,  W ijK = 0 , for ( i, j ) ̸ = (1 , 1) .
Revised: The only difference is that we can further reduce the number of parameters without impacting the result by using only the type-1 features of the keys in the first layer, i.e., [ W K ] k,n k l,m l = 0 , for k ̸ = 1 .

Original: This is due to the same equation involving the inner product between the key and the query we saw above, and the fact that the input query field is of type-1.
Revised: This is due to the same equation involving the inner product between the key and the query we saw above, and the fact that the input query field is of type-1.

Original: In Fig.
Revised: In Fig.

Original: 3, we visualize the equivariance constraint on T , E by using a commutative diagram.
Revised: 3, we visualize the equivariance constraint on T , E by using a commutative diagram.

Original: Additional

Original:  Equivariant Layers : Clearly, the concatenation of multiple attention heads with the same irreducible types, as well as skip connection layers as defined in section A.4, preserve equivariance.
Revised: Additional Equivariant Layers : Clearly, the concatenation of multiple attention heads with the same irreducible types, as well as skip connection layers as defined in section A.9, preserve equivariance.

Original: Since they add the multiplicities of each irreducible type independently, they only change the output representation by increasing the multiplicities of its irreducibles.
Revised: Since they add the multiplicities of each irreducible type independently, they only change the output representation by increasing the multiplicities of its irreducibles.

Original: The subsequent mixing of channels of the same irreducible type by a linear map performed in the multi-headed attention module corresponds to the same operation that the query performs, thus it also preserves equivariance.
Revised: The subsequent mixing of channels of the same irreducible type by a linear map performed in the multi-headed attention module corresponds to the same operation that the query performs, thus it also preserves equivariance.

Original: Finally, the equivariant layer-norm layer also operates on each irreducible type independently.
Revised: Finally, the equivariant layer-norm layer also operates on each irreducible type independently.

Original: Since the irreducibles are unitary transforms—i.e., ∥ f l,m l ∥ 2 = ∥ D l f l,m l ∥ 2 for all f l,m l and all D l —any non-linear operation on the norm of a type-l vector produces a type-0 vector.
Revised: Since the irreducibles are unitary transforms—i.e., ∥ f l,m l ∥ 2 = ∥ D l f l,m l ∥ 2 for all f l,m l and all D l —any non-linear operation on the norm of a type-l vector produces a type-0 vector.

Original: Since f l,m l / ∥ f l,m l ∥ 2 is again a type-l vector, the final result is a type-l vector.
Revised: Since f l,m l / ∥ f l,m l ∥ 2 is again a type-l vector, the final result is a type-l vector.

Original: Attention as a set operation :
Revised: Attention as a set operation :

Original: In addition to SE(3) -equivariance, our model inherits the properties of the standard attention layers.
Revised: In addition to SE(3) -equivariance, our model inherits the properties of the standard attention layers.

Original: Thus, it is equivariant to any permutation of the points in the point cloud, and to the order of the queries.
Revised: Thus, it is equivariant to any permutation of the points in the point cloud, and to the order of the queries.

Original: Moreover, the number of output tokens can vary.
Revised: Moreover, the number of output tokens can vary.

Original: We use this property for scene reconstruction, by reconstructing scenes of a variable number of points (and point clouds) even by training on single objects of a fixed number of points.
Revised: We use this property for scene reconstruction, by reconstructing scenes of a variable number of points (and point clouds) even by training on single objects of a fixed number of points.

Original: The input (key-value) tokens can also change during inference, which we can use to adapt the neighborhoods dynamically during inference.
Revised: The input (key-value) tokens can also change during inference, which we can use to adapt the neighborhoods dynamically during inference.

Original: This can counter that super-sampling a point cloud may reduce the receptive field.
Revised: This can counter that super-sampling a point cloud may reduce the receptive field.

Original: On independent SO(3) rotations.
Revised: On independent SO(3) rotations.

Original: Our local attention neighborhoods and equivariant reconstructions are particularly important properties when reconstructing scenes.
Revised: Our local attention neighborhoods and equivariant reconstructions are particularly important properties when reconstructing scenes.

Original: Here the point clouds can be independently placed in arbitrary positions.
Revised: Here the point clouds can be independently placed in arbitrary positions.

Original: It is natural to ask if we can connect the performance of our model in a particular scene to the performance in another scene, where the same point clouds have been independently roto-translated.
Revised: It is natural to ask if we can connect the performance of our model in a particular scene to the performance in another scene, where the same point clouds have been independently roto-translated.

Original: For a single object, the true occupancy function should be the same under a simultaneous roto-translation of the point cloud and the query.
Revised: For a single object, the true occupancy function should be the same under a simultaneous roto-translation of the point cloud and the query.

Original: Our equivariant pipeline respects this property, by outputting a scalar field.
Revised: Our equivariant pipeline respects this property, by outputting a scalar field.

Original: Thus, the performance of our model is consistent independently of the SE(3) transformations of a single point cloud.
Revised: Thus, the performance of our model is consistent independently of the SE(3) transformations of a single point cloud.

Original: Following a similar approach for scenes with multiple objects, one can associate an independent copy of SO(3) acting on each point cloud, i.e., an action X
Revised: Following a similar approach for scenes with multiple objects, one can associate an independent copy of SO(3) acting on each point cloud, i.e., an action X

Original: i (cid:55)→
Revised: i (cid:55)→

Original: L r i [ X i ] , for all objects i ∈
Revised: L r i [ X i ] , for all objects i ∈

Original: I .
Revised: I .

Original: We can ask if there is a transformation of the query ⃗q , that, after the rotation of the point clouds, results in the same occupancy value.
Revised: We can ask if there is a transformation of the query ⃗q , that, after the rotation of the point clouds, results in the same occupancy value.

Original: Also, can this implemented without knowing the segmentation function that assigns the points to their point clouds?
Revised: Also, can this implemented without knowing the segmentation function that assigns the points to their point clouds?

Original: A reasonable first approach would be to transform the query ⃗q with the rotation matrix R i used to transform its closest neighbor.
Revised: A reasonable first approach would be to transform the query ⃗q with the rotation matrix R i used to transform its closest neighbor.

Original: Unfortunately, this transformation does not correspond to a group action.
Revised: Unfortunately, this transformation does not correspond to a group action.

Original: To understand this, consider two unit spheres at positions (0 , 0 , 0) and (1 , 0 , 0) in three-dimensional space, and the query point ⃗q =
Revised: To understand this, consider two unit spheres at positions (0 , 0 , 0) and (1 , 0 , 0) in three-dimensional space, and the query point ⃗q =

Original: ( − 10 , 0 , 0) .
Revised: ( − 10 , 0 , 0) .

Original: Then, consider the product group action L 1 =
Revised: Then, consider the product group action L 1 =

Original: L r 1 ,r 2 = L (0 , 0 ,π ) ,e that rotates the first sphere around the z -axis by 180 degrees— r 1 = (0 , 0 , π ) —and fixes the second sphere— r 1 = e .
Revised: L r 1 ,r 2 = L (0 , 0 ,π ) ,e that rotates the first sphere around the z -axis by 180 degrees— r 1 = (0 , 0 , π ) —and fixes the second sphere— r 1 = e .

Original: Next, consider the reverse action L 2 =
Revised: Next, consider the reverse action L 2 =

Original: L (0 , 0 , − π ) ,e .
Revised: L (0 , 0 , − π ) ,e .

Original: If we first multiply the corresponding group elements together, we find L =
Revised: If we first multiply the corresponding group elements together, we find L =

Original: L 1 ◦ L 2 =
Revised: L 1 ◦ L 2 =

Original: L e,e
Revised: L e,e

Original: = I .
Revised: = I .

Original: If we then find the nearest sphere and apply the identity action to the query, it will remain at ( − 10 , 0 , 0) .
Revised: If we then find the nearest sphere and apply the identity action to the query, it will remain at ( − 10 , 0 , 0) .

Original: On the other hand, if we first find the nearest neighbor—in this case, the first sphere–and apply the corresponding action, then the query will rotate around the z -axis by 180 -degrees, moving to (10 , 0 , 0) .
Revised: On the other hand, if we first find the nearest neighbor—in this case, the first sphere–and apply the corresponding action, then the query will rotate around the z -axis by 180 -degrees, moving to (10 , 0 , 0) .

Original: If we then apply the second action by finding again the nearest neighbor—now the second sphere—the identity action will be applied and the query point stays at (10 , 0 , 0) .
Revised: If we then apply the second action by finding again the nearest neighbor—now the second sphere—the identity action will be applied and the query point stays at (10 , 0 , 0) .

Original: Thus, this transformation does not satisfy the “compatibility” property and thus it is not a group action.
Revised: Thus, this transformation does not satisfy the “compatibility” property and thus it is not a group action.

Original: Even though the conditions are not met for all query points in space, there are subsets of R 3 that are closed under these transformations and for which these transformations indeed form group actions.
Revised: Even though the conditions are not met for all query points in space, there are subsets of R 3 that are closed under these transformations and for which these transformations indeed form group actions.

Original: We will consider equivariance only in those subsets.
Revised: We will consider equivariance only in those subsets.

Original: We construct them geometrically for two point clouds for simplicity.
Revised: We construct them geometrically for two point clouds for simplicity.

Original: Consider the point clouds X 1 , X 2 rotating around points c 1 , c 2 , respectively.
Revised: Consider the point clouds X 1 , X 2 rotating around points c 1 , c 2 , respectively.

Original: We construct the equivariant zone for the point cloud X 1 .
Revised: We construct the equivariant zone for the point cloud X 1 .

Original: First take the point in X 2 that has the maximum distance (in Euclidean norm), say d 2 , to its center of rotation c 2 .
Revised: First take the point in X 2 that has the maximum distance (in Euclidean norm), say d 2 , to its center of rotation c 2 .

Original: Form the sphere Saround X 2 , with a radius d 2 and center c 2 .
Revised: Form the sphere Saround X 2 , with a radius d 2 and center c 2 .

Original: All of X 2 is contained in S 2 .
Revised: All of X 2 is contained in S 2 .

Original: Connect the centers c 1 , c 2 , and denote the point of intersection of this segment with S 2 as p .
Revised: Connect the centers c 1 , c 2 , and denote the point of intersection of this segment with S 2 as p .

Original: Then, draw the segment between p and c 1 and name the distance of c 1 to the middle point of this segment D 1 .
Revised: Then, draw the segment between p and c 1 and name the distance of c 1 to the middle point of this segment D 1 .

Original: Every point in the sphere
Revised: Every point in the sphere

Original: S 1 of radius D 1 and center c 1 is in the equivariant zone of X 1 , called Z 1 .
Revised: S 1 of radius D 1 and center c 1 is in the equivariant zone of X 1 , called Z 1 .

Original: We prove that in the equivariant zone, our model is equivariant, for any independent rotation of the point clouds.
Revised: We prove that in the equivariant zone, our model is equivariant, for any independent rotation of the point clouds.

Original: By construction, every query ⃗q in the first equivariant zone Z 1 has its closest neighbor in the point cloud X 1 .
Revised: By construction, every query ⃗q in the first equivariant zone Z 1 has its closest neighbor in the point cloud X 1 .

Original: This holds for any rotation of any point cloud.
Revised: This holds for any rotation of any point cloud.

Original: Then, by definition, the action on the query is the same rotation R r 1 that transformed the points in X 1 .
Revised: Then, by definition, the action on the query is the same rotation R r 1 that transformed the points in X 1 .

Original: Since the point cloud and the query both rotate with the same transformation, the query neighborhood constructed within the cross-attention layers is invariant—leading to the same closest neighbor—as we proved for the single object case.
Revised: Since the point cloud and the query both rotate with the same transformation, the query neighborhood constructed within the cross-attention layers is invariant—leading to the same closest neighbor—as we proved for the single object case.

Original: Now, suppose each point in X 1 has its k nearest neighbors in X 1 , for any rotation of the point clouds; this is reasonable for sufficiently dense or separated point clouds.
Revised: Now, suppose each point in X 1 has its k nearest neighbors in X 1 , for any rotation of the point clouds; this is reasonable for sufficiently dense or separated point clouds.

Original: Then these point cloud neighborhoods are also invariant after the individual rotations.
Revised: Then these point cloud neighborhoods are also invariant after the individual rotations.

Original: Under these conditions, a direct product action of SO(3) -s on the point clouds with a simultaneous action of R 1 on the query in the first equivariant zone is viewed by the attention network as a simultaneous rotation of the point cloud X 1 and the query ⃗q .
Revised: Under these conditions, a direct product action of SO(3) -s on the point clouds with a simultaneous action of R 1 on the query in the first equivariant zone is viewed by the attention network as a simultaneous rotation of the point cloud X 1 and the query ⃗q .

Original: This is because N ( ⃗q ) contains only points from X 1 , so the attention module uses only key-value tokens from points in X 1 .
Revised: This is because N ( ⃗q ) contains only points from X 1 , so the attention module uses only key-value tokens from points in X 1 .

Original: Further, the neighborhood of the query is invariant to a simultaneous rotation of X 1 and ⃗q .
Revised: Further, the neighborhood of the query is invariant to a simultaneous rotation of X 1 and ⃗q .

Original: Thus, as we proved in the single-object case, the occupancy value prediction for this transformed query is invariant to the transformation.
Revised: Thus, as we proved in the single-object case, the occupancy value prediction for this transformed query is invariant to the transformation.

Original: Thus, for all queries in an equivariant zone, equivariance to the direct product action of SO(3) holds, i.e., for all r 1 , r 2 , · · · , r I ∈ SO(3) , and for all ⃗q ∈
Revised: Thus, for all queries in an equivariant zone, equivariance to the direct product action of SO(3) holds, i.e., for all r 1 , r 2 , · · · , r I ∈ SO(3) , and for all ⃗q ∈

Original: Z j , with j ∈ [ I ] ,
Revised: Z j , with j ∈ [ I ] ,

Original: [Equation]
Revised: [Equation]

Revised: ] , · · · , L r I [ X I ]) , R j ⃗q ] =

Revised: T [( X 1 , X 2 , · · · , X I ) , ⃗q ] .