Original: P ROTEIN S TRUCTURE R EPRESENTATION L EARNING
Revised: P ROTEIN S TRUCTURE R EPRESENTATION L EARNING

Original: THROUGH
Revised: THROUGH

Original: O RIENTATION -A
Revised: O RIENTATION -A

Original: WARE
Revised: WARE

Original: G RAPH
Revised: G RAPH

Original: N EURAL
Revised: N EURAL

Original: N ETWORKS
Revised: N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: By folding to particular 3D structures, proteins play a key role in living beings.
Revised: By folding to particular 3D structures, proteins play a key role in living beings.

Original: To learn meaningful representation from a protein structure for downstream tasks, not only the global backbone topology but the local fine-grained orientational relations between amino acids should also be considered.
Revised: To learn meaningful representation from a protein structure for downstream tasks, not only the global backbone topology but the local fine-grained orientational relations between amino acids should also be considered.

Original: In this work, we propose the Orientation-Aware Graph Neural Networks (OAGNNs) to better sense the geometric characteristics in protein structure (e.g. inner-residue torsion angles, inter-residue orientations).
Revised: In this work, we propose the Orientation-Aware Graph Neural Networks (OAGNNs) to better sense the geometric characteristics in protein structure (e.g. inner-residue torsion angles, inter-residue orientations).

Original: Extending a single weight from a scalar to a 3D vector, we construct a rich set of geometric-meaningful operations to process both the classical and SO(3) representations of a given structure.
Revised: Extending a single weight from a scalar to a 3D vector, we construct a rich set of geometric-meaningful operations to process both the classical and SO(3) representations of a given structure.

Original: To plug our designed perceptron unit into existing Graph Neural Networks, we further introduce an equivariant message passing paradigm, showing superior versatility in maintaining SO(3) -equivariance at the global scale.
Revised: To plug our designed perceptron unit into existing Graph Neural Networks, we further introduce an equivariant message passing paradigm, showing superior versatility in maintaining SO(3) -equivariance at the global scale.

Original: Experiments have shown that our OAGNNs have a remarkable ability to sense geometric orientational features compared to classical networks.
Revised: Experiments have shown that our OAGNNs have a remarkable ability to sense geometric orientational features compared to classical networks.

Original: OAGNNs have also achieved state-of-theart performance on various computational biology applications related to protein 3D structures.
Revised: OAGNNs have also achieved state-of-theart performance on various computational biology applications related to protein 3D structures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Built from a sequence of amino-acid residues, a protein performs its biological functions by folding to a particular conformation in 3D space.
Revised: Built from a sequence of amino-acid residues, a protein performs its biological functions by folding to a particular conformation in 3D space.

Original: Therefore, untilizing such 3D structures accurately is the key for downstream analysis.
Revised: Therefore, untilizing such 3D structures accurately is the key for downstream analysis.

Original: While we have witnessed remarkable progress in protein structure predictions (Rohl et al., 2004; K ¨ allberg et al., 2012; Baek et al., 2021; Jumper et al., 2021), another thread of tasks with protein 3D structures as input starts to draw a great interest, such as function prediction (Hermosilla et al., 2020; Gligorijevi´c et al., 2021), decoy ranking (Lundstr¨om et al., 2001;
Revised: While we have witnessed remarkable progress in protein structure predictions (Rohl et al., 2004; K ¨ allberg et al., 2012; Baek et al., 2021; Jumper et al., 2021), another thread of tasks with protein 3D structures as input starts to draw a great interest, such as function prediction (Hermosilla et al., 2020; Gligorijevi´c et al., 2021), decoy ranking (Lundstr¨om et al., 2001;

Original: Kwon et al., 2021;
Revised: Kwon et al., 2021;

Original: Wang et al., 2021), protein docking (Duhovny et al., 2002; Shulman-Peleg et al., 2004; Gainza et al., 2020; Sverrisson et al., 2021), and driver mutation identification (Lef ` evre et al., 1997; Antikainen & Martin, 2005; Li et al., 2020; Jankauskait˙e et al., 2019).
Revised: Wang et al., 2021), protein docking (Duhovny et al., 2002; Shulman-Peleg et al., 2004; Gainza et al., 2020; Sverrisson et al., 2021), and driver mutation identification (Lef ` evre et al., 1997; Antikainen & Martin, 2005; Li et al., 2020; Jankauskait˙e et al., 2019).

Original: Most existing works in modeling protein structures directly borrow models designed for other applications, including 3D-CNNs (Ji et al., 2012) in computer vision, Transformers (Vaswani et al., 2017)) from natural language processing, and GNNs (Kipf & Welling, 2016) in data mining.
Revised: Most existing works in modeling protein structures directly borrow models designed for other applications, including 3D-CNNs (Ji et al., 2012) in computer vision, Transformers (Vaswani et al., 2017)) from natural language processing, and GNNs (Kipf & Welling, 2016) in data mining.

Original: Though compatible with general objects, these models have overlooked the subtleties in the fine-grained geometries, which are much more essential in protein structures.
Revised: Though compatible with general objects, these models have overlooked the subtleties in the fine-grained geometries, which are much more essential in protein structures.

Original: For instance, given an amino acid in the protein structure, as shown in Figure 1, the locations of four backbone atoms (carbon, nitrogen, and oxygen) determine a local skeleton, and different residues interact with each other through performing specific orientations between their local frames, either of which have important impacts on the protein structure and its function (Nelson et al., 2008).
Revised: For instance, given an amino acid in the protein structure, as shown in Figure 1, the locations of four backbone atoms (carbon, nitrogen, and oxygen) determine a local skeleton, and different residues interact with each other through performing specific orientations between their local frames, either of which have important impacts on the protein structure and its function (Nelson et al., 2008).

Original: Recent attempts in building geometric-aware neural networks mainly focus on baking 3D rigid transformations into network operations, leading to the area of SO(3) -invariant and equivariant networks.
Revised: Recent attempts in building geometric-aware neural networks mainly focus on baking 3D rigid transformations into network operations, leading to the area of SO(3) -invariant and equivariant networks.

Original: One representative work is the Vector Neuron Network (VNN)
Revised: One representative work is the Vector Neuron Network (VNN)

Original: (Deng et al., 2021), which achieves SO(3) -equivariance on point clouds by generalizing scalar neurons to 3D vectors.
Revised: (Deng et al., 2021), which achieves SO(3) -equivariance on point clouds by generalizing scalar neurons to 3D vectors.

Original: Another work is the GVP-GNN (Jing et al., 2021) that similarly vectorizes hidden neurons in GNN and demonstrates better prediction accuracy on protein design and quality evaluation tasks.
Revised: Another work is the GVP-GNN (Jing et al., 2021) that similarly vectorizes hidden neurons in GNN and demonstrates better prediction accuracy on protein design and quality evaluation tasks.

Original: However, these two models can only adopt linear combinations of input vectors, which significantly limits their modeling capability.
Revised: However, these two models can only adopt linear combinations of input vectors, which significantly limits their modeling capability.

Original: A simple example is that, given two input vector features v 1 and v 2 , the outputs w 1 v 1 + w 2 v 2 through one linear layer is constrained in the 2D plane spanned by v 1 , v 2 even after applying their scalar-product non-linearities.
Revised: A simple example is that, given two input vector features v 1 and v 2 , the outputs w 1 v 1 + w 2 v 2 through one linear layer is constrained in the 2D plane spanned by v 1 , veven after applying their scalar-product non-linearities.

Original: That is, VNN-based models are limited in perceiving orientational features, which have been proven crucial for proteins to perform their functions and interact with other partners ( e.g. inner-residue torsion angles, inter-residue orientations).
Revised: That is, VNN-based models are limited in perceiving orientational features, which have been proven crucial for proteins to perform their functions and interact with other partners ( e.g. inner-residue torsion angles, inter-residue orientations)

Original:  To achieve more sensitive geometric orientation awareness in both global transformations and local relations, we propose a Directed Weight ( ⃗ W ) perceptrons by extending not only the hidden neurons but also the weights from scalars to 3D vectors, naturally saturating the entire network with 3D structure information in the Euclidean space.
Revised: (Nelson et al., 2008; Voet & Voet, 2010; Xu & Berger, 2006; Alford et al., 2017). To achieve more sensitive geometric orientation awareness, we propose a Directed Weight ( ⃗ W ) perceptrons by extending not only the hidden neurons but also the weights from scalars to 3D vectors, naturally saturating the entire network with 3D structure information in the Euclidean space.

Original: Directed weights support a set of geometricmeaningful operations on both the vector neurons (vector-list features) and the classical (scalar-list) latent features, and perform flexible non-linear integration of the hybrid scalar-vector features.
Revised: Directed weights support a set of geometric-meaningful operations on both the vector neurons (vectorlist features) and the classical (scalar-list) latent features, and perform flexible non-linear integration of the hybrid scalar-vector features.

Original: As protein structures are naturally attributed proximity graphs, we introduce a new Equivariant Message Passing Paradigm on protein graphs, to connect the ⃗ W -perceptrons with the graph learning models by using rigid backbone transformations for each amino acid, which provides a versatile framework for bringing the biological suitability and flexibility of the GNN architecture.
Revised: As protein structures are naturally attributed proximity graphs, we introduce a new Equivariant Message Passing Paradigm on protein graphs, to connect the ⃗ Wperceptrons with the graph learning models by using rigid backbone transformations for each amino acid, which provides a versatile framework for bringing the biological suitability and flexibility of the GNN architecture.

Original: To summarize, our key contributions include:
Revised: To summarize, our key contributions include:

Original: • We propose a new network unit based on the Directed Weights for capturing fine-grained geometric relations, especially for the subtle orientational details in proteins.
Revised: • We propose a new network unit based on the Directed Weights for capturing fine-grained geometric relations, especially for the subtle orientational details in proteins.

Original: • We construct an Equivariant Message Passing paradigm based on protein graphs.
Revised: • We construct an Equivariant Message Passing paradigm based on protein graphs.

Original: • Our overall framework, the Orientation-Aware Graph Neural Networks , is versatile in terms of compatibility with existing deep graph learning models, making them biologically suitable with minimal modifications to existing GNN models.
Revised: • Our overall framework, the Orientation- Aware Graph Neural Networks , is versatile in terms of compatibility with existing deep graph learning models, making them biologically suitable with minimal modifications to existing GNN models.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Representation learning on protein 3D structure.
Revised: Representation learning on protein 3D structure.

Original: Early approaches rely on hand-crafted features extracted and statistical methods to predict function annotations (Schaap et al., 2001; Zhang &
Revised: Early approaches rely on hand-crafted features extracted and statistical methods to predict function annotations (Schaap et al., 2001; Zhang &

Original: Zhang, 2010).
Revised: Zhang, 2010).

Original: Deep learning has been found to achieve success then.
Revised: Deep learning has been found to achieve success then.

Original: 3D CNNs are first proposed to process protein 3D structures by scanning atom-level features relying on multiple 3D voxels.
Revised: 3D CNNs are first proposed to process protein 3D structures by scanning atom-level features relying on multiple 3D voxels.

Original: One of the representative works (Derevyanko et al., 2018) adopts a 3D CNN-based model for assessing the quality of the predicted structures.
Revised: One of the representative works (Derevyanko et al., 2018) adopts a 3D CNN-based model for assessing the quality of the predicted structures.

Original: 3D CNNs also shed light on other tasks such as interface prediction (Townshend et al., 2019; Amidi et al., 2018).
Revised: 3D CNNs also shed light on other tasks such as interface prediction (Townshend et al., 2019; Amidi et al., 2018).

Original: People also extend them to spherical convolutions (Gainza et al., 2020; Sverrisson et al., 2021; Hermosilla Casajus et al., 2021), to the Fourier space
Revised: People also extend them to spherical convolutions (Gainza et al., 2020; Sverrisson et al., 2021; Hermosilla Casajus et al., 2021), to the Fourier space

Original: (Zhemchuzhnikov et al., 2021) and the 3D Voronoi Tessellation space (Igashov et al., 2021).
Revised: (Zhemchuzhnikov et al., 2022) and the 3D Voronoi Tessellation space (Igashov et al., 2021).

Original: Graph
Revised: Graph

Original: Convolutional Networks (Kipf & Welling, 2016) have also been adopted to capture geometric and biochemical interactions between residues (Ying et al., 2018; Gao & Ji, 2019; Fout, 2017), and have been shown to achieve great performance on function prediction (Li et al., 2021),
Revised: Convolutional Networks (Kipf & Welling, 2016) have also been adopted to capture geometric and biochemical interactions between residues (Ying et al., 2018; Gao & Ji, 2019; Fout, 2017), and have been shown to achieve great performance on function prediction (Li et al., 2021),

Original: protein design (Strokach et al., 2020) and binding prediction (Vecchio et al., 2021).
Revised: protein design (Strokach et al., 2020) and binding prediction (Vecchio et al., 2021).

Original: Recently, transformer-based methods (Vaswani et al., 2017) have a trend to replace conventional methods in other bioinformatics tasks (Ingraham et al., 2019; Baek et al., 2021; Jumper et al., 2021; Cao et al., 2021).
Revised: Recently, transformer-based methods (Vaswani et al., 2017) have a trend to replace conventional methods in other bioinformatics tasks (Ingraham et al., 2019; Baek et al., 2021; Jumper et al., 2021; Cao et al., 2021).

Original: Equivariant neural networks.
Revised: Equivariant neural networks.

Original: Equivariance is an important property, for generalizing to unseen conditions in geometric learning.
Revised: Equivariance is an important property, for generalizing to unseen conditions in geometric learning.

Original: (Cohen & Welling, 2016; Weiler et al., 2018; K¨ohler et al., 2020;
Revised: (Cohen & Welling, 2016; Weiler et al., 2018; K¨ohler et al., 2020;

Original: Satorras et al., 2021; Thomas et al., 2018; Fuchs et al., 2020; Anderson et al., 2019; Klicpera et al., 2020; Batzner et al., 2021; Eismann et al., 2021).
Revised: Satorras et al., 2021; Thomas et al., 2018; Fuchs et al., 2020; Anderson et al., 2019; Gasteiger et al., 2019b; Batzner et al., 2021; Eismann et al., 2021).

Original: Methods such as Tensor Field Network (Thomas et al., 2018) and Cormorant (Anderson et al., 2019), have been developed to generate irreducible representations for achieving rotation equivariance in 3D.
Revised: Methods such as Tensor Field Network (Thomas et al., 2018) and Cormorant (Anderson et al., 2019), have been developed to generate irreducible representations for achieving rotation equivariance in 3D.

Original: In comparison to Tensor Filed Network with complex tensor products, the Vector Neuron Network (VNN) achieves rotation equivariance in a much simpler way, which generalizes the values of hidden neurons from scalars to 3D vectors (Deng et al., 2021).
Revised: In comparison to Tensor Filed Network with complex tensor products, the Vector Neuron Network (VNN) achieves rotation equivariance in a much simpler way, which generalizes the values of hidden neurons from scalars to 3D vectors (Deng et al., 2021).

Original: GVP-GNN has also been introduced for learning protein representations by featuring geometric factors as vectors (Jing et al., 2020).
Revised: GVP-GNN has also been introduced for learning protein representations by featuring geometric factors as vectors (Jing et al., 2020).

Original: Message passing in GNNs for vector representations using equivariant features have also been explored (Sch¨utt et al., 2021;
Revised: Message passing in GNNs for vector representations using equivariant features have also been explored (Sch¨utt et al., 2021;

Original: Luo et al., 2022).
Revised: Luo et al., 2022).

Original: However, to guarantee rotation equivariance, they can only linearly combine 3D vectors, in essence, limiting their geometric representing capacity.
Revised: However, to guarantee rotation equivariance, they can only linearly combine 3D vectors, in essence, limiting their geometric representing capacity.

Original: Another line of methods condition filters on invariant scalar features for maitaining equivariance (Sch¨utt et al., 2017; Klicpera et al., 2020; Liu et al., 2021).
Revised: Another line of methods condition filters on invariant scalar features for maitaining equivariance (Sch¨utt et al., 2017; Gasteiger et al., 2019a; Liu et al., 2021).

Original:  D IRECTED W EIGHT P ERCEPTRON
Revised:  D IRECTED W EIGHT P ERCEPTRON

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: Output features
Revised: [Figure] [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: latexit
Revised:

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: /ExmRxgxlYJ3F7WZaK8h/tUBObYbwtJPxOEmBxXS8OEwFBoWLOnGPa0ZBDC0gVHN7O6YR0YSCLb1kS/GmK/gLGoc177h2dH1UOT+c1LOEdtAuqiIPnaBzdImuUB1R9ICe0At6dR6dd+fD+RxbZ5xJZhv9GufrG/cbsfg=</latexit> V 0 =( s h

Original: A protein is modeled as a KNN-graph G =
Revised: ! W s )  V h <latexit sha1_base64="UeHEfi8IZX6NBZSVBXKUZMKtiEI=">AAACHXicbVDLTgIxFO34RHyhLt00otEVmVGiLEnYuEQjj4QZSVsKNLSdSdsxIZP5AX/CX3Cre3fGrXHrl9gBFgqepMnJOffm3B4ccaaN6345S8srq2vruY385tb2zm5hb7+pw1gR2iAhD1UbI005k7RhmOG0HSmKBOa0hUe1zG89UKVZKO/MOKKBQAPJ+owgY6Vu4djHImmmp9BnEvoCmSHGyW16n9SsZJigGl6k3ULRLbkTwEXizUgRzFDvFr79XkhiQaUhHGnd8dzIBAlShhFO07wfaxohMkID2rFUIpsTJJPfpPDEKj3YD5V90sCJ+nsjQULrscB2MrtXz3uZ+K+HxVyy6VeChMkoNlSSaXA/5tCEMKsK9piixPCxJYgoZm+HZIgUIsYWmrelePMVLJLmecm7LJVvysVqZVZPDhyCI3AGPHAFquAa1EEDEPAInsELeHWenDfn3fmYji45s50D8AfO5w8Y3aGQ</latexit> V 0 2 R C 0 ⇥<latexit sha1_base64="hNjbOsNKJbZMsS+go8BAJ6ASVXU=">AAACFHicbVC7TgJBFJ31ifhCLWkmEoMV2TVEKUloLNHII2FXMjMMMGFmdjMza0I2W [Table] +o3Nxf4MDBznBobnga8Yf+c8EQ5t1Y0dJaLu/taSf5To+qeM6SnUy90lgPXbGuc5hKDwWWgeC4sZyDXARBmRdgdsyWxhEGIvR5C+X05/j8Yd9udN+3e+15z0K3i2UWv0Gt0hDroBA3QW3SGRoihT+gL+oZuos/R9+hHdLdtrUXVzEv0V0U/fwHyCrLO</latexit> !

Revised: W V 2 R C 0 ⇥ C 0 ⇥

Revised: A protein is modeled as a KNN-graph G =

Original: ( V , E ) where each node u ∈ V corresponds to one amino acid, characterized by a scalar-vector tuple
Revised: ( V , E ) where each node u ∈ V corresponds to one amino acid, characterized by a scalar-vector tuple

Original: h u
Revised: h u

Original: = ( s u , V u ) with s u [Equation] and the edges are constructed spatially by querying its k -nearest neighbours in the space.
Revised: = ( s u , V u ) with s u ∈ R C , V u ∈ R C × 3 , and the edges are constructed spatially by querying its k -nearest neighbours in the space.

Original: The edge features E = { e ij } i ̸
Revised: The edge features E = { e ij } i ̸

Original: = j , which represent edge connected node i and j , are also multi-channel scalar- vector tuples.
Revised: = j , which represent edge connected node i and j , are also multi-channel scalar- vector tuples.

Original: Fornotation simplicity, we set the channel numbers for scalar and vector features as the same, but they can be different in practice .
Revised: For simplicity, we set the channel numbers for scalar and vector features as the same, but they can be different.

Revised: In practice, scalar, vector features are constructed based on a given protein structure and are kept separate.

Revised: To make them compatible, the channel numbers of scalars are larger than the vectors.

Original: Details can be found in the Appendix Section B.1.
Revised: Details can be found in the Appendix Section B.1.

Original: 3. D IRECTED W EIGHTS
Revised: 3. D IRECTED W EIGHTS

Original: Classical neural networks only consider scalar-list features of the form s ∈ R C , with each layer transforming them with a weight matrix W ∈ R C
Revised: Classical neural networks only consider scalar-list features of the form s ∈ R C , with each layer transforming them with a weight matrix W ∈ R C

Original: ′ × C and a bias term b ∈ R C ′ :
Revised: ′ × C and a bias term b ∈ R C ′ :

Original: [Equation]
Revised: [Equation]

Original: Although has been proved to be a universal approximator, such a layer has intrinsically no geometric interpretation in the 3D shape space, and even the simplest 3D rigid transformation on the input can result in unpredictable changes of network outputs.
Revised: Although has been proved to be a universal approximator, such a layer has intrinsically no geometric interpretation in the 3D shape space, and even the simplest 3D rigid transformation on the input can result in unpredictable changes of network outputs.

Original: Recent attempts have lifted neuron features from scalar-lists to vector-lists V ∈ R C × 3 with a linear operation mapping to V ′ ∈ R C ′ × 3 (Deng et al., 2021;
Revised: Recent attempts have lifted neuron features from scalar-lists to vector-lists V ∈ R C × 3 with a linear operation mapping to V ′ ∈ R C ′ × 3 (Deng et al., 2021;

Original: Jing et al., 2020):
Revised: Jing et al., 2020):

Original: [Equation]
Revised: [Equation]

Original: Under this construction, SO(3) -actions in the latent space are simply matrix multiplications, which establishes a clear 3D Euclidean structure.
Revised: Under this construction, SO(3) -actions in the latent space are simply matrix multiplications, which establishes a clear 3D Euclidean structure.

Original: However, to maintain input-output consistency under rigid transformations (e.g. rotation and translation), their operations are still limited to classical linear combinations weighted by W .
Revised: However, to maintain input-output consistency under rigid transformations (e.g. rotation and translation), their operations are still limited to classical linear combinations weighted by W .

Original: To define operations that are more adaptive to the geometrically meaningful features, we introduce the Directed Weights , that we can define any tensor with the last dimension of 3 as a directed weight matrix, for example, ⃗ W ∈ R C ′ ×
Revised: To define operations that are more adaptive to the geometrically meaningful features, we introduce the Directed Weights , that we can define any tensor with the last dimension of 3 as a directed weight matrix, for example, ⃗ W ∈ R C ′ ×

Original: C × 3 , seen as a stack of geometric meaningful vectors lying in the 3D Euclidean Space, and can be acted by SE(3) group from the right.
Revised: C × 3 , seen as a stack of geometric meaningful vectors lying in the 3D Euclidean Space, and can be acted by SE(3) group from the right.

Original: 3. ⃗ W -O PERATORS
Revised: 3. ⃗ W -O PERATORS

Original: With weights and features both equipped with 3D vector representations, we can design a set of geometric operators □ (e.g. · or × ) with learnable ⃗ W as parameters in neural networks, which can operate on both scalar-list features s ∈ R C and vector-list features V ∈ R C ×
Revised: With weights and features both equipped with 3D vector representations, we can design a set of geometric operators □ (e.g. · or × ) with learnable ⃗ W as parameters in neural networks, which can operate on both scalar-list features s ∈ R C and vector-list features V ∈ R C ×

Original: [Equation]
Revised: [Equation]

Original: Geometric vector operations.
Revised: Geometric vector operations.

Original: Let W ∈ R C
Revised: Let W ∈ R C

Original: ×
Revised: ×

Original: C be a conventional scalar weight matrix, ⃗ W 1 ∈
Revised: C be a conventional scalar weight matrix, ⃗ W 1 ∈

Original: R C ′ × C × 3 and ⃗ W 2 ∈ R C × 3 be directed weight tensors.
Revised: R C ′ × C × 3 and ⃗ W 2 ∈ R C × 3 be directed weight tensors.

Original: Beyond linear combinations, We can define two operations that leverage geometric information:
Revised: Beyond linear combinations, We can define two operations that leverage geometric information:

Original: [Equation]
Revised: [Equation]

Original: Here s ′ dot transforms C vector features to C ′ scalars using dot-product with directed weights, detailedly, s ′ i
Revised: Here s ′ dot transforms C vector features to C ′ scalars using dot-product with directed weights, detailedly, s ′ i

Original: = (cid:80) j,k ⃗ W
Revised: = (cid:80) j,k ⃗ W

Original: i,j,k
Revised: i,j,k

Original: V j,k , which explicitly measures angles between vectors, and this operator brings network the ability to accurately sense the orientational features.
Revised: V j,k , which explicitly measures angles between vectors, and this operator brings network the ability to accurately sense the orientational features.

Original: In V ′ crs , a vector crosses a directed weight before being projected onto the hidden space.
Revised: In V ′ crs , a vector crosses a directed weight before being projected onto the hidden space.

Original: While the output of plain linear combinations between two vectors v 1 and v 2 can only lie in the plane w 1 v 1 + w 2 v 2 , cross-product in
Revised: While the output of plain linear combinations between two vectors v 1 and v 2 can only lie in the plane w 1 v 1 + w 2 v 2 , cross-product in

Original: Equation 5 creates a new direction outside the plane, which is crucial in 3D modelling.
Revised: Equation 5 creates a new direction outside the plane, which is crucial in 3D modelling.

Original: For instance, the side-chain angles of a given residue could be largely determined by its C α and C β vectors, but may lie on the direction perpendicular to the space constructed by the two vectors.
Revised: For instance, the side-chain angles of a given residue could be largely determined by its C α and C β vectors, but may lie on the direction perpendicular to the space constructed by the two vectors.

Original: Scalar lifting.
Revised: Scalar lifting.

Original: In addition, a directed weight ⃗ W ∈ R C × 3 can lift scalars to vectors by adopting the following operation, 
Revised: In addition, a directed weight ⃗ W ∈ R C × 3 can lift scalars to vectors by adopting the following operation, the ij th entry of s ⃗W is s i ⃗W ij

Original: [Equation]
Revised: [Equation]

Original: This maps each scalar to a particular vector, enabling inverse transformations or information bottlenecks from R to R 3 .
Revised: This maps each scalar to a particular vector, enabling inverse transformations or information bottlenecks from R to R 3 .

Original: An intuition example is that, we can map a scalar representing the distance between two amino acids to a vector pointing from one amino acid to the other, leading to more biological meaningful representations for the protein fragment.
Revised: An intuition example is that, we can map a scalar representing the distance between two amino acids to a vector pointing from one amino acid to the other, leading to more biological meaningful representations for the protein fragment.

Original: Linear combinations.
Revised: Linear combinations.

Original: In the end, we keep the linear combination operations with scalar weights for both scalar and vector features:
Revised: In the end, we keep the linear combination operations with scalar weights for both scalar and vector features:

Original: [Equation]
Revised: [Equation]

Original: While these operators enables a more flexible network design, equivariance to rigid transformations is broken.
Revised: While these operators enables a more flexible network design, equivariance to rigid transformations is broken if considering more complex functions beyond linear combinations.

Original: We will introduce a globally equivariant paradigm to tackle this issue in Section 4.
Revised: We will introduce a globally equivariant paradigm to tackle this issue in Section 4.

Original: 3. ⃗ W -P ERCEPTRON U NIT
Revised: 3. ⃗ W -P ERCEPTRON U NIT

Original: Now we combine all the ⃗ W -operators together, assembling them into what we call Directed Weight
Revised: Now we combine all the ⃗ W -operators together, assembling them into what we call Directed Weight

Original: Perceptrons ( ⃗ W -perceptrons).
Revised: Perceptrons ( ⃗ W -perceptrons).

Original: A ⃗ W -perceptron unit is a function mapping from u
Revised: A ⃗ W -perceptron unit is a function mapping from u

Original: = ( s , V ) to another scalar-vector tuple u ′ =
Revised: = ( s , V ) to another scalar-vector tuple u ′ =

Original: ( s ′ , V ′ ) , which can be stacked as network layers to form multilayer perceptrons.
Revised: ( s ′ , V ′ ) , which can be stacked as network layers to form multilayer perceptrons.

Original: A single unit comprises three modules in a sequential way: the Directed Linear module, the NonLinearity module, and the Directed Interaction module (Figure 2).
Revised: A single unit comprises three modules in a sequential way: the Directed Linear module, the NonLinearity module, and the Directed Interaction module (Figure 2).

Original: Directed linear module.
Revised: Directed linear module.

Original: The output hidden representations derived from the set of operations introduced previously are concatenated and projected into another hidden space of scalars and vectors separately:
Revised: The output hidden representations derived from the set of operations introduced previously are concatenated and projected into another hidden space of scalars and vectors separately:

Original: [Equation]
Revised: [Equation]

Original: Here W s ∈ R C ′ ×
Revised: Here W s ∈ R C ′ ×

Original: ( C ′ + C ′ )
Revised: ( C ′ + C ′ )

Original: and
Revised: and

Original: W V ∈ R C ′
Revised: W V ∈ R C ′

Original: × ( C ′ + C + C ′ ) are scalar weight matrices.
Revised: × ( C ′ + C + C ′ ) are scalar weight matrices.

Original: In other word, the five separate updating functions allow transformation from scalar to scalar (Equation 7), scalar to vector (Equation 6), vector to scalar (Equation 4), and vector to vector (Equation 8, Equation 5), boosting the model’s capability to reason in 3D space.
Revised: In other word, the five separate updating functions allow transformation from scalar to scalar (Equation 7), scalar to vector (Equation 6), vector to scalar (Equation 4), and vector to vector (Equation 8, Equation 5), boosting the model’s capability to reason in 3D space.

Original: Non-linearity module.
Revised: Non-linearity module.

Original: We then apply the non-linearity module to the hidden representation ( s h , V h ) .
Revised: We then apply the non-linearity module to the hidden representation ( s h , V h ) .

Original: Specifically, we apply standard ReLU non-linearity (Nair & Hinton, 2010) to the scalar components.
Revised: Specifically, we apply standard ReLU non-linearity (Nair & Hinton, 2010) to the scalar components.

Original: For the vector representations, following (Weiler et al., 2018; Jing et al., 2020; Sch¨utt et al., 2021), we compute a sigmoid activation on the L2-norm of each vector and multiply it back to the vector entries accordingly: 
Revised: For the vector representations, following (Weiler et al., 2018; Jing et al., 2020; Sch ¨ utt et al., 2021), we compute a sigmoid activation on the L2-norm of each vector and multiply it back to the vector entries accordingly: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where v hi ∈ R 3 are the vector columns in V h and v hij are their entries.
Revised: where v hi ∈ R 3 are the vector columns in V h and v hij are their entries.

Original: Directed interaction module.
Revised: Directed interaction module.

Original: Finally we introduce the Directed Interaction module, integrating the hidden features s h and V h into the output tuple ( s ′ , V ′ ) 
Revised: Finally we introduce the Directed Interaction module, integrating the hidden features s h and V h into the output tuple ( s ′ , V ′ ) [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here ⃗ W V , ⃗ W s are directed weight matrices with sizes C ′ × C ′ × 3 and C ′ × 3 , respectively, ⊙
Revised: Here ⃗ W V , ⃗ W s are directed weight matrices with sizes C ′ × C ′ × 3 and C ′ × 3 , respectively, ⊙

Original: de-
Revised: de-

Original: notes element-wise multiplication for two matrices.
Revised: notes element-wise multiplication for two matrices.

Original: This module establishes a connection between the scalar and vector feature components, facilitating feature blending.
Revised: This module establishes a connection between the scalar and vector feature components, facilitating feature blending.

Original: Specifically, Equationdynamically determines how much the output should rely on scalar and vector representations, and
Revised: Specifically, Equationdynamically determines how much the output should rely on scalar and vector representations, and

Original: Equation 13 weights a list of vectors using the scalar features as attention scores.
Revised: Equation 13 weights a list of vectors using the scalar features as attention scores.

Original:  O RIENTATION -A WARE G RAPH N EURAL N ETWORKS
Revised:  O RIENTATION -A WARE G RAPH N EURAL N ETWORKS

Original: To achieve SO(3) -equivariance without the loss of modeling capacity, we introduce an equivariant message passing paradigm on protein graphs, to easily plug our versatile ⃗ W -Perceptron into any existing graph learning framework, which makes network architectures free from equivariant constraints (Section 4.1).
Revised: To achieve SO(3) -equivariance without the loss of modeling capacity, we introduce an equivariant message passing paradigm on protein graphs, to easily plug our versatile ⃗ W -Perceptron into any existing graph learning framework, which makes network architectures free from equivariant constraints (Section 4.1).

Original: The integrated models, called Orientation-Aware Graph Neural Networks , can not only accurately model the essential orientational features but also maintain the rotation equivariance efficiently.
Revised: The integrated models, called Orientation-Aware Graph Neural Networks , can not only accurately model the essential orientational features but also maintain the rotation equivariance efficiently.

Original: We also design multiple variants in analogy to other graph neural networks
Revised: We also design multiple variants in analogy to other graph neural networks

Original: Section 4.2.
Revised: Section 4.2.

Original: 4. E QUIVARIANT M ESSAGE P ASSING ON P ROTEINS
Revised: 4. E QUIVARIANT M ESSAGE P ASSING ON P ROTEINS

Original: A function f : R 3 → R 3 is SO(3) -equivariant (rotation-equivariant) if any rotation matrix R ∈ R× 3 applied to the input vector x leads to the same transformation on the output f ( x ) :
Revised: A function f : R 3 → R 3 is SO(3) -equivariant (rotation-equivariant) if any rotation matrix R ∈ R 3 × 3 applied to the input vector x leads to the same transformation on the output f ( x ) :

Original: [Equation]
Revised: [Equation]

Original: Such equivariant property can be achieved on a protein graph with local orientations, which is naturally defined from its biological structure (Ingraham et al., 2019).
Revised: Such equivariant property can be achieved on a protein graph with local orientations, which is naturally defined from its biological structure (Ingraham et al., 2019).

Original: Specifically, each amino acid node u i ∈ V has four backbone heavy atom ( C uα , C u , N u , O u ), defining a local frame O u as:[Equation]
Revised: Specifically, each amino acid node u i ∈ V has four backbone heavy atom ( C uα , C u , N u , O u ), defining a local frame O u as:

Original: [Equation]
Revised: [Equation]

Original: The local frame O u is a rotation matrix that maps a 3D vector from the local to the global coordinate system.
Revised: The local frame O u is a rotation matrix that maps a 3D vector from the local to the global coordinate system.

Original: An equivariant message passing paradigm then emerges through transforming node features back and forth between adjacent local frames.
Revised: An equivariant message passing paradigm then emerges through transforming node features back and forth between adjacent local frames.

Original: Formally, give an amino acid u with hidden representation h
Revised: Formally, give an amino acid u with hidden representation h

Original: = ( s , V ) , let f l and f g be transformations on the vector feature V from and to the global coordinate system:
Revised: = ( s , V ) , let f l and f g be transformations on the vector feature V from and to the global coordinate system:

Original: [Equation]
Revised: [Equation]

Original: The message passing update for node u from layer l to layer l + 1 is performed in the following steps, the [ ] notations indicates the concatenation of different hidden representations along the channel-wise dimension:
Revised: The message passing update for node u from layer l to layer l + 1 is performed in the following steps, the [ ] notations indicates the concatenation of different hidden representations along the channel-wise dimension:

Original: Transform the neighbourhood vector representations of u into its local coordinate system O u . 2.
Revised: Transform the neighbourhood vector representations of u into its local coordinate system O u . 2.

Original: For each adjacent node v , compute the message m l +1 u on edge ( u, v )
Revised: For each adjacent node v , compute the message m l +1 u on edge ( u, v )

Original: ∈ E with an multilayer ⃗ W -perceptron F to the aggregated node and edge feature [ h lu , h lv , e luv ] (Equation 18).
Revised: ∈ E with an multilayer ⃗ W -perceptron F to the aggregated node and edge feature [ h lu , h lv , e luv ] (Equation 18).

Original: Update node feature h l u with another multi-layer ⃗ W -perceptron H , and transform it back to the global coordinate system (Equation 19).
Revised: Update node feature h lu with another multi-layer ⃗ W -perceptron H , and transform it back to the global coordinate system (Equation 19).

Original: This paradigm achieves SO(3) -equivariance in a very general sense with no constraints on F , H .
Revised: This paradigm achieves SO(3) -equivariance in a very general sense with no constraints on F , H .

Original: The formal proof of equivariance is presented in the Appendix Section A.
Revised: The formal proof of equivariance is presented in the Appendix Section A.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 4. V ARIANTS OF O RIENTATION -A WARE G RAPH N EURAL N ETWORKS
Revised: 4. V ARIANTS OF O RIENTATION -A WARE G RAPH N EURAL N ETWORKS

Original: By integrating the ⃗ W -Perceptrons with equivariant message passing, we propose multiple variants of entire Orientation-Aware Graph Neural Networks to boost the design space for different tasks.
Revised: By integrating the ⃗ W -Perceptrons with equivariant message passing, we propose multiple variants of entire Orientation-Aware Graph Neural Networks to boost the design space for different tasks.

Original: OA-GCN .
Revised: OA-GCN .

Original: This is the one described and implemented in the previous subsection with Equationand Equation 19.
Revised: This is the one described and implemented in the previous subsection with Equationand Equation 19.

Original: OA-GIN .
Revised: OA-GIN .

Original: We adopt graph isomorphism operator Xu et al.
Revised: We adopt graph isomorphism operator Xu et al.

Original: (2018) with learnable weighing parameter ε for tunable skip connections,
Revised: (2018) with learnable weighing parameter ε for tunable skip connections,

Original: [Equation]
Revised: [Equation]

Original: OA-GAT .
Revised: OA-GAT .

Original: In comparison to GCN and GIN, it is not trivial to incorporate with the Graph Attention
Revised: In comparison to GCN and GIN, it is not trivial to incorporate with the Graph Attention

Original: Network (GAT) (Veliˇckovi´c et al., 2017), as residues interact with each other unevenly in proteins.
Revised: Network (GAT) (Veliˇckovi´c et al., 2018), as residues interact with each other unevenly in proteins.

Original: In particular, we define separate attentions for the scalar and vector representations
Revised: In particular, we define separate attentions for the scalar and vector representations

Original: [Equation]
Revised: [Equation]

Original: The attention scores α s , α v are the softmax values over the inner products of all neighboring sourcetarget pairs defined as the follows:
Revised: The attention scores α s , α v are the softmax values over the inner products of all neighboring sourcetarget pairs defined as the follows:

Original: [Equation]
Revised: [Equation]

Original: The product for scalars and vectors are standard inner product and Frobenius product, respectively.
Revised: The product for scalars and vectors are standard inner product and Frobenius product, respectively.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To demonstrate the basic rationale in perceiving angular features of our ⃗ W -Perceptron, we design a synthetic task (Section 5.1).
Revised: To demonstrate the basic rationale in perceiving angular features of our ⃗ W -Perceptron, we design a synthetic task (Section 5.1).

Original: We then conduct experiments on multiple benchmarks, including node-level tasks:
Revised: We then conduct experiments on multiple benchmarks, including node-level tasks:

Original: Residue Identification ( RES ) (Section 5.2), Computational Protein Design ( CPD ) (Section 5.3), and graph-level tasks: Model Quality Assessment ( MQA ) (Section 5.4).
Revised: Residue Identification ( RES ) (Section 5.2), Computational Protein Design ( CPD ) (Section 5.3), and graph-level tasks: Model Quality Assessment ( MQA ) (Section 5.4).

Original: Ablation studies are also included (Section 5.5).
Revised: Ablation studies are also included (Section 5.5).

Original: More details are documented in the Appendix Section B.
Revised: More details are documented in the Appendix Section B.

Original: 5. S YNTHETIC T ASK
Revised: 5. S YNTHETIC T ASK

Original: [Figure]
Revised: [Figure]

Original: Datasets.
Revised: Datasets.

Original: We test the performance of different perceptron designs on a synthetic binary classification dataset with angular.
Revised: We test the performance of different perceptron designs on a synthetic binary classification dataset with angular.

Original: We sample a random unit vector v m ∈ R 3 as the anchor vector.
Revised: We sample a random unit vector v m ∈ R 3 as the anchor vector.

Original: Then each vector v i in the space is labeled positive if its angle between the anchor vector a mi ≤ π/ 4 , and negative otherwise.
Revised: Then each vector v i in the space is labeled positive if its angle between the anchor vector a mi ≤ π/ 4 , and negative otherwise.

Original: We generate 100 , 000 random vectors with 50% positive samples and 50% negative samples.
Revised: We generate 100 , 000 random vectors with 50% positive samples and 50% negative samples.

Original: For a fair comparison, we restrict the number of parameters and hyperparameters of different models to approximately the same.
Revised: For a fair comparison, we restrict the number of parameters and hyperparameters of different models to approximately the same.

Original: Results.
Revised: Results.

Original: As shown in Figure 3, in general, our ⃗ W -Perceptron ( DWP ) converges faster than other models and achieves the lowest validation loss.
Revised: As shown in Figure 3, in general, our ⃗ W -Perceptron ( DWP ) converges faster than other models and achieves the lowest validation loss.

Original: We also notice that the Geometric Vector Perceptron ( GVP ) cannot converge given a sufficient number of epochs due to the poor capability of perceiving angulars by only using linear combinations of vector features.
Revised: We also notice that the Geometric Vector Perceptron ( GVP ) cannot converge given a sufficient number of epochs due to the poor capability of perceiving angulars by only using linear combinations of vector features.

Original: In comparison to GVP, Vector Neural
Revised: In comparison to GVP, Vector Neural

Original: Network ( VNN ) performs better based on its non-linear ReLU operation for vector features.
Revised: Network ( VNN ) performs better based on its non-linear ReLU operation for vector features.

Original: Simple Multi-Layer Perceptron ( MLP ) can also capture useful information according to its universal approximation ability.
Revised: Simple Multi-Layer Perceptron ( MLP ) can also capture useful information according to its universal approximation ability.

Original: The superior performance demonstrates that our ⃗ W -Perceptron has a better ability on perceiving potential geometric features in space.
Revised: The superior performance demonstrates that our ⃗ W -Perceptron has a better ability on perceiving potential geometric features in space.

Original: 5. R ESIDUE I DENTIFICATION
Revised: 5. R ESIDUE I DENTIFICATION

Original: Datasets.
Revised: Datasets.

Original: Residue Identification (RES) aims to predict the identity of particular masked amino acid based on its local surrounding structure (Torng & Altman, 2017).
Revised: Residue Identification (RES) aims to predict the identity of particular masked amino acid based on its local surrounding structure (Torng & Altman, 2017).

Original: We download 1000 , 000 ( 10 6 ) substructures from the ATOM3D project (Townshend et al., 2020), which are originally derived from 574 proteins from the PDB (Berman et al., 2000).
Revised: We download 1000 , 000 ( 10 6 ) substructures from the ATOM3D project (Townshend et al., 2021), which are originally derived from 574 proteins from the PDB (Berman et al., 2000).

Original: The entire dataset is split into training, validation, and testing datasets with the ratio of 80% , 10% , and 10% .
Revised: The entire dataset is split into training, validation, and testing datasets with the ratio of 80% , 10% , and 10% .

Original: There are no two proteins with similar structures in the test and non-test datasets based on the CATH 4.2 topology class at the domain level (Orengo et al., 1997).
Revised: There are no two proteins with similar structures in the test and non-test datasets based on the CATH 4.2 topology class at the domain level (Orengo et al., 1997).

Original: Metrics.
Revised: Metrics.

Original: We use classification accuracy to evaluate model performance.
Revised: We use classification accuracy to evaluate model performance.

Original: Baselines.
Revised: Baselines.

Original: The 3D-CNN encodes the positions of the atoms in a voxelized 3D volume.
Revised: The 3D-CNN encodes the positions of the atoms in a voxelized 3D volume.

Original: We also compare GNN variants such as GCN , GIN and GAT with our OA-GCN , OA-GIN and OA-GAT models.
Revised: We also compare GNN variants such as GCN , GIN and GAT with our OA-GCN , OA-GIN and OA-GAT models.

Original: We also include GVP-GNN as well.
Revised: We also include GVP-GNN as well.

Original: Results As shown in Table 1, GCN , GIN and GAT almost cannot accurately identify the type of amino acids, suggesting the lack of ability of conventional GNN models to capture 3D geometric information.
Revised: Results As shown in Table 1, GCN , GIN and GAT almost cannot accurately identify the type of amino acids, suggesting the lack of ability of conventional GNN models to capture 3D geometric information.

Original: 3D-CNN performs better due to the power of the voxelization technique.
Revised: 3D-CNN performs better due to the power of the voxelization technique.

Original: Our models outperform other models, suggesting OA-GNNs could help represent protein substructure geometries better.
Revised: Our models outperform other models, suggesting OA-GNNs could help represent protein substructure geometries better.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Datasets.
Revised: Datasets.

Original: Computational Protein Design (CPD) predicts the native protein sequence of a given backbone structure.
Revised: Computational Protein Design (CPD) predicts the native protein sequence of a given backbone structure.

Original: Specifically, we focus on two databases: CATH 4.2 (Ingraham et al., 2019) organizes proteins in a hierarchical structure (Orengo et al., 1997).
Revised: Specifically, we focus on two databases: CATH 4.2 (Ingraham et al., 2019) organizes proteins in a hierarchical structure (Orengo et al., 1997).

Original: Following prior works, there are 18 , 024 protein chains in the training set, 608 in the validation set
Revised: Following prior works, there are 18 , 024 protein chains in the training set, 608 in the validation set

Original: [Table]
Revised: [Table]

Original: and 1 , 120 in the test.
Revised: and 1 , 120 in the test.

Original: TS dataset (Li et al., 2014) is a relatively old benchmark consisting of only 50 protein structures widely used in biology communities.
Revised: TS dataset (Li et al., 2014) is a relatively old benchmark consisting of only 50 protein structures widely used in biology communities.

Original: Metrics.
Revised: Metrics.

Original: Native Sequence Recovery Rate is adopted to evaluate the prediction performance
Revised: Native Sequence Recovery Rate is adopted to evaluate the prediction performance

Original: [Table]
Revised: [Table]

Original: which compares the predicted sequence with the ground truth sequence at each position and calculates the proportion of the correctly recovered amino acids (Li et al., 2014).
Revised: which compares the predicted sequence with the ground truth sequence at each position and calculates the proportion of the correctly recovered amino acids (Li et al., 2014).

Original: The perplexity score (Jelinek et al., 1977) evaluates whether a model could assign a high likelihood to the test sequences (Ingraham et al., 2019).
Revised: The perplexity score (Jelinek et al., 1977) evaluates whether a model could assign a high likelihood to the test sequences (Ingraham et al., 2019).

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Baselines.
Revised: Baselines.

Original: For the CATH 4.2 dataset, we compare our models with other state-of-the-art models, including GVP-GNN , Structured Transformer and Structured GNN (Ingraham et al., 2019).
Revised: For the CATH 4.2 dataset, we compare our models with other state-of-the-art models, including GVP-GNN , Structured Transformer and Structured GNN (Ingraham et al., 2019).

Original: We compare multiple machine learning approaches in the TS50 dataset, including 3D CNN-based methods ( ProDConn (Zhang et al., 2020), DenseCPD (Qi & Zhang, 2020)), sequential models ( Wang’s model Wang et al.
Revised: We compare multiple machine learning approaches in the TS50 dataset, including 3D CNN-based methods ( ProDConn (Zhang et al., 2020), DenseCPD (Qi & Zhang, 2020)), sequential models ( Wang’s model Wang et al.

Original: , SPIN (Li et al., 2014), SPIN2 (O’Connell et al., 2018)) and GNN-based method (Strokach et al., 2020).
Revised: , SPIN (Li et al., 2014), SPIN2 (O’Connell et al., 2018)) and GNN-based method (Strokach et al., 2020).

Original: We also compare with classical statistic methods (Schaap et al., 2001; Cheng et al., 2019).
Revised: We also compare with classical statistic methods (Schaap et al., 2001; Cheng et al., 2019).

Original: Results.
Revised: Results.

Original: As shown in Table 3, our models achieve significant improvement over other methods on the CATH dataset.
Revised: As shown in Table 3, our models achieve significant improvement over other methods on the CATH dataset.

Original: This result also suggests that the the short-chain and single-chain subsets are more challenging, which is consistent with the result in (Ingraham et al., 2019).
Revised: This result also suggests that the the short-chain and single-chain subsets are more challenging, which is consistent with the result in (Ingraham et al., 2019).

Original: As shown in Table 5, our models also gain superior performance.
Revised: As shown in Table 5, our models also gain superior performance.

Original: These results suggest that adopting OA-GNNs is a more effective way of designing valid protein sequences.
Revised: These results suggest that adopting OA-GNNs is a more effective way of designing valid protein sequences.

Original: 5. M ODEL Q UALITY A SSESSMENT
Revised: 5. M ODEL Q UALITY A SSESSMENT

Original: Datasets.
Revised: Datasets.

Original: Model quality assessment (MQA) evaluates the quality of predicted protein structures (Kwon et al., 2021).
Revised: Model quality assessment (MQA) evaluates the quality of predicted protein structures (Kwon et al., 2021).

Original: The goal is to fit a model approximating the numerical metric used to compare the predicted 3D structure with the native 3D structure.
Revised: The goal is to fit a model approximating the numerical metric used to compare the predicted 3D structure with the native 3D structure.

Original: For this task, We randomly split the targets in CASP5-CASP10 (Moult et al., 2014) and sample 50 decoys for each target for generating the training and validation sets and use the CASP11 stage 2 proteins as the test set to ensure no similar structures are involved.
Revised: For this task, We randomly split the targets in CASP5-CASP10 (Moult et al., 2014) and sample 50 decoys for each target for generating the training and validation sets and use the CASP11 stage 2 proteins as the test set to ensure no similar structures are involved.

Original: In total, there are 508 proteins for training, 56 for validation, and 85 targets with 150 decoys for the test.
Revised: In total, there are 508 proteins for training, 56 for validation, and 85 targets with 150 decoys for the test.

Original: Metrics.
Revised: Metrics.

Original: We regress the global GDT TS score (Zemla, 2003), which evaluates the quality of decoys.
Revised: We regress the global GDT TS score (Zemla, 2003), which evaluates the quality of decoys.

Original: We adopt the correlation between the predicted and real GDT TS values to measure the performance.
Revised: We adopt the correlation between the predicted and real GDT TS values to measure the performance.

Original: We use three statistical correlation metrics: Pearson’s correlation r , Spearman’s ρ , and Kendall’s τ .
Revised: We use three statistical correlation metrics: Pearson’s correlation r , Spearman’s ρ , and Kendall’s τ .

Original: We calculate the correlation for each target and average all the correlations over all the targets.
Revised: We calculate the correlation for each target and average all the correlations over all the targets.

Original: We also calculate the Global correlations by taking the union of all decoy sets without considering the targets (Pag`es et al., 2019).
Revised: We also calculate the Global correlations by taking the union of all decoy sets without considering the targets (Pag`es et al., 2019).

Original: Baselines.
Revised: Baselines.

Original: VoroMQA (Olechnoviˇc & Venclovas, 2017) leverages potential model with protein contact maps, while RWplus (Zhang & Zhang, 2010) relies on physical energy terms.
Revised: VoroMQA (Olechnoviˇc & Venclovas, 2017) leverages potential model with protein contact maps, while RWplus (Zhang & Zhang, 2010) relies on physical energy terms.

Original: SBROD (Karasikov et al., 2019) uses hand-crafted features.
Revised: SBROD (Karasikov et al., 2019) uses hand-crafted features.

Original: Proq3D (Uziela et al., 2017) employs a FCNs for regression.
Revised: Proq3D (Uziela et al., 2017) employs a FCNs for regression.

Original: 3DCNN (Derevyanko et al., 2018) and Ornate (Pag`es et al., 2019) apply 3D CNNs for extracting meaningful features.
Revised: 3DCNN (Derevyanko et al., 2018) and Ornate (Pag`es et al., 2019) apply 3D CNNs for extracting meaningful features.

Original: GraphQA (Baldassarre et al., 2021), DimeNet (Klicpera et al., 2020) and GVP-GNN are the most similar models as ours which adopted GNN-based methods on protein graphs.
Revised: GraphQA (Baldassarre et al., 2021), DimeNet (Gasteiger et al., 2019b) and GVP-GNN are the most similar models as ours which adopted GNN-based methods on protein graphs.

Original: Results.
Revised: Results.

Original: As shown in table 4, our models outperform other methods on both average and global metrics except for Spearman’s correlation ρ where we achieve the second-best, demonstrating our models can not only evaluate the quality for the same protein but also works well across different proteins in a more general way.
Revised: As shown in table 4, our models outperform other methods on both average and global metrics except for Spearman’s correlation ρ where we achieve the second-best, demonstrating our models can not only evaluate the quality for the same protein but also works well across different proteins in a more general way.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: In the end, we conduct ablation experiments to study the contribution of different modules of our model on the tasks of CPD (table 3, 5) and RES (table 2).
Revised: In the end, we conduct ablation experiments to study the contribution of different modules of our model on the tasks of CPD (table 3, 5) and RES (table 2).

Original: More specifically, based on OA-GCN, we replace all the directed weights with the regular linear layer with scalar weights ( No DW ), or remove the Interaction Module ( No Int ), or break the equivariance by canceling coordinate transformations in the step of message passing ( No Equi ).
Revised: More specifically, based on OA-GCN, we replace all the directed weights with the regular linear layer with scalar weights ( No DW ), or remove the Interaction Module ( No Int ), or break the equivariance by canceling coordinate transformations in the step of message passing ( No Equi ).

Original: In general, all three components of our model are important since removing each of them results in a significant performance drop.
Revised: In general, all three components of our model are important since removing each of them results in a significant performance drop.

Original: According to the performance gain, the directed weights are the most important module for the CPD task, as the orientational features are essential for the protein to fold to a given structure.
Revised: According to the performance gain, the directed weights are the most important module for the CPD task, as the orientational features are essential for the protein to fold to a given structure.

Original: For Residue Identity, however, equivariance message passing is the most important component, indicating the necessity of rotation equivariance in learning the representation of local protein structures.
Revised: For Residue Identity, however, equivariance message passing is the most important component, indicating the necessity of rotation equivariance in learning the representation of local protein structures.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: For the first time, we generalize the scalar weights in neural networks to 3D directed vectors for better perceiving geometric features like orientations and angles, which are essential in learning representations from protein 3D structures.
Revised: For the first time, we generalize the scalar weights in neural networks to 3D directed vectors for better perceiving geometric features like orientations and angles, which are essential in learning representations from protein 3D structures.

Original: Based on the directed weights ⃗ W , we provide a toolbox of operators as well as a ⃗ W -Perceptron Unit encouraging efficient scalar-vector feature interactions.
Revised: Based on the directed weights ⃗ W , we provide a toolbox of operators as well as a ⃗ W -Perceptron Unit encouraging efficient scalar-vector feature interactions.

Original: We also enforce global SO(3) -equivariance on a message passing paradigm using the local orientations naturally defined by the rigid property of each amino acid residue, which can be used for designing more powerful networks.
Revised: We also enforce global SO(3) -equivariance on a message passing paradigm using the local orientations naturally defined by the rigid property of each amino acid residue, which can be used for designing more powerful networks.

Original: Our integrated Orientation-Aware Graph Neural Networks achieve comparably better performances on multiple biological tasks in comparison with existing state-of-the-art methods.
Revised: Our integrated Orientation-Aware Graph Neural Networks achieve comparably better performances on multiple biological tasks in comparison with existing state-of-the-art methods.

Original: Limitations and future work.
Revised: Limitations and future work.

Original: While our method shows better awareness to fine geometric details, there remains a huge space for non-linearity designs upon geometric representations.
Revised: While our method shows better awareness to fine geometric details, there remains a huge space for non-linearity designs upon geometric representations.

Original: In the future, we will extend our framework to other 3D learning tasks like small molecules or point clouds by learning local rotation transformations and using directed vector weights.
Revised: In the future, we will extend our framework to other 3D learning tasks like small molecules or point clouds by learning local rotation transformations and using directed vector weights.

Original: A P ROOF OF R OTATION E QUIVARIANCE
Revised: A P ROOF OF R OTATION E QUIVARIANCE

Original: A function f taking a 3D vector x ∈ R 3 as input is rotation equivariance, if applying ant rotation matrix R ∈ R 3 × 3 on x leads to the same transformations of the output f ( x ) .
Revised: A function f taking a 3D vector x ∈ R 3 as input is rotation equivariance, if applying ant rotation matrix R ∈ R 3 × 3 on x leads to the same transformations of the output f ( x ) .

Original: Formally, f : R 3 → Ris rotation equivariance by fulfilling:
Revised: Formally, f : R 3 → R 3 is rotation equivariance by fulfilling:

Original: [Equation]
Revised: [Equation]

Original: For notation consistency, we consider each row of the matrix to be an individual 3D vector and use right-hand matrix multiplication here.
Revised: For notation consistency, we consider each row of the matrix to be an individual 3D vector and use right-hand matrix multiplication here.

Original: When performing equivariant message passing on protein graph for node i with local rotation matrix
Revised: When performing equivariant message passing on protein graph for node i with local rotation matrix

Original: O i , we first transform the vector representations of its neighbors from global reference to its local reference, that is, for a particular neighbor node j with vector feature V j ∈ R C × 3 , apply our DWNN layers f , and transform the updated features back to the global reference.
Revised: O i , we first transform the vector representations of its neighbors from global reference to its local reference, that is, for a particular neighbor node j with vector feature V j ∈ R C × 3 , apply our DWNN layers f , and transform the updated features back to the global reference.

Original: We set only one neighbor node u j for u i for simplicity.
Revised: We set only one neighbor node u j for u i for simplicity.

Original: The updated vector representation V i for node i is
Revised: The updated vector representation V i for node i is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If we apply a global rotation matrix R to all vectors in the global frame, the local rotation matrix O i will be transformed to O ′ i
Revised: If we apply a global rotation matrix R to all vectors in the global frame, the local rotation matrix O i will be transformed to O ′ i

Original: = O i R , and V j
Revised: = O i R , and V j

Original: to be V ′ j = V j R , now the output V
Revised: to be V ′ j = V j R , now the output V

Original: i is
Revised: i is

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: And if instead, we directly apply the rotation matrix to original output V i , we got
Revised: And if instead, we directly apply the rotation matrix to original output V i , we got

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In other words, rotating the input leads to the same transformations to output, so we can preserve rotation equivariance for vector representations in this way.
Revised: In other words, rotating the input leads to the same transformations to output, so we can preserve rotation equivariance for vector representations in this way.

Original: Note that scalar features remain invariant when applying global rotation, so our message-passing paradigm with global and local coordinate transformations is rotation equivariance.
Revised: Note that scalar features remain invariant when applying global rotation, so our message-passing paradigm with global and local coordinate transformations is rotation equivariance.

Original: B E XPERIMENT D ETAILS
Revised: B E XPERIMENT D ETAILS

Original: We represent a protein 3D structure as an attributed graph, with each node and edge attached with scalar and vector features that are geometric-aware.
Revised: We represent a protein 3D structure as an attributed graph, with each node and edge attached with scalar and vector features that are geometric-aware.

Original: We implement our DWNN in the equivariant message passing manner, with 3 layer Directed Weight Perceptrons for all tasks.
Revised: We implement our DWNN in the equivariant message passing manner, with 3 layer Directed Weight Perceptrons for all tasks.

Original: B. P ROTEIN F EATURES
Revised: B. P ROTEIN F EATURES

Original: In this paper, we use an attributed graph G =
Revised: In this paper, we use an attributed graph G =

Original: ( V , E ) to represent the protein structure, where each node corresponds to one particular amino acid in the protein with edges connecting its k-nearest neighbors.
Revised: ( V , E ) to represent the protein structure, where each node corresponds to one particular amino acid in the protein with edges connecting its k-nearest neighbors.

Original: Here we set k = 30 .
Revised: Here we set k = 30 .

Original: The node features V = { v 1 , .., v N } and edge features E = { e ij } i ̸
Revised: The node features V = { v 1 , .., v N } and edge features E = { e ij } i ̸

Original: =
Revised: =

Original: j are both multi-channel scalar-vector tuples with scalar features like distances and dihedral angles and vector features like unit vectors representing particular orientations.
Revised: j are both multi-channel scalar-vector tuples with scalar features like distances and dihedral angles and vector features like unit vectors representing particular orientations.

Original: A node v
Revised: A node v

Original: i represents the i -th residue in the protein with scalar and vector features describing its geometric and chemical properties if available.
Revised: i represents the i -th residue in the protein with scalar and vector features describing its geometric and chemical properties if available.

Original: Therefore, a node in this graph may have multichannel scalar-vector tuples ( s i , V i ) , s
Revised: Therefore, a node in this graph may have multichannel scalar-vector tuples ( s i , V i ) , s

Original: i ∈
Revised: i ∈

Original: R 6 or R 26 , V i ∈ R 3 × 3 as its initial features.
Revised: R 6 or R 26 , V i ∈ R 3 × 3 as its initial features.

Original: • Scalar Feature.
Revised: • Scalar Feature.

Original: The { sin , cos } ◦ { ψ, ω, ϕ } .
Revised: The { sin , cos } ◦ { ψ, ω, ϕ } .

Original: Here { ψ, ω, ϕ } are dihedral angles computed from its four backbone atom positions, Cα i − 1 , N i , Cα i , N i +1 .
Revised: Here { ψ, ω, ϕ } are dihedral angles computed from its four backbone atom positions, Cα i − 1 , N i , Cα i , N i +1 .

Original: • Scalar Feature.
Revised: • Scalar Feature.

Original: A one-hot representation of residue if the identity is available.
Revised: A one-hot representation of residue if the identity is available.

Original: • Vector Feature.
Revised: • Vector Feature.

Original: The unit vectors in the directions of Cα i +1 − Cα i and Cα i − 1 − Cα i .
Revised: The unit vectors in the directions of Cα i +1 − Cα i and Cα i − 1 − Cα i .

Original: • Vector Feature.
Revised: • Vector Feature.

Original: The unit vector in the direction of Cβ i − Cα i corresponds to the side-chain directions.
Revised: The unit vector in the direction of Cβ i − Cα i corresponds to the side-chain directions.

Original: The edge e ij connecting the i -th residue and the j -th residue also has multi-channel scalar-vector tuples as its feature ( s ij , V ij ) , s ij ∈ R 34 , V ij ∈ R 1 ×
Revised: The edge e ij connecting the i -th residue and the j -th residue also has multi-channel scalar-vector tuples as its feature ( s ij , V ij ) , s ij ∈ R 34 , V ij ∈ R 1 ×

Original: • Scalar Feature.
Revised: • Scalar Feature.

Original: The encoding of the distance ∥
Revised: The encoding of the distance ∥

Original: Cα j − Cα i ∥ using 16 Gaussian radial basis functions with centers spaced between 0 to 20 angstroms.
Revised: Cα j − Cα i ∥ using 16 Gaussian radial basis functions with centers spaced between 0 to 20 angstroms.

Original: • Scalar Feature.
Revised: • Scalar Feature.

Original: The positional encoding of j − i corresponding the relative position in the protein sequence.
Revised: The positional encoding of j − i corresponding the relative position in the protein sequence.

Original: • Scalar Feature:
Revised: • Scalar Feature:

Original: The contact signal describes if the two residues contact in the space, 1 if ∥ Cα j − Cα j ∥ ≤ 8 and 0 otherwise.
Revised: The contact signal describes if the two residues contact in the space, 1 if ∥ Cα j − Cα j ∥ ≤ 8 and 0 otherwise.

Original: • Scalar Feature.
Revised: • Scalar Feature.

Original: The H-bond signal describes if there may be a H-bond between the two nodes calculated by backbone distance.
Revised: The H-bond signal describes if there may be a H-bond between the two nodes calculated by backbone distance.

Original: • Vector Feature.
Revised: • Vector Feature.

Original: The unit vector in the direction of Cα j − Cα i .
Revised: The unit vector in the direction of Cα j − Cα i .

Original: B.M ODEL D ETAILS
Revised: B.

Revised: M ODEL D ETAILS

Original: Synthetic Task.
Revised: Synthetic Task.

Revised: We uniformly sample points on the sphere and the ratio of positive and negative samples is 1:1.

Original: We consider the length of each vector as one channel scalar feature ∈ R and the vector itself as one channel vector feature ∈ R 1 × 3 .
Revised: We consider the length of each vector as one channel scalar feature ∈ R and the vector itself as one channel vector feature ∈ R 1 × 3 .

Original: The VNN and GVP models in this experiment are set to 3 layers.
Revised: The VNN and GVP models in this experiment are set to 3 layers.

Original: And our DWP is only 1 layer.
Revised: And our DWP is only 1 layer, consisting 1 Directed Linear Module, 1 NonLinear Module, and 1 Directed Interaction Module.

Original: We also train a 3-layer MLP by concatenating the scalar and vector feature as a four-channel scalar feature ∈ R 4 as input.
Revised: We also train a 3-layer MLP by concatenating the scalar and vector feature as a four-channel scalar feature ∈ R 4 as input.

Original: Specifically, the parameter counts of VNN, GVP, MLP, DWP is 24 , 28 , 24 and 24 .
Revised: Specifically, the parameter counts of VNN, GVP, MLP, DWP is 24 , 28 , 24 and 24 .

Original: We use Binary CrossEntropy Loss for this two-class classification task.
Revised: We use Binary CrossEntropy Loss for this two-class classification task.

Original: CPD.
Revised: For MLP, the scalar and vector features are concatenated along the channel, which means the feature of each point is a vector in R 4 . For GVP and VNN, the input scalar and vector are represented by separate neural network components. For all models we trained in the protein-related task, we use 128 scalar channels and 32 vector channels for each node’s hidden representations, 64 scalar channels, and 16 vector channels for each edge’s hidden representations. There is 4 message passing updations in implemented models, where each passing layer consists of 3 stacked DWPs.

Original:  We train our model in a BERT-style recovery target (Strokach et al., 2020), more specifically, we dynamically mask 85% residues of the whole protein and let the model recover them using structure information from its neighbors using CrossEntropy loss.
Revised: In total there are about 2 . 9 million parameters of our models. CPD. We train our model in a BERT-style recovery target (Strokach et al., 2020), more specifically, we dynamically mask 85% residues of the whole protein and let the model recover them using structure information from its neighbors using CrossEntropy loss.

Original: During testing, we mask all the residues and recover the whole protein sequence in one forward pass, we also try to iteratively predict one residue per forwarding pass.
Revised: During testing, we mask all the residues and recover the whole protein sequence in one forward pass, we also try to iteratively predict one residue per forwarding pass.

Original: MQA.
Revised: MQA.

Original: Because MQA is a regression task, we train our model using MSE loss.
Revised: Because MQA is a regression task, we train our model using MSE loss.

Original: When testing, we compute the different types of correlations between predicted scores and ground truth.
Revised: When testing, we compute the different types of correlations between predicted scores and ground truth.

Original: RES.
Revised: RES.

Original: Residue identity requires the model to classify the center residue from a local substructure of the protein.
Revised: Residue identity requires the model to classify the center residue from a local substructure of the protein.

Original: We construct a subgraph of each local substructure according to Section B.1.
Revised: We construct a subgraph of each local substructure according to Section B.1.

Original: To guarantee equitable comparison, we retrain all the methods based on our protein graph from scratch.
Revised: To guarantee equitable comparison, we retrain all the methods based on our protein graph from scratch.

Original: B. T RAINING D ETAILS
Revised: B. T RAINING D ETAILS

Original: For the synthetic task, we train each model for 1000 epochs with a learning rate of 1 e − 3 and plot the training loss for evaluation.
Revised: For the synthetic task, we train each model for 1000 epochs with a learning rate of 1 e − 3 and plot the training loss for evaluation.

Original: For all models we trained, we use 128 scalar channels and 32 vector channels for each node’s hidden representations, 64 scalar channels, and 16 vector channels for each edge’s hidden representations.

Original: There is 4 message passing updations in implemented models, where each passing layer consists of 3-layer networks.

Original: In total there are about 3 .

Original: 5 to 4 million parameters of our models.

Original: We train our models with learning rate 3 e − 4 for CPD and 2 e − 4 for MQA, a dropout rate of 10% (Srivastava et al., 2014), and Layernorm paradigm.
Revised: We train our models with learning rate 3 e − 4 for CPD and 2 e − 4 for MQA, a dropout rate of 10% (Srivastava et al., 2014), and Layernorm paradigm.

Original: We train all the models on NVIDIA Tesla V100 forepochs for each task.
Revised: We train all the models on NVIDIA Tesla V for 100 epochs for each task.

Original: Our model uses 4-layer message passing, where each message passing consists of 3-layer DWP.
Revised: Our model uses 4-layer message passing, where each message passing consists of 3-layer DWP.

Original: The total model consists of about 2.9 million parameters.
Revised: The total model consists of about 2.9 million parameters.

Original: B. B ASELINE D ETAILS
Revised: B. B ASELINE D ETAILS

Original: We obtain most of our baseline numerical results from other benchmark papers.
Revised: We obtain most of our baseline numerical results from other benchmark papers.

Original: But because some models haven’t been evaluated on some of the datasets we used, we reimplement them following the same model complexity and training hyperparameters as our models.
Revised: But because some models haven’t been evaluated on some of the datasets we used, we reimplement them following the same model complexity and training hyperparameters as our models.

Original: In the synthetic dataset , we download the source code from (Deng et al., 2021) and (Jing et al., 2021), both of which use CrossEntropy loss.
Revised: In the synthetic dataset , we download the source code from (Deng et al., 2021) and (Jing et al., 2021), both of which use CrossEntropy loss.

Original: The number of parameters of VNN, GVP, MLP, and
Revised: The number of parameters of VNN, GVP, MLP, and

Original: DWP is 24 , 28 , 24 , and 24 , respectively.
Revised: DWP is 24 , 28 , 24 , and 24 , respectively.

Original: For the RES task Table 1 , we implement GCN, GIN, and GAT models, which are trained and evaluated with the same settings as our models.
Revised: For the RES task Table 1 , we implement GCN, GIN, and GAT models, which are trained and evaluated with the same settings as our models.

Original: Following the source code from (Townshend et al., 2020; Jing et al., 2021), we also retrain 3D CNN and GVP-GNN.
Revised: Following the source code from (Townshend et al., 2021; Jing et al., 2021), we also retrain 3D CNN and GVP-GNN.

Original: To make a fair comparison, we keep the model complexity (e.g. hidden dimensions, layer numbers) and hyperparameters (epochs, random seed) of these baselines the same as our models.
Revised: To make a fair comparison, we keep the model complexity (e.g. hidden dimensions, layer numbers) and hyperparameters (epochs, random seed) of these baselines the same as our models.

Original: We also include the original GVP performance, denoted as GV P ∗ in 2.

Original: For the CPD task in Table 3 , we report the result of St-Transformer from (Ingraham et al., 2019),
Revised: For the CPD task in Table 3 , we report the result of St-Transformer from (Ingraham et al., 2019),

Original: St-GCN, and GVP-GNN from (Jing et al., 2020).
Revised: St-GCN, and GVP-GNN from (Jing et al., 2020).

Original: We modify the source code from StructedTransformer(Ingraham et al., 2019) to get the result of St-GAT and St-GIN by replacing the original attention layer in the encoder and decoder module with Graph Attention Network and Graph Isomorphism Network.
Revised: We modify the source code from StructedTransformer(Ingraham et al., 2019) to get the result of St-GAT and St-GIN by replacing the original attention layer in the encoder and decoder module with Graph Attention Network and Graph Isomorphism Network.

Original: The hidden dimensions and layer numbers are kept the same as our models.
Revised: The hidden dimensions and layer numbers are kept the same as our models.

Original: In Table 5 , we adopt most of the results from (Jing et al., 2020) except running the St-Transformer model on the TS50 dataset.
Revised: In Table 5 , we adopt most of the results from (Jing et al., 2020) except running the St-Transformer model on the TS50 dataset.

Original: For MQA in Table 4 , we merge the results of the same task from (Jing et al., 2020) and StructedTransformer (Ingraham et al., 2019).
Revised: For MQA in Table 4 , we merge the results of the same task from (Jing et al., 2020) and StructedTransformer (Ingraham et al., 2019).

Original: As GVP-GNN only reports three of six of our metrics, so we reimplement GVP-GNN and DimeNet on our benchmark.
Revised: As GVP-GNN only reports three of six of our metrics, so we reimplement GVP-GNN and DimeNet on our benchmark.

Original: Same as the above model, the hidden dimensions and layer numbers are kept the same as our models.
Revised: Same as the above model, the hidden dimensions and layer numbers are kept the same as our models.

Original: For our ablation models, we concatenate scalar and vector features as input and replace all the directed weights with the regular linear layer with scalar weights to get the No DW model.
Revised: For our ablation models, we concatenate scalar and vector features as input and replace all the directed weights with the regular linear layer with scalar weights to get the No DW model.

Original: By removing the Interaction Module lying in the final layer of each perceptron, we obtain the No Int model.
Revised: By removing the Interaction Module lying in the final layer of each perceptron, we obtain the No Int model.

Original: We also replace local frames with unity matrices to break the equivariance in the message passing part (No equivariance).
Revised: We also replace local frames with unity matrices to break the equivariance in the message passing part (No equivariance).

Original: In the FOLD and REACT tasks, Kipf et al.
Revised: In the FOLD and REACT tasks, Kipf et al.

Original: and Diehl et al.
Revised: and Diehl et al.

Original: construct the input graph using the protein’s contact map and use pair-wise distances of residues as edge features.
Revised: construct the input graph using the protein’s contact map and use pair-wise distances of residues as edge features.

Original: Baldassarre et al.
Revised: Baldassarre et al.

Original: use the spatial features including the dihedral angles, surface accessibility, and secondary structure type of each residue’s node feature.
Revised: use the spatial features including the dihedral angles, surface accessibility, and secondary structure type of each residue’s node feature.

Original: They also use distances, sequence distances, and bond types for edge features.
Revised: They also use distances, sequence distances, and bond types for edge features.

Original: Hermosilla et al. consider more fine-grained atom-level spatial information, e.g. covalent radius, van der Waals radius, and atom mass, which leads to better performance.
Revised: Hermosilla et al. consider more fine-grained atom-level spatial information, e.g. covalent radius, van der Waals radius, and atom mass, which leads to better performance.

Original: Gligorijevi et al.
Revised: Gligorijevi et al.

Original: refer to different types of contact maps and geometric distances for the input of GNN.
Revised: refer to different types of contact maps and geometric distances for the input of GNN.

Original: All GNN and
Revised: All GNN and

Original: CNN baselines make use of 3D information in proteins.
Revised: CNN baselines make use of 3D information in proteins.

Original: C A DDITIONAL R ESULTS C. M ULTIPLE R UNS ON RES T ASK
Revised: C A DDITIONAL R ESULTS C. M ULTIPLE R UNS ON RES T ASK

Original: We conduct different runs on RES dataset for both baseline and our models.
Revised: We conduct different runs on RES dataset for both baseline and our models.

Original: As shown in Table 6, we report the mean and std results of each method across five different random seeds, our models are still doing best among other methods and have stable, robust performance with few deviations.
Revised: As shown in Table 6, we report the mean and std results of each method across five different random seeds, our models are still doing best among other methods and have stable, robust performance with few deviations.

Original: [Table]
Revised: [Table]

Original: C. I TERATIVE P REDICTION ON CPD T ASK
Revised: C. I TERATIVE P REDICTION ON CPD T ASK

Original: We tested the sequential decoding of the prediction from the N-side to the C-side and reported the native sequence recovery as shown in Table 7, 
Revised: We tested the sequential decoding of the prediction from the N-side to the C-side and reported the native sequence recovery as shown in Table 7, [Table]

Original: The results have shown that the sequential decoding approach basically caused a degradation of the results, but because we introduced the directed weight and geometric operations, our results are still better than other benchmarks.
Revised: The results have shown that the sequential decoding approach basically caused a degradation of the results, but because we introduced the directed weight and geometric operations, our results are still better than other benchmarks.

Original: We have added these results in the supplementary material.
Revised: We have added these results in the supplementary material.

Original: [Table]

Original: C. P ROTEIN F UNCTION C LASSIFICATION
Revised: C. P ROTEIN F UNCTION C LASSIFICATION

Original: Datasets.
Revised: Datasets.

Original: Fold Classification (FOLD) predicts the structure category of a given protein structure.
Revised: Fold Classification (FOLD) predicts the structure category of a given protein structure.

Original: We choose the SCOP v1.75 dataset collected by Murzin et al.
Revised: We choose the SCOP v1.75 dataset collected by Murzin et al.

Original: (1995) that organizes structures into 3-layer hierarchical classes.
Revised: (1995) that organizes structures into 3-layer hierarchical classes.

Original: In total, there are 16 , 712 proteins covering 7 major structural types with 1 , 195 identified folds.
Revised: In total, there are 16 , 712 proteins covering 7 major structural types with 1 , 195 identified folds.

Original: We adopt a reduced dataset from Hou et al.
Revised: We adopt a reduced dataset from Hou et al.

Original: (2018), and remove homologous sequences between test and training data sets at Family ( Fam ), Superfamily ( Sup ) or Fold levels, resulting in three different classification tasks.
Revised: (2018), and remove homologous sequences between test and training data sets at Family ( Fam ), Superfamily ( Sup ) or Fold levels, resulting in three different classification tasks.

Original: The Enzyme-Catalyzed Reaction Classification (REACT) study is a very similar task as it requires to classify the EC number Webb et al.
Revised: The Enzyme-Catalyzed Reaction Classification (REACT) study is a very similar task as it requires to classify the EC number Webb et al.

Original: (1992) of a catalyzed enzyme based on the protein structures.
Revised: (1992) of a catalyzed enzyme based on the protein structures.

Original: Therefore, we put these two tasks side-by-side to evaluate the performance of different methods.
Revised: Therefore, we put these two tasks side-by-side to evaluate the performance of different methods.

Original: We use the dataset collected by (Hermosilla Casajus et al., 2021), containing 37 , 428 proteins from 384 types of Enzyme-Catalyzed classes and we split them into training, validation set, test set and ensure no sequence or structure overlaps across different sets.
Revised: We use the dataset collected by (Hermosilla Casajus et al., 2021), containing 37 , 428 proteins from 384 types of Enzyme-Catalyzed classes and we split them into training, validation set, test set and ensure no sequence or structure overlaps across different sets.

Original: In total, there are 29 , 215 structures for training, 2 , 562 for validation, and 5 , 651 for test.
Revised: In total, there are 29 , 215 structures for training, 2 , 562 for validation, and 5 , 651 for test.

Original: Metrics.
Revised: Metrics.

Original: Following standard benchmarks Diehl (2019); Hermosilla Casajus et al.
Revised: Following standard benchmarks Diehl (2019); Hermosilla Casajus et al.

Original: (2021), we use the classification accuracy for evaluating the prediction performance.
Revised: (2021), we use the classification accuracy for evaluating the prediction performance.

Original: Baselines.
Revised: Baselines.

Original: We compare with CNN-based models and GNN-based models which learn the protein annotations using 3D structures from scratch.
Revised: We compare with CNN-based models and GNN-based models which learn the protein annotations using 3D structures from scratch.

Original: For fair comparison, we also do not include LSTMbased or transformer-based methods, as they all pre-train their models using millions of protein sequences and only fine-tune their models on 3D structures Bepler & Berger (2019);
Revised: For fair comparison, we also do not include LSTMbased or transformer-based methods, as they all pre-train their models using millions of protein sequences and only fine-tune their models on 3D structures Bepler & Berger (2018);

Original: Alley et al.
Revised: Alley et al.

Original: ; Rao et al.
Revised: ; Rao et al.

Original: ; Strodthoff et al.
Revised: ; Strodthoff et al.

Original: (2020); Elnaggar et al.
Revised: (2020); Elnaggar et al.

Original: Results.
Revised: Results.

Original: As shown in Table 8, our models demonstrate comparable prediction performance in both tasks across different levels of the hierarchy.
Revised: As shown in Table 8, our models demonstrate comparable prediction performance in both tasks across different levels of the hierarchy.

Original: Classifying the fold and enzyme classes based on the protein structure is one of the key problems in structural biology and our proposed models can be used as new tools to conduct function annotations for new proteins.[Table] et al [Table]
Revised: Classifying the fold and enzyme classes based on the protein structure is one of the key problems in structural biology and our proposed models can be used as new tools to conduct function annotations for new proteins.

Original: [Table]

Original: [Table]

Original: Kipf & Welling (2016)

Original: [Table]

Original: Diehl (2019)

Original: [Table]

Original: Hermosilla Casajus et al.

Original: [Table]

Original: Baldassarre et al.

Original: [Table]

Original: Gligorijevi´c et al.

Original: [Table]

Original: Our method

Original: OA-GCN

Original: [Table]

Original: Our method

Original: OA-GIN

Original: [Table]

Original: Our method

Original: OA-GAT 29.9% [Table] [Table] For FOLD and REACT tasks, all GNN-based baselines use 3D structure information:
Revised: For FOLD and REACT tasks, all GNN-based baselines use 3D structure information:

Original: • Kipf & Welling (2016) and Diehl (2019) construct the input graph using the protein’s contact map and use pair-wise distances of residues as edge features.
Revised: • Kipf & Welling (2016) and Diehl (2019) construct the input graph using the protein’s contact map and use pair-wise distances of residues as edge features.

Original: • Baldassarre et al.
Revised: • Baldassarre et al.

Original: (2021) use the spatial features including the dihedral angles, surface accessibility, and secondary structure type of each residue’s node feature.
Revised: (2021) use the spatial features including the dihedral angles, surface accessibility, and secondary structure type of each residue’s node feature.

Original: They also use distances, sequence distances, and bond types for edge features.
Revised: They also use distances, sequence distances, and bond types for edge features.

Original: • Hermosilla Casajus et al.
Revised: • Hermosilla Casajus et al.

Original: (2021) consider more fine-grained atom-level spatial information, e g. covalent radius, van der Waals radius, and atom mass, which leads to better performance.
Revised: (2021) consider more fine-grained atom-level spatial information, e.g. covalent radius, van der Waals radius, and atom mass, which leads to better performance.

Original: • Gligorijevi´c et al.
Revised: Gligorijevi ´ c et al.

Original: (2021) refer to different types of contact maps and geometric distances for the input of GNN.
Revised: (2021) refer to different types of contact maps and geometric distances for the input of GNN.

Original: Despite not using special training techniques, pre-training tasks, and fine-tuning of hyperparameters, our models demonstrate comparable prediction performance in both tasks across different levels of the hierarchy.
Revised: Despite not using special training techniques, pre-training tasks, and fine-tuning of hyperparameters, our models demonstrate comparable prediction performance in both tasks across different levels of the hierarchy.

Original: Especially with the introduction of our directed weight with equivariant message passing, it is a big improvement over the normal GNN. 
Revised: Especially with the introduction of our directed weight with equivariant message passing, it is a big improvement over the normal GNN. [Table] et al. [Table]

Revised: et al.

Original: D C OMPARISONS TO PRIOR WORKS D. C OMPARISON TO ANGULAR - EXPLICIT MODELS
Revised: [Table] Kipf & Welling (2016) [Table] [Table]

Revised: Hermosilla Casajus et al.

Revised: [Table]

Revised: Baldassarre et al.

Revised: [Table]

Revised: Gligorijevi´c et al.

Revised: [Table]

Revised: Our method

Revised: [Table]

Revised: Our method

Revised: [Table]

Revised: Our method

Revised: [Table]

Revised: D C OMPARISONS TO PRIOR WORKS D. C OMPARISON TO ANGULAR - EXPLICIT MODELS

Original: There are several works that explicitly model angles/torsion angles in GNNs.
Revised: There are several works that explicitly model angles/torsion angles in GNNs.

Original: DimeNet(Klicpera et al., 2020) uses physical-informed RBF and SBF functions to represent distances and angular information.
Revised: DimeNet(Gasteiger et al., 2019b) uses physical-informed RBF and SBF functions to represent distances and angular information.

Original: SphereNet Liu et al.
Revised: SphereNet Liu et al.

Original: (2021) further employs a 3D meaningful spherical coordinate system to represent molecular torsional angles.
Revised: (2021) further employs a 3D meaningful spherical coordinate system to represent molecular torsional angles.

Original: However, these methods rely on invariant scalar input features for remaining equivariance, and during convolution, these scalar features (distances, angles) are just used for updating hidden representations and stay the same in the next layer updating.
Revised: However, these methods rely on invariant scalar input features for remaining equivariance, and during convolution, these scalar features (distances, angles) are just used for updating hidden representations and stay the same in the next layer updating.

Original: We have added the discussions and citations of methods in the revised manuscript.

Original: On the technical level, our model also has several differences from other methods when compared with existing studies to consider backbone torsion angles (DimeNet, SpehreNet, etc.), as follows:
Revised: On the technical level, our model also has several differences from other methods when compared with existing studies to consider backbone torsion angles (DimeNet, SpehreNet, etc.), as follows:

Original: RBF-embedded distances and Trigonometric-embedded angles are adopted in the input feature engineering part (see supplementary material).
Revised: RBF-embedded distances and Trigonometric-embedded angles are adopted in the input feature engineering part (see supplementary material).

Original: To better represent the 3D geometries, we also introduce several normal vectors as input node features and edge characteristics.
Revised: To better represent the 3D geometries, we also introduce several normal vectors as input node features and edge characteristics.

Original: The scalar representations are invariant and vectors are equivariant.
Revised: The scalar representations are invariant and vectors are equivariant.

Original: But these methods don’t consider vector representations and only focus on invariant scalar features.
Revised: But these methods don’t consider vector representations and only focus on invariant scalar features.

Original: We capture fine-grained geometries with the help of vector representations and directed weight perceptrons.
Revised: We capture fine-grained geometries with the help of vector representations and directed weight perceptrons.

Original: And representations are updated layer-by-layer.
Revised: And representations are updated layer-by-layer.

Original: Our directed weights are learnable to be expandable for complex reasoning capabilities.
Revised: Our directed weights are learnable to be expandable for complex reasoning capabilities.

Original: But these methods only embed angles and distances into edge features during message updating and lack geometry meaningful operations.
Revised: But these methods only embed angles and distances into edge features during message updating and lack geometry meaningful operations.

Original: They also don’t have directed weights, which are key contributions to our work.
Revised: They also don’t have directed weights, which are key contributions to our work.

Original: We maintain equivariance with the help of local coordinate transformations on vector representations equipped with rigid residue positions.
Revised: We maintain equivariance with the help of local coordinate transformations on vector representations equipped with rigid residue positions.

Original: But these methods don’t consider equivariant vectors, they only condition on invariant scalars.
Revised: But these methods don’t consider equivariant vectors, they only condition on invariant scalars.

Original: SphereNet does use local spherical systems, but they are used for calculating relative locations of atoms, seen as a feature engineering part.
Revised: SphereNet does use local spherical systems, but they are used for calculating relative locations of atoms, seen as a feature engineering part.

Original: We aim to design a new type of vector neurons for geometric learning and a general message passing paradigm for 3D graphs.
Revised: We aim to design a new type of vector neurons for geometric learning and a general message passing paradigm for 3D graphs.

Original: We have shown our versatility in any existing graph learning frameworks.
Revised: We have shown our versatility in any existing graph learning frameworks.

Original: However, these methods are designed specifically for molecular graphs and are hard to extend for other 3D point sets.
Revised: However, these methods are designed specifically for molecular graphs and are hard to extend for other 3D point sets.

Original: D. C OMPARISON TO OTHER EGNN WORKS
Revised: D. C OMPARISON TO OTHER EGNN WORKS

Original: Local-global frame transformation is an important technique for 3D object learning (Petrelli &
Revised: Local-global frame transformation is an important technique for 3D object learning (Petrelli &

Original: Di Stefano, 2011; Yang et al., 2018; Luo et al., 2022) and is inevitable in the discussions of 3D equivariance.
Revised: Di Stefano, 2011; Yang et al., 2018; Luo et al., 2022) and is inevitable in the discussions of 3D equivariance.

Original: Our proposed framework focuses on its computation in the latent space together with our vector-based feature representations, while many existing works only apply rotations to the primal space of explicit geometric/physical properties (e.g. simple cartesian coordinates).
Revised: Our proposed framework focuses on its computation in the latent space together with our vector-based feature representations, while many existing works only apply rotations to the primal space of explicit geometric/physical properties (e.g. simple cartesian coordinates).

Original: It also provides better flexibility over message-passing operators by eliminating the requirements for specific feature-update arithmetics – this also serves as an adaptor for plugging our designed feature representations into many existing GNN models while guaranteeing equivariance.
Revised: It also provides better flexibility over message-passing operators by eliminating the requirements for specific feature-update arithmetics – this also serves as an adaptor for plugging our designed feature representations into many existing GNN models while guaranteeing equivariance.

Original: Among the EGNNs, we could not agree that our model is a simplified model from others.
Revised: Among the EGNNs, we could not agree that our model is a simplified model from others.

Original: Specifically, some of these EGNNs could be considered a special case of our framework.
Revised: Specifically, some of these EGNNs could be considered a special case of our framework.

Original: For instance,
Revised: For instance,

Original: E(n) GNN (Satorras et al., 2021) combines node coordinates weighted by hidden representations, which could be recognized as linearly updating of simple vector features (coordinate) and unit matrix transformations (in analogy to our back and forth transformations).
Revised: E(n) GNN (Satorras et al., 2021) combines node coordinates weighted by hidden representations, which could be recognized as linearly updating of simple vector features (coordinate) and unit matrix transformations (in analogy to our back and forth transformations).

Original: Additionally, to the best of our knowledge, there is no previous work that integrates vector-based features and directed weighted neural networks into local-global transformation frameworks.
Revised: Additionally, to the best of our knowledge, there is no previous work that integrates vector-based features and directed weighted neural networks into local-global transformation frameworks.

Original: We apply transformations on hidden scalar-vector features back and forth for better representations under equivariance, which turns out to show decent performances in the protein learning tasks.
Revised: We apply transformations on hidden scalar-vector features back and forth for better representations under equivariance, which turns out to show decent performances in the protein learning tasks.

Original: D. C OMPARISON TO OTHER BACKBONE - ORIENTATION MODELS
Revised: D. C OMPARISON TO OTHER BACKBONE - ORIENTATION MODELS

Original: Using sin, cos . . .
Revised: Using sin, cos . . .

Original: ϕ, ψ, ω as features are popular techniques in protein learning tasks (Ingraham et al., 2019; Jing et al., 2021; Ganea et al., 2021), but our models can have a more powerful ability to make use of these features.
Revised: ϕ, ψ, ω as features are popular techniques in protein learning tasks (Ingraham et al., 2019; Jing et al., 2021; Ganea et al., 2021), but our models can have a more powerful ability to make use of these features.

Original: The meaning of Orientation-Aware GNNs is not to use torsional angles as the node and edge features, but for reasoning on them.
Revised: The meaning of Orientation-Aware GNNs is not to use torsional angles as the node and edge features, but for reasoning on them.

Original: Our real contribution is to design a new type of neural network, which has good capability in sensing these geometric features (e.g. angles, orientations) as well as capturing complex interactions.
Revised: Our real contribution is to design a new type of neural network, which has good capability in sensing these geometric features (e.g. angles, orientations) as well as capturing complex interactions.

Original: The synthetic experiment gives an intuitive example of our claims, where our designed directed weight perceptrons have more powerful in sensing and handling angular-related characteristics.
Revised: The synthetic experiment gives an intuitive example of our claims, where our designed directed weight perceptrons have more powerful in sensing and handling angular-related characteristics.

Original: Here we share an even simpler example: Assume the input contains two 3D vectors v 1 and v 2 described by their coordinates [ x 1 , y 1 , z 1 , x 2 , y 2 , z 2 ] , the problem we solved is that how should we let the model know the angle between v 1 and v 2 .
Revised: Here we share an even simpler example: Assume the input contains two 3D vectors v 1 and v 2 described by their coordinates [ x 1 , y 1 , z 1 , x 2 , y 2 , z 2 ] , the problem we solved is that how should we let the model know the angle between v 1 and v 2 .

Original: The most straightforward intuition is that if the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information.
Revised: The most straightforward intuition is that if the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information.

Original: Furthermore, if one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or parallel surfaces based on input coordinates, our employed cross-product operation can easily solve that.
Revised: Furthermore, if one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or parallel surfaces based on input coordinates, our employed cross-product operation can easily solve that.

Original: As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with geometric-meaningful operators are learnable, our models could perform more complex geometric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations.
Revised: As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with geometric-meaningful operators are learnable, our models could perform more complex geometric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations.

Original: The most straightforward intuition is that if the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information.
Revised: The most straightforward intuition is that if the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information.

Original: Furthermore, if one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or parallel surfaces based on input coordinates, our employed cross-product operation can easily solve that.
Revised: Furthermore, if one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or parallel surfaces based on input coordinates, our employed cross-product operation can easily solve that.

Original: As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with geometric-meaningful operators are learnable, our models could perform more complex geometric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations.
Revised: As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with geometric-meaningful operators are learnable, our models could perform more complex geometric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations.

Original: The motivation and function of residue orientations are also different from prior works.
Revised: The motivation and function of residue orientations are also different from prior works.

Original: Instead of using the angles (orientations) as input features, residue backbone orientations define the coordinate frame of the whole system and are used in the equivariant message passing part for coordinate transformation.
Revised: Instead of using the angles (orientations) as input features, residue backbone orientations define the coordinate frame of the whole system and are used in the equivariant message passing part for coordinate transformation.

Original: Rigid orientations play the role of bridging local and global coordinates for equivariance in our message passing part.
Revised: Rigid orientations play the role of bridging local and global coordinates for equivariance in our message passing part.

Original: E E XPLANATIONS TO SOME MODULES E. A BOUT THE s ′ dot
Revised: E E XPLANATIONS TO SOME MODULES E. A BOUT THE s ′ dot

Original: For the s ′ dot features, it can be interpreted as the summation of the channel-wise dot product between the input features (channels of vectors) and the directed weights (a matrix of vectors, so a 3-dimensional tensor).
Revised: For the s ′ dot features, it can be interpreted as the summation of the channel-wise dot product between the input features (channels of vectors) and the directed weights (a matrix of vectors, so a 3-dimensional tensor).

Original: Let us consider the i -th number in the output feature, it can be formulated as:
Revised: Let us consider the i -th number in the output feature, it can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Notice that both are 3D vectors, **the dot product can essentially reflect the cosine value between them after normalization**.
Revised: Notice that both are 3D vectors, **the dot product can essentially reflect the cosine value between them after normalization**.

Original: Essentially, our method utilizes the dot product operation between the directed weights and the input coordinate features, and more importantly, the weight vectors are learnable.
Revised: Essentially, our method utilizes the dot product operation between the directed weights and the input coordinate features, and more importantly, the weight vectors are learnable.

Original: We also recommend you pay attention to other operations we introduced for reasoning in complex geometric relations and encouraging efficient scalar-vector feature interactions.
Revised: We also recommend you pay attention to other operations we introduced for reasoning in complex geometric relations and encouraging efficient scalar-vector feature interactions.

Original: For instance, the cross-product operation can create new orthogonal orientations; the interaction modules can facilitate feature blending.
Revised: For instance, the cross-product operation can create new orthogonal orientations; the interaction modules can facilitate feature blending.

Original: E. A BOUT THE S CALR -L IFTING P ART
Revised: E. A BOUT THE S CALR -L IFTING P ART

Original: The initial motivation of scalar lifting is to enable scalar-vector interactions, which are not well addressed in previous works (for instance, VNN (Deng et al., 2021) and GVP (Jing et al., 2020) can only perform vector-to-scalar mappings with a very limited choice of operations).
Revised: The initial motivation of scalar lifting is to enable scalar-vector interactions, which are not well addressed in previous works (for instance, VNN (Deng et al., 2021) and GVP (Jing et al., 2020) can only perform vector-to-scalar mappings with a very limited choice of operations).

Original: Unprojecting from lower dimensions to higher dimensions is indeed non-bijective on the test data themselves, but it is a widely adopted approach like the diluted convolution or the transposed convolution that succeeds in the projection onto higher dimensional features.
Revised: Unprojecting from lower dimensions to higher dimensions is indeed non-bijective on the test data themselves, but it is a widely adopted approach like the diluted convolution or the transposed convolution that succeeds in the projection onto higher dimensional features.

Original: Mapping from high dimensions to lower space and then projecting back is also a popular technique in FCNs (Devlin et al., 2018; Jumper et al., 2021), Generative Models (Wang et al., 2014), and Manifold Learning, which can be regarded as feature compression and information bottleneck.
Revised: Mapping from high dimensions to lower space and then projecting back is also a popular technique in FCNs (Kenton & Toutanova, 2019; Jumper et al., 2021), Generative Models (Wang et al., 2014), and Manifold Learning, which can be regarded as feature compression and information bottleneck.

Original: Therefore, though not unique, the learnable directed weights endow the model with the expressiveness to recover useful information hopefully after training on many data.
Revised: Therefore, though not unique, the learnable directed weights endow the model with the expressiveness to recover useful information hopefully after training on many data.

Original: More intuitively, the scalar lifting operator generates fixed directional vectors with different lengths during inference, which can potentially provide more information during the next round of feature processing and facilitate information flow between scalar and vector features.
Revised: More intuitively, the scalar lifting operator generates fixed directional vectors with different lengths during inference, which can potentially provide more information during the next round of feature processing and facilitate information flow between scalar and vector features.

Original: The scalar lifting module serves to pass scalar information to the vector representation in our model.
Revised: The scalar lifting module serves to pass scalar information to the vector representation in our model.

Original: Though we give a geometric interpretation of the intuition, our directed weight is learnable, the operation of the model is complex and its final geometric meaning is non-linear, which is difficult to define simply.
Revised: Though we give a geometric interpretation of the intuition, our directed weight is learnable, the operation of the model is complex and its final geometric meaning is non-linear, which is difficult to define simply.

Original: The function of the mapping is learned according to the needs of the downstream task and is not limited to reconstruction or compression.
Revised: The function of the mapping is learned according to the needs of the downstream task and is not limited to reconstruction or compression.

Original: From the experimental results, the network we designed does achieve excellent performance, demonstrating our strong geometric perception and representation capabilities.
Revised: From the experimental results, the network we designed does achieve excellent performance, demonstrating our strong geometric perception and representation capabilities.