Original: A N E XPLORATION OF C ONDITIONING M ETHODS IN G RAPH N EURAL N ETWORKS
Revised: A N E XPLORATION OF C ONDITIONING M ETHODS IN G RAPH N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The flexibility and effectiveness of message passing based graph neural networks (GNNs) induced considerable advances in deep learning on graph-structured data.
Revised: The flexibility and effectiveness of message passing based graph neural networks (GNNs) induced considerable advances in deep learning on graph-structured data.

Original: In such approaches, GNNs recursively update node representations based on their neighbors and they gain expressivity through the use of node and edge attribute vectors.
Revised: In such approaches, GNNs recursively update node representations based on their neighbors and they gain expressivity through the use of node and edge attribute vectors.

Original: E.g., in computational tasks such as physics and chemistry usage of edge attributes such as relative position or distance proved to be essential.
Revised: E.g., in computational tasks such as physics and chemistry usage of edge attributes such as relative position or distance proved to be essential.

Original: In this work, we address not what kind of attributes to use, but how to condition on this information to improve model performance.
Revised: In this work, we address not what kind of attributes to use, but how to condition on this information to improve model performance.

Original: We consider three types of conditioning; weak, strong, and pure, which respectively relate to concatenation-based conditioning, gating, and transformations that are causally dependent on the attributes.
Revised: We consider three types of conditioning; weak, strong, and pure, which respectively relate to concatenation-based conditioning, gating, and transformations that are causally dependent on the attributes.

Original: This categorization provides a unifying viewpoint on different classes of GNNs, from separable convolutions to various forms of message passing networks.
Revised: This categorization provides a unifying viewpoint on different classes of GNNs, from separable convolutions to various forms of message passing networks.

Original: We provide an empirical study on the effect of conditioning methods in several tasks in computational chemistry.
Revised: We provide an empirical study on the effect of conditioning methods in several tasks in computational chemistry.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph neural networks (GNNs) are a family of neural networks that can learn from graph-structured data.
Revised: Graph neural networks (GNNs) are a family of neural networks that can learn from graph-structured data.

Original: Starting with the success of GCN (Kipf & Welling, 2016) in achieving state-of-the-art performance on semi-supervised classification, several variants of GNNs have been developed for this task, including Graph-SAGE (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2017), GATv2 (Brody et al., 2021), EGNN (Satorras et al., 2021) to name a few most recent ones.
Revised: Starting with the success of GCN (Kipf & Welling, 2016) in achieving state-of-the-art performance on semi-supervised classification, several variants of GNNs have been developed for this task, including Graph-SAGE (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2017), GATv2 (Brody et al., 2021), EGNN (Satorras et al., 2021) to name a few most recent ones.

Original: Most of the models based on the message-passing framework utilize conditional linear layers.
Revised: Most of the models based on the message-passing framework utilize conditional linear layers.

Original: We define ”conditioning” as using additional information together with feature vectors from its neighbors’ nodes.
Revised: We define ”conditioning” as using additional information together with feature vectors from its neighbors’ nodes.

Original: For example, EGNN (Satorras et al., 2021) conditions message vectors on the distance between two nodes or DimeNet (Gasteiger et al., 2020b) additionally utilizes angle information.
Revised: For example, EGNN (Satorras et al., 2021) conditions message vectors on the distance between two nodes or DimeNet (Gasteiger et al., 2020b) additionally utilizes angle information.

Original: Many neural network models use conditioning in their layers without exploring their different variants.
Revised: Many neural network models use conditioning in their layers without exploring their different variants.

Original: Therefore, improving upon the type of conditioning could still improve most state-of-the-art models.
Revised: Therefore, improving upon the type of conditioning could still improve most state-of-the-art models.

Original: We believe that this is the first work that analyzes different conditioning methods in GNNs.
Revised: We believe that this is the first work that analyzes different conditioning methods in GNNs.

Original: In this paper, we categorize three conditioning methods: weak, strong, and pure.
Revised: In this paper, we categorize three conditioning methods: weak, strong, and pure.

Original: They differ in the level of dependency on the a given quantity, such as an edge attributes, and differ in complexity.
Revised: They differ in the level of dependency on the a given quantity, such as an edge attributes, and differ in complexity.

Original: Message passing neural network (MPNN) using weak conditioning method concatenates attributes with node features.
Revised: Message passing neural network (MPNN) using weak conditioning method concatenates attributes with node features.

Original: In this scenario, linear layers effectively gain an attribute-dependent bias, which we consider a weak type of conditioning as this does not guarantee that the attribute is actually utilized, i.e., it could be ignored.
Revised: In this scenario, linear layers effectively gain an attribute-dependent bias, which we consider a weak type of conditioning as this does not guarantee that the attribute is actually utilized, i.e., it could be ignored.

Original: On the other hand, we have pure conditioning method which forces the model to always use the attributes by letting them causally parametrize transformation matrices.
Revised: On the other hand, we have pure conditioning method which forces the model to always use the attributes by letting them causally parametrize transformation matrices.

Original: However, from a practical perspective pure conditioning is computationally expensive and it can be simplified to a strong conditioning method, which corresponds to an attribute-dependent gating of the outputs of linear layers.
Revised: However, from a practical perspective pure conditioning is computationally expensive and it can be simplified to a strong conditioning method, which corresponds to an attribute-dependent gating of the outputs of linear layers.

Original: We experiment with these three conditioning methods in variations of the EGNN model (Satorras et al., 2021) on computational chemistry datasets QM9 (Ramakrishnan et al., 2014) and MD17 (Chmiela et al., 2017) and show the advantage of strong conditioning over weak conditioning in performance, and over pure conditioning in training time.
Revised: We experiment with these three conditioning methods in variations of the EGNN model (Satorras et al., 2021) on computational chemistry datasets QM9 (Ramakrishnan et al., 2014) and MD17 (Chmiela et al., 2017) and show the advantage of strong conditioning over weak conditioning in performance, and over pure conditioning in training time.

Original: The main contributions of this paper are:
Revised: The main contributions of this paper are:

Original: (i)
Revised: (i)

Original: A unifying analysis of geometric message passing by formulating conditional transformations in terms of various forms of conditional linear layers.
Revised: A unifying analysis of geometric message passing by formulating conditional transformations in terms of various forms of conditional linear layers.

Original: (ii)
Revised: (ii)

Original: An intuitive exposition of different conditioning methods in the context of convolutional message passing.
Revised: An intuitive exposition of different conditioning methods in the context of convolutional message passing.

Original: (iii) Empirical studies that show the benefit of strong conditioning methods, as well as the benefit of deep conditioning in multi-layer perceptron-based message functions.
Revised: (iii) Empirical studies that show the benefit of strong conditioning methods, as well as the benefit of deep conditioning in multi-layer perceptron-based message functions.

Original: In this work, we address not what kind of attributes to use, but how to condition on this information to improve model performance.
Revised: In this work, we address not what kind of attributes to use, but how to condition on this information to improve model performance.

Original: As such, we focus on an intuitive analysis and in the experimental section we do not intend to achieve the best performance, but focus on ablation studies in order to obtain general take-home messages.
Revised: As such, we focus on an intuitive analysis and in the experimental section we do not intend to achieve the best performance, but focus on ablation studies in order to obtain general take-home messages.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we introduce the relevant materials on graph neural networks on top of which we will later complement our analysis and definitions of conditioning methods.
Revised: In this section, we introduce the relevant materials on graph neural networks on top of which we will later complement our analysis and definitions of conditioning methods.

Original: 2. G RAPH N EURAL N ETWORK
Revised: 2. G RAPH N EURAL N ETWORK

Original: In this work, we consider the graph regression task as an example.
Revised: In this work, we consider the graph regression task as an example.

Original: A graph is represented by
Revised: A graph is represented by

Original: G =
Revised: G =

Original: ( V , E ) with nodes v i ∈ V and edges e ij ∈ E .
Revised: ( V , E ) with nodes v i ∈ V and edges e ij ∈ E .

Original: A typical message passing layer (Gilmer et al., 2017) is defined as:
Revised: A typical message passing layer (Gilmer et al., 2017) is defined as:

Original: [Equation]
Revised: [Equation]

Original: Where h li is the embedding of node v i at layer l , a ij is the edge attribute of nodes v i and v j , and
Revised: Where h li is the embedding of node v i at layer l , a ij is the edge attribute of nodes v i and v j , and

Original: N ( i ) is the set of neighbors of the node v i .
Revised: N ( i ) is the set of neighbors of the node v i .

Original: Finally, ϕ e and ϕ h are the message (edge) and update (node) functions respectively which are commonly parametrized by Multilayer Perceptrons (MLPs)
Revised: Finally, ϕ e and ϕ h are the message (edge) and update (node) functions respectively which are commonly parametrized by Multilayer Perceptrons (MLPs)

Original: . 2. G EOMETRIC G RAPH N EURAL N ETWORKS
Revised: . 2. G EOMETRIC G RAPH N EURAL N ETWORKS

Original: When the graphs have an embedding in Euclidean space, i.e., each node v i has an associated position x i ∈ R n , we want to leverage this geometric information whilst preserving stability/invariance to rigid-body transformations.
Revised: When the graphs have an embedding in Euclidean space, i.e., each node v i has an associated position x i ∈ R n , we want to leverage this geometric information whilst preserving stability/invariance to rigid-body transformations.

Original: That is, many tasks are invariant to Euclidean distance preserving transformations in E ( n ) .
Revised: That is, many tasks are invariant to Euclidean distance preserving transformations in E ( n ) .

Original: E.g., the prediction of energy of a system of atoms in invariant to its global position and orientation in space.
Revised: E.g., the prediction of energy of a system of atoms in invariant to its global position and orientation in space.

Original: Several works have shown how to build equivariant message passing based graph neural networks for such geometric graphs.
Revised: Several works have shown how to build equivariant message passing based graph neural networks for such geometric graphs.

Original: Central in those works is the conditioning of the message and update function on invariant geometric attributes, such as the pairwise distance a ij = ∥
Revised: Central in those works is the conditioning of the message and update function on invariant geometric attributes, such as the pairwise distance a ij = ∥

Original: x j − x i ∥ , as popularized in (Satorras et al., 2021), or covariant spherical/circular harmonic embeddings of relative position a ij =
Revised: x j − x i ∥ , as popularized in (Satorras et al., 2021), or covariant spherical/circular harmonic embeddings of relative position a ij =

Original: Y ( x j − x i ) as is common steerable group convolution-based graph NNs (Brandstetter et al., 2021).
Revised: Y ( x j − x i ) as is common steerable group convolution-based graph NNs (Brandstetter et al., 2021).

Original: Here we consider attributes that transforms predictably via representations of E ( n ) as covariants , and those that remain invariant as invariants .
Revised: Here we consider attributes that transforms predictably via representations of E ( n ) as covariants , and those that remain invariant as invariants .

Original: Such covariants typically contain more (directional) information, but require specialized operations such as the Clebsch-Gordan tensor product (Thomas et al., 2018;
Revised: Such covariants typically contain more (directional) information, but require specialized operations such as the Clebsch-Gordan tensor product (Thomas et al., 2018;

Original: Anderson et al., 2019) in order to preserve equivariance of the graph NNs.
Revised: Anderson et al., 2019) in order to preserve equivariance of the graph NNs.

Original: Satorras et al.
Revised: Satorras et al.

Original: (2021) show that with a simple recipe based on invariant attributes, one can often obtain equally powerful graph NNs.
Revised: (2021) show that with a simple recipe based on invariant attributes, one can often obtain equally powerful graph NNs.

Original: As such, we focus this paper on the use of a ij =
Revised: As such, we focus this paper on the use of a ij =

Original: ∥ x
Revised: ∥ x

Original: j − x i ∥ as a sufficiently expressive attribute, and model the message and update functions ϕ e and ϕ h as regular MLPs.
Revised: j − x i ∥ as a sufficiently expressive attribute, and model the message and update functions ϕ e and ϕ h as regular MLPs.

Original: Our objective then is to understand what is the most effective way of utilizing attributes in geometric graph NNs.
Revised: Our objective then is to understand what is the most effective way of utilizing attributes in geometric graph NNs.

Original: To make this notion of conditioning explicit, we will denote the message and update function as
Revised: To make this notion of conditioning explicit, we will denote the message and update function as

Original: [Equation]
Revised: [Equation]

Original: where we note that, although uncommon, it is possible to define invariant or covariant geometric node attributes a i (Brandstetter et al., 2021).
Revised: where we note that, although uncommon, it is possible to define invariant or covariant geometric node attributes a i (Brandstetter et al., 2021).

Original:  A NALYSIS OF C ONDITIONING M ETHODS
Revised:  A NALYSIS OF C ONDITIONING M ETHODS

Original: In the subsequent, we unify several conditioning methods used in literature through the notion of conditional linear layers, and by discussing them in relation to the prevalent convolution layer and its variations.
Revised: In the subsequent, we unify several conditioning methods used in literature through the notion of conditional linear layers, and by discussing them in relation to the prevalent convolution layer and its variations.

Original: As a starting point, we use the fact that convolution is a simple form of message passing with linear message functions conditioned on relative position, i.e.,
Revised: As a starting point, we use the fact that convolution is a simple form of message passing with linear message functions conditioned on relative position, i.e.,

Original: [Equation]
Revised: [Equation]

Original: and the following update function typically is the application of a point-wise activation function σ , i.e., ϕ h ( f i ) =
Revised: and the following update function typically is the application of a point-wise activation function σ , i.e., ϕ h ( f i ) =

Original: σ ( f i ) , possibly with a skip connection as in ResNets
Revised: σ ( f i ) , possibly with a skip connection as in ResNets

Original: He et al.
Revised: He et al.

Original: In general, geometric graph NN do not just linearly transform node features, but generally do this non-linearly via message/update functions parametrized by MLPs, leading to a notion of non-linear convolutions when the attributes are invariant/covariant quantities (Brandstetter et al., 2021).
Revised: In general, geometric graph NN do not just linearly transform node features, but generally do this non-linearly via message/update functions parametrized by MLPs, leading to a notion of non-linear convolutions when the attributes are invariant/covariant quantities (Brandstetter et al., 2021).

Original: Importantly, these MLPs themselves are parametrized by linear layers, intertwined with non-linear activation functions, and nothing prevents from conditioning each of these linear layers on the attributes.
Revised: Importantly, these MLPs themselves are parametrized by linear layers, intertwined with non-linear activation functions, and nothing prevents from conditioning each of these linear layers on the attributes.

Original: It is the purpose of this paper to categorize several options for conditioning and understand what effect this has on performance.
Revised: It is the purpose of this paper to categorize several options for conditioning and understand what effect this has on performance.

Original: 3. C ONDITIONAL LINEAR LAYERS
Revised: 3. C ONDITIONAL LINEAR LAYERS

Original: We propose the following modifications of the linear layer, as to make them conditional on attributes
Revised: We propose the following modifications of the linear layer, as to make them conditional on attributes

Original: [Equation]
Revised: [Equation]

Original: no
Revised: no

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: weak
Revised: weak

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: strong
Revised: strong

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: pure
Revised: pure

Original: [Equation]
Revised: [Equation]

Original: 8) where to keep the similarity to the common notation for linear transformations, we use the notation
Revised: 8) where to keep the similarity to the common notation for linear transformations, we use the notation

Original: W a to denote that the linear transformation is conditioned on a .
Revised: W a to denote that the linear transformation is conditioned on a .

Original: We further use to notation h ⊕ a to denote the concatenation of vectors h and a , and use ⊙ to denote element-wise multiplication, i.e., a ⊙ b =
Revised: We further use to notation h ⊕ a to denote the concatenation of vectors h and a , and use ⊙ to denote element-wise multiplication, i.e., a ⊙ b =

Original: ( a 1 b 1 ,
Revised: ( a 1 b 1 ,

Original: a 2 b
Revised: a 2 b

Original: T .
Revised: T .

Original: We stress the hierarchy in terms of the dependence of the transformation on a .
Revised: We stress the hierarchy in terms of the dependence of the transformation on a .

Original: The most direct dependence is in the pure method, in which the transformation is causally parametrized by a , followed by strong conditioning in which a standard unconditional transformation W h h is gated by a vector
Revised: The most direct dependence is in the pure method, in which the transformation is causally parametrized by a , followed by strong conditioning in which a standard unconditional transformation W h h is gated by a vector

Original: W a a .
Revised: W a a .

Original: Both the pure and strong methods are by construction forced to utilize the attribute, as, in particular in the strong case, the transformation would not exists without attribute a .
Revised: Both the pure and strong methods are by construction forced to utilize the attribute, as, in particular in the strong case, the transformation would not exists without attribute a .

Original: We refer to equation 6 as weak conditioning, as in principle the transformation would still exist in the absence of the attribute, and moreover, if the dimensionality of h is much larger than that of a , the transformation of the attribute only contributes to a small extent to the output of this layer.
Revised: We refer to equation 6 as weak conditioning, as in principle the transformation would still exist in the absence of the attribute, and moreover, if the dimensionality of h is much larger than that of a , the transformation of the attribute only contributes to a small extent to the output of this layer.

Original: We hypothesize that this hierarchy correlates with performance and experimentally test this hypothesis in Sec. 5.
Revised: We hypothesize that this hierarchy correlates with performance and experimentally test this hypothesis in Sec. 5.

Original: What follows is a brief analysis of these types of conditioning.
Revised: What follows is a brief analysis of these types of conditioning.

Original: 3. P URE CONDITIONING CORRESPONDS TO BI - LINEAR LAYERS
Revised: 3. P URE CONDITIONING CORRESPONDS TO BI - LINEAR LAYERS

Original: As common for implementations of convolutions, one typically expands the convolution kernel
Revised: As common for implementations of convolutions, one typically expands the convolution kernel

Original: W (
Revised: W (

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: d i , i.e., a transformation matrix with elements W oi ( x j − x i ) that depend on relative position, in a basis { ϕ b : R n → R } d b b via
Revised: d i , i.e., a transformation matrix with elements W oi ( x j − x i ) that depend on relative position, in a basis { ϕ b : R n → R } d b b via

Original: [Equation]
Revised: [Equation]

Original: The basis could be the usual 3 × 3 pixel basis, or it could be a continuous basis for when the continuous structure of the data is to be respected, such as circular or spherical harmonics (Worrall et al., 2017; Weiler & Cesa, 2019; Thomas et al., 2018; Anderson et al., 2019), B-splines (Bekkers, 2020; Fey et al., 2018), or hermite polynomials (Sosnovik et al., 2020).
Revised: The basis could be the usual 3 × 3 pixel basis, or it could be a continuous basis for when the continuous structure of the data is to be respected, such as circular or spherical harmonics (Worrall et al., 2017; Weiler & Cesa, 2019; Thomas et al., 2018; Anderson et al., 2019), B-splines (Bekkers, 2020; Fey et al., 2018), or hermite polynomials (Sosnovik et al., 2020).

Original: In recent works on the parametrization of continuous functions as Neural Fields (Xie et al., 2022) or in transformer-based methods (Vaswani et al., 2017), such basis functions are often referred to as coordinate embeddings or position encodings .
Revised: In recent works on the parametrization of continuous functions as Neural Fields (Xie et al., 2022) or in transformer-based methods (Vaswani et al., 2017), such basis functions are often referred to as coordinate embeddings or position encodings .

Original: An important observation is that, given such a parametrization through basis functions, the pure conditioning layer corresponds to a bi-linear layer
Revised: An important observation is that, given such a parametrization through basis functions, the pure conditioning layer corresponds to a bi-linear layer

Original: [Equation]
Revised: [Equation]

Original: where we use i to index the input feature vector, o the output feature vector, and b the basis functions.
Revised: where we use i to index the input feature vector, o the output feature vector, and b the basis functions.

Original: Such bilinear layers are often implicitly used in convolutional architectures, where the expansion in the basis function is typically hard-coded or pre-computed.
Revised: Such bilinear layers are often implicitly used in convolutional architectures, where the expansion in the basis function is typically hard-coded or pre-computed.

Original: On continuous data, such as point cloud methods, the bilinear layer is ubiquitous.
Revised: On continuous data, such as point cloud methods, the bilinear layer is ubiquitous.

Original: Notably, in the context of steerable group equivariant convolutions, the transformations happen via bilinear operators called the Clebsch-Gordan tensor product, in combination with spherical harmonic embeddings of relative position (Brandstetter et al., 2021).
Revised: Notably, in the context of steerable group equivariant convolutions, the transformations happen via bilinear operators called the Clebsch-Gordan tensor product, in combination with spherical harmonic embeddings of relative position (Brandstetter et al., 2021).

Original: Outside of the (group) convolution literature, bilinear layers are often used to explicitly model conditioning on geometric attributes, for which a relevant example to our current work is DimeNet (Gasteiger et al., 2020b).
Revised: Outside of the (group) convolution literature, bilinear layers are often used to explicitly model conditioning on geometric attributes, for which a relevant example to our current work is DimeNet (Gasteiger et al., 2020b).

Original: DimeNet uses an advanced message passing framework in which messages and updates are conditioned on geometric quantities (embedded as spherical harmonics and radial basis functions) via combinations of weak, strong, and pure conditioning.
Revised: DimeNet uses an advanced message passing framework in which messages and updates are conditioned on geometric quantities (embedded as spherical harmonics and radial basis functions) via combinations of weak, strong, and pure conditioning.

Original: 3. S TRONG CONDITIONING CORRESPONDS TO DEPTH - WISE SEPARABLE CONVOLUTIONS
Revised: 3. S TRONG CONDITIONING CORRESPONDS TO DEPTH - WISE SEPARABLE CONVOLUTIONS

Original: In later works, DimeNet was improved by DimeNet++ (Gasteiger et al., 2020a), both in performance and speed, by replacing the compute-heavy bilinear layers (pure conditioning) with the more efficient gating type conditioning (strong conditioning).
Revised: In later works, DimeNet was improved by DimeNet++ (Gasteiger et al., 2020a), both in performance and speed, by replacing the compute-heavy bilinear layers (pure conditioning) with the more efficient gating type conditioning (strong conditioning).

Original: The computational bottleneck of pure conditioning motives the use of strong conditioning, a route that has proven successful in convolutional architectures computer vision as well, via the use of so-called depth-wise separable convolutions (Sifre & Mallat, 2014; Chollet, 2017).
Revised: The computational bottleneck of pure conditioning motives the use of strong conditioning, a route that has proven successful in convolutional architectures computer vision as well, via the use of so-called depth-wise separable convolutions (Sifre & Mallat, 2014; Chollet, 2017).

Original: Chollet (2017) shows huge efficiency gains, both in terms of computing and performance, when factorizing convolution kernels in a partthat does the channel mixing,and which does not depend on  relative position, and a part that does depend on the relative position which scales/gates the output.
Revised: Chollet (2017) shows huge efficiency gains, both in terms of computing and performance, when factorizing convolution kernels in two parts. One part does the channel mixing, which does not depend on the relative position, while another part depends on the relative position which scales/gates the output.

Original: That is, if the kernel is given by
Revised: That is if the kernel is given by

Original: [Equation]
Revised: [Equation]

Original: the convolution boils down to message passing with conditional linear layers of the strong type, as we can write
Revised: the convolution boils down to message passing with conditional linear layers of the strong type, as we can write

Original: [Equation]
Revised: [Equation]

Original: where we can define W a ( x j − x i )
Revised: where we can define W a ( x j − x i )

Original: =
Revised: =

Original: W a a ij as the linear transformation of a coordinate embedding a ij = ϕ
Revised: W a a ij as the linear transformation of a coordinate embedding a ij = ϕ

Original: ( x j − x i ) if we want the connection to equation 7 explicit.
Revised: ( x j − x i ) if we want the connection to equation 7 explicit.

Original: The fact that the transformation overall is linear and that it splits into a part that does and a part that doesn’t depend on pair-wise attributes allows for very efficient implementations that first perform a group-wise convolution, followed by channel mixing.
Revised: The fact that the transformation overall is linear and that it splits into a part that does and a part that doesn’t depend on pair-wise attributes allows for very efficient implementations that first perform a group-wise convolution, followed by channel mixing.

Original: This principle is at the core of the recently popularized
Revised: This principle is at the core of the recently popularized

Original: ConvNeXt architecture (Liu et al., 2022) and plays an essential role in equivariant graph NNs on (molecular) point clouds in order to be able to scale up (Thomas et al., 2018).
Revised: ConvNeXt architecture (Liu et al., 2022) and plays an essential role in equivariant graph NNs on (molecular) point clouds in order to be able to scale up (Thomas et al., 2018).

Original: Separability recently also proved necessary in order for equivariant convolutional NNs to scale up to large groups, such as the scale-rotation-translation group (Knigge et al., 2022).
Revised: Separability recently also proved necessary in order for equivariant convolutional NNs to scale up to large groups, such as the scale-rotation-translation group (Knigge et al., 2022).

Original: In the context of conditional NN to parametrize neural fields (Xie et al., 2022), strong conditioning commonly appear in the form of so-called film layers Perez et al.
Revised: In the context of conditional NN to parametrize neural fields (Xie et al., 2022), strong conditioning commonly appear in the form of so-called film layers Perez et al.

Original: 3. W EAK CONDITIONING CORRESPONDS TO LINEAR LAYERS WITH A CONDITIONAL BIAS
Revised: 3. W EAK CONDITIONING CORRESPONDS TO LINEAR LAYERS WITH A CONDITIONAL BIAS

Original: Weak conditioning (equation 6) corresponds to a standard linear layer with an adaptive bias:
Revised: Weak conditioning (equation 6) corresponds to a standard linear layer with an adaptive bias:

Original: [Equation]
Revised: [Equation]

Original: no condition
Revised: no condition

Original: [Equation]
Revised: [Equation]

Original: where W ′ are the first d h rows of W that are applied to the h ∈ R d h of the concatenated vector, and W ′′ are the last d a rows of W that are applied to a ∈ R d a .
Revised: where W ′ are the first d h rows of W that are applied to the h ∈ R d h of the concatenated vector, and W ′′ are the last d a rows of W that are applied to a ∈ R d a .

Original: This simple form of conditioning is most used in literature to condition any type of NN on the conditioning vector a , from message passing methods (Gilmer et al., 2017;
Revised: This simple form of conditioning is most used in literature to condition any type of NN on the conditioning vector a , from message passing methods (Gilmer et al., 2017;

Original: Satorras et al., 2021), to conditional neural fields, e.g. as a simple but effective form of modulation in sirens (Dupont et al., 2022), to conditional variational auto-encoders (Sohn et al., 2015).
Revised: Satorras et al., 2021), to conditional neural fields, e.g. as a simple but effective form of modulation in sirens (Dupont et al., 2022), to conditional variational auto-encoders (Sohn et al., 2015).

Original: 3. C ONDITIONAL MLP S
Revised: 3. C ONDITIONAL MLP S

Original: With the various forms of conditional linear layers given in equations 6, 7, and 8 we can build conditional MLPs, by simply replacing the usual linear layers with the conditional ones.
Revised: With the various forms of conditional linear layers given in equations 6, 7, and 8 we can build conditional MLPs, by simply replacing the usual linear layers with the conditional ones.

Original: Such
Revised: Such

Original: MLPs could e.g. be denoted with MLP( h | a ) .
Revised: MLPs could e.g. be denoted with MLP( h | a ) .

Original: Usually, only the first layer of such a conditional
Revised: Usually, only the first layer of such a conditional

Original: MLP is conditional and usually of the weak type, as with EGNN (Satorras et al., 2021).
Revised: MLP is conditional and usually of the weak type, as with EGNN (Satorras et al., 2021).

Original: In the experiments we show that this simple choice is sub-optimal in the context of geometric message passing `a la EGNN, and show that improvements can be made by either by conditioning more layers, or switching to strong conditioning.
Revised: In the experiments we show that this simple choice is sub-optimal in the context of geometric message passing `a la EGNN, and show that improvements can be made by either by conditioning more layers, or switching to strong conditioning.

Original:  R ELATED WORK : G EOMETRIC M ESSAGE P ASSING FOR C OMPUTATIONAL C HEMISTRY
Revised:  R ELATED WORK : G EOMETRIC M ESSAGE P ASSING FOR C OMPUTATIONAL C HEMISTRY

Original: In the previous section, we discussed several options for conditioning message/update functions for us in message passing graph neural networks, as well as their use in other fields of deep learning.
Revised: In the previous section, we discussed several options for conditioning message/update functions for us in message passing graph neural networks, as well as their use in other fields of deep learning.

Original: In our experiments we benchmark the three conditioning methods (equations 6, 7, 8) in the context geometric message passing for computational chemistry.
Revised: In our experiments we benchmark the three conditioning methods (equations 6, 7, 8) in the context geometric message passing for computational chemistry.

Original: Recent works in this category, and the types of conditioning used in those works, are as follows.
Revised: Recent works in this category, and the types of conditioning used in those works, are as follows.

Original: EGNN (Satorras et al., 2021) is a message passing neural network (MPNN) that uses a weak conditioning method to utilize the distance between nodes.
Revised: EGNN (Satorras et al., 2021) is a message passing neural network (MPNN) that uses a weak conditioning method to utilize the distance between nodes.

Original: DimeNet (Gasteiger et al., 2020b) is the type of MPNN, where message embeddings interact based on the distance between atoms and the angle directions.
Revised: DimeNet (Gasteiger et al., 2020b) is the type of MPNN, where message embeddings interact based on the distance between atoms and the angle directions.

Original: Pure conditioning is adapted to utilize angle directions in message update and aggregation.
Revised: Pure conditioning is adapted to utilize angle directions in message update and aggregation.

Original: Tensor Field Network (Thomas et al., 2018) is a neural network for 3D point clouds.
Revised: Tensor Field Network (Thomas et al., 2018) is a neural network for 3D point clouds.

Original: Each point in TFN is associated with a vector in a representation of SO (3) .
Revised: Each point in TFN is associated with a vector in a representation of SO (3) .

Original: To condition one representation on another, a tensor product of representations is used which in its general form corresponds to pure conditioning.
Revised: To condition one representation on another, a tensor product of representations is used which in its general form corresponds to pure conditioning.

Original: However, many works of the steerable message passing kind, including TFN and
Revised: However, many works of the steerable message passing kind, including TFN and

Original: NequIP (Batzner et al., 2022), implement a separable variation ( strong conditioning) for the sake of computational efficiency.
Revised: NequIP (Batzner et al., 2022), implement a separable variation ( strong conditioning) for the sake of computational efficiency.

Original: The exception is steerable EGNN (Brandstetter et al., 2021), which uses steerable MLPs with pure conditioning in each layer, i.e., not only the first layer of the message function as in EGNN.
Revised: The exception is steerable EGNN (Brandstetter et al., 2021), which uses steerable MLPs with pure conditioning in each layer, i.e., not only the first layer of the message function as in EGNN.

Original: DimeNet++ (Klicpera et al., 2020) is an extension of DimeNet which changed conditioning method from pure to strong to increase efficiency.
Revised: DimeNet++ (Klicpera et al., 2020) is an extension of DimeNet which changed conditioning method from pure to strong to increase efficiency.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2020) showed that changing the conditioning method decreased the runtime time of the original DimeNet by a factor of 5.
Revised: (2020) showed that changing the conditioning method decreased the runtime time of the original DimeNet by a factor of 5.

Original: SchNet (Sch ¨ utt et al., 2017) is another example of MPNN that utilizes atom locations using a strong conditioning method.
Revised: SchNet (Sch ¨ utt et al., 2017) is another example of MPNN that utilizes atom locations using a strong conditioning method.

Original: PaiNN (Sch ¨ utt et al., 2021) further extends SchNet by projecting the interatomic distances via radial basis functions and iteratively updating the vectors along with the scalar features but also using a strong conditioning method.
Revised: PaiNN (Sch ¨ utt et al., 2021) further extends SchNet by projecting the interatomic distances via radial basis functions and iteratively updating the vectors along with the scalar features but also using a strong conditioning method.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we design experiments to evaluate the effectiveness of three conditioning methods shown in equation 6, 7, and 8 on two real-world datasets: QM9 (Ramakrishnan et al., 2014) and
Revised: In this section, we design experiments to evaluate the effectiveness of three conditioning methods shown in equation 6, 7, and 8 on two real-world datasets: QM9 (Ramakrishnan et al., 2014) and

Original: MD17 (Chmiela et al., 2017).
Revised: MD17 (Chmiela et al., 2017).

Original: We want to demonstrate the effect of the choice of the conditioning method and present the results of some other models on benchmarks to provide context.
Revised: We want to demonstrate the effect of the choice of the conditioning method and present the results of some other models on benchmarks to provide context.

Original: In our experiments, we report Mean Absolute Error (MAE) between model predictions and ground truth.
Revised: In our experiments, we report Mean Absolute Error (MAE) between model predictions and ground truth.

Original: Implementation details As a baseline architecture, we use the EGNN model that consists of 7 layers, 128 features per hidden layer, and a 2-layer message (edge) function ϕ e =
Revised: Implementation details As a baseline architecture, we use the EGNN model that consists of 7 layers, 128 features per hidden layer, and a 2-layer message (edge) function ϕ e =

Original: MLP( h i ⊕ h
Revised: MLP( h i ⊕ h

Original: j | ∥ x
Revised: j | ∥ x

Original: j − x i ∥ 2 ) with only weak conditioning via equation 6 in the first layer.
Revised: j − x i ∥ 2 ) with only weak conditioning via equation 6 in the first layer.

Original: We will test modifications of this model by changing the weak conditioning, to strong or pure, and explore the effect of conditioning multiple layers in the edge function MLP.
Revised: We will test modifications of this model by changing the weak conditioning, to strong or pure, and explore the effect of conditioning multiple layers in the edge function MLP.

Original: We found it useful to pass geometric distances through two-layer MLP, as a form of vectorization/coordinate embedding of the pairwise distance, for the QM9 dataset and Random Fourier Feature layer (Rahimi & Recht, 2007) for the MD17 dataset.
Revised: We found it useful to pass geometric distances through two-layer MLP, as a form of vectorization/coordinate embedding of the pairwise distance, for the QM9 dataset and Random Fourier Feature layer (Rahimi & Recht, 2007) for the MD17 dataset.

Original: We use the same values for hyperparameters from the EGNN paper (Satorras et al., 2021) for both datasets: we trained each model on QM9/MD17 dataset for a total of 1.000/500 epochs, used Adam optimizer, batch size 96 (64 for pure-EGNN), weight decay 1e-16, and cosine decay for the learning rate starting at 5e-4.
Revised: We use the same values for hyperparameters from the EGNN paper (Satorras et al., 2021) for both datasets: we trained each model on QM9/MD17 dataset for a total of 1.000/500 epochs, used Adam optimizer, batch size 96 (64 for pure-EGNN), weight decay 1e-16, and cosine decay for the learning rate starting at 5e-4.

Original: QM9 The QM9 (Ramakrishnan et al., 2014) dataset consists of small molecules represented as a set of atoms (up to 29 atoms per molecule), each atom having a 3D position associated and a fivedimensional one-hot node embedding that describe the atom type (H, C, N, O, F).
Revised: QM9 The QM9 (Ramakrishnan et al., 2014) dataset consists of small molecules represented as a set of atoms (up to 29 atoms per molecule), each atom having a 3D position associated and a fivedimensional one-hot node embedding that describe the atom type (H, C, N, O, F).

Original: The QM9 dataset has 3D coordinate locations of each atom and we use the distance between two atoms (nodes) as an edge attribute.
Revised: The QM9 dataset has 3D coordinate locations of each atom and we use the distance between two atoms (nodes) as an edge attribute.

Original: The dataset comprises 12 quantum properties for each of the molecules.
Revised: The dataset comprises 12 quantum properties for each of the molecules.

Original: We use 100k molecules for training, 18k for validation, and 13k for testing.
Revised: We use 100k molecules for training, 18k for validation, and 13k for testing.

Original: Table 1 shows the Mean Absolute Error for the prediction of 12 molecular properties for weak and strong conditioning methods.
Revised: Table 1 shows the Mean Absolute Error for the prediction of 12 molecular properties for weak and strong conditioning methods.

Original: We can clearly see that using strong conditioning shows lower
Revised: We can clearly see that using strong conditioning shows lower

Original: MAE than weak conditioning for all properties by an average of 10% .
Revised: MAE than weak conditioning for all properties by an average of 10% .

Original: Due to a large number of data points and our limited computational resources we limit our experiments to weak and strong conditioning methods.
Revised: Due to a large number of data points and our limited computational resources we limit our experiments to weak and strong conditioning methods.

Original: Training one epoch of pure-EGNN was on average 267x longer than training strong-EGNN, which made it intractable to validate pure conditioning.
Revised: Training one epoch of pure-EGNN was on average 267x longer than training strong-EGNN, which made it intractable to validate pure conditioning.

Original: [Table]
Revised: [Table]

Original: The advantage in the performance of strong conditioning over weak conditioning is clearly shown in Table 1.
Revised: The advantage in the performance of strong conditioning over weak conditioning is clearly shown in Table 1.

Original: We also hypothesize that the advantage of the strong conditioning method will be more evident in constrained settings of shallow networks.
Revised: We also hypothesize that the advantage of the strong conditioning method will be more evident in constrained settings of shallow networks.

Original: In order to test whether strong conditioning provide a more efficient parametrization then weak conditioning, we repeat experiments from Table 1 on three molecular properties, but with a more shallow network.
Revised: In order to test whether strong conditioning provide a more efficient parametrization then weak conditioning, we repeat experiments from Table 1 on three molecular properties, but with a more shallow network.

Original: Table 2a shows the MAE for the prediction of three molecular properties for weak and strong conditioning methods for models with 7 and 3 layers.
Revised: Table 2a shows the MAE for the prediction of three molecular properties for weak and strong conditioning methods for models with 7 and 3 layers.

Original: On average improvement of strong conditioning over weak conditioning was more highlighted for the shallow network than for the deep network (10.8% vs 7.7%).
Revised: On average improvement of strong conditioning over weak conditioning was more highlighted for the shallow network than for the deep network (10.8% vs 7.7%).

Original: It demonstrates the importance of conditioning methods in models with less capacity.
Revised: It demonstrates the importance of conditioning methods in models with less capacity.

Original: We define conditioning depth as the number of conditional layers in the message function.
Revised: We define conditioning depth as the number of conditional layers in the message function.

Original: In our experiments, the conditioning depth was 2 as we conditioned all two layers in the message function.
Revised: In our experiments, the conditioning depth was 2 as we conditioned all two layers in the message function.

Original: We test the effect of conditioning depth for different conditioning methods.
Revised: We test the effect of conditioning depth for different conditioning methods.

Original: Table 2b shows the
Revised: Table 2b shows the

Original: MAE for the prediction of five molecular properties for weak and strong conditioning methods for models with conditioning depths 1 and 2.
Revised: MAE for the prediction of five molecular properties for weak and strong conditioning methods for models with conditioning depths 1 and 2.

Original: The weak-EGNN shows better performance with single conditional layer while strong-EGNN benefits more from two conditional layers.
Revised: The weak-EGNN shows better performance with single conditional layer while strong-EGNN benefits more from two conditional layers.

Original: MD17 MD

Revised: MD17 MD

Original: (Chmiela et al., 2017) is a dataset of eight small organic molecules containing up to 17 total atoms composed of the atoms H, C, N, O, F.
Revised: (Chmiela et al., 2017) is a dataset of eight small organic molecules containing up to 17 total atoms composed of the atoms H, C, N, O, F.

Original: For each molecule, an ab-initio molecular dynamics simulation was run using DFT to calculate the ground state energy and forces.
Revised: For each molecule, an ab-initio molecular dynamics simulation was run using DFT to calculate the ground state energy and forces.

Original: At intermittent timesteps, the energy, forces, and configuration (positions of each atom) were recorded.
Revised: At intermittent timesteps, the energy, forces, and configuration (positions of each atom) were recorded.

Original: We uniformly sample 50k molecules for training, 10k for validation, and 10k for testing.
Revised: We uniformly sample 50k molecules for training, 10k for validation, and 10k for testing.

Original: Table 3 shows the Mean Absolute Error for the prediction of energies for 8 molecules for three conditioning methods.
Revised: Table 3 shows the Mean Absolute Error for the prediction of energies for 8 molecules for three conditioning methods.

Original: From Table 3 we can see that strong conditioning improved the performance of 5 molecules by an average of 14.5%.
Revised: From Table 3 we can see that strong conditioning improved the performance of 5 molecules by an average of 14.5%.

Original: It did not show improvement in molecules that initially had high MAE.
Revised: It did not show improvement in molecules that initially had high MAE.

Original: In experiments with pure conditioning, to decrease computational cost we used half of the model: 2 → 1 conditional layers, hidden embedding size of 128 → 64 , and 7 → 3 layers.
Revised: In experiments with pure conditioning, to decrease computational cost we used half of the model: 2 → 1 conditional layers, hidden embedding size of 128 → 64 , and 7 → 3 layers.

Original: With
Revised: With

Original: [Table]
Revised: [Table]

Original: these changes, a pure-EGNN model was 14.3x and 16x times slower than strong-EGNN and weakEGNN respectively.
Revised: these changes, a pure-EGNN model was 14.3x and 16x times slower than strong-EGNN and weakEGNN respectively.

Original: The negative impact of decreasing the number of layers on the performance of strong-EGNN on the QM9 dataset can be seen in Table 2a.
Revised: The negative impact of decreasing the number of layers on the performance of strong-EGNN on the QM9 dataset can be seen in Table 2a.

Original: Considering that we reduced by half the number of layers, hidden embedding size, and conditioning depth, pure-EGNN shows competitive performance with weak/strong-EGNN on 5 molecules and outperforms it on some of them.
Revised: Considering that we reduced by half the number of layers, hidden embedding size, and conditioning depth, pure-EGNN shows competitive performance with weak/strong-EGNN on 5 molecules and outperforms it on some of them.

Original: This potential gain in performance might, however, not outweigh the computation costs of pure over strong or weak conditioning.
Revised: This potential gain in performance might, however, not outweigh the computation costs of pure over strong or weak conditioning.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work
Revised: In this work

Original: [Table]
Revised: [Table]

Original: we explore how graph neural networks can recursively update node embeddings with edge attribute information such as geometric distance.
Revised: we explore how graph neural networks can recursively update node embeddings with edge attribute information such as geometric distance.

Original: We provide a unifying analysis of several works in literature that utilize attributes, through a notion of conditional linear layers.
Revised: We provide a unifying analysis of several works in literature that utilize attributes, through a notion of conditional linear layers.

Original: We present three conditioning methods to this end: weak, strong [Table] and pure.
Revised: We present three conditioning methods to this end: weak, strong, and pure.

Original: Weak conditioning method concatenates edge attributes to node features, strong conditioning method gates node features
Revised: Weak conditioning method concatenates edge attributes to node features, strong conditioning method gates node features

Original: [Table]
Revised: [Table]

Original: and in pure conditioning, edge attributes causally parametrize transformation matrices.
Revised: and in pure conditioning, edge attributes causally parametrize transformation matrices.

Original: We explain the intuition of each method and apply them to the EGNN model to empirically show their difference in performance and computational cost on QM9 and MD17 datasets.
Revised: We explain the intuition of each method and apply them to the EGNN model to empirically show their difference in performance and computational cost on QM9 and MD17 datasets.

Original: Our conclusion is that strong conditioning (gating) generally beats weak conditioning (concatenation) in the message passing framework.
Revised: Our conclusion is that strong conditioning (gating) generally beats weak conditioning (concatenation) in the message passing framework.

Original: We also conclude that pure conditioning is computationally prohibitive in geometric message passing, whilst it can achieve competitive performance with weak and strong conditioning methods with a smaller network.
Revised: We also conclude that pure conditioning is computationally prohibitive in geometric message passing, whilst it can achieve competitive performance with weak and strong conditioning methods with a smaller network.

Original: This confirms the impact observed by other works on separable convolutions, s.a. in Depth-wise separable convolutions (Chollet, 2017) and ConvNeXt (Liu et al., 2022), and justifies the use of separable (group) convolutions of steerable tensor-based methods such as TFN (Thomas et al., 2018).
Revised: This confirms the impact observed by other works on separable convolutions, s.a. in Depth-wise separable convolutions (Chollet, 2017) and ConvNeXt (Liu et al., 2022), and justifies the use of separable (group) convolutions of steerable tensor-based methods such as TFN (Thomas et al., 2018).

Original: While these methods can be formulated as linear message passing methods of the convolutional form, we show that performance gains can be achieved through multi-layer conditional message functions, as a form of non-linear convolution (Brandstetter et al., 2021).
Revised: While these methods can be formulated as linear message passing methods of the convolutional form, we show that performance gains can be achieved through multi-layer conditional message functions, as a form of non-linear convolution (Brandstetter et al., 2021).

Original: In this setting, we show that it can be beneficial to condition all layers in the conditional MLP, rather than only condition the first layer as is the convention.
Revised: In this setting, we show that it can be beneficial to condition all layers in the conditional MLP, rather than only condition the first layer as is the convention.

Original: We believe that our categorization of conditioning methods, combined with our empirical findings can be used as a guideline in designing the next generation of geometric graph neural network architectures.
Revised: We believe that our categorization of conditioning methods, combined with our empirical findings can be used as a guideline in designing the next generation of geometric graph neural network architectures.