Original: L EARNING FROM P ROTEIN S TRUCTURE WITH G EOMETRIC V ECTOR P ERCEPTRONS
Revised: L EARNING FROM P ROTEIN S TRUCTURE WITH G EOMETRIC V ECTOR P ERCEPTRONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning on 3D structures of large biomolecules is emerging as a distinct area in machine learning, but there has yet to emerge a unifying network architecture that simultaneously leverages the geometric and relational aspects of the problem domain.
Revised: Learning on 3D structures of large biomolecules is emerging as a distinct area in machine learning, but there has yet to emerge a unifying network architecture that simultaneously leverages the geometric and relational aspects of the problem domain.

Original: To address this gap, we introduce geometric vector perceptrons , which extend standard dense layers to operate on collections of Euclidean vectors.
Revised: To address this gap, we introduce geometric vector perceptrons , which extend standard dense layers to operate on collections of Euclidean vectors.

Original: Graph neural networks equipped with such layers are able to perform both geometric and relational reasoning on efﬁcient representations of macromolecules.
Revised: Graph neural networks equipped with such layers are able to perform both geometric and relational reasoning on efﬁcient representations of macromolecules.

Original: We demonstrate our approach on two important problems in learning from protein structure: model quality assessment and computational protein design.
Revised: We demonstrate our approach on two important problems in learning from protein structure: model quality assessment and computational protein design.

Original: Our approach improves over existing classes of architectures on both problems, including state-ofthe-art convolutional neural networks and graph neural networks.
Revised: Our approach improves over existing classes of architectures on both problems, including state-ofthe-art convolutional neural networks and graph neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many efforts in structural biology aim to predict, or derive insights from, the structure of a macromolecule (such as a protein, RNA, or DNA), represented as a set of positions associated with atoms or groups of atoms in 3D Euclidean space.
Revised: Many efforts in structural biology aim to predict, or derive insights from, the structure of a macromolecule (such as a protein, RNA, or DNA), represented as a set of positions associated with atoms or groups of atoms in 3D Euclidean space.

Original: These problems can often be framed as functions mapping the input domain of structures to some property of interest—for example, predicting the quality of a structural model or determining whether two molecules will bind.
Revised: These problems can often be framed as functions mapping the input domain of structures to some property of interest—for example, predicting the quality of a structural model or determining whether two molecules will bind.

Original: Thanks to their importance and difﬁculty, such problems, which we broadly refer to as learning from structure , have recently developed into an exciting and promising application area for deep learning (Graves et al., 2020;
Revised: Thanks to their importance and difﬁculty, such problems, which we broadly refer to as learning from structure , have recently developed into an exciting and promising application area for deep learning (Graves et al., 2020;

Original: Ingraham et al., 2019; Pereira et al., 2016; Townshend et al., 2019; Won et al., 2019).
Revised: Ingraham et al., 2019; Pereira et al., 2016; Townshend et al., 2019; Won et al., 2019).

Original: Successful applications of deep learning are often driven by techniques that leverage the problem structure of the domain—for example, convolutions in computer vision (Cohen & Shashua, 2016) and attention in natural language processing (Vaswani et al., 2017).
Revised: Successful applications of deep learning are often driven by techniques that leverage the problem structure of the domain—for example, convolutions in computer vision (Cohen & Shashua, 2017) and attention in natural language processing (Vaswani et al., 2017).

Original: What are the relevant considerations in the domain of learning from structure?
Revised: What are the relevant considerations in the domain of learning from structure?

Original: Using proteins as the most common example, we have on the one hand the arrangement and orientation of the amino acids in space, which govern the dynamics and function of the molecule (Berg et al., 2002).
Revised: Using proteins as the most common example, we have on the one hand the arrangement and orientation of the amino acids in space, which govern the dynamics and function of the molecule (Berg et al., 2002).

Original: On the other hand, proteins also possess relational structure in terms of their amino-acid sequence and the residue-residue interactions that mediate the aforementioned protein properties (Hammes-Schiffer & Benkovic, 2006).
Revised: On the other hand, proteins also possess relational structure in terms of their amino-acid sequence and the residue-residue interactions that mediate the aforementioned protein properties (Hammes-Schiffer & Benkovic, 2006).

Original: We refer to these as the geometric and relational aspects of the problem domain, respectively.
Revised: We refer to these as the geometric and relational aspects of the problem domain, respectively.

Original: Recent state-of-the-art methods for learning from structure are successful by leveraging one of these two aspects.
Revised: Recent state-of-the-art methods for learning from structure are successful by leveraging one of these two aspects.

Original: Commonly, such methods either employ graph neural networks (GNNs), which are expressive in terms of relational reasoning (Battaglia et al., 2018), or convolutional neural networks (CNNs), which operate directly on the geometry of the structure.
Revised: Commonly, such methods either employ graph neural networks (GNNs), which are expressive in terms of relational reasoning (Battaglia et al., 2018), or convolutional neural networks (CNNs), which operate directly on the geometry of the structure.

Original: Here, we present a unifying architecture that bridges these two families of methods to leverage both aspects of the problem domain.
Revised: Here, we present a unifying architecture that bridges these two families of methods to leverage both aspects of the problem domain.

Original: We do so by introducing geometric vector perceptrons (GVPs), a drop-in replacement for standard multi-layer perceptrons (MLPs) in aggregation and feed-forward layers of GNNs.
Revised: We do so by introducing geometric vector perceptrons (GVPs), a drop-in replacement for standard multi-layer perceptrons (MLPs) in aggregation and feed-forward layers of GNNs.

Original: GVPs operate directly on both scalar and geometric features—features that transform as a vector under a rotation of spatial coordinates.
Revised: GVPs operate directly on both scalar and geometric features—features that transform as a vector under a rotation of spatial coordinates.

Original: GVPs therefore allow for the embedding of geometric information at nodes and edges without reducing such information to scalars that may not fully capture complex geometry.
Revised: GVPs therefore allow for the embedding of geometric information at nodes and edges without reducing such information to scalars that may not fully capture complex geometry.

Original: We postulate that our approach makes it easier for a GNN to learn functions whose signiﬁcant features are both geometric and relational.
Revised: We postulate that our approach makes it easier for a GNN to learn functions whose signiﬁcant features are both geometric and relational.

Original: Our method (GVP-GNN) can be applied to any problem where the input domain is a structure of a single macromolecule or of molecules bound to one another.
Revised: Our method (GVP-GNN) can be applied to any problem where the input domain is a structure of a single macromolecule or of molecules bound to one another.

Original: In this work, we speciﬁcally demonstrate our approach on two problems connected to protein structure: model quality assessment and computational protein design.
Revised: In this work, we speciﬁcally demonstrate our approach on two problems connected to protein structure: model quality assessment and computational protein design.

Original: Model quality assessment (MQA) aims to select the best structural model of a protein from a large pool of candidate structures and is a crucial step in protein structure prediction (Cheng et al., 2019).
Revised: Model quality assessment (MQA) aims to select the best structural model of a protein from a large pool of candidate structures and is a crucial step in protein structure prediction (Cheng et al., 2019).

Original: Computational protein design (CPD) is the conceptual inverse of structure prediction, aiming to infer an amino acid sequence that will fold into a given structure.
Revised: Computational protein design (CPD) is the conceptual inverse of structure prediction, aiming to infer an amino acid sequence that will fold into a given structure.

Original: Our method outperforms existing methods on both tasks.
Revised: Our method outperforms existing methods on both tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: ML methods for learning from protein structure largely fall into one of three types, operating on sequential , voxelized , or graph-structured representations of proteins.
Revised: ML methods for learning from protein structure largely fall into one of three types, operating on sequential , voxelized , or graph-structured representations of proteins.

Original: We brieﬂy discuss each type and introduce state-of-the-art examples for MQA and CPD to set the stage for our experiments later.
Revised: We brieﬂy discuss each type and introduce state-of-the-art examples for MQA and CPD to set the stage for our experiments later.

Original: Sequential representations In traditional models of learning from protein structure, each amino acid is represented as a feature vector using hand-crafted representations of the 3D structural environment, such as contact-based features (Olechnoviˇc & Venclovas, 2017), orientations or positions collectively projected to local coordinates (Karasikov et al., 2019; Wang et al., 2018), and physicsinspired energy terms (O’Connell et al., 2018; Uziela et al., 2017).
Revised: Sequential representations In traditional models of learning from protein structure, each amino acid is represented as a feature vector using hand-crafted representations of the 3D structural environment. These representations include residue contacts (Olechnovi ˇ c & Venclovas, 2017), orientations or positions collectively projected to local coordinates (Karasikov et al., 2019), physicsinspired energy terms (O’Connell et al., 2018; Uziela et al., 2017), or context-free grammars of protein topology (Greener et al., 2018).

Original: The structure is then viewed as a sequence  of such feature vectors which can be fed into a 1D convolutional network, RNN, or dense feedforward network.
Revised: The structure is then viewed as a sequence or collection of such features which can be fed into a 1D convolutional network, RNN, or dense feedforward network.

Original: Although these methods only indirectly represent the full 3D structure of the protein, a number of them, such as ProQ3D (Uziela et al., 2017), VoroMQA (Olechnovi ˇ c & Venclovas, 2017), and SBROD (Karasikov et al., 2019), are competitive in assessments of MQA.
Revised: Although these methods only indirectly represent the full 3D structure of the protein, a number of them, such as ProQ4 (Hurtado et al., 2018), VoroMQA (Olechnoviˇc & Venclovas, 2017), and SBROD (Karasikov et al., 2019), are competitive in assessments of MQA.

Original: Voxelized representations In lieu of hand-crafted representations of structure, 3D convolutional neural networks (CNNs) can operate directly on the positions of atoms in space, encoded as occupancy maps in a voxelized 3D volume.
Revised: Voxelized representations In lieu of hand-crafted representations of structure, 3D convolutional neural networks (CNNs) can operate directly on the positions of atoms in space, encoded as occupancy maps in a voxelized 3D volume.

Original: The hierarchical convolutions of such networks are easily compatible with the detection of structural motifs, binding pockets, and the speciﬁc shapes of other important structural features, leveragingwhat we call the geometric aspect of the domain.
Revised: The hierarchical convolutions of such networks are easily compatible with the detection of structural motifs, binding pockets, and the speciﬁc shapes of other important structural features, leveraging the geometric aspect of the domain.

Original: The MQA methods 3DCNN (Derevyanko et al., 2018) and Ornate (Pag ` es et al., 2019) and a number of CPD methods (Anand et al., 2020;
Revised: The MQA methods 3DCNN (Derevyanko et al., 2018) and Ornate (Pag ` es et al., 2019) and a number of CPD methods (Anand et al., 2020;

Original: Zhang et al.,  2019) exemplify the power of this approach.
Revised: Zhang et al., 2019; Shroff et al., 2019) exemplify the power of this approach.

Original: Graph-structured representations A3D protein structure can also be represented as a proximity graph over amino acid nodes, reducing the challenging task of representing a collective structural neighborhood in a single feature vector to that of representing individual edges.
Revised: Graph-structured representations A protein structure can also be represented as a proximity graph over amino acid nodes, reducing the challenge of representing a collective structural neighborhood in a single feature vector to that of representing individual edges.

Original: Graph neural networks (GNNs) can then perform complex relational reasoning over structures (Battaglia et al., 2018)—perhaps identifying key relationships amonga set of amino acids, or ﬂexible structural motifsbest described as a connectivity pattern rather than a rigid shape.
Revised: Graph neural networks (GNNs) can then perform complex relational reasoning over structures (Battaglia et al. , 2018)—for example, identifying key relationships among amino acids, or ﬂexible structural motifs described as a connectivity pattern rather than a rigid shape.

Original: While some GNN methods, such as GraphQA(MQA) (Baldassarre et al., 2020), simply represent edges by their length, others, such as Structured Transformer (CPD) (Ingraham et al., 2019), indirectly encode the 3D geometry of the proximity graph in terms of relative orientations and other scalar features.
Revised: Recent state-of-the-art GNNs include GraphQA (Baldassarre et al., 2020) on MQA, Structured Transformer (Ingraham et al., 2019) on CPD, and ProteinSolver (Strokach et al., 2020) on CPD and mutation stability prediction. These methods vary in their representation of geometry: while some, such as GraphQA and ProteinSolver, represent edges as a function of their length, others, such as Structured Transformer, indirectly encode the 3D geometry of the proximity graph in terms of relative orientations and other scalar features.

Original:  M ETHODS
Revised:  M ETHODS

Revised: Our architecture seeks to combine the strengths of CNN and GNN methods in learning from biomolecular structure by improving the latter’s ability to reason geometrically .

Original: The GNNs described in the previous section encode the 3D geometry of the protein by encoding vector features (such as node orientations and edge directions) in terms of rotation-invariant scalars , often by deﬁning a local coordinate system at each node.
Revised: The GNNs described in the previous section encode the 3D geometry of the protein by encoding vector features (such as node orientations and edge directions) in terms of rotation-invariant scalars , often by deﬁning a local coordinate system at each node.

Original: We instead propose that these features be directly represented as geometric vectors —features in R 3 which transform appropriately under a change of spatial coordinates—at all steps of graph propagation.
Revised: We instead propose that these features be directly represented as geometric vectors —features in R 3 which transform appropriately under a change of spatial coordinates—at all steps of graph propagation.

Original: This conceptual shift has two important ramiﬁcations.
Revised: This conceptual shift has two important ramiﬁcations.

Original: First, the input representation is more efﬁcient: instead of encoding the orientation of a node by its relative orientation with all of its neighbors, we only have to represent one absolute orientation per node.
Revised: First, the input representation is more efﬁcient: instead of encoding the orientation of a node by its relative orientation with all of its neighbors, we only have to represent one absolute orientation per node.

Original: Second, it standardizes a global coordinate system across the entire structure, which allows geometric features to be directly propagated without transforming between local coordinates.
Revised: Second, it standardizes a global coordinate system across the entire structure, which allows geometric features to be directly propagated without transforming between local coordinates.

Revised: For example, representations of arbitrary positions in space—including points that are not themselves nodes—can be easily propagated across the graph by Euclidean vector addition.

Original: We postulate this allows the GNN to more easily access global geometric properties of the structure.
Revised: We postulate this allows the GNN to more easily access global geometric properties of the structure.

Original: The key challenge with such a representation, however, is to perform graph propagation in a way that simultaneously preserves the full expressive power of the original
Revised: The key challenge with this representation, however, is to perform graph propagation in a way that simultaneously preserves the full expressive power of the original

Original: GNN while maintaining the rotation invariance provided by the scalar representations.
Revised: GNN while maintaining the rotation invariance provided by the scalar representations.

Original: We do so by introducing a new module, the geometric vector perceptron , to replace dense layers in a GNN.
Revised: We do so by introducing a new module, the geometric vector perceptron , to replace dense layers in a GNN.

Original: 3. G EOMETRIC V ECTOR P ERCEPTRONS
Revised: 3. G EOMETRIC V ECTOR P ERCEPTRONS

Original: The geometric vector perceptron is a simple module for learning vector-valued and scalar-valued functions over geometric vectors and scalars.
Revised: The geometric vector perceptron is a simple module for learning vector-valued and scalar-valued functions over geometric vectors and scalars.

Original: That is, given a tuple ( s , V ) of scalar features s ∈ R n and vector features V ∈ R ν × 3 , we compute new features ( s (cid:48) , V (cid:48) )
Revised: That is, given a tuple ( s , V ) of scalar features s ∈ R n and vector features V ∈ R ν × 3 , we compute new features ( s (cid:48) , V (cid:48) )

Original: ∈
Revised: ∈

Original: R m × R µ × 3 .
Revised: R m × R µ × 3 .

Original: The computation is formally described in Algorithm 1 and illustrated in Figure 1A.
Revised: The computation is formally described in Algorithm 1 and illustrated in Figure 1A.

Original: At its core, the GVP consists of two separate linear transformations W m , W h for the scalar and vector features, followed by nonlinearities σ, σ + .
Revised: At its core, the GVP consists of two separate linear transformations W m , W h for the scalar and vector features, followed by nonlinearities σ, σ + .

Original: However, before the scalar features are transformed, we concatenate the L 2 norm of the transformed vector features V h ; this allows us to extract rotation-invariant information from the input vectors V .
Revised: However, before the scalar features are transformed, we concatenate the L 2 norm of the transformed vector features V h ; this allows us to extract rotation-invariant information from the input vectors V .

Original: An additional linear transformation W µ is inserted just before the vector nonlinearity to control the output dimensionality independently of the number of norms extracted.
Revised: An additional linear transformation W µ is inserted just before the vector nonlinearity to control the output dimensionality independently of the number of norms extracted.

Original: Input : Scalar and vector features (
Revised: Input : Scalar and vector features (

Original: s , V ) ∈ R n × R ν × 3 .
Revised: s , V ) ∈ R n × R ν × 3 .

Original: Output : Scalar and vector features ( s (cid:48) , V (cid:48) )
Revised: Output : Scalar and vector features ( s (cid:48) , V (cid:48) )

Original: ∈
Revised: ∈

Original: R m × R µ × 3 .
Revised: R m × R µ × 3 .

Original: h ← max ( ν, µ )
Revised: h ← max [Equation]

Revised: [Equation]

Original: GVP :
Revised: GVP :

Original: V h ← W h V ∈ R h × V µ ← W µ V h ∈ R µ ×s h ← (cid:107) V h (cid:107) 2 (
Revised: [Equation] V µ ← W µ V h ∈ R µ ×s h ← (cid:107) V h (cid:107) 2 (

Original: row-wise ) ∈
Revised: row-wise ) ∈

Original: R h v µ ← (cid:107) V µ (cid:107) 2 ( row-wise ) ∈ R µ s h + n ← concat ( s h , s )
Revised: R h v µ ← (cid:107) V µ (cid:107) 2 ( row-wise ) ∈ R µ s h + n ← concat ( s h , s )

Original: ∈
Revised: ∈

Original: R h + n s m ← W m s h + n + b ∈ R m s (cid:48) ← σ
Revised: R h + n s m ← W m s h + n + b ∈ R m s (cid:48) ← σ

Original: ( s m ) ∈
Revised: ( s m ) ∈

Original: R m
Revised: R m

Original: V (cid:48) ← σ + ( v µ ) (cid:12) V µ ,
Revised: V (cid:48) ← σ + ( v µ ) (cid:12) V µ ,

Original: ( row-wise multiplication ) ∈ R µ ×return ( s (cid:48) , V (cid:48) )
Revised: ( row-wise multiplication ) ∈ R µ ×return ( s (cid:48) , V (cid:48) )

Original: The GVP is conceptually simple, yet provably possesses the desired properties of invariance/equivariance and expressiveness.
Revised: The GVP is conceptually simple, yet provably possesses the desired properties of invariance/equivariance and expressiveness.

Original: First, the vector and scalar outputs of the GVP are equivariant and invariant, respectively, with respect to an arbitrary composition  of rotations and reﬂections in 3D Euclidean space described by R i.e. , 
Revised: First, the vector and scalar outputs of the GVP are equivariant and invariant, respectively, with respect to an arbitrary composition R of rotations and reﬂections in 3D Euclidean space — i.e. , if GVP ( s , V ) = ( s (cid:48) , V (cid:48) ) then

Original: [Equation]
Revised: [Equation]

Original: This is due to the fact that the only operations on vector-valued inputs are scalar multiplication, linear combination, and the L 2 norm.
Revised: This is due to the fact that the only operations on vector-valued inputs are scalar multiplication, linear combination, and the L 2 norm.

Original: 1 We include a formal proof in Appendix A.
Revised: 1 We include a formal proof in Appendix A.

Original: In addition, a GVP can approximate any continuous rotation- and reﬂection-invariant scalar-valued function of V .
Revised: In addition, a GVP can approximate any continuous rotation- and reﬂection-invariant scalar-valued function of V .

Original: More precisely, let G s be a GVP deﬁned with n, µ = 0 —that is, the part of a GVP that transforms vector features to scalar features.
Revised: More precisely, let G s be a GVP deﬁned with n, µ = 0 —that is, the part of a GVP that transforms vector features to scalar features.

Original: Then G s is able to (cid:15) -approximate[-a function F : R ν × 
-]
Revised: Then G s is able to (cid:15) -approximate[-a function F : R ν × 
-]

Original:  → R that is invariant with respect to rotations and reﬂections in 3D under mild assumptions.
Revised: any function F : R ν × 3 → R that is invariant with respect to rotations and reﬂections in 3D under mild assumptions.

Original: Theorem.
Revised: Theorem.

Original: Let R describe an arbitrary rotation or reﬂection in R 3 .
Revised: Let R describe an arbitrary rotation and/or reﬂection in R 3 .

Original: For ν ≥ 3 let Ω ν ⊂ R ν × 3 be the set of all V =
Revised: For ν ≥ 3 let Ω ν ⊂ R ν ×be the set of all V =

Original: [ v 1 , . .
Revised: [ v 1 , . .

Original: . , v ν ] T ∈ R ν × 3 such that v 1 , v 2 , v 3 are linearly independent and 0 <
Revised: . , v ν ] T ∈ R ν × 3 such that v 1 , v 2 , v 3 are linearly independent and 0 <

Original: || v i ||
Revised: || v i ||

Original: 2 ≤ b for all i and some ﬁnite b > 0 .
Revised: 2 ≤ b for all i and some ﬁnite b > 0 .

Original: Then for any continuous F : Ω ν
Revised: Then for any continuous F : Ω ν

Original: → R such that F ( R ( V ))
Revised: → R such that F ( R ( V ))

Original: =
Revised: =

Original: F ( V ) and for any (cid:15) > 0 , there exists a form f ( V ) =
Revised: F ( V ) and for any (cid:15) > 0 , there exists a form f ( V ) =

Original: w T G s ( V ) such that | F ( V ) − f
Revised: w T G s ( V ) such that | F ( V ) − f

Original: ( V )
Revised: ( V )

Original: | < (cid:15) for all V ∈ Ω ν .
Revised: | < (cid:15) for all V ∈ Ω ν .

Original: We include a formal proof in Appendix A. As a corollary, a GVP with nonzero n, µ is also able to approximate similarly-deﬁned functions over the full input domain R n × R ν × 3 .
Revised: We include a formal proof in Appendix A. As a corollary, a GVP with nonzero n, µ is also able to approximate similarly-deﬁned functions over the full input domain R n × R ν × 3 .

Original: In addition to the GVP layer itself, we use a version of dropout that drops entire vector channels at random (as opposed to coordinates within vector channels).
Revised: In addition to the GVP layer itself, we use a version of dropout that drops entire vector channels at random (as opposed to coordinates within vector channels).

Original: We also introduce layer normalization for the vector features as
Revised: We also introduce layer normalization for the vector features as

Original: [Equation]
Revised: [Equation]

Original: j || v j ||∀ i , (2) where v i ∈ R 3 are theindividual row vectors of the vector feature matrix R ν × 3 .
Revised: That is, we scale the row vectors of V such that their root-mean-square norm is one.

Original: This vector layer norm has no trainable parameters, but we continue to use normal layer normalization on scalar channels with trainable parameters γ, β .
Revised: This vector layer norm has no trainable parameters, but we continue to use normal layer normalization on scalar channels with trainable parameters γ, β .

Original: We study our hypothesis that GVPs augment the geometric reasoning ability of GNNs on a synthetic dataset (Appendix B).
Revised: We study our hypothesis that GVPs augment the geometric reasoning ability of GNNs on a synthetic dataset (Appendix B).

Original: The synthetic dataset allows us to control the function underlying the groundtruth label in order to explicitly separate geometric and relational aspects in different tasks.
Revised: The synthetic dataset allows us to control the function underlying the groundtruth label in order to explicitly separate geometric and relational aspects in different tasks.

Original: The
Revised: The

Original: GVP-GNN matches a CNN on a geometric task and a  GNN on a relational task.
Revised: GVP-augmented GNN (or GVP-GNN) matches a CNN on a geometric task and a standard GNN on a relational task.

Original: However, when we combine the two tasks in one objective, the GVP-GNN does signiﬁcantly better than either a GNN or a CNN.
Revised: However, when we combine the two tasks in one objective, the GVP-GNN does signiﬁcantly better than either a GNN or a CNN

Original: 3. R EPRESENTATIONS OF P ROTEINS
Revised: 3. R EPRESENTATIONS OF P ROTEINS

Original: The main empirical validation of our architecture is its performance on two real-world tasks: model quality assessment (MQA) and computational protein design (CPD).
Revised: The main empirical validation of our architecture is its performance on two real-world tasks: model quality assessment (MQA) and computational protein design (CPD).

Original: These tasks, as illustrated in
Revised: These tasks, as illustrated in

Original: Figure 1B and described in detail in Section 4, are complementary in that one (MQA) predicts a global property and the other (CPD) predicts a property for each amino acid.
Revised: Figure 1B and described in detail in Section 4, are complementary in that one (MQA) predicts a global property and the other (CPD) predicts a property for each amino acid.

Original: We represent a protein structure input as a proximity graph with a minimal number of scalar and vector features to fully specify the 3D structure of the molecule.
Revised: We represent a protein structure input as a proximity graph with a minimal number of scalar and vector features to fully specify the 3D structure of the molecule.

Original: A protein structure is a sequence of amino acids, where each amino acid consists of four backbone atoms 2 and a set of sidechain atoms located in 3D Euclidean space.
Revised: A protein structure is a sequence of amino acids, where each amino acid consists of four backbone atoms 2 and a set of sidechain atoms located in 3D Euclidean space.

Original: Here we represent only the backbone because our MQA benchmark corresponds to the assessment of backbone structure.
Revised: Here we represent only the backbone because our MQA benchmark corresponds to the assessment of backbone structure.

Original: In CPD, the sidechains are by deﬁnition unknown.
Revised: In CPD, the sidechains are by deﬁnition unknown.

Original: Let X i be the position of atom X in the i th amino acid.
Revised: Let X i be the position of atom X in the i th amino acid (e.g. N i is the position of the nitrogen atom in the i th amino acid).

Original: We represent backbone structure as a graph G =
Revised: We represent backbone structure as a graph G =

Original: ( V , E ) where each node v i ∈ V corresponds to an amino acid and has embedding h ( i ) v with the following features:
Revised: ( V , E ) where each node v i ∈ V corresponds to an amino acid and has embedding h ( i ) v with the following features:

Original: • Scalar features { sin , cos }◦{ φ, ψ, ω } , where φ, ψ, ω are the dihedral angles computed from C
Revised: • Scalar features { sin , cos }◦{ φ, ψ, ω } , where φ, ψ, ω are the dihedral angles computed from C

Original: i − 1 , N i , C α i , C i , and N i 
Revised: i − 1 , N i , C α i , C i , and N i +1 .

Original: • The forward and reverse unit vectors in the directions of C α
Revised: • The forward and reverse unit vectors in the directions of C α

Original: i +1 − C
Revised: i +1 − C

Original: α i and C α
Revised: α i and C α

Original: i − 1 − C α i , respectively.
Revised: i − 1 − C α i , respectively.

Original: • The unit vector in the imputed direction of C β
Revised: • The unit vector in the imputed direction of C β

Original: i − C α i .
Revised: i − C α i .

Original: This is computed by assuming tetrahedral geometry and normalizing
Revised: This is computed by assuming tetrahedral geometry and normalizing

Original: [Equation]
Revised: [Equation]

Original: where n = N i − C α i and c =
Revised: where n = N i − C α i and c =

Original: C i − C
Revised: C i − C

Original: α
Revised: α

Original: i .
Revised: i .

Original: This vector, along with the forward and reverse unit vectors, unambiguously deﬁne the orientation of each amino acid residue.
Revised: This vector, along with the forward and reverse unit vectors, unambiguously deﬁne the orientation of each amino acid residue.

Original: • A one-hot representation of amino acid identity, when available.
Revised: • A one-hot representation of amino acid identity, when available.

Original: The set of edges is E = { e i → j } i (cid:54) =
Revised: The set of edges is E = { e i → j } i (cid:54) =

Original: j for all i, j where v i is among the k = 30 nearest neighbors of v j as measured by the distance between their C α atoms.
Revised: j for all i, j where v i is among the k = 30 nearest neighbors of v j as measured by the distance between their C α atoms.

Original: Each edge has an embedding h ( i → j ) e with the following features:
Revised: Each edge has an embedding h ( i → j ) e with the following features:

Original: • The unit vector in the direction of C α
Revised: • The unit vector in the direction of C α

Original: i − C
Revised: i − C

Original: α j . • Theradial basis encoding of the distanceRBF ( || C α
Revised: α j . • The encoding of the distance || C α

Original: i − C
Revised: i − C

Original: α j || 2 ) .
Revised: α j || 2 in terms of Gaussian radial basis functions.•

Original: • A sinusoidal encoding of i − j as described in Vaswani et al.
Revised: A sinusoidal encoding of i − j as described in Vaswani et al.

Original: (2017), representing distance along the backbone.
Revised: (2017), representing distance along the backbone.

Original: In our notation, each feature vector h is a concatenation of scalar and vector features as described above.
Revised: In our notation, each feature vector h is a concatenation of scalar and vector features as described above.

Original: Collectively, these features are sufﬁcient for a complete description of the protein backbone.
Revised: Collectively, these features are sufﬁcient for a complete description of the protein backbone.

Original: 3. N ETWORK ARCHITECTURE
Revised: 3. N ETWORK A RCHITECTURE

Original: Our architecture is a GVP-augmented graph neural network (or GVP-GNN) that takes as input the protein graph deﬁned above and performs graph propagation steps to update the node embeddings:
Revised: Our architecture (GVP-GNN) leverages message passing (Gilmer et al., 2017) in which messages from neighboring nodes and edges are used to update node embeddings at each graph propagation step. More explicitly, the architecture takes as input the protein graph deﬁned above and performs graph propagation steps according to:

Original: [Equation]
Revised: [Equation]

Original: Here, g is a sequence of three GVPs.
Revised: Here, g is a sequence of three GVPs, h ( i ) v

Original: We do not update edge embeddings  and do not use a global graph embedding.
Revised: and h ( i → j ) e are the embeddings of the node i and edge ( i → j ) as above, h ( i → j ) m represents the message passed from node i to node j . k (cid:48) is the number of incoming messages, which is equal to k unless the protein contains fewer than k amino acid residues.

Original: Between graph propagation steps, we also use a feed-forward point-wise  update layer:
Revised: Between graph propagation steps, we also use a feed-forward point-wise layer to update the node embeddings at all nodes j :

Original: [Equation]
Revised: [Equation]

Original: where g is a sequence of two GVPs.
Revised: where g is a sequence of two GVPs.

Original: These graph propagation and feed-forward steps update the vector features at each node in addition to its scalar features.
Revised: These graph propagation and feed-forward steps update the vector features at each node in addition to its scalar features.

Original: In model quality assessment, we use three graph propagation steps and perform regression against the true quality score of a candidate structure, a global scalar property.
Revised: In model quality assessment, we use three graph propagation steps and perform regression against the true quality score of a candidate structure, a global scalar property.

Original: To obtain a single global representation, we apply a node-wise GVP to reduce all node embeddings to scalars.
Revised: To obtain a single global representation, we apply a node-wise GVP to reduce all node embeddings to scalars.

Original: We then average the representations across all nodes4 and apply a ﬁnal dense feed-forward network to output the network’s prediction.
Revised: We then average the representations across all nodes and apply a ﬁnal dense feed-forward network to output the network’s prediction.

Original: In computational protein design, the network learns a generative model over the space of protein sequences conditioned on the given backbone structure.
Revised: In computational protein design, the network learns a generative model over the space of protein sequences conditioned on the given backbone structure.

Original: Following Ingraham et al.
Revised: Following Ingraham et al.

Original: (2019), we frame this as an autoregressive task and use a masked encoder-decoder architecture to capture the joint distribution over all positions: for each i , the network models the distribution at i based on the complete structure graph, as well as the sequence information at positions j < i .
Revised: (2019), we frame this as an autoregressive task and use a masked encoder-decoder architecture to capture the joint distribution over all positions: for each i , the network models the distribution at i based on the complete structure graph, as well as the sequence information at positions j < i .

Original: The encoder ﬁrst performs three graph propagation steps on the structural information only.
Revised: The encoder ﬁrst performs three graph propagation steps on the structural information only.

Original: Then, sequence information is added to the graph, and the decoder performs three further graph propagation steps where incoming messages h ( i → j ) m for i ≥ j are computed only with the encoder embeddings.
Revised: Then, sequence information is added to the graph, and the decoder performs three further graph propagation steps where incoming messages h ( i → j ) m for i ≥ j are computed only with the encoder embeddings.

Original: Finally, we use one last GVP with 20-way scalar softmax output to predict the probability of the amino acids.
Revised: Finally, we use one last GVP with 20-way scalar softmax output to predict the probability of the amino acids.

Original: Further details regarding training and hyperparameters can be found in Appendix D.
Revised: Further details regarding training and hyperparameters can be found in Appendix D.

Original:  D ATASET AND EVALUATION
Revised:  D ATASET AND E VALUATION

Original: In this section, we set up the background and methodology for benchmarking the architecture on model quality assessment and protein design.
Revised: In this section, we set up the background and methodology for benchmarking the architecture on model quality assessment and protein design.

Original: Model quality assessment Model quality assessment aims to select the best structural model of a protein from a large pool of candidate structures.
Revised: Model quality assessment Model quality assessment aims to select the best structural model of a protein from a large pool of candidate structures.

Original: The performance of different MQA methods is evaluated every two years in the community-wide Critical Assessment of Structure Prediction (CASP) (Cheng et al., 2019).
Revised: The performance of different MQA methods is evaluated every two years in the community-wide Critical Assessment of Structure Prediction (CASP) (Cheng et al., 2019).

Original: For a number of recently solved but unreleased structures, called targets , structure generation programs produce a large number of candidate structures.
Revised: For a number of recently solved but unreleased structures, called targets , structure generation programs produce a large number of candidate structures.

Original: MQA methods are evaluated by how well they predict the GDT-TS score of a candidate structure compared to the experimentally solved structure for that target.
Revised: MQA methods are evaluated by how well they predict the GDT-TS score of a candidate structure compared to the experimentally solved structure for that target.

Original: GDT-TS is a scalar measure of how similar two protein backbones are after global alignment (Zemla et al., 2001).
Revised: GDT-TS is a scalar measure of how similar two protein backbones are after global alignment (Zemla et al., 2001).

Original: In addition to accurately predicting the absolute quality of a candidate structure, a good MQA method should also be able to accurately assess the relative model qualities among a pool of candidates for a given target so that the best ones can be selected, perhaps for further reﬁnement.
Revised: In addition to accurately predicting the absolute quality of a candidate structure, a good MQA method should also be able to accurately assess the relative model qualities among a pool of candidates for a given target so that the best ones can be selected, perhaps for further reﬁnement.

Original: Therefore, MQA methods are commonly evaluated on two metrics: a global correlation between the predicted and ground truth scores, pooled across all targets, and the average per-target correlation among only the candidate structures for a speciﬁc target (Cao & Cheng, 2016; Derevyanko et al., 2018; Pag ` es et al., 2019).
Revised: Therefore, MQA methods are commonly evaluated on two metrics: a global correlation between the predicted and ground truth scores, pooled across all targets, and the average per-target correlation among only the candidate structures for a speciﬁc target (Cao & Cheng, 2016; Derevyanko et al., 2018; Pag ` es et al., 2019).

Original: We follow this convention in our experiments.
Revised: We follow this convention in our experiments.

Original: We train and validate on the set of candidate structures generated in the CASP 5-10 assessments, which collectively contain 528 targets and 79200 candidate structures.
Revised: We train and validate on 79200 candidate structures for 528 targets submitted to CASP 5-10. We then test GVP-GNN on two MQA datasets.

Original: For testing, we predict model quality for a total of 20880 stage 1 and stage 2 candidate structures from CASP 11 (84 targets) and 12 (40 targets). 
Revised: First, we score 20880 stage 1 and stage 2 candidate structures from CASP 11 (84 targets) and 12 (40 targets). This benchmark was ﬁrst established by Karasikov et al.

Revised: and has been used by many recently published methods.

Revised: However, to compare with a larger number of methods on more recent structural data, we also score 1472 stage 2 candidate structures from all 20 publicly available targets in CASP 13.

Revised: We add the CASP 11structures to our training set to evaluate on CASP 13.

Original: Further detailson the dataset can be found in Appendix C.
Revised: Further details can be found in Appendix C.

Original: Protein design Computational protein design is the conceptual inverse of structure prediction, aiming to infer an amino acid sequence that will fold into a given structure.
Revised: Protein design Computational protein design is the conceptual inverse of structure prediction, aiming to infer an amino acid sequence that will fold into a given structure.

Original: In comparison to model quality assessment, computational protein design is more difﬁcult to unambiguously benchmark, as some structures may correspond to a large space of sequences and others may correspond to none at all.
Revised: In comparison to model quality assessment, computational protein design is more difﬁcult to unambiguously benchmark, as some structures may correspond to a large space of sequences and others may correspond to none at all.

Original: Therefore, the proxy metric of native sequence recovery —splitting the set of all known native structures in the PDB and attempting to design the sequences corresponding to held-out structures— is typically used to benchmark CPD models (Li et al., 2014; O’Connell et al., 2018;
Revised: Therefore, the proxy metric of native sequence recovery —splitting the set of all known native structures in the PDB and attempting to design the sequences corresponding to held-out structures— is typically used to benchmark CPD models (Li et al., 2014; O’Connell et al., 2018;

Original: Wang et al., 2018).
Revised: Wang et al., 2018).

Original: Drawing an analogy between sequence design and language modelling, Ingraham et al.
Revised: Drawing an analogy between sequence design and language modelling, Ingraham et al.

Original: (2019) also evaluate the model perplexity on held-out native sequences.
Revised: (2019) also evaluate the model perplexity on held-out native sequences.

Original: Both metrics rest on the implicit assumption that native sequences are optimized for their structures (Kuhlman & Baker, 2000) and should be assigned high probabilities. 
Revised: Both metrics rest on the implicit assumption that native sequences are optimized for their structures (Kuhlman & Baker, 2000) and should be assigned high probabilities. [Table]

Original: To best approximate real-world applications that may require design of novel structures, the heldout evaluation set should bear minimal similarity to the training structures.
Revised: To best approximate real-world applications that may require design of novel structures, the heldout evaluation set should bear minimal similarity to the training structures.

Original: We use the CATH 4.
Revised: We use the CATH 4.

Original: dataset curated by Ingraham et al.
Revised: dataset curated by Ingraham et al.

Original: (2019) in which all available structures with 40% nonredudancy are partitioned by their CATH (class, architecture, topology/fold, homologous superfamily) classiﬁcation.
Revised: (2019) in which all available structures with 40% nonredudancy are partitioned by their CATH (class, architecture, topology/fold, homologous superfamily) classiﬁcation.

Original: The canonical training, validation, and test splits consist of 18204, 608, and 1120 structures, respectively.
Revised: The canonical training, validation, and test splits consist of 18204, 608, and 1120 structures, respectively.

Original: We also report results on TS50, an older test set of 50 native structures ﬁrst introduced by Li et al.
Revised: We also report results on TS50, an older test set of 50 native structures ﬁrst introduced by Li et al.

Original: The smaller size of this benchmark  allows a comparison to the computationally expensive physics-based calculations of the fixbb protocol in Rosetta, a software suite well-established in the structural biology community (Das & Baker, 2008).
Revised: The smaller size of this benchmark also allows a comparison to the computationally expensive physics-based calculations of the fixbb protocol in Rosetta, a software suite well-established in the structural biology community (Das & Baker, 2008).

Original: No canonical training and validation sets exist for TS50.
Revised: No canonical training and validation sets exist for TS50.

Original: In order to evaluate on TS50, we remove sequences with more than 30% similarity with any structure in TS50 from our training and validation sets and retrain our model.
Revised: In order to evaluate on TS50, we remove sequences with more than 30% similarity with any structure in TS50 from the CATH 4.2 training and validation sets and retrain our model.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Model quality assessment We compare our method against state-of-the-art methods on the CASP 11-12 test set in Table 1.
Revised: Model quality assessment We compare GVP-GNN against state-of-the-art methods on the CASP 11-12 test set in Table 1.

Original: These include the CNN methods 3DCNN (Derevyanko et al., 2018)

Original: [Table] and Ornate (Pag ` es et al., 2019), the GNN method GraphQA (Baldassarre et al., 2020) [Table] and three methods that use sequential representations—VoroMQA (Olechnovi ˇ c & Venclovas, 2017), SBROD (Karasikov et al., 2019), and ProQ3D (Uziela et al., 2017).
Revised: These include the CNN methods 3DCNN (Derevyanko et al., 2018) and Ornate (Pag`es et al., 2019), the GNN method GraphQA (Baldassarre et al., 2020), and three methods that use sequential representations—VoroMQA (Olechnovi ˇ c & Venclovas, 2017), SBROD (Karasikov et al., 2019), and ProQ3D (Uziela et al., 2017).

Original: All of these methods learn solely from protein structure, 6 with the exception of ProQ3D, which in addition uses sequence information on related proteins, which is not always available.
Revised: All of these methods learn solely from protein structure, 6 with the exception of ProQ3D, which in addition uses sequence proﬁles based on alignments, which are not always available.

Original: We include ProQ3D because it is an improved version of the best single-model method in CASP 11 and CASP 12 (Uziela et al., 2017).
Revised: We include ProQ3D because it is an improved version of the best single-model method in CASP 11 and CASP 12 (Uziela et al., 2017).

Original: Our method outperforms all other structural methods in both global and per-target correlation, and even performs better than ProQ3D on all but one benchmark.
Revised: GVP-GNN outperforms all other structural methods in both global and per-target correlation, and even performs better than ProQ3D on all but one benchmark.

Original: We also evaluate DimeNet, a recent 3D-aware GNN architecture which achieves state-of-the-art on many small-molecule tasks (Klicpera et al., 2020), on CASP 11-12.
Revised: We also evaluate DimeNet, a recent 3D-aware GNN architecture which achieves state-of-the-art on many small-molecule tasks (Klicpera et al., 2019), on CASP 11-12.

Original: DimeNet differs from our method in that it uses relative edge orientations to indirectly encode geometry into its messagepassing operations.
Revised: DimeNet differs from GVP-GNN in that it uses relative edge orientations to indirectly encode geometry into its messagepassing operations.

Original: While this paradigm appears well-suited for the domain of learning from small molecules, DimeNet does not outperform any of the models in Table 1 on model quality assessment.
Revised: While this paradigm appears well-suited for the domain of learning from small molecules, DimeNet does not outperform any of the models in Table 1 on model quality assessment.

Original: See Appendix F for detailed results and discussion.
Revised: See Appendix E for detailed results and discussion.

Original: Recent MQA methods have most extensively been benchmarked on the CASP 11-12 datasets. However, for completeness, we also evaluate our method on CASP 13 (Table 2). Because of its recency, many target structures remain publicly unavailable.
Revised: We compare GVP-GNN with all 23 single-structure MQA methods participating in CASPfor which complete predictions are available. Seven of these methods were highlighted as bestperforming by the CASP organizers in Cheng et al.

Original: We use the stage 2 candidate structures of a subset of 20 targets previously used for benchmarking (Baldassarre et al., 2020). Our method achieves improved results over all other methods, including ProteinGCN (Sanyal et al., 2020), a more recent
Revised: and are shown along with GVP-GNN [Table] in Table 2.

Original: GNN method which was evaluated only on CASP 13.
Revised: These include four methods learning solely from structural features and three also using sequence proﬁles.

Original: Because of the small sample size, we emphasize that these results, although promising, should be considered preliminary until further structures for CASP 13 are available.
Revised: SASHAN learns a linear model over secondary structure and contact-based features (Cheng et al., 2019). FaeNNz 7 (Studer et al., 2020), ProQ3D (Uziela et al., 2017), and VoroMQA 8 (Olechnovi ˇ c & Venclovas, 2017) learn a multi-layer perceptron or statistical potential on top of such structural features. Finally, MULTICOM-NOVEL (Hou et al., 2019) and ProQ (Hurtado et al., 2018) employ one-dimensional deep convolutional networks on top of sequential representations.

Revised: GVP-GNN outperforms all methods in terms of global correlation and outperforms all structure-only methods in per-target correlation (see Appendix E for more details).

Revised: Because of the small size of the available CASP 13 dataset, we emphasize that these results, although promising, should be considered preliminary until experimental structures of further targets from CASPbecome publicly available.

Original: Finally, because our architecture updates vector features along with scalar features at each node embedding, it is possible to visualize learned vector features in the intermediate layers of the trained
Revised: Finally, because our architecture updates vector features along with scalar features at each node embedding, it is possible to visualize learned vector features in the intermediate layers of the trained

Original: MQA network.
Revised: MQA network.

Original: We show and discuss the interpretability of such features in Appendix E.
Revised: We show and discuss the interpretability of such features in Appendix F.

Original: Protein design Our method achieves state-of-the-art performance on CATH 4.2, representing a substantial improvement both in terms of perplexity and sequence recovery over Structured Transformer (Ingraham et al., 2019), a GNN method which was trained using the same training and validation sets (Table 4).
Revised: Protein design GVP-GNN achieves state-of-the-art performance on CATH 4.2, representing a substantial improvement both in terms of perplexity and sequence recovery over Structured Transformer (Ingraham et al., 2019), a GNN method which was trained using the same training and validation sets (Table 3).

Original: Following Ingraham et al.
Revised: Following Ingraham et al.

Original: (2019), we report evaluation on short (100 or fewer amino acid residues) and single-chain subsets of the CATH 4.2 test set, containing 94 andproteins, respectively, in addition to the full test set.
Revised: (2019), we report evaluation on short (100 or fewer amino acid residues) and single-chain subsets of the CATH 4.2 test set, containing 94 andproteins, respectively, in addition to the full test set.

Original: Although Structured Transformer leverages an attention mechanism on top of a graph-structured representation of proteins, the authors note in ablation studies that removing attention appeared to increase performance.We therefore retrain and [Table]
Revised: Although Structured Transformer leverages an attention mechanism on top of a graph-structured representation of proteins, the authors note in ablation studies that removing attention appeared to increase performance.

Original: [Table] [Table] compare against a version of Structured Transformer with the attention layers replaced with standard graph propagation operations (Structured GNN).
Revised: We therefore retrain and compare against a version of Structured Transformer with the attention layers replaced with standard graph propagation operations (Structured GNN).

Original: Our method also improves upon this model.
Revised: Our method also improves upon this model.

Original: On the smaller test set TS50, we achieve 44.9% recovery compared to Rosetta’s 30% and outperform methods based on each of the three classes of structural representations.
Revised: On the smaller test set TS50, we achieve 44.9% recovery compared to Rosetta’s 30% and outperform methods based on each of the three classes of structural representations.

Original: Overall, we place 2nd out of 8 methods in terms of recovery (see Appendix F).
Revised: Overall, we place 2nd out of 9 methods in terms of recovery (see Appendix E).

Original: However, the results for this test set should be taken with a grain of salt, given that the different methods did not use canonical training datasets.
Revised: However, the results for this test set should be taken with a grain of salt, given that the different methods did not use canonical training datasets.

Original: Ablation studies TheMQA and CPD methods we have compared against include a number of 
Revised: Ablation studies The methods we have compared against include a number of GNNs (GraphQA,

Original: GNNs (GraphQA, ProteinGCN, Structured Transformer/GNN).
Revised: Structured Transformer/GNN, ProteinSolver).

Original: We perform ablation studies to identify the aspects of the GVP which most contribute to our  improvement over these state-of-the-art models (Table 3).
Revised: We train a number of ablated models for MQA and CPD to identify the aspects of the GVP which most contribute to our performance improvement over these GNNs (Table 4).

Original: Replacing the GVP with a vanilla MLP layer eliminates the view of geometric information as geometric vectors , and propagating only scalar features forces the model to learn scalar-valued, indirect representations of geometry.
Revised: Replacing the GVP with a vanilla MLP layer or propagating only scalar features both remove direct access to geometric information, forcing the model to learn scalarvalued, indirect representations of geometry.

Original: Both modiﬁcations result in considerableand similar decreases  in performance, underscoring the importance of direct access to geometric information.
Revised: These modiﬁcations result in considerable decreases [Table] [Table] in performance, underscoring the importance of direct access to geometric information.

Revised: Propagating only the vector features results in an even larger decrease as it both eliminates important scalar input features (such as torsion angles and amino acid identity) and the part of the GVP with approximation guarantees.

Revised: Therefore, the dual scalar/vector design of the GVP is essential: without either, the best ablated model only matches GraphQA on MQA and falls short of Structured GNN on CPD.

Original: Eliminating the intermediate transformation W h results in a slightbut non-negligible decrease in performance.
Revised: Finally, eliminating the second vector transformation W µ results in a slight decrease in performance.

Original: All of these elements contributed to our improvement over Structured
Revised: Therefore, all architectural elements contributed to our improvement over state-of-the-art.

Original: GNN (on CPD) and GraphQA (on MQA).

Original: See Appendix F for a more detailed discussion.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we developed the ﬁrst architecture designed speciﬁcally for learning on dual relational and geometric representations of 3D macromolecular structure.
Revised: In this work, we developed the ﬁrst architecture designed speciﬁcally for learning on dual relational and geometric representations of 3D macromolecular structure.

Original: At its core, our method, GVP-GNN, augments graph neural networks with computationally simple layers that perform expressive geometric reasoning over Euclidean vector features.
Revised: At its core, our method, GVP-GNN, augments graph neural networks with computationally simple layers that perform expressive geometric reasoning over Euclidean vector features.

Original: Our method possesses desirable theoretical properties and empirically outperforms existing architectures on learning quality scores and sequence designs, respectively, from protein structure.
Revised: Our method possesses desirable theoretical properties and empirically outperforms existing architectures on learning quality scores and sequence designs, respectively, from protein structure.

Original: In further work, we hope to apply our architecture to other important structural biology problem areas, including protein complexes, RNA structure, and protein-ligand interactions.
Revised: In further work, we hope to apply our architecture to other important structural biology problem areas, including protein complexes, RNA structure, and protein-ligand interactions.

Original: A P ROPERTIES OF G EOMETRIC V ECTOR P ERCEPTRONS A. E QUIVARIANCE AND INVARIANCE
Revised: A P ROPERTIES OF G EOMETRIC V ECTOR P ERCEPTRONS A. E QUIVARIANCE AND I NVARIANCE

Original: The vector and scalar outputs of the GVP are equivariant and invariant, respectively, with respect to an arbitrary composition of rotations and reﬂections in 3D Euclidean space described by R i.e. ,GVP (( s , R ( V )))
Revised: The vector and scalar outputs of the GVP are equivariant and invariant, respectively, with respect to an arbitrary composition of rotations and reﬂections in 3D Euclidean space described by R i.e. ,

Original: = ( s (cid:48) , R ( V (cid:48) ))
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We can write the transformation described by R as multiplying V with a unitary matrix
Revised: We can write the transformation described by R as multiplying V with a unitary matrix

Original: U ∈ R 3 × 3 from the right.
Revised: U ∈ R 3 × 3 from the right.

Original: The L 2 -norm, scalar multiplications, and nonlinearities are deﬁned rowwise as in Algorithm 1.
Revised: The L 2 -norm, scalar multiplications, and nonlinearities are deﬁned rowwise as in Algorithm 1.

Original: We consider scalar and vector outputs separately.
Revised: We consider scalar and vector outputs separately.

Original: The scalar output, as a function of the inputs, is
Revised: The scalar output, as a function of the inputs, is

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) W h
Revised: Since (cid:107) W h

Original: VU (cid:107) 

Revised: VU (cid:107)

Original: = (cid:107) W h V (cid:107) 2 , we conclude s (cid:48) is invariant.
Revised: = (cid:107) W h V (cid:107) 2 , we conclude s (cid:48) is invariant.

Original: Similarly the vector output is
Revised: Similarly the vector output is

Original: [Equation]
Revised: [Equation]

Original: The row-wise scaling can also be viewed as left-multiplication by a diagonal matrix D .
Revised: The row-wise scaling can also be viewed as left-multiplication by a diagonal matrix D .

Original: Since (cid:107) W µ W h V (cid:107) 2 = (cid:107) W µ W h VU (cid:107) 2 , D is invariant.
Revised: Since (cid:107) W µ W h V (cid:107) 2 = (cid:107) W µ W h VU (cid:107) 2 , D is invariant.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we conclude that V (cid:48) is equivariant.
Revised: we conclude that V (cid:48) is equivariant.

Original: A. T HE ABILITY TO APPROXIMATE ROTATION - INVARIANT FUNCTIONS
Revised: A. T HE A BILITY TO A PPROXIMATE R OTATION -I NVARIANT F UNCTIONS

Original: The GVP inherits an analogue of the Universal Approximation property Cybenko (1989) of standard dense layers.
Revised: The GVP inherits an analogue of the Universal Approximation property Cybenko (1989) of standard dense layers.

Original: If R describes an arbitrary rotation or reﬂection in 3D Euclidean space, 7 we show that a GVP can approximate arbitrary scalar-valued functions invariant under R and deﬁned over Ω ν ⊂ R ν × 3 , the bounded subset of R ν × 3 whose elements can be canonically oriented based on three linearly independent vector entries.
Revised: If R describes an arbitrary rotation or reﬂection in 3D Euclidean space, 9 we show that a GVP can approximate arbitrary scalar-valued functions invariant under R and deﬁned over Ω ν ⊂ R ν × 3 , the bounded subset of R ν × 3 whose elements can be canonically oriented based on three linearly independent vector entries.

Original: Without loss of generality, we assume the ﬁrst three vector entries can be used.
Revised: Without loss of generality, we assume the ﬁrst three vector entries can be used.

Original: The machinery corresponding to such approximations corresponds to a GVP G s with only vector inputs, only scalar outputs, and a sigmoidal nonlinearity σ ; followed by a dense layer.
Revised: The machinery corresponding to such approximations corresponds to a GVP G s with only vector inputs, only scalar outputs, and a sigmoidal nonlinearity σ ; followed by a dense layer.

Original: This can also be viewed as the sequence of matrix multiplication with W h , taking the L 2 -norm, and a dense network with one hidden layer.
Revised: This can also be viewed as the sequence of matrix multiplication with W h , taking the L 2 -norm, and a dense network with one hidden layer.

Original: Such machinery can be extracted from any two consecutive GVPs (assuming a sigmoidal σ ).
Revised: Such machinery can be extracted from any two consecutive GVPs (assuming a sigmoidal σ ).

Original: Theorem.
Revised: Theorem.

Original: Let R describe an arbitrary rotation or reﬂection in R 3 .
Revised: Let R describe an arbitrary rotation and/or reﬂection in R 3 .

Original: For ν ≥ 3 let Ω ν ⊂ R ν ×be the set of all V =
Revised: For ν ≥ 3 let Ω ν ⊂ R ν ×be the set of all V =

Original: [ v 1 , . .
Revised: [ v 1 , . .

Original: . , v ν ] T ∈ R ν × 3 such that v 1 , v 2 , v 3 are linearly independent and 0 < (cid:107) v
Revised: . , v ν ] T ∈ R ν × 3 such that v 1 , v 2 , v 3 are linearly independent and 0 < (cid:107) v

Original: i (cid:107) 2 ≤ b for all i and some ﬁnite b > 0 .
Revised: i (cid:107) 2 ≤ b for all i and some ﬁnite b > 0 .

Original: Then for any continuous F : Ω ν
Revised: Then for any continuous F : Ω ν

Original: → R such that F ( R ( V ))
Revised: → R such that F ( R ( V ))

Original: =
Revised: =

Original: F ( V ) and for any (cid:15) > 0 , there exists a form f ( V ) =
Revised: F ( V ) and for any (cid:15) > 0 , there exists a form f ( V ) =

Original: w T G s ( V ) such that | F ( V ) − f
Revised: w T G s ( V ) such that | F ( V ) − f

Original: ( V )
Revised: ( V )

Original: | < (cid:15) for all V ∈ Ω .
Revised: | < (cid:15) for all V ∈ Ω .

Original: Proof.
Revised: Proof.

Original: The idea is to write F as a composition
Revised: The idea is to write F as a composition

Original: F = ˜
Revised: F = ˜

Original: F ◦ ω and ω
Revised: F ◦ ω and ω

Original: =
Revised: =

Original: h ◦ y .
Revised: h ◦ y .

Original: We show that multiplication with W h and and taking the L 2 -norm can compute y , and that the dense network with one hidden layer can approximate ˜ F ◦ h .
Revised: We show that multiplication with W h and and taking the L 2 -norm can compute y , and that the dense network with one hidden layer can approximate ˜ F ◦ h .

Original: Call an element V ∈ Ω ν oriented if v 1 = x 1 e x , v 2 = x 2 e x + y 2 e y , and v 3 = x 3 e x + y 3 e y + z 3 e z , with x 1 , y 2 , z 3 > 0 .
Revised: Call an element V ∈ Ω ν oriented if v 1 = x 1 e x , v 2 = x 2 e x + y 2 e y , and v 3 = x 3 e x + y 3 e y + z 3 e z , with x 1 , y 2 , z 3 > 0 .

Original: Deﬁne ω : Ω ν → R 3 ν − 3 to be the orientation function that orients its input and then extracts the vector of 3 ν − 3 coefﬁcients, [ x 1 , x 2 , y 2 , x 3 , y 3 , z 3 , . . .
Revised: Deﬁne ω : Ω ν → R 3 ν − 3 to be the orientation function that orients its input and then extracts the vector of 3 ν − 3 coefﬁcients, [ x 1 , x 2 , y 2 , x 3 , y 3 , z 3 , . . .

Original: , x i , y i , z i , . . . ]
Revised: , x i , y i , z i , . . . ]

Original: T .
Revised: T .

Original: These elements can be written as
Revised: These elements can be written as

Original: [Equation]
Revised: [Equation]

Original: and are invariant under rotation and reﬂection, because they are deﬁned using only the norms and inner products of the v i .
Revised: and are invariant under rotation and reﬂection, because they are deﬁned using only the norms and inner products of the v i .

Original: Then F = ˜ F ◦ ω , where ˜ F : [ − b, b ] 3 ν − 3 → R .
Revised: Then F = ˜ F ◦ ω , where ˜ F : [ − b, b ] 3 ν − 3 → R .

Original: The key insight is that if we construct W h such that the rows of W h V are the original vectors v i , ∀
Revised: The key insight is that if we construct W h such that the rows of W h V are the original vectors v i , ∀

Original: i and all differences v i − v j , ∀
Revised: i and all differences v i − v j , ∀

Original: i, j ≤ min( i, 3) , then we can compute ω ( V ) from the row-wise norms of W h V .
Revised: i, j ≤ min( i, 3) , then we can compute ω ( V ) from the row-wise norms of W h V .

Original: That is, ω = h ◦ y where
Revised: That is, ω = h ◦ y where

Original: y = y ( V ) = (cid:107) .
Revised: y = y ( V ) = (cid:107) .

Original: (cid:107) 2 (cid:12) ( W h V ) ∈
Revised: (cid:107) 2 (cid:12) ( W h V ) ∈

Original: R 4 ν − 6 and h is an application of the cosine law.
Revised: R 4 ν − 6 and h is an application of the cosine law.

Original: The GVP precisely computes y as an intermediate step: G s ( V )
Revised: The GVP precisely computes y as an intermediate step: G s ( V )

Original: = σ (cid:12) ( W m y + b ) .
Revised: = σ (cid:12) ( W m y + b ) .

Original: It remains to show that there exists a form ˜ f
Revised: It remains to show that there exists a form ˜ f

Original: ( y ) =
Revised: ( y ) =

Original: w T [ σ (cid:12) ( W h y + b )] that (cid:15) -approximates
Revised: w T [ σ (cid:12) ( W h y + b )] that (cid:15) -approximates

Original: ˜
Revised: ˜

Original: F ◦ h : [ − 2 b, 2 b ] 4 ν − 6 → R .
Revised: F ◦ h : [ − 2 b, 2 b ] 4 ν − 6 → R .

Original: Up to a translation and uniform scaling of the hypercube, this is the result of the Universal Approximation Theorem (Cybenko, 1989).
Revised: Up to a translation and uniform scaling of the hypercube, this is the result of the Universal Approximation Theorem (Cybenko, 1989).

Original: B S YNTHETIC T ASKS
Revised: B S YNTHETIC T ASKS

Original: We perform controlled experiments on a synthetic dataset in order to analyze the beneﬁts of the
Revised: We perform controlled experiments on a synthetic dataset in order to analyze the beneﬁts of the

Original: GVP architecture and determine if it indeed improves the geometric and joint geometric-relational reasoning abilities of GNNs.
Revised: GVP architecture and determine if it indeed improves the geometric and joint geometric-relational reasoning abilities of GNNs.

Original: Dataset The synthetic dataset is designed to mimic the essential qualities of the domain of protein structures.
Revised: Dataset The synthetic dataset is designed to mimic the essential qualities of the domain of protein structures.

Original: Each ”structure” consists of n = 100 random points in R 3 , distributed uniformly in the ball of radius r = 10 , with the constraint that no two points are less than distance d = 2 apart.
Revised: Each “structure” consists of n = 100 random points in R 3 , distributed uniformly in the ball of radius r = 10 , with the constraint that no two points are less than distance d = 2 apart.

Original: Each position is also associated with a random unit vector (a ”sidechain”) to endow it with an orientation.
Revised: Each position is also associated with a random unit vector (a “sidechain”) to endow it with an orientation.

Original: Three points are randomly chosen and are labelled as ”special”; these will be used to deﬁne the learning tasks.
Revised: Three points are randomly chosen and are labelled as “special”; these will be used to deﬁne the learning tasks.

Original: We generate 20k ”structures” and split them 80% train : 10% validation : 10% test.
Revised: We generate 20k “structures” and split them 80% train : 10% validation : 10% test.

Original: In the voxelized representation of a data point, the volume is voxelized into unit cubes.
Revised: In the voxelized representation of a data point, the volume is voxelized into unit cubes.

Original: Each point is partitioned into a neighborhood of eight voxels by trilinear interpolation, such that exact coordinate information is retained.
Revised: Each point is partitioned into a neighborhood of eight voxels by trilinear interpolation, such that exact coordinate information is retained.

Original: Separate channels are used for the special and non-special points.
Revised: Separate channels are used for the special and non-special points.

Original: The ”sidechains” are represented with a set of n = 100 points located at the ends of the unit vectors and mapped into a third channel.
Revised: The “sidechains” are represented with a set of n = 100 points located at the ends of the unit vectors and mapped into a third channel.

Original: In the graph-structured representation of a data point, a proximity graph is drawn with k =nearest neighbors.
Revised: In the graph-structured representation of a data point, a proximity graph is drawn with k =nearest neighbors.

Original: Each node is labelled with a one-hot encoding of its type (”special” or ”nonspecial”) and each edge with its Euclidean length.
Revised: Each node is labelled with a one-hot encoding of its type (“special” or “nonspecial”) and each edge with its Euclidean length.

Original: In the vanilla GNN, orientation information is encoded by additionally including all three dot products in each edge embedding.
Revised: In the vanilla GNN, orientation information is encoded by additionally including all three dot products in each edge embedding.

Original: In the GVP-GNN, each node embedding contains the node’s ”sidechain” vector, and each edge embedding contains a unit vector indicating the direction of the edge.
Revised: In the GVP-GNN, each node embedding contains the node’s “sidechain” vector, and each edge embedding contains a unit vector indicating the direction of the edge.

Revised: A link to our synthetic dataset can be found here: https://drive.google.com/drive/ folders/1XuCyPFTM2Ro9kfCP_tWOF1uwBB-Hn3LH

Original: Tasks We identify two regression tasks to exemplify geometric and relational reasoning, respectively.
Revised: Tasks We identify two regression tasks to exemplify geometric and relational reasoning, respectively.

Original: In the ”Off-center” task, the network predicts the distance from the centroid of the three special points to the centroid of the entire structure.
Revised: In the “Off-center” task, the network predicts the distance from the centroid of the three special points to the centroid of the entire structure.

Original: In the ”Perimeter” task, the network predicts the perimeter of the triangle deﬁned by the three special points.
Revised: In the “Perimeter” task, the network predicts the perimeter of the triangle deﬁned by the three special points.

Original: We characterize the former as primarily geometric, as it requires reasoning about global properties of the 3D shape ,  and the latter as primarily relational, as it involves distances between three  pairs of nodes.
Revised: We characterize the former as primarily geometric, as it requires reasoning about global properties of the 3D shape , in particular points in space that are not themselves nodes, and the latter as primarily relational, as it involves distances between three speciﬁc pairs of nodes.

Original: Finally, to represent a problem with geometric and relational aspects, in the ”Combined” task we attempt to predict the difference of the (normalized) off-center and perimeter objectives.
Revised: Finally, to represent a problem with geometric and relational aspects, in the “Combined” task we attempt to predict the difference of the (normalized) off-center and perimeter objectives.

Original: Models We train a 3-layer shallow CNN and a 3-layer GNN with a single-layer feed-forward network.
Revised: Models We train a 3-layer shallow CNN and a 3-layer GNN with a single-layer feed-forward network.

Original: These are compared against a GVP-GNN that is otherwise identical to the standard GNN.
Revised: These are compared against a GVP-GNN that is otherwise identical to the standard GNN.

Original: To reﬂect the spirit of the synthetic experiment, all models have the same intermediate dimensionality of 32 (4 vector and 20 scalar channels in the GVP-GNN), we use the same training procedure for all models, and no hyperparameter tuning or architecture search is performed.
Revised: To reﬂect the spirit of the synthetic experiment, all models have the same intermediate dimensionality of 32 (4 vector and 20 scalar channels in the GVP-GNN), we use the same training procedure for all models, and no hyperparameter tuning or architecture search is performed.

Original: Results
Revised: Results

Original: The results of the synthetic experiments are shown in Table 5.
Revised: The results of the synthetic experiments are shown in Table 5.

Original: The vanilla GNN signiﬁcantly outperforms the CNN on the perimeter task, while the CNN signiﬁcantly outperforms the
Revised: The vanilla GNN signiﬁcantly outperforms the CNN on the perimeter task, while the CNN signiﬁcantly outperforms the

Original: GNN on the off-center task, supporting our conceptual framework of the relative strengths of the two architectures.
Revised: GNN on the off-center task, supporting our conceptual framework of the relative strengths of the two architectures.

Original: However, the GVP-GNN matches (and even outperforms) the CNN on the geometric task while maintaining the GNN’s performance on the relational task.
Revised: However, the GVP-GNN matches (and even outperforms) the CNN on the geometric task while maintaining the GNN’s performance on the relational task.

Original: It additionally signiﬁcantly outperforms both models on the combined task.
Revised: It additionally signiﬁcantly outperforms both models on the combined task.

Original: On the basis of these results, the GVP appears successful in combining the strengths of the CNN and GNN into a single architecture.
Revised: On the basis of these results, the GVP appears successful in combining the strengths of the CNN and GNN into a single architecture.

Original: [Table]
Revised: [Table]

Original: C D ATASETS
Revised: C D ATASETS

Original: Model quality assessment For training and validation, we use 528 targets from CASP 5-10 and 150 candidate structures per target.
Revised: Model quality assessment For training and validation, we use 528 targets from CASP 5-10 and 150 candidate structures per target.

Original: These targets are partitioned at random into 480 training targets and 48 validation targets.
Revised: These targets are partitioned at random into 480 training targets and 48 validation targets.

Original: We include native structures for training and validation to make use of the greatest range of GDT-TS scores available.
Revised: We include native structures for training and validation to make use of the greatest range of GDT-TS scores.

Revised: We do not include native structures for testing in order to mimic

Revised: CASP and real-world applications and because other methods were not tested on native structures.

Original: In the CASP assessments, stage 1 refers to a set of 20 candidate structures per target and stage 2 to a set of 150 candidate structures per target (5 from each structure prediction server).
Revised: In the CASP assessments, stage 1 refers to a set of 20 candidate structures per target and stage 2 to a set of 150 candidate structures per target (5 from each structure prediction server).

Original: Both sets are pre-designated by the CASP organizers.
Revised: Both sets are pre-designated by the CASP organizers.

Original: There has been slight inconsistency in the literature with regards to the exact composition of the
Revised: There has been slight inconsistency in the literature with regards to the exact composition of the

Original: CASP 11 and 12 test sets.
Revised: CASP 11 and 12 test sets.

Original: We use the list established by Karasikov et al.
Revised: We use the list established by Karasikov et al.

Original: (2019) because nearly all recent methods have been benchmarked on this set at some point.
Revised: (2019) because nearly all recent methods have been benchmarked on this set at some point.

Original: We do not include native structures for testing to best mimic CASP and real-world application scenarios and because other methods were not tested on native structures. All structures are relaxed with SCWRL4 (Krivov et al., 2009) to eliminate artifacts from the various prediction servers.
Revised: For the CASP 13 test set, we include all stage 2 candidate structures for which the ground-truth GDT-TS is available and for which the mapping between candidate structure names and IDs is provided, enabling the candidate structure set to be matched with predictions from methods which ofﬁcially participated in CASP 13.

Original: The exact numbers of targets and structures in each set can be found in Table 6. [Table] Validation [Table]
Revised: There are 1472 such structures in total on 20 targets: T0950, T0951, T0953s1, T0953s2, T0954,

Original: Yes CASP 11 stage No CASP
Revised: T0955, T0957s1, T0957s2, T0958, T0960, T0963, T0966, T0968s1, T0968s2, T1003, T1005,

Original: [Table] [Table] No CASP 12 stage
Revised: T1008, T1009, T1011, and T1016.

Original: No CASP [Table] [Table]
Revised: This information is obtained from the CASP download center as described by Baldassarre et al.

Original: No CASP [Table] No
Revised: The exact numbers of targets and structures in each set can be found in Table 6.

Original: Protein design As described in the main text, we use the CATH 4.2 dataset and splits as curated by Ingraham et al.
Revised: Protein design As described in the main text, we use the CATH 4.2 dataset and splits as curated by Ingraham et al.

Original: and the TS50 test set as curated by Li et al.
Revised: and the TS50 test set as curated by Li et al.

Original: To evaluate on TS50, we remove sequences with more than 30% similarity as computed by PSIBLAST to any protein in
Revised: To evaluate on TS50, we remove sequences with more than 30% similarity as computed by PSIBLAST to any protein in

Original: TS50 from CATH 4.2.
Revised: TS50 from CATH 4.2.

Original: D T RAINING AND H YPERPARAMETERS
Revised: D T RAINING AND H YPERPARAMETERS

Original: To train the MQA model to perform regression against the model quality score, we use a sum of an absolute loss and a pairwise loss.
Revised: To train the MQA model to perform regression against the model quality score, we use a sum of an absolute loss and a pairwise loss.

Original: That is, for each training step we intake pairs i, j where i, j arecandidate structures for the same target and compute
Revised: That is, for each training step we intake pairs i, j where i, j are

Original: [Equation]
Revised: [Table]

Revised: candidate structures for the same target and compute

Original: [Equation]
Revised: [Equation]

Original: where H is the Huber loss.
Revised: where H is the Huber loss.

Original: When reshufﬂing at the beginning of each epoch, we also randomly pair up the candidate structures for each target.
Revised: When reshufﬂing at the beginning of each epoch, we also randomly pair up the candidate structures for each target.

Original: Interestingly, adding the pairwise term also improves global correlation,  because the much larger number of possible pairs 
Revised: Interestingly, adding the pairwise term also improves global correlation, likely because the much larger number of possible pairs makes it more difﬁcult to overﬁt.

Original: To train the CPD model to perform classiﬁcation / discrete generative modelling, we use the crossentropy / negative log likelihood loss.
Revised: To train the CPD model to perform classiﬁcation / discrete generative modelling, we use the crossentropy / negative log likelihood loss.

Original: We also experimented with label smoothing, but this did not improve performance.
Revised: We also experimented with label smoothing, but this did not improve performance.

Original: For both the MQA and CPD model, we use node and hidden embeddings with 16 vector andscalar channels and edge embeddings with 1 vector and 32 scalar channels.
Revised: For both the MQA and CPD model, we use node and hidden embeddings with 16 vector andscalar channels and edge embeddings with 1 vector and 32 scalar channels.

Original: The input node and edge features are ﬁrst transformed by a sequence of GVPs to these dimensionalities before graph propagation.
Revised: The input node and edge features are ﬁrst transformed by a sequence of GVPs to these dimensionalities before graph propagation.

Original: In all training runs, we use the Adam optimizer to perform mini-batch gradient descent.
Revised: In all training runs, we use the Adam optimizer to perform mini-batch gradient descent.

Original: Batches are constructed by grouping structures of similar size to have a maximum of 1800 residues per batch for CPD and 3000 residues per batch for MQA.
Revised: Batches are constructed by grouping structures of similar size to have a maximum of 1800 residues per batch for CPD and 3000 residues per batch for MQA.

Original: We also tune the following hyperparameters over a total of 70 training runs:
Revised: We also tune the following hyperparameters over a total of 70 training runs:

Original: • Learning rate in the range of 10 − 4 to 10 −• Dropout probability in the range of 10 − 4 to 10 −• Number of graph propagation layers in the range of 3 to• Relative weight of the MQA pairwise loss in the range of [-
-] to
Revised: • Learning rate in the range of 10 − 4 to 10 −• Dropout probability in the range of 10 − 4 to 10 −• Number of graph propagation layers in the range of 3 to• Relative weight of the MQA pairwise loss in the range of [-
-]0 to

Original: All models are implemented in TensorFlow 2.1 and trained for a maximum of 100 epochs.
Revised: All models are implemented in TensorFlow 2.1 and trained for a maximum of 100 epochs.

Original: This takes around two days for both models on a single Titan X GPU.
Revised: This takes around two days for both models on a single Titan X GPU.

Original: However, we note that the GPU memory, not compute power, is the bottleneck when training, based on the the average volatile GPU usage.
Revised: However, we note that the GPU memory, not compute power, is the bottleneck when training, based on the the average volatile GPU usage.

Original: We therefore anticipate that the runtime can be further optimized.
Revised: We therefore anticipate that the runtime can be further optimized.

Original: E V ISUALIZATION AND I NTERPRETATION OF L EARNED F EATURES

Original: The geometric vector perceptron updates the vector features, in addition to scalar features, at node embeddings during graph propagation.

Original: Therefore, while the input vector channels represent the forward and reverse directions at each amino acid, the intermediate layers represent learned vector features.

Original: Could some of these features correspond to interpretable properties of the structure?

Original: Among a total of 64 intermediate vector channels learned by the MQA model, a few appeared visually interpretable and are shown on selected structures in Figure 2.

Original: We caution against generalizing from the necessarily small number of images that could be manually inspected, but ﬁnd these preliminary visualizations intriguing.

Original: F A DDITIONAL RESULTS F. D IME N ET ON MQA
Revised: E A DDITIONAL R ESULTS E. D IME N ET ON MQA

Original: DimeNet (Klicpera et al., 2020) is a recent GNN architecture designed to incorporate the 3D geometry of small molecule graphs by updating edge embeddings with incoming messages from neighboring edges, where the relative orientation of each edge is encoded in a local spherical Bessel basis.
Revised: DimeNet (Klicpera et al., 2019) is a recent GNN architecture designed to incorporate the 3D geometry of small molecule graphs by updating edge embeddings with incoming messages from neighboring edges, where the relative orientation of each edge is encoded in a local spherical Bessel basis.

Original: DimeNet and our architecture are similar in that both seek to leverage geometric aspects of a problem domain on top of graph-structured representations.
Revised: DimeNet and our architecture are similar in that both seek to leverage geometric aspects of a problem domain on top of graph-structured representations.

Original: However, unlike our architecture, it speciﬁcally updates edge embeddings and must represent each pair of neighboring edges, and therefore does not scale well to large protein structure graphs.
Revised: However, unlike our architecture, it speciﬁcally updates edge embeddings and must represent each pair of neighboring edges, and therefore does not scale well to large protein structure graphs.

Original: In evaluating DimeNet on MQA, we could only extend the distance cutoff to 7.5 angstroms, while 30 neighbors corresponds to roughlyangstroms.
Revised: In evaluating DimeNet on MQA, we could only extend the distance cutoff to 7.5 angstroms, while 30 neighbors corresponds to roughlyangstroms.

Original: DimeNet does not perform comparably to our model, or to previous GNNs designed for learning from structure such as GraphQA.
Revised: DimeNet does not perform comparably to our model, or to previous GNNs designed for learning from structure such as GraphQA.

Original: (Table 7).
Revised: (Table 7).

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: No

Revised: ProQ 0.

Revised: No

Revised: ProQ3D-TM 0.

Revised: No

Revised: MASS 0.

Revised: Yes

Revised: VoroMQA-A 0.

Revised: Yes

Revised: VoroMQA-B 0.

Revised: Yes

Revised: MASS 0.

Revised: Yes

Revised: MULTICOM-NOVEL 0.

Revised: No

Revised: ProQ 0.

Revised: No

Revised: PLU-AngularQA 0.

Revised: Yes

Revised: Bhattacharya-Server 0.

Revised: No

Revised: Bhattacharya-SingQ 0.

Revised: No

Revised: Kiharalab 0.

Revised: No

Revised: PLU-TopQA 0.

Original: F. CPD: R ESULTS ON TS50 TEST SET
Revised: Yes Jagodzinski-Cao-QA 0. Yes E. MQA: R ESULTS ON CASP

Original: A number of methods  for computational protein design only report results for the test set TS50 and not the CATH 4.2 test set.
Revised: We report results for all 23 single-structure methods assessed in CASP 13 for which scores on all 20 targets are available (Table 8). The following 7 methods were excluded because they do not report results for some targets: LamoureuxLab, SBROD-server, SBROD, 3DCNN, MESHI-server, SBROD-plus, FALCON-QA, and Grudinin. We do include comparisons with LamoureuxLab (previously 3DCNN), 3DCNN (previously Ornate), and SBROD on CASP 11-12.

Original: Our comparison with these methods is shown in Table 8.
Revised: All of the methods highlighted as top-performing by the CASP organizers in Cheng et al. (2019) are in our comparison for CASP 13.

Original: No canonical training and validation sets exist for TS50.
Revised: All predictions were obtained from the CASP download center as described by

Original: In order to evaluate onTS50, we remove sequences with more than 30% similarity from the CATH 4.2 training and validation sets and retrain our model.
Revised: Baldassarre et al. E. CPD: R ESULTS ON TS50 T EST S ET We compare against a number of recent CPD methods on the TS50 test set in Table 9.

Original: The methods to which we compare include two CNNs (ProDCoNN and DenseCPD), a distance-map method (SBROF), and sequential representation methods (Wang’s model and SPIN2). 
Revised: These include two CNNs (ProDCoNN and DenseCPD), a distance-map method (SBROF), and sequential representation methods (Wang’s model and SPIN2). We also evaluate the GNN ProteinSolver on

Revised: TS50 by sampling 100 sequences with temperature 1 (the default setting) for each structure using the public web server.

Revised: No canonical training and validation sets exist for TS50.

Revised: Therefore, in order to evaluate on TS50, we remove sequences with more than 30% similarity from the CATH 4.2 training and validation sets and retrain our model.

Original: We outperform all other methods with the exception of DenseCPD, a CNN method with canonical orientations.
Revised: We outperform all other methods with the exception of DenseCPD, a CNN method with canonical orientations.

Original: Interestingly, DenseCPD leverages the same underlying representation as ProDCoNN, yet achieves remarkably better performance.
Revised: Interestingly, DenseCPD leverages the same underlying representation as ProDCoNN, yet achieves remarkably better performance.

Original: The main difference between the two methods is that ProDCoNN has 4 convolutional layers and DenseCPD 21 layers organized into dense residual blocks.
Revised: The main difference between the two methods is that ProDCoNN has 4 convolutional layers and DenseCPD 21 layers organized into dense residual blocks.

Original: [Table]
Revised: [Table]

Original: F. A BLATION STUDIES
Revised: F V ISUALIZATION AND I NTERPRETATION OF L EARNED F EATURES

Original: In order to determine which aspects of the GVP contribute most to our improvement over existing state-of-the-art GNNs
Revised: The geometric vector perceptron updates the vector features, in addition to scalar features, at node embeddings during graph propagation.

Original: [Table] we train a number of ablated models for MQA and CPD and compare them against the baseline GVP models. In addition to Table 3 in the main text [Table]
Revised: Therefore, while the input vector channels represent the forward and reverse directions at each amino acid, the intermediate layers represent learned vector features.

Original: we report a more detailed breakdown of performance metrics in Table 9. Replacing the GVP with a vanilla MLP layer or propagating only scalar features both remove direct access to geometric information, hurting the performance of the model. Propagating only the vector features results in an even more signiﬁcant decrease as it both eliminates important scalar input features (such as torsion angles and amino acid identity) and the part of the GVP with approximation guarantees. Therefore, the dual scalar/vector design of the GVP is essential.
Revised: Could some of these features correspond to interpretable properties of the structure?

Original: Finally, eliminating the intermediate transformation W h , and instead taking the norms of  the ﬁnal transformed vectors V µ , results in a slight but non-negligible decrease in performance. The results suggest that each aspect of the GVP contributed to our improvement over state-of-the-art GNNs. [Table]
Revised: Among a total of 64 intermediate vector channels learned by the MQA model, a few appeared visually interpretable and are shown on selected structures in Figure 2. We caution against generalizing from the necessarily small number of images that could be manually inspected, but ﬁnd these preliminary visualizations intriguing.